diff --git a/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md new file mode 100644 index 0000000..623d7d1 --- /dev/null +++ b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md @@ -0,0 +1,111 @@ +--- +date: 2026-10-06 +branch: docs/pm-agent-registry-pull-prereq +title: The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin +--- + +## What changed + +The image pull for `pm-agent` failed TLS verification, and the previous entry on this branch +honestly recorded it as **not diagnosed**. The operator asked for the diagnosis before any +registry design. This is it, and the fix turns out to be a pattern this repository already +has. + +`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins +`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the +Keycloak issuer pin that is there for the identical reason. **An operator must run the +`node_baseline` play for it to reach the nodes** — writing `/etc/hosts` on a node is not +something Flux or an agent can do. + +The docs that described this as an undiagnosed blocker now describe the cause and the fix: +`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and +the play), and `docs/plans/pm-agent-issue-to-pr.md`. + +Also recorded, in the two places an operator reads when creating the credential: **the agent +authenticates to Gitea with a local account and a scoped PAT, by decision** (operator, +2026-10-06), not with a federated identity. Written down because the alternative looks +obviously better until you check whether it is possible — see below. + +## What I verified + +Every step measured, read-only, in this order. + +**The failing certificate is the router's, not the Gateway's.** + +- containerd's error names TLS, not the digest: `tls: failed to verify certificate: x509: + certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev`. + "not valid for **any** names" means the presented certificate carries no SANs at all. +- `192.168.2.20` is the shared Gateway's own LoadBalancer VIP + (`gateway/cilium-gateway-shared`, from the Cilium pool `192.168.2.20-30`), and its `https` + listener serves `*.home.keithjackson.dev` with `wildcard-home-tls`. From the operator's + workstation the name resolves there and serves the real Let's Encrypt + `CN=*.home.keithjackson.dev` with correct SANs. **So the certificate and the Gateway are + healthy.** +- **The node asks a different resolver and gets a different answer.** Prod's + `node_baseline_dns_servers` is the LAN gateway, not Pi-hole — deliberately, as that file + explains. Public DNS for this zone is a CNAME to `wan.home.keithjackson.dev`: measured via + both `1.1.1.1` and `8.8.8.8`, `gitea.home.keithjackson.dev` → + `wan.home.keithjackson.dev.` → `136.51.115.252`, and the apex answers the same. +- **That address serves a certificate with no names.** Measured from the workstation against + the WAN address with the right SNI: + + subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D + issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D + No extensions in certificate + + No extensions means no `subjectAltName`, which is exactly "not valid for any names". The + `CN=%D` is the router's placeholder. + +**This repository had already measured the same thing, for Keycloak, and I did not need to +re-derive it.** `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` says, above the pin: +"Since platform/gateway/gateway.yaml set an external-dns `target`, every public name is a +CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from +the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from +k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18)." Same cause, same symptom, +one name later. The pin exists precisely for node processes that must reach the Gateway +directly; containerd is one, and nobody had added it. + +**The parity the pin has to satisfy:** `192.168.2.20` equals the live +`GATEWAY_VIP`, and `k3s_base_domain` equals the live `BASE_DOMAIN` — both read from the +cluster, not from the comment that claims prod serves `prod.home.keithjackson.dev` (it does +not; `flux-system/cluster-vars` reports `BASE_DOMAIN=home.keithjackson.dev` and +`GATEWAY_VIP=192.168.2.20`, and Ansible's `k3s_base_domain` is `home.keithjackson.dev`). +That comment is stale history. + +**The credential decision, and why the obvious alternative is not available.** Gitea's API +accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2 +provider Gitea implements **only the Authorization Code Grant** (plus PKCE/OIDC), so there +is no client-credentials grant, and nothing in Gitea validates a token issued by a +third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to +Gitea's API or to git-over-HTTP. Read from Gitea's own documentation +(`docs.gitea.com/development/api-usage`, `/development/oauth2-provider`), not assumed. + +**Gate:** `./ci/gate.sh` in the pinned toolbox from a §0 worktree — see the branch's run; rc 3 +with `gitea-settings` the only stage that did not run. + +## What I did NOT verify + +- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the + nodes is written by the `node_baseline` play, which is an operator action. Until that runs + and an image exists, "the node now reaches the Gateway" is a design claim. The evidence + that it *will* is the Keycloak pin working for the same cause — which I also did not + re-measure. +- **Why public DNS answers at all for an `internal` service.** `catalog/gitea.yaml` sets no + `exposure`, so it inherits `internal`, and the renderer contract says an internal service's + HTTPRoute carries `external-dns.alpha.kubernetes.io/controller: none` and publishes no + public record. Yet `gitea.home.keithjackson.dev` resolves publicly to the WAN CNAME. I did + not chase that: it may be a zone-level wildcard rather than a published record, and it did + not change the diagnosis. It is worth someone's attention, because "internal services + publish no public DNS" is an L0-checked claim and this looks like a case where the *name* + resolves regardless of what the renderer withheld. +- **Registry authentication.** Unchanged and still open: an unauthenticated client gets + nothing from this registry, and no `imagePullSecret` exists anywhere in the tree. With the + credential model now decided (local Gitea accounts with PATs) the shape is a + `dockerconfigjson` for an account whose token carries `read:package`, referenced from the + Deployment and the committed worker Job template — and the worker Job template's shape is + pinned by admission, so whether the policy should also pin `imagePullSecrets` deserves a + deliberate look rather than a quiet addition. Not built: minting that token is an operator + action, and I did not want to guess at a second credential's scope. +- **Anything about runtime.** The image still does not exist. No dispatcher has + authenticated, no issue has been triaged, no worker Job has been created, and the admission + policy — live, `Enforce`, six rules — has still never admitted or refused a real Job. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index 37d7830..567ac69 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -236,12 +236,14 @@ about runtime behaviour in this document is a design claim, not a measurement. 1. **Make the registry pullable, THEN build, push, pin.** Measured after the merge, and neither half is declared anywhere in this repository (`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`): - (i) the cluster's nodes cannot verify the registry's TLS certificate — containerd reports - `x509: certificate is not valid for any names` for `gitea.home.keithjackson.dev`, while - that name serves a healthy Let's Encrypt wildcard from the LAN; (ii) an unauthenticated - client gets nothing from that registry and no `imagePullSecret` exists in the tree, so - either one is added or anonymous package reads are allowed. Only then does the digest - matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both + (i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the + router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning + `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same + pin the Keycloak issuer already carries; an operator must run the `node_baseline` play; + (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with + the credential decision now being local Gitea accounts with PATs, that means a + `dockerconfigjson` for an account whose token carries `read:package`. Only then does the + digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both placeholder digests. Expect the build itself to need iteration too: the nodejs module stream and the npm install have never run against this base. The same two blockers sit in front of `conformance-runner`, which shares the registry path and has never been pulled. diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index 4813196..2b865a6 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -327,6 +327,29 @@ node_baseline_hosts_pins: - ip: "192.168.2.20" names: - "keycloak.{{ k3s_base_domain }}" + # gitea: the SECOND node process to need this pin, for the identical reason, found + # 2026-10-06. containerd pulls images as a NODE process, so it resolves through the + # public upstreams above and gets the CNAME to wan.home.keithjackson.dev — the WAN IP, + # which this router does not hairpin. The pull then fails on the router's own + # placeholder certificate rather than on anything to do with the image: + # + # failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:…": + # tls: failed to verify certificate: x509: certificate is not valid for any names, + # but wanted to match gitea.home.keithjackson.dev + # + # Measured from the workstation the same day, against the WAN address with that SNI: + # subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D + # No extensions in certificate <- no SANs at all, hence "not valid for ANY names" + # while the same name via PiHole answers 192.168.2.20 and serves the real + # Let's Encrypt *.home.keithjackson.dev. So the certificate and the Gateway are fine; + # only the node's ANSWER is wrong. Same root cause, same fix, as the issuer pin above. + # + # WHO NEEDS IT: any workload whose image lives in the in-cluster Gitea registry — + # today platform/pm-agent (agent-runner) and conformance/incluster + # (conformance-runner), neither of which has ever been pulled. A pull is still + # UNAUTHENTICATED after this pin, which is a separate thing to settle; this only makes + # the node reach the right endpoint. + - "gitea.{{ k3s_base_domain }}" # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) -------------------- # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/ diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index ec68fb2..baedb1d 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -22,6 +22,25 @@ requires. --- +## Why the agent authenticates to Gitea with a local account and a PAT + +Not Keycloak, and that is a decision rather than an omission (operator, 2026-10-06): the +agents only ever talk to Gitea, so a local Gitea account with a scoped personal access token +is the whole requirement, and it is drastically simpler than federating a machine identity. + +It is also what Gitea can actually do. Keycloak service accounts were considered and are not +available for this: Gitea's API accepts basic auth, PATs, SSH signatures and **its own** +OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization +Code Grant** — there is no client-credentials grant, and nothing in Gitea validates a token +issued by a third-party OIDC provider. So a Keycloak service-account token cannot +authenticate to Gitea's API or to git-over-HTTP at all. Keycloak remains what it is for human +access: the browser login source registered on Gitea (`catalog/gitea.yaml`). + +What still matters, and is enforced by the hand-off rather than by Keycloak: **the account +must not be an administrator** — `infra/gitea/approval-policy.json` authorises +`/approve-override` on live repository permission alone, so an admin-permission bot could +waive the sensitive-path human review on its own pull request. + ## It does nothing at all until an operator does two things Both are deliberate, and both fail **visibly** rather than silently: @@ -38,17 +57,24 @@ Both are deliberate, and both fail **visibly** rather than silently: path. Details and the exact commands: [`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md). - - **The nodes must be able to verify the registry's TLS certificate.** containerd - currently fails with `x509: certificate is not valid for any names, but wanted to match - gitea.home.keithjackson.dev`, while the same name from the LAN serves a healthy - Let's Encrypt `*.home.keithjackson.dev`. Correcting the digest will not clear this. - - **The pull must be authenticated.** An unauthenticated client gets nothing from this - Gitea's registry (its anonymous token carries no access scope; every path 401s), and - there is no `imagePullSecret` anywhere in the tree. Either add one — a - `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the - Deployment *and* from the committed worker Job template — or allow anonymous reads of - that package in Gitea, which makes the image readable to anything that can reach the - host. That is a posture decision, so it is yours rather than an agent's. + - **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is + the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a + *node* process, so it resolves through the node's public upstreams + (`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public + answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN + IP — which this router does not hairpin; it answers :443 with its own placeholder + certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely + `x509: certificate is not valid for any names`. The Gateway, the wildcard certificate + and the digest were never implicated. The fix is the pin this repository already uses + for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now + in `node_baseline_hosts_pins`. **An operator must run the `node_baseline` play for it + to reach the nodes.** + - **The pull is still unauthenticated** once the node reaches the right endpoint. An + unauthenticated client gets nothing from this Gitea's registry, and there is no + `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision + (local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea + account whose token carries `read:package`, referenced from the Deployment and the + committed worker Job template — not built yet, because minting that token is yours. The same two apply to `conformance-runner`: same registry path, same push-then-pin model, never pulled either. diff --git a/platform/pm-agent/pm-agent-credentials.sops.yaml.example b/platform/pm-agent/pm-agent-credentials.sops.yaml.example index a7fe9fd..dd2a601 100644 --- a/platform/pm-agent/pm-agent-credentials.sops.yaml.example +++ b/platform/pm-agent/pm-agent-credentials.sops.yaml.example @@ -51,6 +51,15 @@ # (2) MINT THE GITEA TOKEN as that user (log in as `pm-agent`, or use the admin API to # create a token for it): Settings -> Applications -> Generate New Token. # +# A LOCAL GITEA ACCOUNT WITH A PAT IS THE DECIDED MODEL (operator, 2026-10-06), not +# Keycloak. The agents only ever talk to Gitea, so this is the whole requirement and it +# is far simpler than federating a machine identity. It is also the only thing that +# works: Gitea's API accepts basic auth, PATs, SSH signatures and its OWN OAuth2 bearer +# tokens, and as a provider Gitea implements only the Authorization Code Grant — no +# client-credentials grant, and nothing that validates a third-party OIDC token. A +# Keycloak service-account token therefore cannot authenticate to Gitea's API or to +# git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that. +# # Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT # `write:admin`, NOT `write:organization`, NOT `write:user`. # diff --git a/runbooks/pm-agent.md b/runbooks/pm-agent.md index 4a6a5f6..3a4c759 100644 --- a/runbooks/pm-agent.md +++ b/runbooks/pm-agent.md @@ -68,7 +68,7 @@ Work through these in order; the first three are by far the most likely. |---|---|---| | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` | | `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first | -| `ImagePullBackOff` with `x509: certificate is not valid for any names` | the **node** cannot verify the registry's TLS certificate. Nothing to do with the digest: a correct digest fails identically. The same name serves a healthy Let's Encrypt wildcard from the LAN, so what the node reaches is not what you reach | not diagnosed — see `decisions/2026-10-05-pm-agent-registry-pull-prereq.md` for everything measured. Check the nodes' resolution of `gitea.home.keithjackson.dev` and their registry trust (`/etc/rancher/k3s/registries.yaml`), neither of which this repository declares | +| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. **Run the `node_baseline` play** for it to reach the nodes. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | | `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours | | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` | | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |