From 523a463fd7e2a9e09d49cd165d244137340ead87 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 6 Oct 2026 00:35:18 -0500 Subject: [PATCH] =?UTF-8?q?fix(node):=20pin=20gitea=20to=20the=20Gateway?= =?UTF-8?q?=20VIP=20=E2=80=94=20containerd=20was=20resolving=20it=20to=20t?= =?UTF-8?q?he=20non-hairpinned=20WAN?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pm-agent image pull failed TLS verification, and the previous entry on this branch recorded it honestly as not diagnosed. Diagnosed now, and it is not about the image. containerd pulls as a NODE process, so it resolves through node_baseline_dns_servers (the LAN gateway, deliberately not PiHole) and gets the public answer: a CNAME to wan.home.keithjackson.dev -> 136.51.115.252, measured via both 1.1.1.1 and 8.8.8.8. This router does not hairpin, so WAN:443 answers with its own placeholder certificate — measured with the right SNI from the workstation: subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D No extensions in certificate No extensions means no subjectAltName, which is exactly 'certificate is not valid for any names'. Meanwhile 192.168.2.20 IS the shared Gateway's VIP and its https listener serves the real Let's Encrypt *.home.keithjackson.dev. The Gateway, the wildcard cert and the digest were never implicated — only the node's answer. This file already documents the identical failure for the Keycloak issuer, measured from k3s-prod-2/-3 on 2026-10-03 (curl exit 60, ssl_verify_result=18), and already carries the mechanism that fixes it. gitea is simply the second node process to need it; containerd is a node process and nobody had added it. An operator must run the node_baseline play. Also records, where an operator reads them, that the agent authenticates to Gitea with a local account and a scoped PAT by decision — and why the federated alternative is not merely unchosen but unavailable: Gitea implements only the Authorization Code Grant and validates no third-party OIDC token, so a Keycloak service-account token cannot authenticate to its API or to git-over-HTTP. Co-Authored-By: Claude Opus 5 (1M context) --- ...-06-pm-agent-node-resolves-gitea-to-wan.md | 111 ++++++++++++++++++ docs/plans/pm-agent-issue-to-pr.md | 14 ++- .../inventory/group_vars/k3s_prod/vars.yml | 23 ++++ platform/pm-agent/README.md | 48 ++++++-- .../pm-agent-credentials.sops.yaml.example | 9 ++ runbooks/pm-agent.md | 2 +- 6 files changed, 189 insertions(+), 18 deletions(-) create mode 100644 decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md diff --git a/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md new file mode 100644 index 0000000..623d7d1 --- /dev/null +++ b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md @@ -0,0 +1,111 @@ +--- +date: 2026-10-06 +branch: docs/pm-agent-registry-pull-prereq +title: The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin +--- + +## What changed + +The image pull for `pm-agent` failed TLS verification, and the previous entry on this branch +honestly recorded it as **not diagnosed**. The operator asked for the diagnosis before any +registry design. This is it, and the fix turns out to be a pattern this repository already +has. + +`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins +`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the +Keycloak issuer pin that is there for the identical reason. **An operator must run the +`node_baseline` play for it to reach the nodes** — writing `/etc/hosts` on a node is not +something Flux or an agent can do. + +The docs that described this as an undiagnosed blocker now describe the cause and the fix: +`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and +the play), and `docs/plans/pm-agent-issue-to-pr.md`. + +Also recorded, in the two places an operator reads when creating the credential: **the agent +authenticates to Gitea with a local account and a scoped PAT, by decision** (operator, +2026-10-06), not with a federated identity. Written down because the alternative looks +obviously better until you check whether it is possible — see below. + +## What I verified + +Every step measured, read-only, in this order. + +**The failing certificate is the router's, not the Gateway's.** + +- containerd's error names TLS, not the digest: `tls: failed to verify certificate: x509: + certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev`. + "not valid for **any** names" means the presented certificate carries no SANs at all. +- `192.168.2.20` is the shared Gateway's own LoadBalancer VIP + (`gateway/cilium-gateway-shared`, from the Cilium pool `192.168.2.20-30`), and its `https` + listener serves `*.home.keithjackson.dev` with `wildcard-home-tls`. From the operator's + workstation the name resolves there and serves the real Let's Encrypt + `CN=*.home.keithjackson.dev` with correct SANs. **So the certificate and the Gateway are + healthy.** +- **The node asks a different resolver and gets a different answer.** Prod's + `node_baseline_dns_servers` is the LAN gateway, not Pi-hole — deliberately, as that file + explains. Public DNS for this zone is a CNAME to `wan.home.keithjackson.dev`: measured via + both `1.1.1.1` and `8.8.8.8`, `gitea.home.keithjackson.dev` → + `wan.home.keithjackson.dev.` → `136.51.115.252`, and the apex answers the same. +- **That address serves a certificate with no names.** Measured from the workstation against + the WAN address with the right SNI: + + subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D + issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D + No extensions in certificate + + No extensions means no `subjectAltName`, which is exactly "not valid for any names". The + `CN=%D` is the router's placeholder. + +**This repository had already measured the same thing, for Keycloak, and I did not need to +re-derive it.** `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` says, above the pin: +"Since platform/gateway/gateway.yaml set an external-dns `target`, every public name is a +CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from +the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from +k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18)." Same cause, same symptom, +one name later. The pin exists precisely for node processes that must reach the Gateway +directly; containerd is one, and nobody had added it. + +**The parity the pin has to satisfy:** `192.168.2.20` equals the live +`GATEWAY_VIP`, and `k3s_base_domain` equals the live `BASE_DOMAIN` — both read from the +cluster, not from the comment that claims prod serves `prod.home.keithjackson.dev` (it does +not; `flux-system/cluster-vars` reports `BASE_DOMAIN=home.keithjackson.dev` and +`GATEWAY_VIP=192.168.2.20`, and Ansible's `k3s_base_domain` is `home.keithjackson.dev`). +That comment is stale history. + +**The credential decision, and why the obvious alternative is not available.** Gitea's API +accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2 +provider Gitea implements **only the Authorization Code Grant** (plus PKCE/OIDC), so there +is no client-credentials grant, and nothing in Gitea validates a token issued by a +third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to +Gitea's API or to git-over-HTTP. Read from Gitea's own documentation +(`docs.gitea.com/development/api-usage`, `/development/oauth2-provider`), not assumed. + +**Gate:** `./ci/gate.sh` in the pinned toolbox from a §0 worktree — see the branch's run; rc 3 +with `gitea-settings` the only stage that did not run. + +## What I did NOT verify + +- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the + nodes is written by the `node_baseline` play, which is an operator action. Until that runs + and an image exists, "the node now reaches the Gateway" is a design claim. The evidence + that it *will* is the Keycloak pin working for the same cause — which I also did not + re-measure. +- **Why public DNS answers at all for an `internal` service.** `catalog/gitea.yaml` sets no + `exposure`, so it inherits `internal`, and the renderer contract says an internal service's + HTTPRoute carries `external-dns.alpha.kubernetes.io/controller: none` and publishes no + public record. Yet `gitea.home.keithjackson.dev` resolves publicly to the WAN CNAME. I did + not chase that: it may be a zone-level wildcard rather than a published record, and it did + not change the diagnosis. It is worth someone's attention, because "internal services + publish no public DNS" is an L0-checked claim and this looks like a case where the *name* + resolves regardless of what the renderer withheld. +- **Registry authentication.** Unchanged and still open: an unauthenticated client gets + nothing from this registry, and no `imagePullSecret` exists anywhere in the tree. With the + credential model now decided (local Gitea accounts with PATs) the shape is a + `dockerconfigjson` for an account whose token carries `read:package`, referenced from the + Deployment and the committed worker Job template — and the worker Job template's shape is + pinned by admission, so whether the policy should also pin `imagePullSecrets` deserves a + deliberate look rather than a quiet addition. Not built: minting that token is an operator + action, and I did not want to guess at a second credential's scope. +- **Anything about runtime.** The image still does not exist. No dispatcher has + authenticated, no issue has been triaged, no worker Job has been created, and the admission + policy — live, `Enforce`, six rules — has still never admitted or refused a real Job. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index 37d7830..567ac69 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -236,12 +236,14 @@ about runtime behaviour in this document is a design claim, not a measurement. 1. **Make the registry pullable, THEN build, push, pin.** Measured after the merge, and neither half is declared anywhere in this repository (`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`): - (i) the cluster's nodes cannot verify the registry's TLS certificate — containerd reports - `x509: certificate is not valid for any names` for `gitea.home.keithjackson.dev`, while - that name serves a healthy Let's Encrypt wildcard from the LAN; (ii) an unauthenticated - client gets nothing from that registry and no `imagePullSecret` exists in the tree, so - either one is added or anonymous package reads are allowed. Only then does the digest - matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both + (i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the + router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning + `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same + pin the Keycloak issuer already carries; an operator must run the `node_baseline` play; + (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with + the credential decision now being local Gitea accounts with PATs, that means a + `dockerconfigjson` for an account whose token carries `read:package`. Only then does the + digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both placeholder digests. Expect the build itself to need iteration too: the nodejs module stream and the npm install have never run against this base. The same two blockers sit in front of `conformance-runner`, which shares the registry path and has never been pulled. diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index 4813196..2b865a6 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -327,6 +327,29 @@ node_baseline_hosts_pins: - ip: "192.168.2.20" names: - "keycloak.{{ k3s_base_domain }}" + # gitea: the SECOND node process to need this pin, for the identical reason, found + # 2026-10-06. containerd pulls images as a NODE process, so it resolves through the + # public upstreams above and gets the CNAME to wan.home.keithjackson.dev — the WAN IP, + # which this router does not hairpin. The pull then fails on the router's own + # placeholder certificate rather than on anything to do with the image: + # + # failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:…": + # tls: failed to verify certificate: x509: certificate is not valid for any names, + # but wanted to match gitea.home.keithjackson.dev + # + # Measured from the workstation the same day, against the WAN address with that SNI: + # subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D + # No extensions in certificate <- no SANs at all, hence "not valid for ANY names" + # while the same name via PiHole answers 192.168.2.20 and serves the real + # Let's Encrypt *.home.keithjackson.dev. So the certificate and the Gateway are fine; + # only the node's ANSWER is wrong. Same root cause, same fix, as the issuer pin above. + # + # WHO NEEDS IT: any workload whose image lives in the in-cluster Gitea registry — + # today platform/pm-agent (agent-runner) and conformance/incluster + # (conformance-runner), neither of which has ever been pulled. A pull is still + # UNAUTHENTICATED after this pin, which is a separate thing to settle; this only makes + # the node reach the right endpoint. + - "gitea.{{ k3s_base_domain }}" # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) -------------------- # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/ diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index ec68fb2..baedb1d 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -22,6 +22,25 @@ requires. --- +## Why the agent authenticates to Gitea with a local account and a PAT + +Not Keycloak, and that is a decision rather than an omission (operator, 2026-10-06): the +agents only ever talk to Gitea, so a local Gitea account with a scoped personal access token +is the whole requirement, and it is drastically simpler than federating a machine identity. + +It is also what Gitea can actually do. Keycloak service accounts were considered and are not +available for this: Gitea's API accepts basic auth, PATs, SSH signatures and **its own** +OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization +Code Grant** — there is no client-credentials grant, and nothing in Gitea validates a token +issued by a third-party OIDC provider. So a Keycloak service-account token cannot +authenticate to Gitea's API or to git-over-HTTP at all. Keycloak remains what it is for human +access: the browser login source registered on Gitea (`catalog/gitea.yaml`). + +What still matters, and is enforced by the hand-off rather than by Keycloak: **the account +must not be an administrator** — `infra/gitea/approval-policy.json` authorises +`/approve-override` on live repository permission alone, so an admin-permission bot could +waive the sensitive-path human review on its own pull request. + ## It does nothing at all until an operator does two things Both are deliberate, and both fail **visibly** rather than silently: @@ -38,17 +57,24 @@ Both are deliberate, and both fail **visibly** rather than silently: path. Details and the exact commands: [`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md). - - **The nodes must be able to verify the registry's TLS certificate.** containerd - currently fails with `x509: certificate is not valid for any names, but wanted to match - gitea.home.keithjackson.dev`, while the same name from the LAN serves a healthy - Let's Encrypt `*.home.keithjackson.dev`. Correcting the digest will not clear this. - - **The pull must be authenticated.** An unauthenticated client gets nothing from this - Gitea's registry (its anonymous token carries no access scope; every path 401s), and - there is no `imagePullSecret` anywhere in the tree. Either add one — a - `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the - Deployment *and* from the committed worker Job template — or allow anonymous reads of - that package in Gitea, which makes the image readable to anything that can reach the - host. That is a posture decision, so it is yours rather than an agent's. + - **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is + the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a + *node* process, so it resolves through the node's public upstreams + (`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public + answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN + IP — which this router does not hairpin; it answers :443 with its own placeholder + certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely + `x509: certificate is not valid for any names`. The Gateway, the wildcard certificate + and the digest were never implicated. The fix is the pin this repository already uses + for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now + in `node_baseline_hosts_pins`. **An operator must run the `node_baseline` play for it + to reach the nodes.** + - **The pull is still unauthenticated** once the node reaches the right endpoint. An + unauthenticated client gets nothing from this Gitea's registry, and there is no + `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision + (local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea + account whose token carries `read:package`, referenced from the Deployment and the + committed worker Job template — not built yet, because minting that token is yours. The same two apply to `conformance-runner`: same registry path, same push-then-pin model, never pulled either. diff --git a/platform/pm-agent/pm-agent-credentials.sops.yaml.example b/platform/pm-agent/pm-agent-credentials.sops.yaml.example index a7fe9fd..dd2a601 100644 --- a/platform/pm-agent/pm-agent-credentials.sops.yaml.example +++ b/platform/pm-agent/pm-agent-credentials.sops.yaml.example @@ -51,6 +51,15 @@ # (2) MINT THE GITEA TOKEN as that user (log in as `pm-agent`, or use the admin API to # create a token for it): Settings -> Applications -> Generate New Token. # +# A LOCAL GITEA ACCOUNT WITH A PAT IS THE DECIDED MODEL (operator, 2026-10-06), not +# Keycloak. The agents only ever talk to Gitea, so this is the whole requirement and it +# is far simpler than federating a machine identity. It is also the only thing that +# works: Gitea's API accepts basic auth, PATs, SSH signatures and its OWN OAuth2 bearer +# tokens, and as a provider Gitea implements only the Authorization Code Grant — no +# client-credentials grant, and nothing that validates a third-party OIDC token. A +# Keycloak service-account token therefore cannot authenticate to Gitea's API or to +# git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that. +# # Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT # `write:admin`, NOT `write:organization`, NOT `write:user`. # diff --git a/runbooks/pm-agent.md b/runbooks/pm-agent.md index 4a6a5f6..3a4c759 100644 --- a/runbooks/pm-agent.md +++ b/runbooks/pm-agent.md @@ -68,7 +68,7 @@ Work through these in order; the first three are by far the most likely. |---|---|---| | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` | | `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first | -| `ImagePullBackOff` with `x509: certificate is not valid for any names` | the **node** cannot verify the registry's TLS certificate. Nothing to do with the digest: a correct digest fails identically. The same name serves a healthy Let's Encrypt wildcard from the LAN, so what the node reaches is not what you reach | not diagnosed — see `decisions/2026-10-05-pm-agent-registry-pull-prereq.md` for everything measured. Check the nodes' resolution of `gitea.home.keithjackson.dev` and their registry trust (`/etc/rancher/k3s/registries.yaml`), neither of which this repository declares | +| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. **Run the `node_baseline` play** for it to reach the nodes. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | | `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours | | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` | | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |