--- date: 2026-10-06 branch: docs/pm-agent-registry-pull-prereq title: The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin --- ## What changed The image pull for `pm-agent` failed TLS verification, and the previous entry on this branch honestly recorded it as **not diagnosed**. The operator asked for the diagnosis before any registry design. This is it, and the fix turns out to be a pattern this repository already has. `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the Keycloak issuer pin that is there for the identical reason. **It reaches the nodes the way every other node setting in this project does — through Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs, OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the *only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command: cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \ -e k3s_hosts=k3s_prod --tags hosts `--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other baseline setting. Running the play is an operator action (it has become/root on prod), but the configuration itself is declared, reviewed and version-controlled — not typed on a node. The docs that described this as an undiagnosed blocker now describe the cause and the fix: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and the play), and `docs/plans/pm-agent-issue-to-pr.md`. Also recorded, in the two places an operator reads when creating the credential: **the agent authenticates to Gitea with a local account and a scoped PAT, by decision** (operator, 2026-10-06), not with a federated identity. Written down because the alternative looks obviously better until you check whether it is possible — see below. ## What I verified Every step measured, read-only, in this order. **The failing certificate is the router's, not the Gateway's.** - containerd's error names TLS, not the digest: `tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev`. "not valid for **any** names" means the presented certificate carries no SANs at all. - `192.168.2.20` is the shared Gateway's own LoadBalancer VIP (`gateway/cilium-gateway-shared`, from the Cilium pool `192.168.2.20-30`), and its `https` listener serves `*.home.keithjackson.dev` with `wildcard-home-tls`. From the operator's workstation the name resolves there and serves the real Let's Encrypt `CN=*.home.keithjackson.dev` with correct SANs. **So the certificate and the Gateway are healthy.** - **The node asks a different resolver and gets a different answer.** Prod's `node_baseline_dns_servers` is the LAN gateway, not Pi-hole — deliberately, as that file explains. Public DNS for this zone is a CNAME to `wan.home.keithjackson.dev`: measured via both `1.1.1.1` and `8.8.8.8`, `gitea.home.keithjackson.dev` → `wan.home.keithjackson.dev.` → `136.51.115.252`, and the apex answers the same. - **That address serves a certificate with no names.** Measured from the workstation against the WAN address with the right SNI: subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D No extensions in certificate No extensions means no `subjectAltName`, which is exactly "not valid for any names". The `CN=%D` is the router's placeholder. **This repository had already measured the same thing, for Keycloak, and I did not need to re-derive it.** `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` says, above the pin: "Since platform/gateway/gateway.yaml set an external-dns `target`, every public name is a CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18)." Same cause, same symptom, one name later. The pin exists precisely for node processes that must reach the Gateway directly; containerd is one, and nobody had added it. **The parity the pin has to satisfy:** `192.168.2.20` equals the live `GATEWAY_VIP`, and `k3s_base_domain` equals the live `BASE_DOMAIN` — both read from the cluster, not from the comment that claims prod serves `prod.home.keithjackson.dev` (it does not; `flux-system/cluster-vars` reports `BASE_DOMAIN=home.keithjackson.dev` and `GATEWAY_VIP=192.168.2.20`, and Ansible's `k3s_base_domain` is `home.keithjackson.dev`). That comment is stale history. **The credential decision, and why the obvious alternative is not available.** Gitea's API accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2 provider Gitea implements **only the Authorization Code Grant** (plus PKCE/OIDC), so there is no client-credentials grant, and nothing in Gitea validates a token issued by a third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to Gitea's API or to git-over-HTTP. Read from Gitea's own documentation (`docs.gitea.com/development/api-usage`, `/development/oauth2-provider`), not assumed. **Gate:** `./ci/gate.sh` in the pinned toolbox from a §0 worktree — see the branch's run; rc 3 with `gitea-settings` the only stage that did not run. ## What I did NOT verify - **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml` renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and `state: absent` when the list empties, so a removed entry really leaves the node. Until that runs and an image exists, "the node now reaches the Gateway" is a design claim. The evidence that it *will* is the Keycloak pin working for the same cause — which I also did not re-measure. - **Why public DNS answers at all for an `internal` service.** `catalog/gitea.yaml` sets no `exposure`, so it inherits `internal`, and the renderer contract says an internal service's HTTPRoute carries `external-dns.alpha.kubernetes.io/controller: none` and publishes no public record. Yet `gitea.home.keithjackson.dev` resolves publicly to the WAN CNAME. I did not chase that: it may be a zone-level wildcard rather than a published record, and it did not change the diagnosis. It is worth someone's attention, because "internal services publish no public DNS" is an L0-checked claim and this looks like a case where the *name* resolves regardless of what the renderer withheld. - **Registry authentication.** Unchanged and still open: an unauthenticated client gets nothing from this registry, and no `imagePullSecret` exists anywhere in the tree. With the credential model now decided (local Gitea accounts with PATs) the shape is a `dockerconfigjson` for an account whose token carries `read:package`, referenced from the Deployment and the committed worker Job template — and the worker Job template's shape is pinned by admission, so whether the policy should also pin `imagePullSecrets` deserves a deliberate look rather than a quiet addition. Not built: minting that token is an operator action, and I did not want to guess at a second credential's scope. - **Anything about runtime.** The image still does not exist. No dispatcher has authenticated, no issue has been triaged, no worker Job has been created, and the admission policy — live, `Enforce`, six rules — has still never admitted or refused a real Job.