home-network-k3s-impl/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md

date, branch, title

date	branch	title
2026-10-06	docs/pm-agent-registry-pull-prereq	The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin

What changed

The image pull for pm-agent failed TLS verification, and the previous entry on this branch honestly recorded it as not diagnosed. The operator asked for the diagnosis before any registry design. This is it, and the fix turns out to be a pattern this repository already has.

infra/ansible/inventory/group_vars/k3s_prod/vars.yml now pins gitea.{{ k3s_base_domain }} → 192.168.2.20 in node_baseline_hosts_pins, beside the Keycloak issuer pin that is there for the identical reason.

It reaches the nodes the way every other node setting in this project does — through Ansible, from the inventory. Node configuration is Ansible's (and, for the dev VMs, OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the only surface: infra/PROD-PROVISIONING.md records that prod is 3× bare metal and that the dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command:

cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
  -e k3s_hosts=k3s_prod --tags hosts

--tags hosts runs ONLY the /etc/hosts task, so applying this pin cannot drift any other baseline setting. Running the play is an operator action (it has become/root on prod), but the configuration itself is declared, reviewed and version-controlled — not typed on a node.

The docs that described this as an undiagnosed blocker now describe the cause and the fix: platform/pm-agent/README.md, runbooks/pm-agent.md (its symptom row now names the pin and the play), and docs/plans/pm-agent-issue-to-pr.md.

Also recorded, in the two places an operator reads when creating the credential: the agent authenticates to Gitea with a local account and a scoped PAT, by decision (operator, 2026-10-06), not with a federated identity. Written down because the alternative looks obviously better until you check whether it is possible — see below.

What I verified

Every step measured, read-only, in this order.

The failing certificate is the router's, not the Gateway's.

• containerd's error names TLS, not the digest: tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev. "not valid for any names" means the presented certificate carries no SANs at all.

• 192.168.2.20 is the shared Gateway's own LoadBalancer VIP (gateway/cilium-gateway-shared, from the Cilium pool 192.168.2.20-30), and its https listener serves *.home.keithjackson.dev with wildcard-home-tls. From the operator's workstation the name resolves there and serves the real Let's Encrypt CN=*.home.keithjackson.dev with correct SANs. So the certificate and the Gateway are healthy.

• The node asks a different resolver and gets a different answer. Prod's node_baseline_dns_servers is the LAN gateway, not Pi-hole — deliberately, as that file explains. Public DNS for this zone is a CNAME to wan.home.keithjackson.dev: measured via both 1.1.1.1 and 8.8.8.8, gitea.home.keithjackson.dev → wan.home.keithjackson.dev. → 136.51.115.252, and the apex answers the same.

• That address serves a certificate with no names. Measured from the workstation against the WAN address with the right SNI:

  subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
  issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
  No extensions in certificate
  

  No extensions means no subjectAltName, which is exactly "not valid for any names". The CN=%D is the router's placeholder.

This repository had already measured the same thing, for Keycloak, and I did not need to re-derive it. infra/ansible/inventory/group_vars/k3s_prod/vars.yml says, above the pin: "Since platform/gateway/gateway.yaml set an external-dns target, every public name is a CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from k3s-prod-2/-3 2026-10-03: curl exit 60, ssl_verify_result=18)." Same cause, same symptom, one name later. The pin exists precisely for node processes that must reach the Gateway directly; containerd is one, and nobody had added it.

The parity the pin has to satisfy: 192.168.2.20 equals the live GATEWAY_VIP, and k3s_base_domain equals the live BASE_DOMAIN — both read from the cluster, not from the comment that claims prod serves prod.home.keithjackson.dev (it does not; flux-system/cluster-vars reports BASE_DOMAIN=home.keithjackson.dev and GATEWAY_VIP=192.168.2.20, and Ansible's k3s_base_domain is home.keithjackson.dev). That comment is stale history.

The credential decision, and why the obvious alternative is not available. Gitea's API accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2 provider Gitea implements only the Authorization Code Grant (plus PKCE/OIDC), so there is no client-credentials grant, and nothing in Gitea validates a token issued by a third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to Gitea's API or to git-over-HTTP. Read from Gitea's own documentation (docs.gitea.com/development/api-usage, /development/oauth2-provider), not assumed.

Gate: ./ci/gate.sh in the pinned toolbox from a §0 worktree — see the branch's run; rc 3 with gitea-settings the only stage that did not run.

What I did NOT verify

• That the pin fixes the pull. It is IaC, and nothing has applied it: /etc/hosts on the nodes is written by the node_baseline play's hosts-tagged task, which is an operator action. I did confirm the mechanism is not inert — roles/node_baseline/tasks/main.yml renders node_baseline_hosts_pins into a marked blockinfile with backup: true, and state: absent when the list empties, so a removed entry really leaves the node. Until that runs and an image exists, "the node now reaches the Gateway" is a design claim. The evidence that it will is the Keycloak pin working for the same cause — which I also did not re-measure.
• Why public DNS answers at all for an internal service. catalog/gitea.yaml sets no exposure, so it inherits internal, and the renderer contract says an internal service's HTTPRoute carries external-dns.alpha.kubernetes.io/controller: none and publishes no public record. Yet gitea.home.keithjackson.dev resolves publicly to the WAN CNAME. I did not chase that: it may be a zone-level wildcard rather than a published record, and it did not change the diagnosis. It is worth someone's attention, because "internal services publish no public DNS" is an L0-checked claim and this looks like a case where the name resolves regardless of what the renderer withheld.
• Registry authentication. Unchanged and still open: an unauthenticated client gets nothing from this registry, and no imagePullSecret exists anywhere in the tree. With the credential model now decided (local Gitea accounts with PATs) the shape is a dockerconfigjson for an account whose token carries read:package, referenced from the Deployment and the committed worker Job template — and the worker Job template's shape is pinned by admission, so whether the policy should also pin imagePullSecrets deserves a deliberate look rather than a quiet addition. Not built: minting that token is an operator action, and I did not want to guess at a second credential's scope.
• Anything about runtime. The image still does not exist. No dispatcher has authenticated, no issue has been triaged, no worker Job has been created, and the admission policy — live, Enforce, six rules — has still never admitted or refused a real Job.