home-network-k3s-impl/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md

---
date: 2026-10-06
branch: docs/pm-agent-registry-pull-prereq
title: The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin
---

## What changed

The image pull for `pm-agent` failed TLS verification, and the previous entry on this branch
honestly recorded it as **not diagnosed**. The operator asked for the diagnosis before any
registry design. This is it, and the fix turns out to be a pattern this repository already
has.

`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins
`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the
Keycloak issuer pin that is there for the identical reason.

**It reaches the nodes the way every other node setting in this project does — through
Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs,
OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the
*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the
dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no
cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command:

    cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
      -e k3s_hosts=k3s_prod --tags hosts

`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other
baseline setting. Running the play is an operator action (it has become/root on prod), but
the configuration itself is declared, reviewed and version-controlled — not typed on a node.

The docs that described this as an undiagnosed blocker now describe the cause and the fix:
`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and
the play), and `docs/plans/pm-agent-issue-to-pr.md`.

Also recorded, in the two places an operator reads when creating the credential: **the agent
authenticates to Gitea with a local account and a scoped PAT, by decision** (operator,
2026-10-06), not with a federated identity. Written down because the alternative looks
obviously better until you check whether it is possible — see below.

## What I verified

Every step measured, read-only, in this order.

**The failing certificate is the router's, not the Gateway's.**

- containerd's error names TLS, not the digest: `tls: failed to verify certificate: x509:
  certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev`.
  "not valid for **any** names" means the presented certificate carries no SANs at all.
- `192.168.2.20` is the shared Gateway's own LoadBalancer VIP
  (`gateway/cilium-gateway-shared`, from the Cilium pool `192.168.2.20-30`), and its `https`
  listener serves `*.home.keithjackson.dev` with `wildcard-home-tls`. From the operator's
  workstation the name resolves there and serves the real Let's Encrypt
  `CN=*.home.keithjackson.dev` with correct SANs. **So the certificate and the Gateway are
  healthy.**
- **The node asks a different resolver and gets a different answer.** Prod's
  `node_baseline_dns_servers` is the LAN gateway, not Pi-hole — deliberately, as that file
  explains. Public DNS for this zone is a CNAME to `wan.home.keithjackson.dev`: measured via
  both `1.1.1.1` and `8.8.8.8`, `gitea.home.keithjackson.dev` →
  `wan.home.keithjackson.dev.` → `136.51.115.252`, and the apex answers the same.
- **That address serves a certificate with no names.** Measured from the workstation against
  the WAN address with the right SNI:

      subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
      issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
      No extensions in certificate

  No extensions means no `subjectAltName`, which is exactly "not valid for any names". The
  `CN=%D` is the router's placeholder.

**This repository had already measured the same thing, for Keycloak, and I did not need to
re-derive it.** `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` says, above the pin:
"Since platform/gateway/gateway.yaml set an external-dns `target`, every public name is a
CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from
the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18)." Same cause, same symptom,
one name later. The pin exists precisely for node processes that must reach the Gateway
directly; containerd is one, and nobody had added it.

**The parity the pin has to satisfy:** `192.168.2.20` equals the live
`GATEWAY_VIP`, and `k3s_base_domain` equals the live `BASE_DOMAIN` — both read from the
cluster, not from the comment that claims prod serves `prod.home.keithjackson.dev` (it does
not; `flux-system/cluster-vars` reports `BASE_DOMAIN=home.keithjackson.dev` and
`GATEWAY_VIP=192.168.2.20`, and Ansible's `k3s_base_domain` is `home.keithjackson.dev`).
That comment is stale history.

**The credential decision, and why the obvious alternative is not available.** Gitea's API
accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2
provider Gitea implements **only the Authorization Code Grant** (plus PKCE/OIDC), so there
is no client-credentials grant, and nothing in Gitea validates a token issued by a
third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to
Gitea's API or to git-over-HTTP. Read from Gitea's own documentation
(`docs.gitea.com/development/api-usage`, `/development/oauth2-provider`), not assumed.

**Gate:** `./ci/gate.sh` in the pinned toolbox from a §0 worktree — see the branch's run; rc 3
with `gitea-settings` the only stage that did not run.

## What I did NOT verify

- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the
  nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator
  action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml`
  renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and
  `state: absent` when the list empties, so a removed entry really leaves the node. Until that runs
  and an image exists, "the node now reaches the Gateway" is a design claim. The evidence
  that it *will* is the Keycloak pin working for the same cause — which I also did not
  re-measure.
- **Why public DNS answers at all for an `internal` service.** `catalog/gitea.yaml` sets no
  `exposure`, so it inherits `internal`, and the renderer contract says an internal service's
  HTTPRoute carries `external-dns.alpha.kubernetes.io/controller: none` and publishes no
  public record. Yet `gitea.home.keithjackson.dev` resolves publicly to the WAN CNAME. I did
  not chase that: it may be a zone-level wildcard rather than a published record, and it did
  not change the diagnosis. It is worth someone's attention, because "internal services
  publish no public DNS" is an L0-checked claim and this looks like a case where the *name*
  resolves regardless of what the renderer withheld.
- **Registry authentication.** Unchanged and still open: an unauthenticated client gets
  nothing from this registry, and no `imagePullSecret` exists anywhere in the tree. With the
  credential model now decided (local Gitea accounts with PATs) the shape is a
  `dockerconfigjson` for an account whose token carries `read:package`, referenced from the
  Deployment and the committed worker Job template — and the worker Job template's shape is
  pinned by admission, so whether the policy should also pin `imagePullSecrets` deserves a
  deliberate look rather than a quiet addition. Not built: minting that token is an operator
  action, and I did not want to guess at a second credential's scope.
- **Anything about runtime.** The image still does not exist. No dispatcher has
  authenticated, no issue has been triaged, no worker Job has been created, and the admission
  policy — live, `Enforce`, six rules — has still never admitted or refused a real Job.