fix(node): pin gitea to the Gateway VIP — containerd was resolving it to the non-hairpinned WAN
gate / gate (pull_request) Successful in 1m52s
gate / conformance (pull_request) Has been skipped

The pm-agent image pull failed TLS verification, and the previous entry on this branch
recorded it honestly as not diagnosed. Diagnosed now, and it is not about the image.

containerd pulls as a NODE process, so it resolves through node_baseline_dns_servers (the
LAN gateway, deliberately not PiHole) and gets the public answer: a CNAME to
wan.home.keithjackson.dev -> 136.51.115.252, measured via both 1.1.1.1 and 8.8.8.8. This
router does not hairpin, so WAN:443 answers with its own placeholder certificate —
measured with the right SNI from the workstation:

    subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
    No extensions in certificate

No extensions means no subjectAltName, which is exactly 'certificate is not valid for any
names'. Meanwhile 192.168.2.20 IS the shared Gateway's VIP and its https listener serves
the real Let's Encrypt *.home.keithjackson.dev. The Gateway, the wildcard cert and the
digest were never implicated — only the node's answer.

This file already documents the identical failure for the Keycloak issuer, measured from
k3s-prod-2/-3 on 2026-10-03 (curl exit 60, ssl_verify_result=18), and already carries the
mechanism that fixes it. gitea is simply the second node process to need it; containerd is
a node process and nobody had added it. An operator must run the node_baseline play.

Also records, where an operator reads them, that the agent authenticates to Gitea with a
local account and a scoped PAT by decision — and why the federated alternative is not
merely unchosen but unavailable: Gitea implements only the Authorization Code Grant and
validates no third-party OIDC token, so a Keycloak service-account token cannot
authenticate to its API or to git-over-HTTP.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-06 00:35:18 -05:00
1 parent a209a618fd
commit 523a463fd7
6 files changed
+189 -18

No files matched your search

+37 -11
View File
@@ -22,6 +22,25 @@ requires.
---
## Why the agent authenticates to Gitea with a local account and a PAT
Not Keycloak, and that is a decision rather than an omission (operator, 2026-10-06): the
agents only ever talk to Gitea, so a local Gitea account with a scoped personal access token
is the whole requirement, and it is drastically simpler than federating a machine identity.
It is also what Gitea can actually do. Keycloak service accounts were considered and are not
available for this: Gitea's API accepts basic auth, PATs, SSH signatures and **its own**
OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization
Code Grant** — there is no client-credentials grant, and nothing in Gitea validates a token
issued by a third-party OIDC provider. So a Keycloak service-account token cannot
authenticate to Gitea's API or to git-over-HTTP at all. Keycloak remains what it is for human
access: the browser login source registered on Gitea (`catalog/gitea.yaml`).
What still matters, and is enforced by the hand-off rather than by Keycloak: **the account
must not be an administrator** — `infra/gitea/approval-policy.json` authorises
`/approve-override` on live repository permission alone, so an admin-permission bot could
waive the sensitive-path human review on its own pull request.
## It does nothing at all until an operator does two things
Both are deliberate, and both fail **visibly** rather than silently:
@@ -38,17 +57,24 @@ Both are deliberate, and both fail **visibly** rather than silently:
path. Details and the exact commands:
[`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md).
- **The nodes must be able to verify the registry's TLS certificate.** containerd
currently fails with `x509: certificate is not valid for any names, but wanted to match
gitea.home.keithjackson.dev`, while the same name from the LAN serves a healthy
Let's Encrypt `*.home.keithjackson.dev`. Correcting the digest will not clear this.
- **The pull must be authenticated.** An unauthenticated client gets nothing from this
Gitea's registry (its anonymous token carries no access scope; every path 401s), and
there is no `imagePullSecret` anywhere in the tree. Either add one — a
`dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the
Deployment *and* from the committed worker Job template — or allow anonymous reads of
that package in Gitea, which makes the image readable to anything that can reach the
host. That is a posture decision, so it is yours rather than an agent's.
- **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
*node* process, so it resolves through the node's public upstreams
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
IP — which this router does not hairpin; it answers :443 with its own placeholder
certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
and the digest were never implicated. The fix is the pin this repository already uses
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
in `node_baseline_hosts_pins`. **An operator must run the `node_baseline` play for it
to reach the nodes.**
- **The pull is still unauthenticated** once the node reaches the right endpoint. An
unauthenticated client gets nothing from this Gitea's registry, and there is no
`imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
(local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea
account whose token carries `read:package`, referenced from the Deployment and the
committed worker Job template — not built yet, because minting that token is yours.
The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
never pulled either.
@@ -51,6 +51,15 @@
# (2) MINT THE GITEA TOKEN as that user (log in as `pm-agent`, or use the admin API to
# create a token for it): Settings -> Applications -> Generate New Token.
#
# A LOCAL GITEA ACCOUNT WITH A PAT IS THE DECIDED MODEL (operator, 2026-10-06), not
# Keycloak. The agents only ever talk to Gitea, so this is the whole requirement and it
# is far simpler than federating a machine identity. It is also the only thing that
# works: Gitea's API accepts basic auth, PATs, SSH signatures and its OWN OAuth2 bearer
# tokens, and as a provider Gitea implements only the Authorization Code Grant — no
# client-credentials grant, and nothing that validates a third-party OIDC token. A
# Keycloak service-account token therefore cannot authenticate to Gitea's API or to
# git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that.
#
# Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT
# `write:admin`, NOT `write:organization`, NOT `write:user`.
#