fix(node): pin gitea to the Gateway VIP — containerd was resolving it to the non-hairpinned WAN
The pm-agent image pull failed TLS verification, and the previous entry on this branch
recorded it honestly as not diagnosed. Diagnosed now, and it is not about the image.
containerd pulls as a NODE process, so it resolves through node_baseline_dns_servers (the
LAN gateway, deliberately not PiHole) and gets the public answer: a CNAME to
wan.home.keithjackson.dev -> 136.51.115.252, measured via both 1.1.1.1 and 8.8.8.8. This
router does not hairpin, so WAN:443 answers with its own placeholder certificate —
measured with the right SNI from the workstation:
subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
No extensions in certificate
No extensions means no subjectAltName, which is exactly 'certificate is not valid for any
names'. Meanwhile 192.168.2.20 IS the shared Gateway's VIP and its https listener serves
the real Let's Encrypt *.home.keithjackson.dev. The Gateway, the wildcard cert and the
digest were never implicated — only the node's answer.
This file already documents the identical failure for the Keycloak issuer, measured from
k3s-prod-2/-3 on 2026-10-03 (curl exit 60, ssl_verify_result=18), and already carries the
mechanism that fixes it. gitea is simply the second node process to need it; containerd is
a node process and nobody had added it. An operator must run the node_baseline play.
Also records, where an operator reads them, that the agent authenticates to Gitea with a
local account and a scoped PAT by decision — and why the federated alternative is not
merely unchosen but unavailable: Gitea implements only the Authorization Code Grant and
validates no third-party OIDC token, so a Keycloak service-account token cannot
authenticate to its API or to git-over-HTTP.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
a209a618fd
commit
523a463fd7
6 files changed
+189
-18
No files matched your search
+37
-11
@@ -22,6 +22,25 @@ requires.
|
||||
|
||||
---
|
||||
|
||||
## Why the agent authenticates to Gitea with a local account and a PAT
|
||||
|
||||
Not Keycloak, and that is a decision rather than an omission (operator, 2026-10-06): the
|
||||
agents only ever talk to Gitea, so a local Gitea account with a scoped personal access token
|
||||
is the whole requirement, and it is drastically simpler than federating a machine identity.
|
||||
|
||||
It is also what Gitea can actually do. Keycloak service accounts were considered and are not
|
||||
available for this: Gitea's API accepts basic auth, PATs, SSH signatures and **its own**
|
||||
OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization
|
||||
Code Grant** — there is no client-credentials grant, and nothing in Gitea validates a token
|
||||
issued by a third-party OIDC provider. So a Keycloak service-account token cannot
|
||||
authenticate to Gitea's API or to git-over-HTTP at all. Keycloak remains what it is for human
|
||||
access: the browser login source registered on Gitea (`catalog/gitea.yaml`).
|
||||
|
||||
What still matters, and is enforced by the hand-off rather than by Keycloak: **the account
|
||||
must not be an administrator** — `infra/gitea/approval-policy.json` authorises
|
||||
`/approve-override` on live repository permission alone, so an admin-permission bot could
|
||||
waive the sensitive-path human review on its own pull request.
|
||||
|
||||
## It does nothing at all until an operator does two things
|
||||
|
||||
Both are deliberate, and both fail **visibly** rather than silently:
|
||||
@@ -38,17 +57,24 @@ Both are deliberate, and both fail **visibly** rather than silently:
|
||||
path. Details and the exact commands:
|
||||
[`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md).
|
||||
|
||||
- **The nodes must be able to verify the registry's TLS certificate.** containerd
|
||||
currently fails with `x509: certificate is not valid for any names, but wanted to match
|
||||
gitea.home.keithjackson.dev`, while the same name from the LAN serves a healthy
|
||||
Let's Encrypt `*.home.keithjackson.dev`. Correcting the digest will not clear this.
|
||||
- **The pull must be authenticated.** An unauthenticated client gets nothing from this
|
||||
Gitea's registry (its anonymous token carries no access scope; every path 401s), and
|
||||
there is no `imagePullSecret` anywhere in the tree. Either add one — a
|
||||
`dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the
|
||||
Deployment *and* from the committed worker Job template — or allow anonymous reads of
|
||||
that package in Gitea, which makes the image readable to anything that can reach the
|
||||
host. That is a posture decision, so it is yours rather than an agent's.
|
||||
- **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
|
||||
the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
|
||||
*node* process, so it resolves through the node's public upstreams
|
||||
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
|
||||
answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
|
||||
IP — which this router does not hairpin; it answers :443 with its own placeholder
|
||||
certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
|
||||
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
|
||||
and the digest were never implicated. The fix is the pin this repository already uses
|
||||
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
|
||||
in `node_baseline_hosts_pins`. **An operator must run the `node_baseline` play for it
|
||||
to reach the nodes.**
|
||||
- **The pull is still unauthenticated** once the node reaches the right endpoint. An
|
||||
unauthenticated client gets nothing from this Gitea's registry, and there is no
|
||||
`imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
|
||||
(local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea
|
||||
account whose token carries `read:package`, referenced from the Deployment and the
|
||||
committed worker Job template — not built yet, because minting that token is yours.
|
||||
|
||||
The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
|
||||
never pulled either.
|
||||
|
||||
@@ -51,6 +51,15 @@
|
||||
# (2) MINT THE GITEA TOKEN as that user (log in as `pm-agent`, or use the admin API to
|
||||
# create a token for it): Settings -> Applications -> Generate New Token.
|
||||
#
|
||||
# A LOCAL GITEA ACCOUNT WITH A PAT IS THE DECIDED MODEL (operator, 2026-10-06), not
|
||||
# Keycloak. The agents only ever talk to Gitea, so this is the whole requirement and it
|
||||
# is far simpler than federating a machine identity. It is also the only thing that
|
||||
# works: Gitea's API accepts basic auth, PATs, SSH signatures and its OWN OAuth2 bearer
|
||||
# tokens, and as a provider Gitea implements only the Authorization Code Grant — no
|
||||
# client-credentials grant, and nothing that validates a third-party OIDC token. A
|
||||
# Keycloak service-account token therefore cannot authenticate to Gitea's API or to
|
||||
# git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that.
|
||||
#
|
||||
# Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT
|
||||
# `write:admin`, NOT `write:organization`, NOT `write:user`.
|
||||
#
|
||||
|
||||
Reference in new issue
Block a user