fix(node): pin gitea to the Gateway VIP — the failing certificate was the router's, not the Gateway's #59
No files matched your search
@@ -0,0 +1,111 @@
|
||||
---
|
||||
date: 2026-10-06
|
||||
branch: docs/pm-agent-registry-pull-prereq
|
||||
title: The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The image pull for `pm-agent` failed TLS verification, and the previous entry on this branch
|
||||
honestly recorded it as **not diagnosed**. The operator asked for the diagnosis before any
|
||||
registry design. This is it, and the fix turns out to be a pattern this repository already
|
||||
has.
|
||||
|
||||
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins
|
||||
`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the
|
||||
Keycloak issuer pin that is there for the identical reason. **An operator must run the
|
||||
`node_baseline` play for it to reach the nodes** — writing `/etc/hosts` on a node is not
|
||||
something Flux or an agent can do.
|
||||
|
||||
The docs that described this as an undiagnosed blocker now describe the cause and the fix:
|
||||
`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and
|
||||
the play), and `docs/plans/pm-agent-issue-to-pr.md`.
|
||||
|
||||
Also recorded, in the two places an operator reads when creating the credential: **the agent
|
||||
authenticates to Gitea with a local account and a scoped PAT, by decision** (operator,
|
||||
2026-10-06), not with a federated identity. Written down because the alternative looks
|
||||
obviously better until you check whether it is possible — see below.
|
||||
|
||||
## What I verified
|
||||
|
||||
Every step measured, read-only, in this order.
|
||||
|
||||
**The failing certificate is the router's, not the Gateway's.**
|
||||
|
||||
- containerd's error names TLS, not the digest: `tls: failed to verify certificate: x509:
|
||||
certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev`.
|
||||
"not valid for **any** names" means the presented certificate carries no SANs at all.
|
||||
- `192.168.2.20` is the shared Gateway's own LoadBalancer VIP
|
||||
(`gateway/cilium-gateway-shared`, from the Cilium pool `192.168.2.20-30`), and its `https`
|
||||
listener serves `*.home.keithjackson.dev` with `wildcard-home-tls`. From the operator's
|
||||
workstation the name resolves there and serves the real Let's Encrypt
|
||||
`CN=*.home.keithjackson.dev` with correct SANs. **So the certificate and the Gateway are
|
||||
healthy.**
|
||||
- **The node asks a different resolver and gets a different answer.** Prod's
|
||||
`node_baseline_dns_servers` is the LAN gateway, not Pi-hole — deliberately, as that file
|
||||
explains. Public DNS for this zone is a CNAME to `wan.home.keithjackson.dev`: measured via
|
||||
both `1.1.1.1` and `8.8.8.8`, `gitea.home.keithjackson.dev` →
|
||||
`wan.home.keithjackson.dev.` → `136.51.115.252`, and the apex answers the same.
|
||||
- **That address serves a certificate with no names.** Measured from the workstation against
|
||||
the WAN address with the right SNI:
|
||||
|
||||
subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
|
||||
issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
|
||||
No extensions in certificate
|
||||
|
||||
No extensions means no `subjectAltName`, which is exactly "not valid for any names". The
|
||||
`CN=%D` is the router's placeholder.
|
||||
|
||||
**This repository had already measured the same thing, for Keycloak, and I did not need to
|
||||
re-derive it.** `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` says, above the pin:
|
||||
"Since platform/gateway/gateway.yaml set an external-dns `target`, every public name is a
|
||||
CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from
|
||||
the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
|
||||
k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18)." Same cause, same symptom,
|
||||
one name later. The pin exists precisely for node processes that must reach the Gateway
|
||||
directly; containerd is one, and nobody had added it.
|
||||
|
||||
**The parity the pin has to satisfy:** `192.168.2.20` equals the live
|
||||
`GATEWAY_VIP`, and `k3s_base_domain` equals the live `BASE_DOMAIN` — both read from the
|
||||
cluster, not from the comment that claims prod serves `prod.home.keithjackson.dev` (it does
|
||||
not; `flux-system/cluster-vars` reports `BASE_DOMAIN=home.keithjackson.dev` and
|
||||
`GATEWAY_VIP=192.168.2.20`, and Ansible's `k3s_base_domain` is `home.keithjackson.dev`).
|
||||
That comment is stale history.
|
||||
|
||||
**The credential decision, and why the obvious alternative is not available.** Gitea's API
|
||||
accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2
|
||||
provider Gitea implements **only the Authorization Code Grant** (plus PKCE/OIDC), so there
|
||||
is no client-credentials grant, and nothing in Gitea validates a token issued by a
|
||||
third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to
|
||||
Gitea's API or to git-over-HTTP. Read from Gitea's own documentation
|
||||
(`docs.gitea.com/development/api-usage`, `/development/oauth2-provider`), not assumed.
|
||||
|
||||
**Gate:** `./ci/gate.sh` in the pinned toolbox from a §0 worktree — see the branch's run; rc 3
|
||||
with `gitea-settings` the only stage that did not run.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the
|
||||
nodes is written by the `node_baseline` play, which is an operator action. Until that runs
|
||||
and an image exists, "the node now reaches the Gateway" is a design claim. The evidence
|
||||
that it *will* is the Keycloak pin working for the same cause — which I also did not
|
||||
re-measure.
|
||||
- **Why public DNS answers at all for an `internal` service.** `catalog/gitea.yaml` sets no
|
||||
`exposure`, so it inherits `internal`, and the renderer contract says an internal service's
|
||||
HTTPRoute carries `external-dns.alpha.kubernetes.io/controller: none` and publishes no
|
||||
public record. Yet `gitea.home.keithjackson.dev` resolves publicly to the WAN CNAME. I did
|
||||
not chase that: it may be a zone-level wildcard rather than a published record, and it did
|
||||
not change the diagnosis. It is worth someone's attention, because "internal services
|
||||
publish no public DNS" is an L0-checked claim and this looks like a case where the *name*
|
||||
resolves regardless of what the renderer withheld.
|
||||
- **Registry authentication.** Unchanged and still open: an unauthenticated client gets
|
||||
nothing from this registry, and no `imagePullSecret` exists anywhere in the tree. With the
|
||||
credential model now decided (local Gitea accounts with PATs) the shape is a
|
||||
`dockerconfigjson` for an account whose token carries `read:package`, referenced from the
|
||||
Deployment and the committed worker Job template — and the worker Job template's shape is
|
||||
pinned by admission, so whether the policy should also pin `imagePullSecrets` deserves a
|
||||
deliberate look rather than a quiet addition. Not built: minting that token is an operator
|
||||
action, and I did not want to guess at a second credential's scope.
|
||||
- **Anything about runtime.** The image still does not exist. No dispatcher has
|
||||
authenticated, no issue has been triaged, no worker Job has been created, and the admission
|
||||
policy — live, `Enforce`, six rules — has still never admitted or refused a real Job.
|
||||
@@ -236,12 +236,14 @@ about runtime behaviour in this document is a design claim, not a measurement.
|
||||
1. **Make the registry pullable, THEN build, push, pin.** Measured after the merge, and
|
||||
neither half is declared anywhere in this repository
|
||||
(`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`):
|
||||
(i) the cluster's nodes cannot verify the registry's TLS certificate — containerd reports
|
||||
`x509: certificate is not valid for any names` for `gitea.home.keithjackson.dev`, while
|
||||
that name serves a healthy Let's Encrypt wildcard from the LAN; (ii) an unauthenticated
|
||||
client gets nothing from that registry and no `imagePullSecret` exists in the tree, so
|
||||
either one is added or anonymous package reads are allowed. Only then does the digest
|
||||
matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
|
||||
(i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the
|
||||
router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning
|
||||
`gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
|
||||
pin the Keycloak issuer already carries; an operator must run the `node_baseline` play;
|
||||
(ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
|
||||
the credential decision now being local Gitea accounts with PATs, that means a
|
||||
`dockerconfigjson` for an account whose token carries `read:package`. Only then does the
|
||||
digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
|
||||
placeholder digests. Expect the build itself to need iteration too: the nodejs module
|
||||
stream and the npm install have never run against this base. The same two blockers sit in
|
||||
front of `conformance-runner`, which shares the registry path and has never been pulled.
|
||||
|
||||
@@ -327,6 +327,29 @@ node_baseline_hosts_pins:
|
||||
- ip: "192.168.2.20"
|
||||
names:
|
||||
- "keycloak.{{ k3s_base_domain }}"
|
||||
# gitea: the SECOND node process to need this pin, for the identical reason, found
|
||||
# 2026-10-06. containerd pulls images as a NODE process, so it resolves through the
|
||||
# public upstreams above and gets the CNAME to wan.home.keithjackson.dev — the WAN IP,
|
||||
# which this router does not hairpin. The pull then fails on the router's own
|
||||
# placeholder certificate rather than on anything to do with the image:
|
||||
#
|
||||
# failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:…":
|
||||
# tls: failed to verify certificate: x509: certificate is not valid for any names,
|
||||
# but wanted to match gitea.home.keithjackson.dev
|
||||
#
|
||||
# Measured from the workstation the same day, against the WAN address with that SNI:
|
||||
# subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
|
||||
# No extensions in certificate <- no SANs at all, hence "not valid for ANY names"
|
||||
# while the same name via PiHole answers 192.168.2.20 and serves the real
|
||||
# Let's Encrypt *.home.keithjackson.dev. So the certificate and the Gateway are fine;
|
||||
# only the node's ANSWER is wrong. Same root cause, same fix, as the issuer pin above.
|
||||
#
|
||||
# WHO NEEDS IT: any workload whose image lives in the in-cluster Gitea registry —
|
||||
# today platform/pm-agent (agent-runner) and conformance/incluster
|
||||
# (conformance-runner), neither of which has ever been pulled. A pull is still
|
||||
# UNAUTHENTICATED after this pin, which is a separate thing to settle; this only makes
|
||||
# the node reach the right endpoint.
|
||||
- "gitea.{{ k3s_base_domain }}"
|
||||
|
||||
# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
|
||||
# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
|
||||
|
||||
+37
-11
@@ -22,6 +22,25 @@ requires.
|
||||
|
||||
---
|
||||
|
||||
## Why the agent authenticates to Gitea with a local account and a PAT
|
||||
|
||||
Not Keycloak, and that is a decision rather than an omission (operator, 2026-10-06): the
|
||||
agents only ever talk to Gitea, so a local Gitea account with a scoped personal access token
|
||||
is the whole requirement, and it is drastically simpler than federating a machine identity.
|
||||
|
||||
It is also what Gitea can actually do. Keycloak service accounts were considered and are not
|
||||
available for this: Gitea's API accepts basic auth, PATs, SSH signatures and **its own**
|
||||
OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization
|
||||
Code Grant** — there is no client-credentials grant, and nothing in Gitea validates a token
|
||||
issued by a third-party OIDC provider. So a Keycloak service-account token cannot
|
||||
authenticate to Gitea's API or to git-over-HTTP at all. Keycloak remains what it is for human
|
||||
access: the browser login source registered on Gitea (`catalog/gitea.yaml`).
|
||||
|
||||
What still matters, and is enforced by the hand-off rather than by Keycloak: **the account
|
||||
must not be an administrator** — `infra/gitea/approval-policy.json` authorises
|
||||
`/approve-override` on live repository permission alone, so an admin-permission bot could
|
||||
waive the sensitive-path human review on its own pull request.
|
||||
|
||||
## It does nothing at all until an operator does two things
|
||||
|
||||
Both are deliberate, and both fail **visibly** rather than silently:
|
||||
@@ -38,17 +57,24 @@ Both are deliberate, and both fail **visibly** rather than silently:
|
||||
path. Details and the exact commands:
|
||||
[`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md).
|
||||
|
||||
- **The nodes must be able to verify the registry's TLS certificate.** containerd
|
||||
currently fails with `x509: certificate is not valid for any names, but wanted to match
|
||||
gitea.home.keithjackson.dev`, while the same name from the LAN serves a healthy
|
||||
Let's Encrypt `*.home.keithjackson.dev`. Correcting the digest will not clear this.
|
||||
- **The pull must be authenticated.** An unauthenticated client gets nothing from this
|
||||
Gitea's registry (its anonymous token carries no access scope; every path 401s), and
|
||||
there is no `imagePullSecret` anywhere in the tree. Either add one — a
|
||||
`dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the
|
||||
Deployment *and* from the committed worker Job template — or allow anonymous reads of
|
||||
that package in Gitea, which makes the image readable to anything that can reach the
|
||||
host. That is a posture decision, so it is yours rather than an agent's.
|
||||
- **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
|
||||
the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
|
||||
*node* process, so it resolves through the node's public upstreams
|
||||
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
|
||||
answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
|
||||
IP — which this router does not hairpin; it answers :443 with its own placeholder
|
||||
certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
|
||||
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
|
||||
and the digest were never implicated. The fix is the pin this repository already uses
|
||||
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
|
||||
in `node_baseline_hosts_pins`. **An operator must run the `node_baseline` play for it
|
||||
to reach the nodes.**
|
||||
- **The pull is still unauthenticated** once the node reaches the right endpoint. An
|
||||
unauthenticated client gets nothing from this Gitea's registry, and there is no
|
||||
`imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
|
||||
(local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea
|
||||
account whose token carries `read:package`, referenced from the Deployment and the
|
||||
committed worker Job template — not built yet, because minting that token is yours.
|
||||
|
||||
The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
|
||||
never pulled either.
|
||||
|
||||
@@ -51,6 +51,15 @@
|
||||
# (2) MINT THE GITEA TOKEN as that user (log in as `pm-agent`, or use the admin API to
|
||||
# create a token for it): Settings -> Applications -> Generate New Token.
|
||||
#
|
||||
# A LOCAL GITEA ACCOUNT WITH A PAT IS THE DECIDED MODEL (operator, 2026-10-06), not
|
||||
# Keycloak. The agents only ever talk to Gitea, so this is the whole requirement and it
|
||||
# is far simpler than federating a machine identity. It is also the only thing that
|
||||
# works: Gitea's API accepts basic auth, PATs, SSH signatures and its OWN OAuth2 bearer
|
||||
# tokens, and as a provider Gitea implements only the Authorization Code Grant — no
|
||||
# client-credentials grant, and nothing that validates a third-party OIDC token. A
|
||||
# Keycloak service-account token therefore cannot authenticate to Gitea's API or to
|
||||
# git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that.
|
||||
#
|
||||
# Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT
|
||||
# `write:admin`, NOT `write:organization`, NOT `write:user`.
|
||||
#
|
||||
|
||||
@@ -68,7 +68,7 @@ Work through these in order; the first three are by far the most likely.
|
||||
|---|---|---|
|
||||
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
|
||||
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
|
||||
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | the **node** cannot verify the registry's TLS certificate. Nothing to do with the digest: a correct digest fails identically. The same name serves a healthy Let's Encrypt wildcard from the LAN, so what the node reaches is not what you reach | not diagnosed — see `decisions/2026-10-05-pm-agent-registry-pull-prereq.md` for everything measured. Check the nodes' resolution of `gitea.home.keithjackson.dev` and their registry trust (`/etc/rancher/k3s/registries.yaml`), neither of which this repository declares |
|
||||
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. **Run the `node_baseline` play** for it to reach the nodes. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
||||
| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
|
||||
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
|
||||
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
|
||||
|
||||
Reference in new issue
Block a user