From 6253d546f876684d93562decd02751bfd780554a Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 5 Oct 2026 23:22:52 -0500 Subject: [PATCH 1/4] =?UTF-8?q?docs(pm-agent):=20the=20registry=20cannot?= =?UTF-8?q?=20be=20pulled=20from=20yet=20=E2=80=94=20name=20both=20prerequ?= =?UTF-8?q?isites=20where=20the=20operator=20reads=20them?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pm-agent reconciled exactly as designed after #57: Flux applied main@sha1:95597bd to platform-config, pm-agent and apps-wave-0 (all Ready), the ClusterPolicy pm-agent-job-shape is live with Enforce + failurePolicy Fail and all six rules, SOPS decryption works through the new overlay patch (checked by key names and value lengths, never values), and pm-agent-worker really is bound to nothing. Measuring it turned up two prerequisites for the NEXT operator step that nothing in this repository declares, and that #57's handoff could not have known — it listed the image pull as unverified, and that is exactly where it bit: - the nodes cannot verify the registry's TLS certificate (containerd: 'x509: certificate is not valid for any names'), while the same name serves a healthy Let's Encrypt wildcard from the LAN. A correct digest fails identically; - an unauthenticated client gets nothing from this Gitea's registry (its anonymous token carries no access scope) and no imagePullSecret exists anywhere in the tree. Not specific to pm-agent: conformance-runner shares the registry path and has never been pulled either. The dispatcher is the first workload in the cluster ever to reference it. Documentation only — no manifest, no policy, no code. Both fixes need an operator action and one of them is a posture decision, so the docs present both rather than pick. Co-Authored-By: Claude Opus 5 (1M context) --- ...026-10-05-pm-agent-registry-pull-prereq.md | 123 ++++++++++++++++++ docs/plans/pm-agent-issue-to-pr.md | 15 ++- platform/pm-agent/README.md | 21 +++ runbooks/pm-agent.md | 4 +- 4 files changed, 159 insertions(+), 4 deletions(-) create mode 100644 decisions/2026-10-05-pm-agent-registry-pull-prereq.md diff --git a/decisions/2026-10-05-pm-agent-registry-pull-prereq.md b/decisions/2026-10-05-pm-agent-registry-pull-prereq.md new file mode 100644 index 0000000..b4c4a58 --- /dev/null +++ b/decisions/2026-10-05-pm-agent-registry-pull-prereq.md @@ -0,0 +1,123 @@ +--- +date: 2026-10-05 +branch: docs/pm-agent-registry-pull-prereq +title: pm-agent reconciled exactly as designed — and is the first workload ever to pull from the in-cluster registry, which does not work yet +--- + +## What changed + +Documentation only. No manifest, no policy, no code. + +PR #57 merged and `pm-agent` reconciled. I measured what landed, and the measurement turned +up **two prerequisites for the next operator step that nothing in this repository +declares** — and that the handoff entry for #57 could not have known, because it listed the +image pull as unverified and that is exactly where it bit. + +So this entry records the post-merge measurement, and +`platform/pm-agent/README.md`, `docs/plans/pm-agent-issue-to-pr.md` and +`runbooks/pm-agent.md` now name both prerequisites where they say "build, push, pin the +digest". Before this, that step read as one action; it is three, and the two new ones are +not an agent's to perform. + +**The finding is not specific to pm-agent.** `conformance-runner` has the identical +registry path (`gitea.home.keithjackson.dev/home-network/...`), the identical "operator +pushes, a digest goes in a PR" model, and has never been pulled either — so the same two +blockers sit in front of the L2/L3 conformance wiring. Measured: **no pod anywhere in the +cluster references that registry** except the pm-agent dispatcher, which is therefore the +first workload ever to try it. + +## What I verified + +Read-only, through the pinned toolbox, against the live prod cluster. + +**The merge reconciled, completely and exactly as designed.** + +- `GitRepository/flux-system` artifact revision: `main@sha1:95597bd6883637…` — the merge + commit. `platform-config`, `pm-agent` and `apps-wave-0` all report that same + `lastAppliedRevision` with `Ready=True`. Compared field to field, per CLAUDE.md §2b, + rather than assumed from the merge. +- **The security boundary is live.** `ClusterPolicy/pm-agent-job-shape`: + `validationFailureAction=Enforce`, `failurePolicy=Fail`, `Ready=True`, and all six rules + present (`worker-must-be-the-committed-shape`, + `worker-must-be-one-container-with-no-args`, + `worker-volumes-must-be-workspace-or-tmp-emptydirs`, + `worker-may-not-declare-init-or-ephemeral-containers`, + `worker-env-names-must-be-the-committed-set`, `worker-may-not-use-envfrom`). This is the + check that mattered most: a ClusterPolicy added to the wrong kustomization list ships + green and never reaches the cluster, which is what happened to the conformance boundary + on 2026-09-29. +- Namespace contents: both ServiceAccounts, `pm-agent-config` (17 keys), the + `pm-agent-credentials` Secret, the `default-deny` NetworkPolicy, and both + CiliumNetworkPolicies with `VALID=True`. +- **SOPS decryption works through the new overlay patch.** Checked by KEY NAMES and VALUE + LENGTHS only, never values (CLAUDE.md §6): keys `gitea-token`, `claude-code-oauth-token`, + lengths 76 and 52 bytes — which are the base64 lengths of the committed placeholder + strings, so the Secret holds the decrypted placeholders and not ciphertext. A + ciphertext apply succeeds silently and would have left `ENC[AES256_GCM,…]` values an + order of magnitude longer. +- **`pm-agent-worker` really is bound to nothing.** The namespace has exactly one + RoleBinding (`pm-agent` → Role `pm-agent`, subject `ServiceAccount/pm-agent`), and no + ClusterRoleBinding anywhere names either pm-agent ServiceAccount. +- `gitea-allow-pm-agent-ingress` is applied in the `gitea` namespace, matching + `io.kubernetes.pod.namespace: pm-agent`. +- The dispatcher is `ImagePullBackOff` on the all-zero placeholder digest — the designed, + visible failure. + +**The two prerequisites, measured.** + +1. **The nodes cannot verify the registry's TLS certificate.** containerd's error on the + pull is not about the digest: + + failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:000…": + tls: failed to verify certificate: x509: certificate is not valid for any names, + but wanted to match gitea.home.keithjackson.dev + + "not valid for **any** names" means the certificate presented to the node carries no + matching SAN at all. From the operator's workstation the same name is healthy: it + resolves to `192.168.2.20` and serves a Let's Encrypt certificate + `CN=*.home.keithjackson.dev` with SANs `*.home.keithjackson.dev, home.keithjackson.dev` + — and `Certificate gateway/wildcard-home` is `Ready=True` for exactly those names. So + the certificate that *should* be served is fine, and what the NODE reaches is not what + the workstation reaches. **Correcting the digest will not clear this.** +2. **Anonymous pulls from this registry are refused.** Not inferred from a bare 401 — that + would have been wrong, because this Gitea answers 401 for *everything* anonymously: + `/v2/home-network/agent-runner/manifests/latest`, `/v2/home-network/conformance-runner/…`, + a randomly-named nonexistent repository, and `nosuchowner/nosuchimage` all return 401, + and `/api/v1/packages/Home-Network` returns `{"message":"token is required"}`. The + registry *does* issue a token to an anonymous client, but its `access` claim is `null` — + no pull scope. So the conclusion that holds is the simple one: **an unauthenticated + client gets nothing from this registry**, so a pull with no credential cannot succeed. + And `grep` over the tree finds **no `imagePullSecrets` anywhere** — not in + `platform/pm-agent/deployment.yaml`, not in `infra/agent/worker-job-template.yaml`, not + in `conformance/incluster/`. + +## What I did NOT verify + +- **Why the node sees a nameless certificate.** I measured that it does, and that the LAN + does not. I did not diagnose it: I did not read the nodes' resolver or + `/etc/rancher/k3s/registries.yaml` (node files, not reachable read-only through the API), + and I did not check whether the node resolves the name to the same `192.168.2.20` the + workstation does or to something else. Note `cluster-vars` has + `GATEWAY_VIP: 192.168.1.53` while the nodes are `192.168.2.10-12` and the name resolves + to `192.168.2.20` — three different subnets in one sentence is a reason to measure rather + than to theorise, and I have not. +- **Whether the package would be pullable once authenticated.** Nothing has ever been + pushed to that path, so "a credential is sufficient" is untested. The only thing + established is that no credential is insufficient. +- **Which fix the operator wants.** Two routes, and they are not equivalent: + (a) an `imagePullSecret` — a `dockerconfigjson` holding a Gitea token, SOPS-encrypted + like `pm-agent-credentials`, referenced from the Deployment *and* from the committed + worker Job template; note the Job template's shape is pinned by admission, and + `imagePullSecrets` is a field that policy does not currently constrain, so adding one is + a template change that needs a deliberate look at whether the policy should pin it too. + (b) allowing anonymous reads of that package in Gitea — no manifest change at all, but it + makes the image world-readable to anything that can reach the host. + I have implemented neither: both need an operator action I must not take, and (b) is a + posture decision. The docs now present both. +- **Everything the #57 entry listed as unverified about RUNTIME remains unverified.** The + image has still never been built or pushed. No dispatcher has authenticated to Gitea, no + issue has been triaged, no worker Job has been created — so the admission policy, though + now provably applied and Enforce, has still never refused or admitted a real Job. The + network policies have not carried traffic. +- **I did not re-run the merge gate against `main` after the merge.** This branch's own gate + run is the evidence for this change; the merged state's is CI run on PR #57. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index f6f42e1..37d7830 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -233,9 +233,18 @@ about runtime behaviour in this document is a design claim, not a measurement. ## Follow-ups, in the order they are worth doing -1. **Build, push, pin.** Operator runs `infra/agent/build-image.sh`, pushes, and one PR - replaces both placeholder digests. Expect the build to need iteration: the nodejs module - stream and the npm install have never run against this base. +1. **Make the registry pullable, THEN build, push, pin.** Measured after the merge, and + neither half is declared anywhere in this repository + (`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`): + (i) the cluster's nodes cannot verify the registry's TLS certificate — containerd reports + `x509: certificate is not valid for any names` for `gitea.home.keithjackson.dev`, while + that name serves a healthy Let's Encrypt wildcard from the LAN; (ii) an unauthenticated + client gets nothing from that registry and no `imagePullSecret` exists in the tree, so + either one is added or anonymous package reads are allowed. Only then does the digest + matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both + placeholder digests. Expect the build itself to need iteration too: the nodejs module + stream and the npm install have never run against this base. The same two blockers sit in + front of `conformance-runner`, which shares the registry path and has never been pulled. 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the bot's permission is `write`, not `admin`** — that measurement is the control that keeps the agent out of its own approval gate. diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index ddbbbbd..732ea30 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -31,6 +31,27 @@ Both are deliberate, and both fail **visibly** rather than silently: `infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the operator, then update **both** places with the digest `podman push --digestfile` returned, in one pull request. Until then: `ImagePullBackOff`. + + **Two things have to be true before any digest will pull, and neither is declared in this + repository.** Both were measured after PR #57 merged — this service is the first workload + in the cluster ever to reference the in-cluster registry, so nothing had exercised the + path. Details and the exact commands: + [`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md). + + - **The nodes must be able to verify the registry's TLS certificate.** containerd + currently fails with `x509: certificate is not valid for any names, but wanted to match + gitea.home.keithjackson.dev`, while the same name from the LAN serves a healthy + Let's Encrypt `*.home.keithjackson.dev`. Correcting the digest will not clear this. + - **The pull must be authenticated.** An unauthenticated client gets nothing from this + Gitea's registry (its anonymous token carries no access scope; every path 401s), and + there is no `imagePullSecret` anywhere in the tree. Either add one — a + `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the + Deployment *and* from the committed worker Job template — or allow anonymous reads of + that package in Gitea, which makes the image readable to anything that can reach the + host. That is a posture decision, so it is yours rather than an agent's. + + The same two apply to `conformance-runner`: same registry path, same push-then-pin model, + never pulled either. 2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off — which Gitea account to create, which scopes, and why it must *not* be an administrator — diff --git a/runbooks/pm-agent.md b/runbooks/pm-agent.md index b5b852d..4a6a5f6 100644 --- a/runbooks/pm-agent.md +++ b/runbooks/pm-agent.md @@ -67,7 +67,9 @@ Work through these in order; the first three are by far the most likely. | Symptom in the log | Cause | Fix | |---|---|---| | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` | -| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) | +| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first | +| `ImagePullBackOff` with `x509: certificate is not valid for any names` | the **node** cannot verify the registry's TLS certificate. Nothing to do with the digest: a correct digest fails identically. The same name serves a healthy Let's Encrypt wildcard from the LAN, so what the node reaches is not what you reach | not diagnosed — see `decisions/2026-10-05-pm-agent-registry-pull-prereq.md` for everything measured. Check the nodes' resolution of `gitea.home.keithjackson.dev` and their registry trust (`/etc/rancher/k3s/registries.yaml`), neither of which this repository declares | +| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours | | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` | | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob | | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) | -- 2.54.0 From a209a618fdcf2a382d85018467d4c9a445ac1d2a Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 5 Oct 2026 23:40:55 -0500 Subject: [PATCH 2/4] =?UTF-8?q?fix(pm-agent):=20the=20agent=20is=20IN=20th?= =?UTF-8?q?e=20cluster=20=E2=80=94=20one=20git=20host,=20the=20cluster-loc?= =?UTF-8?q?al=20service=20name?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An in-cluster consumer reaches a service by its *.svc.cluster.local name, which is by design a different path from how a client outside the cluster reaches the same service. pm-worker's host allowlist accepted TWO values: the cluster-local name and the external hostname gitea.home.keithjackson.dev. The second is removed. It was never needed — nothing sets it — it would route in-cluster traffic out through the Gateway and back for no reason, and as an ACCEPTED value it widened the allowlist to a host reachable over the namespace's world:443 egress rule, which is precisely the injection shape the allowlist exists to refuse. probe-validation.sh now pins that: 20 refusals, including the external hostname, the cluster-local name over https, and the cluster-local name on the wrong port. Also corrects a remark in the handoff entry: I cited cluster-vars' GATEWAY_VIP (192.168.1.53) as suspicious next to the nodes' 192.168.2.x. That was noise — those base cluster-vars values are dev-era leftovers for a decommissioned cluster and say nothing about prod. None of the measured facts depended on it. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-10-05-pm-agent-registry-pull-prereq.md | 12 ++++++++---- infra/agent/pm-worker | 14 ++++++++++++-- infra/agent/probe-validation.sh | 16 +++++++++++++--- platform/pm-agent/README.md | 2 +- 4 files changed, 34 insertions(+), 10 deletions(-) diff --git a/decisions/2026-10-05-pm-agent-registry-pull-prereq.md b/decisions/2026-10-05-pm-agent-registry-pull-prereq.md index b4c4a58..bb57d27 100644 --- a/decisions/2026-10-05-pm-agent-registry-pull-prereq.md +++ b/decisions/2026-10-05-pm-agent-registry-pull-prereq.md @@ -97,10 +97,14 @@ Read-only, through the pinned toolbox, against the live prod cluster. does not. I did not diagnose it: I did not read the nodes' resolver or `/etc/rancher/k3s/registries.yaml` (node files, not reachable read-only through the API), and I did not check whether the node resolves the name to the same `192.168.2.20` the - workstation does or to something else. Note `cluster-vars` has - `GATEWAY_VIP: 192.168.1.53` while the nodes are `192.168.2.10-12` and the name resolves - to `192.168.2.20` — three different subnets in one sentence is a reason to measure rather - than to theorise, and I have not. + workstation does or to something else. + + **A correction to a remark I first wrote here:** I cited + `cluster-vars`'s `GATEWAY_VIP: 192.168.1.53` as suspicious next to the nodes' + `192.168.2.x`. That was noise — the operator confirms those base `cluster-vars` values are + **dev-era leftovers for a cluster that is decommissioned**, so the 192.168.1.x addresses in + that file say nothing about prod and should not be read as evidence about anything. The + measured facts stand on their own and none of them depended on that knob. - **Whether the package would be pullable once authenticated.** Nothing has ever been pushed to that path, so "a credential is sufficient" is untested. The only thing established is that no credential is insufficient. diff --git a/infra/agent/pm-worker b/infra/agent/pm-worker index 5f7be31..fe8663d 100755 --- a/infra/agent/pm-worker +++ b/infra/agent/pm-worker @@ -90,9 +90,19 @@ esac # pointing at an external host would be REACHABLE: it could clone a repository that is not # this one and push this repository's work somewhere else. The allowlist is baked in here, # where changing it requires rebuilding the image whose digest admission pins. +# +# ONE ENTRY, AND IT IS THE CLUSTER-LOCAL SERVICE NAME. This agent runs INSIDE the cluster, so +# it reaches Gitea the way every other in-cluster consumer does — `*.svc.cluster.local` — +# which is by design a different path from how a client outside the cluster reaches the same +# service. The external hostname `gitea.home.keithjackson.dev` used to be a second accepted +# value here; it is removed deliberately. It was never needed (nothing sets it), it routes +# in-cluster traffic out through the Gateway and back for no reason, and as an ACCEPTED value +# it widened this allowlist to a host reachable over the `world:443` egress rule — which is +# the exact shape of injection the allowlist exists to refuse. The cluster-local name also +# depends on none of public DNS, Cloudflare, the Gateway or the certificate. case "${PM_GITEA_URL}" in - http://gitea.gitea.svc.cluster.local:3000|https://gitea.home.keithjackson.dev) ;; - *) die "PM_GITEA_URL '${PM_GITEA_URL}' is not an allowed git host (see this script's header)" ;; + http://gitea.gitea.svc.cluster.local:3000) ;; + *) die "PM_GITEA_URL '${PM_GITEA_URL}' is not the allowed in-cluster git host (see this script's header)" ;; esac [[ "${PM_GITEA_OWNER}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,38}$ ]] || die "PM_GITEA_OWNER is malformed" [[ "${PM_GITEA_REPO}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$ ]] || die "PM_GITEA_REPO is malformed" diff --git a/infra/agent/probe-validation.sh b/infra/agent/probe-validation.sh index 2690e8c..042d004 100755 --- a/infra/agent/probe-validation.sh +++ b/infra/agent/probe-validation.sh @@ -86,12 +86,22 @@ refuses "branch equal to the base branch" "must not be the base branch" \ # --- where code comes FROM and goes TO ------------------------------------- # The namespace's egress policy permits tcp/443 to the world (the model needs it), so an # injected host would be REACHABLE. These are the cases that matter most. -refuses "a redirected git host" "is not an allowed git host" \ +refuses "a redirected git host" "is not the allowed in-cluster git host" \ PM_GITEA_URL=https://evil.invalid -refuses "a real, reachable, WRONG git host" "is not an allowed git host" \ +refuses "a real, reachable, WRONG git host" "is not the allowed in-cluster git host" \ PM_GITEA_URL=https://github.com -refuses "the right host on a smuggled path" "is not an allowed git host" \ +refuses "the right host on a smuggled path" "is not the allowed in-cluster git host" \ PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000/../.. +# This agent is IN the cluster, so it reaches Gitea by its cluster-local service name. The +# EXTERNAL hostname is refused too — it used to be a second accepted value, and as an +# accepted value it widened this allowlist to a host reachable over the `world:443` egress +# rule, which is the injection shape the allowlist exists to refuse. +refuses "this Gitea's EXTERNAL hostname" "is not the allowed in-cluster git host" \ + PM_GITEA_URL=https://gitea.home.keithjackson.dev +refuses "the cluster-local name over https" "is not the allowed in-cluster git host" \ + PM_GITEA_URL=https://gitea.gitea.svc.cluster.local:3000 +refuses "the cluster-local name, wrong port" "is not the allowed in-cluster git host" \ + PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:8080 refuses "a malformed owner" "PM_GITEA_OWNER is malformed" \ 'PM_GITEA_OWNER=../other-org' refuses "a malformed repository" "PM_GITEA_REPO is malformed" \ diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index 732ea30..ec68fb2 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -117,7 +117,7 @@ Job from a committed template. **The model's output is data, never a command.** | **cannot** | merge anything — the gate is the approver, and the service makes no merge call | | **cannot** | reach the Kubernetes API: no ServiceAccount token is mounted, the `pm-agent-worker` identity is bound to nothing, and the egress policy does not include the API server. `kubectl` is on its `PATH` (the gate's toolchain needs it) and is inert | | **cannot** | reach anything except this Gitea and tcp/443 — no package registry, no GitHub | -| **cannot** | clone from or push to any host but this Gitea: `pm-worker` validates `PM_GITEA_URL` against an allowlist baked into the image | +| **cannot** | clone from or push to anywhere but `gitea.gitea.svc.cluster.local:3000`: `pm-worker` validates `PM_GITEA_URL` against a one-entry allowlist baked into the image. In-cluster consumers reach a service by its cluster-local name — a different path by design from how a client outside the cluster reaches it — so the external hostname is not an accepted value | | **cannot** | authenticate a `git push` **during the model session**: `GIT_ASKPASS` is unset for its duration, so the restriction is structural rather than a request in a prompt | | **cannot** | be created in any shape but the committed one — see the admission policy | -- 2.54.0 From 523a463fd7e2a9e09d49cd165d244137340ead87 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 6 Oct 2026 00:35:18 -0500 Subject: [PATCH 3/4] =?UTF-8?q?fix(node):=20pin=20gitea=20to=20the=20Gatew?= =?UTF-8?q?ay=20VIP=20=E2=80=94=20containerd=20was=20resolving=20it=20to?= =?UTF-8?q?=20the=20non-hairpinned=20WAN?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pm-agent image pull failed TLS verification, and the previous entry on this branch recorded it honestly as not diagnosed. Diagnosed now, and it is not about the image. containerd pulls as a NODE process, so it resolves through node_baseline_dns_servers (the LAN gateway, deliberately not PiHole) and gets the public answer: a CNAME to wan.home.keithjackson.dev -> 136.51.115.252, measured via both 1.1.1.1 and 8.8.8.8. This router does not hairpin, so WAN:443 answers with its own placeholder certificate — measured with the right SNI from the workstation: subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D No extensions in certificate No extensions means no subjectAltName, which is exactly 'certificate is not valid for any names'. Meanwhile 192.168.2.20 IS the shared Gateway's VIP and its https listener serves the real Let's Encrypt *.home.keithjackson.dev. The Gateway, the wildcard cert and the digest were never implicated — only the node's answer. This file already documents the identical failure for the Keycloak issuer, measured from k3s-prod-2/-3 on 2026-10-03 (curl exit 60, ssl_verify_result=18), and already carries the mechanism that fixes it. gitea is simply the second node process to need it; containerd is a node process and nobody had added it. An operator must run the node_baseline play. Also records, where an operator reads them, that the agent authenticates to Gitea with a local account and a scoped PAT by decision — and why the federated alternative is not merely unchosen but unavailable: Gitea implements only the Authorization Code Grant and validates no third-party OIDC token, so a Keycloak service-account token cannot authenticate to its API or to git-over-HTTP. Co-Authored-By: Claude Opus 5 (1M context) --- ...-06-pm-agent-node-resolves-gitea-to-wan.md | 111 ++++++++++++++++++ docs/plans/pm-agent-issue-to-pr.md | 14 ++- .../inventory/group_vars/k3s_prod/vars.yml | 23 ++++ platform/pm-agent/README.md | 48 ++++++-- .../pm-agent-credentials.sops.yaml.example | 9 ++ runbooks/pm-agent.md | 2 +- 6 files changed, 189 insertions(+), 18 deletions(-) create mode 100644 decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md diff --git a/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md new file mode 100644 index 0000000..623d7d1 --- /dev/null +++ b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md @@ -0,0 +1,111 @@ +--- +date: 2026-10-06 +branch: docs/pm-agent-registry-pull-prereq +title: The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin +--- + +## What changed + +The image pull for `pm-agent` failed TLS verification, and the previous entry on this branch +honestly recorded it as **not diagnosed**. The operator asked for the diagnosis before any +registry design. This is it, and the fix turns out to be a pattern this repository already +has. + +`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins +`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the +Keycloak issuer pin that is there for the identical reason. **An operator must run the +`node_baseline` play for it to reach the nodes** — writing `/etc/hosts` on a node is not +something Flux or an agent can do. + +The docs that described this as an undiagnosed blocker now describe the cause and the fix: +`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and +the play), and `docs/plans/pm-agent-issue-to-pr.md`. + +Also recorded, in the two places an operator reads when creating the credential: **the agent +authenticates to Gitea with a local account and a scoped PAT, by decision** (operator, +2026-10-06), not with a federated identity. Written down because the alternative looks +obviously better until you check whether it is possible — see below. + +## What I verified + +Every step measured, read-only, in this order. + +**The failing certificate is the router's, not the Gateway's.** + +- containerd's error names TLS, not the digest: `tls: failed to verify certificate: x509: + certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev`. + "not valid for **any** names" means the presented certificate carries no SANs at all. +- `192.168.2.20` is the shared Gateway's own LoadBalancer VIP + (`gateway/cilium-gateway-shared`, from the Cilium pool `192.168.2.20-30`), and its `https` + listener serves `*.home.keithjackson.dev` with `wildcard-home-tls`. From the operator's + workstation the name resolves there and serves the real Let's Encrypt + `CN=*.home.keithjackson.dev` with correct SANs. **So the certificate and the Gateway are + healthy.** +- **The node asks a different resolver and gets a different answer.** Prod's + `node_baseline_dns_servers` is the LAN gateway, not Pi-hole — deliberately, as that file + explains. Public DNS for this zone is a CNAME to `wan.home.keithjackson.dev`: measured via + both `1.1.1.1` and `8.8.8.8`, `gitea.home.keithjackson.dev` → + `wan.home.keithjackson.dev.` → `136.51.115.252`, and the apex answers the same. +- **That address serves a certificate with no names.** Measured from the workstation against + the WAN address with the right SNI: + + subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D + issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D + No extensions in certificate + + No extensions means no `subjectAltName`, which is exactly "not valid for any names". The + `CN=%D` is the router's placeholder. + +**This repository had already measured the same thing, for Keycloak, and I did not need to +re-derive it.** `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` says, above the pin: +"Since platform/gateway/gateway.yaml set an external-dns `target`, every public name is a +CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from +the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from +k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18)." Same cause, same symptom, +one name later. The pin exists precisely for node processes that must reach the Gateway +directly; containerd is one, and nobody had added it. + +**The parity the pin has to satisfy:** `192.168.2.20` equals the live +`GATEWAY_VIP`, and `k3s_base_domain` equals the live `BASE_DOMAIN` — both read from the +cluster, not from the comment that claims prod serves `prod.home.keithjackson.dev` (it does +not; `flux-system/cluster-vars` reports `BASE_DOMAIN=home.keithjackson.dev` and +`GATEWAY_VIP=192.168.2.20`, and Ansible's `k3s_base_domain` is `home.keithjackson.dev`). +That comment is stale history. + +**The credential decision, and why the obvious alternative is not available.** Gitea's API +accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2 +provider Gitea implements **only the Authorization Code Grant** (plus PKCE/OIDC), so there +is no client-credentials grant, and nothing in Gitea validates a token issued by a +third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to +Gitea's API or to git-over-HTTP. Read from Gitea's own documentation +(`docs.gitea.com/development/api-usage`, `/development/oauth2-provider`), not assumed. + +**Gate:** `./ci/gate.sh` in the pinned toolbox from a §0 worktree — see the branch's run; rc 3 +with `gitea-settings` the only stage that did not run. + +## What I did NOT verify + +- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the + nodes is written by the `node_baseline` play, which is an operator action. Until that runs + and an image exists, "the node now reaches the Gateway" is a design claim. The evidence + that it *will* is the Keycloak pin working for the same cause — which I also did not + re-measure. +- **Why public DNS answers at all for an `internal` service.** `catalog/gitea.yaml` sets no + `exposure`, so it inherits `internal`, and the renderer contract says an internal service's + HTTPRoute carries `external-dns.alpha.kubernetes.io/controller: none` and publishes no + public record. Yet `gitea.home.keithjackson.dev` resolves publicly to the WAN CNAME. I did + not chase that: it may be a zone-level wildcard rather than a published record, and it did + not change the diagnosis. It is worth someone's attention, because "internal services + publish no public DNS" is an L0-checked claim and this looks like a case where the *name* + resolves regardless of what the renderer withheld. +- **Registry authentication.** Unchanged and still open: an unauthenticated client gets + nothing from this registry, and no `imagePullSecret` exists anywhere in the tree. With the + credential model now decided (local Gitea accounts with PATs) the shape is a + `dockerconfigjson` for an account whose token carries `read:package`, referenced from the + Deployment and the committed worker Job template — and the worker Job template's shape is + pinned by admission, so whether the policy should also pin `imagePullSecrets` deserves a + deliberate look rather than a quiet addition. Not built: minting that token is an operator + action, and I did not want to guess at a second credential's scope. +- **Anything about runtime.** The image still does not exist. No dispatcher has + authenticated, no issue has been triaged, no worker Job has been created, and the admission + policy — live, `Enforce`, six rules — has still never admitted or refused a real Job. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index 37d7830..567ac69 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -236,12 +236,14 @@ about runtime behaviour in this document is a design claim, not a measurement. 1. **Make the registry pullable, THEN build, push, pin.** Measured after the merge, and neither half is declared anywhere in this repository (`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`): - (i) the cluster's nodes cannot verify the registry's TLS certificate — containerd reports - `x509: certificate is not valid for any names` for `gitea.home.keithjackson.dev`, while - that name serves a healthy Let's Encrypt wildcard from the LAN; (ii) an unauthenticated - client gets nothing from that registry and no `imagePullSecret` exists in the tree, so - either one is added or anonymous package reads are allowed. Only then does the digest - matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both + (i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the + router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning + `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same + pin the Keycloak issuer already carries; an operator must run the `node_baseline` play; + (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with + the credential decision now being local Gitea accounts with PATs, that means a + `dockerconfigjson` for an account whose token carries `read:package`. Only then does the + digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both placeholder digests. Expect the build itself to need iteration too: the nodejs module stream and the npm install have never run against this base. The same two blockers sit in front of `conformance-runner`, which shares the registry path and has never been pulled. diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index 4813196..2b865a6 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -327,6 +327,29 @@ node_baseline_hosts_pins: - ip: "192.168.2.20" names: - "keycloak.{{ k3s_base_domain }}" + # gitea: the SECOND node process to need this pin, for the identical reason, found + # 2026-10-06. containerd pulls images as a NODE process, so it resolves through the + # public upstreams above and gets the CNAME to wan.home.keithjackson.dev — the WAN IP, + # which this router does not hairpin. The pull then fails on the router's own + # placeholder certificate rather than on anything to do with the image: + # + # failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:…": + # tls: failed to verify certificate: x509: certificate is not valid for any names, + # but wanted to match gitea.home.keithjackson.dev + # + # Measured from the workstation the same day, against the WAN address with that SNI: + # subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D + # No extensions in certificate <- no SANs at all, hence "not valid for ANY names" + # while the same name via PiHole answers 192.168.2.20 and serves the real + # Let's Encrypt *.home.keithjackson.dev. So the certificate and the Gateway are fine; + # only the node's ANSWER is wrong. Same root cause, same fix, as the issuer pin above. + # + # WHO NEEDS IT: any workload whose image lives in the in-cluster Gitea registry — + # today platform/pm-agent (agent-runner) and conformance/incluster + # (conformance-runner), neither of which has ever been pulled. A pull is still + # UNAUTHENTICATED after this pin, which is a separate thing to settle; this only makes + # the node reach the right endpoint. + - "gitea.{{ k3s_base_domain }}" # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) -------------------- # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/ diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index ec68fb2..baedb1d 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -22,6 +22,25 @@ requires. --- +## Why the agent authenticates to Gitea with a local account and a PAT + +Not Keycloak, and that is a decision rather than an omission (operator, 2026-10-06): the +agents only ever talk to Gitea, so a local Gitea account with a scoped personal access token +is the whole requirement, and it is drastically simpler than federating a machine identity. + +It is also what Gitea can actually do. Keycloak service accounts were considered and are not +available for this: Gitea's API accepts basic auth, PATs, SSH signatures and **its own** +OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization +Code Grant** — there is no client-credentials grant, and nothing in Gitea validates a token +issued by a third-party OIDC provider. So a Keycloak service-account token cannot +authenticate to Gitea's API or to git-over-HTTP at all. Keycloak remains what it is for human +access: the browser login source registered on Gitea (`catalog/gitea.yaml`). + +What still matters, and is enforced by the hand-off rather than by Keycloak: **the account +must not be an administrator** — `infra/gitea/approval-policy.json` authorises +`/approve-override` on live repository permission alone, so an admin-permission bot could +waive the sensitive-path human review on its own pull request. + ## It does nothing at all until an operator does two things Both are deliberate, and both fail **visibly** rather than silently: @@ -38,17 +57,24 @@ Both are deliberate, and both fail **visibly** rather than silently: path. Details and the exact commands: [`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md). - - **The nodes must be able to verify the registry's TLS certificate.** containerd - currently fails with `x509: certificate is not valid for any names, but wanted to match - gitea.home.keithjackson.dev`, while the same name from the LAN serves a healthy - Let's Encrypt `*.home.keithjackson.dev`. Correcting the digest will not clear this. - - **The pull must be authenticated.** An unauthenticated client gets nothing from this - Gitea's registry (its anonymous token carries no access scope; every path 401s), and - there is no `imagePullSecret` anywhere in the tree. Either add one — a - `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the - Deployment *and* from the committed worker Job template — or allow anonymous reads of - that package in Gitea, which makes the image readable to anything that can reach the - host. That is a posture decision, so it is yours rather than an agent's. + - **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is + the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a + *node* process, so it resolves through the node's public upstreams + (`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public + answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN + IP — which this router does not hairpin; it answers :443 with its own placeholder + certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely + `x509: certificate is not valid for any names`. The Gateway, the wildcard certificate + and the digest were never implicated. The fix is the pin this repository already uses + for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now + in `node_baseline_hosts_pins`. **An operator must run the `node_baseline` play for it + to reach the nodes.** + - **The pull is still unauthenticated** once the node reaches the right endpoint. An + unauthenticated client gets nothing from this Gitea's registry, and there is no + `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision + (local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea + account whose token carries `read:package`, referenced from the Deployment and the + committed worker Job template — not built yet, because minting that token is yours. The same two apply to `conformance-runner`: same registry path, same push-then-pin model, never pulled either. diff --git a/platform/pm-agent/pm-agent-credentials.sops.yaml.example b/platform/pm-agent/pm-agent-credentials.sops.yaml.example index a7fe9fd..dd2a601 100644 --- a/platform/pm-agent/pm-agent-credentials.sops.yaml.example +++ b/platform/pm-agent/pm-agent-credentials.sops.yaml.example @@ -51,6 +51,15 @@ # (2) MINT THE GITEA TOKEN as that user (log in as `pm-agent`, or use the admin API to # create a token for it): Settings -> Applications -> Generate New Token. # +# A LOCAL GITEA ACCOUNT WITH A PAT IS THE DECIDED MODEL (operator, 2026-10-06), not +# Keycloak. The agents only ever talk to Gitea, so this is the whole requirement and it +# is far simpler than federating a machine identity. It is also the only thing that +# works: Gitea's API accepts basic auth, PATs, SSH signatures and its OWN OAuth2 bearer +# tokens, and as a provider Gitea implements only the Authorization Code Grant — no +# client-credentials grant, and nothing that validates a third-party OIDC token. A +# Keycloak service-account token therefore cannot authenticate to Gitea's API or to +# git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that. +# # Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT # `write:admin`, NOT `write:organization`, NOT `write:user`. # diff --git a/runbooks/pm-agent.md b/runbooks/pm-agent.md index 4a6a5f6..3a4c759 100644 --- a/runbooks/pm-agent.md +++ b/runbooks/pm-agent.md @@ -68,7 +68,7 @@ Work through these in order; the first three are by far the most likely. |---|---|---| | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` | | `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first | -| `ImagePullBackOff` with `x509: certificate is not valid for any names` | the **node** cannot verify the registry's TLS certificate. Nothing to do with the digest: a correct digest fails identically. The same name serves a healthy Let's Encrypt wildcard from the LAN, so what the node reaches is not what you reach | not diagnosed — see `decisions/2026-10-05-pm-agent-registry-pull-prereq.md` for everything measured. Check the nodes' resolution of `gitea.home.keithjackson.dev` and their registry trust (`/etc/rancher/k3s/registries.yaml`), neither of which this repository declares | +| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. **Run the `node_baseline` play** for it to reach the nodes. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | | `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours | | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` | | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob | -- 2.54.0 From 90a79d29d340c50b22ffe59a3a13cb5d2186d4c8 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 6 Oct 2026 00:44:26 -0500 Subject: [PATCH 4/4] docs(node): name the exact Ansible command that applies the pin, not 'run the play' MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Node configuration in this project is Ansible's — declared in the inventory, reviewed, version-controlled — and for prod it is the ONLY surface: infra/PROD-PROVISIONING.md records prod as 3x bare metal with the dev OpenTofu -> Proxmox -> cloud-init path explicitly not applying, so there is no tofu/cloud-init half to keep in parity for an /etc/hosts pin. The pin was already in the right place; what the docs lacked was the command. All four places now carry the scoped, idempotent one: cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \ -e k3s_hosts=k3s_prod --tags hosts --tags hosts runs ONLY the /etc/hosts task, so applying this pin cannot drift any other baseline setting. Also verified the mechanism is not inert, which is the failure mode this repo keeps meeting: roles/node_baseline/tasks/main.yml renders node_baseline_hosts_pins into a marked blockinfile with backup: true, and state: absent when the list empties — so a removed entry really does leave the node. Co-Authored-By: Claude Opus 5 (1M context) --- ...-06-pm-agent-node-resolves-gitea-to-wan.md | 23 +++++++++++++++---- docs/plans/pm-agent-issue-to-pr.md | 3 ++- platform/pm-agent/README.md | 6 +++-- runbooks/pm-agent.md | 2 +- 4 files changed, 26 insertions(+), 8 deletions(-) diff --git a/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md index 623d7d1..45ae653 100644 --- a/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md +++ b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md @@ -13,9 +13,21 @@ has. `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the -Keycloak issuer pin that is there for the identical reason. **An operator must run the -`node_baseline` play for it to reach the nodes** — writing `/etc/hosts` on a node is not -something Flux or an agent can do. +Keycloak issuer pin that is there for the identical reason. + +**It reaches the nodes the way every other node setting in this project does — through +Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs, +OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the +*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the +dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no +cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command: + + cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \ + -e k3s_hosts=k3s_prod --tags hosts + +`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other +baseline setting. Running the play is an operator action (it has become/root on prod), but +the configuration itself is declared, reviewed and version-controlled — not typed on a node. The docs that described this as an undiagnosed blocker now describe the cause and the fix: `platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and @@ -86,7 +98,10 @@ with `gitea-settings` the only stage that did not run. ## What I did NOT verify - **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the - nodes is written by the `node_baseline` play, which is an operator action. Until that runs + nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator + action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml` + renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and + `state: absent` when the list empties, so a removed entry really leaves the node. Until that runs and an image exists, "the node now reaches the Gateway" is a design claim. The evidence that it *will* is the Keycloak pin working for the same cause — which I also did not re-measure. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index 567ac69..ad87805 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -239,7 +239,8 @@ about runtime behaviour in this document is a design claim, not a measurement. (i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same - pin the Keycloak issuer already carries; an operator must run the `node_baseline` play; + pin the Keycloak issuer already carries, applied by + `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`; (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with the credential decision now being local Gitea accounts with PATs, that means a `dockerconfigjson` for an account whose token carries `read:package`. Only then does the diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index baedb1d..b201f37 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -67,8 +67,10 @@ Both are deliberate, and both fail **visibly** rather than silently: `x509: certificate is not valid for any names`. The Gateway, the wildcard certificate and the digest were never implicated. The fix is the pin this repository already uses for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now - in `node_baseline_hosts_pins`. **An operator must run the `node_baseline` play for it - to reach the nodes.** + in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the + inventory, never typed on a node. Apply it scoped to that one task: + `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod + --tags hosts`. - **The pull is still unauthenticated** once the node reaches the right endpoint. An unauthenticated client gets nothing from this Gitea's registry, and there is no `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision diff --git a/runbooks/pm-agent.md b/runbooks/pm-agent.md index 3a4c759..7f30319 100644 --- a/runbooks/pm-agent.md +++ b/runbooks/pm-agent.md @@ -68,7 +68,7 @@ Work through these in order; the first three are by far the most likely. |---|---|---| | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` | | `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first | -| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. **Run the `node_baseline` play** for it to reach the nodes. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | +| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | | `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours | | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` | | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob | -- 2.54.0