diff --git a/decisions/2026-10-05-pm-agent-registry-pull-prereq.md b/decisions/2026-10-05-pm-agent-registry-pull-prereq.md new file mode 100644 index 0000000..bb57d27 --- /dev/null +++ b/decisions/2026-10-05-pm-agent-registry-pull-prereq.md @@ -0,0 +1,127 @@ +--- +date: 2026-10-05 +branch: docs/pm-agent-registry-pull-prereq +title: pm-agent reconciled exactly as designed — and is the first workload ever to pull from the in-cluster registry, which does not work yet +--- + +## What changed + +Documentation only. No manifest, no policy, no code. + +PR #57 merged and `pm-agent` reconciled. I measured what landed, and the measurement turned +up **two prerequisites for the next operator step that nothing in this repository +declares** — and that the handoff entry for #57 could not have known, because it listed the +image pull as unverified and that is exactly where it bit. + +So this entry records the post-merge measurement, and +`platform/pm-agent/README.md`, `docs/plans/pm-agent-issue-to-pr.md` and +`runbooks/pm-agent.md` now name both prerequisites where they say "build, push, pin the +digest". Before this, that step read as one action; it is three, and the two new ones are +not an agent's to perform. + +**The finding is not specific to pm-agent.** `conformance-runner` has the identical +registry path (`gitea.home.keithjackson.dev/home-network/...`), the identical "operator +pushes, a digest goes in a PR" model, and has never been pulled either — so the same two +blockers sit in front of the L2/L3 conformance wiring. Measured: **no pod anywhere in the +cluster references that registry** except the pm-agent dispatcher, which is therefore the +first workload ever to try it. + +## What I verified + +Read-only, through the pinned toolbox, against the live prod cluster. + +**The merge reconciled, completely and exactly as designed.** + +- `GitRepository/flux-system` artifact revision: `main@sha1:95597bd6883637…` — the merge + commit. `platform-config`, `pm-agent` and `apps-wave-0` all report that same + `lastAppliedRevision` with `Ready=True`. Compared field to field, per CLAUDE.md §2b, + rather than assumed from the merge. +- **The security boundary is live.** `ClusterPolicy/pm-agent-job-shape`: + `validationFailureAction=Enforce`, `failurePolicy=Fail`, `Ready=True`, and all six rules + present (`worker-must-be-the-committed-shape`, + `worker-must-be-one-container-with-no-args`, + `worker-volumes-must-be-workspace-or-tmp-emptydirs`, + `worker-may-not-declare-init-or-ephemeral-containers`, + `worker-env-names-must-be-the-committed-set`, `worker-may-not-use-envfrom`). This is the + check that mattered most: a ClusterPolicy added to the wrong kustomization list ships + green and never reaches the cluster, which is what happened to the conformance boundary + on 2026-09-29. +- Namespace contents: both ServiceAccounts, `pm-agent-config` (17 keys), the + `pm-agent-credentials` Secret, the `default-deny` NetworkPolicy, and both + CiliumNetworkPolicies with `VALID=True`. +- **SOPS decryption works through the new overlay patch.** Checked by KEY NAMES and VALUE + LENGTHS only, never values (CLAUDE.md §6): keys `gitea-token`, `claude-code-oauth-token`, + lengths 76 and 52 bytes — which are the base64 lengths of the committed placeholder + strings, so the Secret holds the decrypted placeholders and not ciphertext. A + ciphertext apply succeeds silently and would have left `ENC[AES256_GCM,…]` values an + order of magnitude longer. +- **`pm-agent-worker` really is bound to nothing.** The namespace has exactly one + RoleBinding (`pm-agent` → Role `pm-agent`, subject `ServiceAccount/pm-agent`), and no + ClusterRoleBinding anywhere names either pm-agent ServiceAccount. +- `gitea-allow-pm-agent-ingress` is applied in the `gitea` namespace, matching + `io.kubernetes.pod.namespace: pm-agent`. +- The dispatcher is `ImagePullBackOff` on the all-zero placeholder digest — the designed, + visible failure. + +**The two prerequisites, measured.** + +1. **The nodes cannot verify the registry's TLS certificate.** containerd's error on the + pull is not about the digest: + + failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:000…": + tls: failed to verify certificate: x509: certificate is not valid for any names, + but wanted to match gitea.home.keithjackson.dev + + "not valid for **any** names" means the certificate presented to the node carries no + matching SAN at all. From the operator's workstation the same name is healthy: it + resolves to `192.168.2.20` and serves a Let's Encrypt certificate + `CN=*.home.keithjackson.dev` with SANs `*.home.keithjackson.dev, home.keithjackson.dev` + — and `Certificate gateway/wildcard-home` is `Ready=True` for exactly those names. So + the certificate that *should* be served is fine, and what the NODE reaches is not what + the workstation reaches. **Correcting the digest will not clear this.** +2. **Anonymous pulls from this registry are refused.** Not inferred from a bare 401 — that + would have been wrong, because this Gitea answers 401 for *everything* anonymously: + `/v2/home-network/agent-runner/manifests/latest`, `/v2/home-network/conformance-runner/…`, + a randomly-named nonexistent repository, and `nosuchowner/nosuchimage` all return 401, + and `/api/v1/packages/Home-Network` returns `{"message":"token is required"}`. The + registry *does* issue a token to an anonymous client, but its `access` claim is `null` — + no pull scope. So the conclusion that holds is the simple one: **an unauthenticated + client gets nothing from this registry**, so a pull with no credential cannot succeed. + And `grep` over the tree finds **no `imagePullSecrets` anywhere** — not in + `platform/pm-agent/deployment.yaml`, not in `infra/agent/worker-job-template.yaml`, not + in `conformance/incluster/`. + +## What I did NOT verify + +- **Why the node sees a nameless certificate.** I measured that it does, and that the LAN + does not. I did not diagnose it: I did not read the nodes' resolver or + `/etc/rancher/k3s/registries.yaml` (node files, not reachable read-only through the API), + and I did not check whether the node resolves the name to the same `192.168.2.20` the + workstation does or to something else. + + **A correction to a remark I first wrote here:** I cited + `cluster-vars`'s `GATEWAY_VIP: 192.168.1.53` as suspicious next to the nodes' + `192.168.2.x`. That was noise — the operator confirms those base `cluster-vars` values are + **dev-era leftovers for a cluster that is decommissioned**, so the 192.168.1.x addresses in + that file say nothing about prod and should not be read as evidence about anything. The + measured facts stand on their own and none of them depended on that knob. +- **Whether the package would be pullable once authenticated.** Nothing has ever been + pushed to that path, so "a credential is sufficient" is untested. The only thing + established is that no credential is insufficient. +- **Which fix the operator wants.** Two routes, and they are not equivalent: + (a) an `imagePullSecret` — a `dockerconfigjson` holding a Gitea token, SOPS-encrypted + like `pm-agent-credentials`, referenced from the Deployment *and* from the committed + worker Job template; note the Job template's shape is pinned by admission, and + `imagePullSecrets` is a field that policy does not currently constrain, so adding one is + a template change that needs a deliberate look at whether the policy should pin it too. + (b) allowing anonymous reads of that package in Gitea — no manifest change at all, but it + makes the image world-readable to anything that can reach the host. + I have implemented neither: both need an operator action I must not take, and (b) is a + posture decision. The docs now present both. +- **Everything the #57 entry listed as unverified about RUNTIME remains unverified.** The + image has still never been built or pushed. No dispatcher has authenticated to Gitea, no + issue has been triaged, no worker Job has been created — so the admission policy, though + now provably applied and Enforce, has still never refused or admitted a real Job. The + network policies have not carried traffic. +- **I did not re-run the merge gate against `main` after the merge.** This branch's own gate + run is the evidence for this change; the merged state's is CI run on PR #57. diff --git a/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md new file mode 100644 index 0000000..45ae653 --- /dev/null +++ b/decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md @@ -0,0 +1,126 @@ +--- +date: 2026-10-06 +branch: docs/pm-agent-registry-pull-prereq +title: The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin +--- + +## What changed + +The image pull for `pm-agent` failed TLS verification, and the previous entry on this branch +honestly recorded it as **not diagnosed**. The operator asked for the diagnosis before any +registry design. This is it, and the fix turns out to be a pattern this repository already +has. + +`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins +`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the +Keycloak issuer pin that is there for the identical reason. + +**It reaches the nodes the way every other node setting in this project does — through +Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs, +OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the +*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the +dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no +cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command: + + cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \ + -e k3s_hosts=k3s_prod --tags hosts + +`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other +baseline setting. Running the play is an operator action (it has become/root on prod), but +the configuration itself is declared, reviewed and version-controlled — not typed on a node. + +The docs that described this as an undiagnosed blocker now describe the cause and the fix: +`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and +the play), and `docs/plans/pm-agent-issue-to-pr.md`. + +Also recorded, in the two places an operator reads when creating the credential: **the agent +authenticates to Gitea with a local account and a scoped PAT, by decision** (operator, +2026-10-06), not with a federated identity. Written down because the alternative looks +obviously better until you check whether it is possible — see below. + +## What I verified + +Every step measured, read-only, in this order. + +**The failing certificate is the router's, not the Gateway's.** + +- containerd's error names TLS, not the digest: `tls: failed to verify certificate: x509: + certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev`. + "not valid for **any** names" means the presented certificate carries no SANs at all. +- `192.168.2.20` is the shared Gateway's own LoadBalancer VIP + (`gateway/cilium-gateway-shared`, from the Cilium pool `192.168.2.20-30`), and its `https` + listener serves `*.home.keithjackson.dev` with `wildcard-home-tls`. From the operator's + workstation the name resolves there and serves the real Let's Encrypt + `CN=*.home.keithjackson.dev` with correct SANs. **So the certificate and the Gateway are + healthy.** +- **The node asks a different resolver and gets a different answer.** Prod's + `node_baseline_dns_servers` is the LAN gateway, not Pi-hole — deliberately, as that file + explains. Public DNS for this zone is a CNAME to `wan.home.keithjackson.dev`: measured via + both `1.1.1.1` and `8.8.8.8`, `gitea.home.keithjackson.dev` → + `wan.home.keithjackson.dev.` → `136.51.115.252`, and the apex answers the same. +- **That address serves a certificate with no names.** Measured from the workstation against + the WAN address with the right SNI: + + subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D + issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D + No extensions in certificate + + No extensions means no `subjectAltName`, which is exactly "not valid for any names". The + `CN=%D` is the router's placeholder. + +**This repository had already measured the same thing, for Keycloak, and I did not need to +re-derive it.** `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` says, above the pin: +"Since platform/gateway/gateway.yaml set an external-dns `target`, every public name is a +CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from +the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from +k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18)." Same cause, same symptom, +one name later. The pin exists precisely for node processes that must reach the Gateway +directly; containerd is one, and nobody had added it. + +**The parity the pin has to satisfy:** `192.168.2.20` equals the live +`GATEWAY_VIP`, and `k3s_base_domain` equals the live `BASE_DOMAIN` — both read from the +cluster, not from the comment that claims prod serves `prod.home.keithjackson.dev` (it does +not; `flux-system/cluster-vars` reports `BASE_DOMAIN=home.keithjackson.dev` and +`GATEWAY_VIP=192.168.2.20`, and Ansible's `k3s_base_domain` is `home.keithjackson.dev`). +That comment is stale history. + +**The credential decision, and why the obvious alternative is not available.** Gitea's API +accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2 +provider Gitea implements **only the Authorization Code Grant** (plus PKCE/OIDC), so there +is no client-credentials grant, and nothing in Gitea validates a token issued by a +third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to +Gitea's API or to git-over-HTTP. Read from Gitea's own documentation +(`docs.gitea.com/development/api-usage`, `/development/oauth2-provider`), not assumed. + +**Gate:** `./ci/gate.sh` in the pinned toolbox from a §0 worktree — see the branch's run; rc 3 +with `gitea-settings` the only stage that did not run. + +## What I did NOT verify + +- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the + nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator + action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml` + renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and + `state: absent` when the list empties, so a removed entry really leaves the node. Until that runs + and an image exists, "the node now reaches the Gateway" is a design claim. The evidence + that it *will* is the Keycloak pin working for the same cause — which I also did not + re-measure. +- **Why public DNS answers at all for an `internal` service.** `catalog/gitea.yaml` sets no + `exposure`, so it inherits `internal`, and the renderer contract says an internal service's + HTTPRoute carries `external-dns.alpha.kubernetes.io/controller: none` and publishes no + public record. Yet `gitea.home.keithjackson.dev` resolves publicly to the WAN CNAME. I did + not chase that: it may be a zone-level wildcard rather than a published record, and it did + not change the diagnosis. It is worth someone's attention, because "internal services + publish no public DNS" is an L0-checked claim and this looks like a case where the *name* + resolves regardless of what the renderer withheld. +- **Registry authentication.** Unchanged and still open: an unauthenticated client gets + nothing from this registry, and no `imagePullSecret` exists anywhere in the tree. With the + credential model now decided (local Gitea accounts with PATs) the shape is a + `dockerconfigjson` for an account whose token carries `read:package`, referenced from the + Deployment and the committed worker Job template — and the worker Job template's shape is + pinned by admission, so whether the policy should also pin `imagePullSecrets` deserves a + deliberate look rather than a quiet addition. Not built: minting that token is an operator + action, and I did not want to guess at a second credential's scope. +- **Anything about runtime.** The image still does not exist. No dispatcher has + authenticated, no issue has been triaged, no worker Job has been created, and the admission + policy — live, `Enforce`, six rules — has still never admitted or refused a real Job. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index f6f42e1..ad87805 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -233,9 +233,21 @@ about runtime behaviour in this document is a design claim, not a measurement. ## Follow-ups, in the order they are worth doing -1. **Build, push, pin.** Operator runs `infra/agent/build-image.sh`, pushes, and one PR - replaces both placeholder digests. Expect the build to need iteration: the nodejs module - stream and the npm install have never run against this base. +1. **Make the registry pullable, THEN build, push, pin.** Measured after the merge, and + neither half is declared anywhere in this repository + (`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`): + (i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the + router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning + `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same + pin the Keycloak issuer already carries, applied by + `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`; + (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with + the credential decision now being local Gitea accounts with PATs, that means a + `dockerconfigjson` for an account whose token carries `read:package`. Only then does the + digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both + placeholder digests. Expect the build itself to need iteration too: the nodejs module + stream and the npm install have never run against this base. The same two blockers sit in + front of `conformance-runner`, which shares the registry path and has never been pulled. 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the bot's permission is `write`, not `admin`** — that measurement is the control that keeps the agent out of its own approval gate. diff --git a/infra/agent/pm-worker b/infra/agent/pm-worker index 5f7be31..fe8663d 100755 --- a/infra/agent/pm-worker +++ b/infra/agent/pm-worker @@ -90,9 +90,19 @@ esac # pointing at an external host would be REACHABLE: it could clone a repository that is not # this one and push this repository's work somewhere else. The allowlist is baked in here, # where changing it requires rebuilding the image whose digest admission pins. +# +# ONE ENTRY, AND IT IS THE CLUSTER-LOCAL SERVICE NAME. This agent runs INSIDE the cluster, so +# it reaches Gitea the way every other in-cluster consumer does — `*.svc.cluster.local` — +# which is by design a different path from how a client outside the cluster reaches the same +# service. The external hostname `gitea.home.keithjackson.dev` used to be a second accepted +# value here; it is removed deliberately. It was never needed (nothing sets it), it routes +# in-cluster traffic out through the Gateway and back for no reason, and as an ACCEPTED value +# it widened this allowlist to a host reachable over the `world:443` egress rule — which is +# the exact shape of injection the allowlist exists to refuse. The cluster-local name also +# depends on none of public DNS, Cloudflare, the Gateway or the certificate. case "${PM_GITEA_URL}" in - http://gitea.gitea.svc.cluster.local:3000|https://gitea.home.keithjackson.dev) ;; - *) die "PM_GITEA_URL '${PM_GITEA_URL}' is not an allowed git host (see this script's header)" ;; + http://gitea.gitea.svc.cluster.local:3000) ;; + *) die "PM_GITEA_URL '${PM_GITEA_URL}' is not the allowed in-cluster git host (see this script's header)" ;; esac [[ "${PM_GITEA_OWNER}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,38}$ ]] || die "PM_GITEA_OWNER is malformed" [[ "${PM_GITEA_REPO}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$ ]] || die "PM_GITEA_REPO is malformed" diff --git a/infra/agent/probe-validation.sh b/infra/agent/probe-validation.sh index 2690e8c..042d004 100755 --- a/infra/agent/probe-validation.sh +++ b/infra/agent/probe-validation.sh @@ -86,12 +86,22 @@ refuses "branch equal to the base branch" "must not be the base branch" \ # --- where code comes FROM and goes TO ------------------------------------- # The namespace's egress policy permits tcp/443 to the world (the model needs it), so an # injected host would be REACHABLE. These are the cases that matter most. -refuses "a redirected git host" "is not an allowed git host" \ +refuses "a redirected git host" "is not the allowed in-cluster git host" \ PM_GITEA_URL=https://evil.invalid -refuses "a real, reachable, WRONG git host" "is not an allowed git host" \ +refuses "a real, reachable, WRONG git host" "is not the allowed in-cluster git host" \ PM_GITEA_URL=https://github.com -refuses "the right host on a smuggled path" "is not an allowed git host" \ +refuses "the right host on a smuggled path" "is not the allowed in-cluster git host" \ PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000/../.. +# This agent is IN the cluster, so it reaches Gitea by its cluster-local service name. The +# EXTERNAL hostname is refused too — it used to be a second accepted value, and as an +# accepted value it widened this allowlist to a host reachable over the `world:443` egress +# rule, which is the injection shape the allowlist exists to refuse. +refuses "this Gitea's EXTERNAL hostname" "is not the allowed in-cluster git host" \ + PM_GITEA_URL=https://gitea.home.keithjackson.dev +refuses "the cluster-local name over https" "is not the allowed in-cluster git host" \ + PM_GITEA_URL=https://gitea.gitea.svc.cluster.local:3000 +refuses "the cluster-local name, wrong port" "is not the allowed in-cluster git host" \ + PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:8080 refuses "a malformed owner" "PM_GITEA_OWNER is malformed" \ 'PM_GITEA_OWNER=../other-org' refuses "a malformed repository" "PM_GITEA_REPO is malformed" \ diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index 4813196..2b865a6 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -327,6 +327,29 @@ node_baseline_hosts_pins: - ip: "192.168.2.20" names: - "keycloak.{{ k3s_base_domain }}" + # gitea: the SECOND node process to need this pin, for the identical reason, found + # 2026-10-06. containerd pulls images as a NODE process, so it resolves through the + # public upstreams above and gets the CNAME to wan.home.keithjackson.dev — the WAN IP, + # which this router does not hairpin. The pull then fails on the router's own + # placeholder certificate rather than on anything to do with the image: + # + # failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:…": + # tls: failed to verify certificate: x509: certificate is not valid for any names, + # but wanted to match gitea.home.keithjackson.dev + # + # Measured from the workstation the same day, against the WAN address with that SNI: + # subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D + # No extensions in certificate <- no SANs at all, hence "not valid for ANY names" + # while the same name via PiHole answers 192.168.2.20 and serves the real + # Let's Encrypt *.home.keithjackson.dev. So the certificate and the Gateway are fine; + # only the node's ANSWER is wrong. Same root cause, same fix, as the issuer pin above. + # + # WHO NEEDS IT: any workload whose image lives in the in-cluster Gitea registry — + # today platform/pm-agent (agent-runner) and conformance/incluster + # (conformance-runner), neither of which has ever been pulled. A pull is still + # UNAUTHENTICATED after this pin, which is a separate thing to settle; this only makes + # the node reach the right endpoint. + - "gitea.{{ k3s_base_domain }}" # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) -------------------- # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/ diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index ddbbbbd..b201f37 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -22,6 +22,25 @@ requires. --- +## Why the agent authenticates to Gitea with a local account and a PAT + +Not Keycloak, and that is a decision rather than an omission (operator, 2026-10-06): the +agents only ever talk to Gitea, so a local Gitea account with a scoped personal access token +is the whole requirement, and it is drastically simpler than federating a machine identity. + +It is also what Gitea can actually do. Keycloak service accounts were considered and are not +available for this: Gitea's API accepts basic auth, PATs, SSH signatures and **its own** +OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization +Code Grant** — there is no client-credentials grant, and nothing in Gitea validates a token +issued by a third-party OIDC provider. So a Keycloak service-account token cannot +authenticate to Gitea's API or to git-over-HTTP at all. Keycloak remains what it is for human +access: the browser login source registered on Gitea (`catalog/gitea.yaml`). + +What still matters, and is enforced by the hand-off rather than by Keycloak: **the account +must not be an administrator** — `infra/gitea/approval-policy.json` authorises +`/approve-override` on live repository permission alone, so an admin-permission bot could +waive the sensitive-path human review on its own pull request. + ## It does nothing at all until an operator does two things Both are deliberate, and both fail **visibly** rather than silently: @@ -31,6 +50,36 @@ Both are deliberate, and both fail **visibly** rather than silently: `infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the operator, then update **both** places with the digest `podman push --digestfile` returned, in one pull request. Until then: `ImagePullBackOff`. + + **Two things have to be true before any digest will pull, and neither is declared in this + repository.** Both were measured after PR #57 merged — this service is the first workload + in the cluster ever to reference the in-cluster registry, so nothing had exercised the + path. Details and the exact commands: + [`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md). + + - **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is + the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a + *node* process, so it resolves through the node's public upstreams + (`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public + answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN + IP — which this router does not hairpin; it answers :443 with its own placeholder + certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely + `x509: certificate is not valid for any names`. The Gateway, the wildcard certificate + and the digest were never implicated. The fix is the pin this repository already uses + for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now + in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the + inventory, never typed on a node. Apply it scoped to that one task: + `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod + --tags hosts`. + - **The pull is still unauthenticated** once the node reaches the right endpoint. An + unauthenticated client gets nothing from this Gitea's registry, and there is no + `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision + (local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea + account whose token carries `read:package`, referenced from the Deployment and the + committed worker Job template — not built yet, because minting that token is yours. + + The same two apply to `conformance-runner`: same registry path, same push-then-pin model, + never pulled either. 2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off — which Gitea account to create, which scopes, and why it must *not* be an administrator — @@ -96,7 +145,7 @@ Job from a committed template. **The model's output is data, never a command.** | **cannot** | merge anything — the gate is the approver, and the service makes no merge call | | **cannot** | reach the Kubernetes API: no ServiceAccount token is mounted, the `pm-agent-worker` identity is bound to nothing, and the egress policy does not include the API server. `kubectl` is on its `PATH` (the gate's toolchain needs it) and is inert | | **cannot** | reach anything except this Gitea and tcp/443 — no package registry, no GitHub | -| **cannot** | clone from or push to any host but this Gitea: `pm-worker` validates `PM_GITEA_URL` against an allowlist baked into the image | +| **cannot** | clone from or push to anywhere but `gitea.gitea.svc.cluster.local:3000`: `pm-worker` validates `PM_GITEA_URL` against a one-entry allowlist baked into the image. In-cluster consumers reach a service by its cluster-local name — a different path by design from how a client outside the cluster reaches it — so the external hostname is not an accepted value | | **cannot** | authenticate a `git push` **during the model session**: `GIT_ASKPASS` is unset for its duration, so the restriction is structural rather than a request in a prompt | | **cannot** | be created in any shape but the committed one — see the admission policy | diff --git a/platform/pm-agent/pm-agent-credentials.sops.yaml.example b/platform/pm-agent/pm-agent-credentials.sops.yaml.example index a7fe9fd..dd2a601 100644 --- a/platform/pm-agent/pm-agent-credentials.sops.yaml.example +++ b/platform/pm-agent/pm-agent-credentials.sops.yaml.example @@ -51,6 +51,15 @@ # (2) MINT THE GITEA TOKEN as that user (log in as `pm-agent`, or use the admin API to # create a token for it): Settings -> Applications -> Generate New Token. # +# A LOCAL GITEA ACCOUNT WITH A PAT IS THE DECIDED MODEL (operator, 2026-10-06), not +# Keycloak. The agents only ever talk to Gitea, so this is the whole requirement and it +# is far simpler than federating a machine identity. It is also the only thing that +# works: Gitea's API accepts basic auth, PATs, SSH signatures and its OWN OAuth2 bearer +# tokens, and as a provider Gitea implements only the Authorization Code Grant — no +# client-credentials grant, and nothing that validates a third-party OIDC token. A +# Keycloak service-account token therefore cannot authenticate to Gitea's API or to +# git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that. +# # Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT # `write:admin`, NOT `write:organization`, NOT `write:user`. # diff --git a/runbooks/pm-agent.md b/runbooks/pm-agent.md index b5b852d..7f30319 100644 --- a/runbooks/pm-agent.md +++ b/runbooks/pm-agent.md @@ -67,7 +67,9 @@ Work through these in order; the first three are by far the most likely. | Symptom in the log | Cause | Fix | |---|---|---| | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` | -| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) | +| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first | +| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` | +| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours | | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` | | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob | | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |