diff --git a/decisions/2026-10-05-pm-agent-registry-pull-prereq.md b/decisions/2026-10-05-pm-agent-registry-pull-prereq.md new file mode 100644 index 0000000..b4c4a58 --- /dev/null +++ b/decisions/2026-10-05-pm-agent-registry-pull-prereq.md @@ -0,0 +1,123 @@ +--- +date: 2026-10-05 +branch: docs/pm-agent-registry-pull-prereq +title: pm-agent reconciled exactly as designed — and is the first workload ever to pull from the in-cluster registry, which does not work yet +--- + +## What changed + +Documentation only. No manifest, no policy, no code. + +PR #57 merged and `pm-agent` reconciled. I measured what landed, and the measurement turned +up **two prerequisites for the next operator step that nothing in this repository +declares** — and that the handoff entry for #57 could not have known, because it listed the +image pull as unverified and that is exactly where it bit. + +So this entry records the post-merge measurement, and +`platform/pm-agent/README.md`, `docs/plans/pm-agent-issue-to-pr.md` and +`runbooks/pm-agent.md` now name both prerequisites where they say "build, push, pin the +digest". Before this, that step read as one action; it is three, and the two new ones are +not an agent's to perform. + +**The finding is not specific to pm-agent.** `conformance-runner` has the identical +registry path (`gitea.home.keithjackson.dev/home-network/...`), the identical "operator +pushes, a digest goes in a PR" model, and has never been pulled either — so the same two +blockers sit in front of the L2/L3 conformance wiring. Measured: **no pod anywhere in the +cluster references that registry** except the pm-agent dispatcher, which is therefore the +first workload ever to try it. + +## What I verified + +Read-only, through the pinned toolbox, against the live prod cluster. + +**The merge reconciled, completely and exactly as designed.** + +- `GitRepository/flux-system` artifact revision: `main@sha1:95597bd6883637…` — the merge + commit. `platform-config`, `pm-agent` and `apps-wave-0` all report that same + `lastAppliedRevision` with `Ready=True`. Compared field to field, per CLAUDE.md §2b, + rather than assumed from the merge. +- **The security boundary is live.** `ClusterPolicy/pm-agent-job-shape`: + `validationFailureAction=Enforce`, `failurePolicy=Fail`, `Ready=True`, and all six rules + present (`worker-must-be-the-committed-shape`, + `worker-must-be-one-container-with-no-args`, + `worker-volumes-must-be-workspace-or-tmp-emptydirs`, + `worker-may-not-declare-init-or-ephemeral-containers`, + `worker-env-names-must-be-the-committed-set`, `worker-may-not-use-envfrom`). This is the + check that mattered most: a ClusterPolicy added to the wrong kustomization list ships + green and never reaches the cluster, which is what happened to the conformance boundary + on 2026-09-29. +- Namespace contents: both ServiceAccounts, `pm-agent-config` (17 keys), the + `pm-agent-credentials` Secret, the `default-deny` NetworkPolicy, and both + CiliumNetworkPolicies with `VALID=True`. +- **SOPS decryption works through the new overlay patch.** Checked by KEY NAMES and VALUE + LENGTHS only, never values (CLAUDE.md §6): keys `gitea-token`, `claude-code-oauth-token`, + lengths 76 and 52 bytes — which are the base64 lengths of the committed placeholder + strings, so the Secret holds the decrypted placeholders and not ciphertext. A + ciphertext apply succeeds silently and would have left `ENC[AES256_GCM,…]` values an + order of magnitude longer. +- **`pm-agent-worker` really is bound to nothing.** The namespace has exactly one + RoleBinding (`pm-agent` → Role `pm-agent`, subject `ServiceAccount/pm-agent`), and no + ClusterRoleBinding anywhere names either pm-agent ServiceAccount. +- `gitea-allow-pm-agent-ingress` is applied in the `gitea` namespace, matching + `io.kubernetes.pod.namespace: pm-agent`. +- The dispatcher is `ImagePullBackOff` on the all-zero placeholder digest — the designed, + visible failure. + +**The two prerequisites, measured.** + +1. **The nodes cannot verify the registry's TLS certificate.** containerd's error on the + pull is not about the digest: + + failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:000…": + tls: failed to verify certificate: x509: certificate is not valid for any names, + but wanted to match gitea.home.keithjackson.dev + + "not valid for **any** names" means the certificate presented to the node carries no + matching SAN at all. From the operator's workstation the same name is healthy: it + resolves to `192.168.2.20` and serves a Let's Encrypt certificate + `CN=*.home.keithjackson.dev` with SANs `*.home.keithjackson.dev, home.keithjackson.dev` + — and `Certificate gateway/wildcard-home` is `Ready=True` for exactly those names. So + the certificate that *should* be served is fine, and what the NODE reaches is not what + the workstation reaches. **Correcting the digest will not clear this.** +2. **Anonymous pulls from this registry are refused.** Not inferred from a bare 401 — that + would have been wrong, because this Gitea answers 401 for *everything* anonymously: + `/v2/home-network/agent-runner/manifests/latest`, `/v2/home-network/conformance-runner/…`, + a randomly-named nonexistent repository, and `nosuchowner/nosuchimage` all return 401, + and `/api/v1/packages/Home-Network` returns `{"message":"token is required"}`. The + registry *does* issue a token to an anonymous client, but its `access` claim is `null` — + no pull scope. So the conclusion that holds is the simple one: **an unauthenticated + client gets nothing from this registry**, so a pull with no credential cannot succeed. + And `grep` over the tree finds **no `imagePullSecrets` anywhere** — not in + `platform/pm-agent/deployment.yaml`, not in `infra/agent/worker-job-template.yaml`, not + in `conformance/incluster/`. + +## What I did NOT verify + +- **Why the node sees a nameless certificate.** I measured that it does, and that the LAN + does not. I did not diagnose it: I did not read the nodes' resolver or + `/etc/rancher/k3s/registries.yaml` (node files, not reachable read-only through the API), + and I did not check whether the node resolves the name to the same `192.168.2.20` the + workstation does or to something else. Note `cluster-vars` has + `GATEWAY_VIP: 192.168.1.53` while the nodes are `192.168.2.10-12` and the name resolves + to `192.168.2.20` — three different subnets in one sentence is a reason to measure rather + than to theorise, and I have not. +- **Whether the package would be pullable once authenticated.** Nothing has ever been + pushed to that path, so "a credential is sufficient" is untested. The only thing + established is that no credential is insufficient. +- **Which fix the operator wants.** Two routes, and they are not equivalent: + (a) an `imagePullSecret` — a `dockerconfigjson` holding a Gitea token, SOPS-encrypted + like `pm-agent-credentials`, referenced from the Deployment *and* from the committed + worker Job template; note the Job template's shape is pinned by admission, and + `imagePullSecrets` is a field that policy does not currently constrain, so adding one is + a template change that needs a deliberate look at whether the policy should pin it too. + (b) allowing anonymous reads of that package in Gitea — no manifest change at all, but it + makes the image world-readable to anything that can reach the host. + I have implemented neither: both need an operator action I must not take, and (b) is a + posture decision. The docs now present both. +- **Everything the #57 entry listed as unverified about RUNTIME remains unverified.** The + image has still never been built or pushed. No dispatcher has authenticated to Gitea, no + issue has been triaged, no worker Job has been created — so the admission policy, though + now provably applied and Enforce, has still never refused or admitted a real Job. The + network policies have not carried traffic. +- **I did not re-run the merge gate against `main` after the merge.** This branch's own gate + run is the evidence for this change; the merged state's is CI run on PR #57. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index f6f42e1..37d7830 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -233,9 +233,18 @@ about runtime behaviour in this document is a design claim, not a measurement. ## Follow-ups, in the order they are worth doing -1. **Build, push, pin.** Operator runs `infra/agent/build-image.sh`, pushes, and one PR - replaces both placeholder digests. Expect the build to need iteration: the nodejs module - stream and the npm install have never run against this base. +1. **Make the registry pullable, THEN build, push, pin.** Measured after the merge, and + neither half is declared anywhere in this repository + (`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`): + (i) the cluster's nodes cannot verify the registry's TLS certificate — containerd reports + `x509: certificate is not valid for any names` for `gitea.home.keithjackson.dev`, while + that name serves a healthy Let's Encrypt wildcard from the LAN; (ii) an unauthenticated + client gets nothing from that registry and no `imagePullSecret` exists in the tree, so + either one is added or anonymous package reads are allowed. Only then does the digest + matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both + placeholder digests. Expect the build itself to need iteration too: the nodejs module + stream and the npm install have never run against this base. The same two blockers sit in + front of `conformance-runner`, which shares the registry path and has never been pulled. 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the bot's permission is `write`, not `admin`** — that measurement is the control that keeps the agent out of its own approval gate. diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index ddbbbbd..732ea30 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -31,6 +31,27 @@ Both are deliberate, and both fail **visibly** rather than silently: `infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the operator, then update **both** places with the digest `podman push --digestfile` returned, in one pull request. Until then: `ImagePullBackOff`. + + **Two things have to be true before any digest will pull, and neither is declared in this + repository.** Both were measured after PR #57 merged — this service is the first workload + in the cluster ever to reference the in-cluster registry, so nothing had exercised the + path. Details and the exact commands: + [`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md). + + - **The nodes must be able to verify the registry's TLS certificate.** containerd + currently fails with `x509: certificate is not valid for any names, but wanted to match + gitea.home.keithjackson.dev`, while the same name from the LAN serves a healthy + Let's Encrypt `*.home.keithjackson.dev`. Correcting the digest will not clear this. + - **The pull must be authenticated.** An unauthenticated client gets nothing from this + Gitea's registry (its anonymous token carries no access scope; every path 401s), and + there is no `imagePullSecret` anywhere in the tree. Either add one — a + `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the + Deployment *and* from the committed worker Job template — or allow anonymous reads of + that package in Gitea, which makes the image readable to anything that can reach the + host. That is a posture decision, so it is yours rather than an agent's. + + The same two apply to `conformance-runner`: same registry path, same push-then-pin model, + never pulled either. 2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off — which Gitea account to create, which scopes, and why it must *not* be an administrator — diff --git a/runbooks/pm-agent.md b/runbooks/pm-agent.md index b5b852d..4a6a5f6 100644 --- a/runbooks/pm-agent.md +++ b/runbooks/pm-agent.md @@ -67,7 +67,9 @@ Work through these in order; the first three are by far the most likely. | Symptom in the log | Cause | Fix | |---|---|---| | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` | -| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) | +| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first | +| `ImagePullBackOff` with `x509: certificate is not valid for any names` | the **node** cannot verify the registry's TLS certificate. Nothing to do with the digest: a correct digest fails identically. The same name serves a healthy Let's Encrypt wildcard from the LAN, so what the node reaches is not what you reach | not diagnosed — see `decisions/2026-10-05-pm-agent-registry-pull-prereq.md` for everything measured. Check the nodes' resolution of `gitea.home.keithjackson.dev` and their registry trust (`/etc/rancher/k3s/registries.yaml`), neither of which this repository declares | +| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours | | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` | | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob | | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |