--- date: 2026-10-05 branch: docs/pm-agent-registry-pull-prereq title: pm-agent reconciled exactly as designed — and is the first workload ever to pull from the in-cluster registry, which does not work yet --- ## What changed Documentation only. No manifest, no policy, no code. PR #57 merged and `pm-agent` reconciled. I measured what landed, and the measurement turned up **two prerequisites for the next operator step that nothing in this repository declares** — and that the handoff entry for #57 could not have known, because it listed the image pull as unverified and that is exactly where it bit. So this entry records the post-merge measurement, and `platform/pm-agent/README.md`, `docs/plans/pm-agent-issue-to-pr.md` and `runbooks/pm-agent.md` now name both prerequisites where they say "build, push, pin the digest". Before this, that step read as one action; it is three, and the two new ones are not an agent's to perform. **The finding is not specific to pm-agent.** `conformance-runner` has the identical registry path (`gitea.home.keithjackson.dev/home-network/...`), the identical "operator pushes, a digest goes in a PR" model, and has never been pulled either — so the same two blockers sit in front of the L2/L3 conformance wiring. Measured: **no pod anywhere in the cluster references that registry** except the pm-agent dispatcher, which is therefore the first workload ever to try it. ## What I verified Read-only, through the pinned toolbox, against the live prod cluster. **The merge reconciled, completely and exactly as designed.** - `GitRepository/flux-system` artifact revision: `main@sha1:95597bd6883637…` — the merge commit. `platform-config`, `pm-agent` and `apps-wave-0` all report that same `lastAppliedRevision` with `Ready=True`. Compared field to field, per CLAUDE.md §2b, rather than assumed from the merge. - **The security boundary is live.** `ClusterPolicy/pm-agent-job-shape`: `validationFailureAction=Enforce`, `failurePolicy=Fail`, `Ready=True`, and all six rules present (`worker-must-be-the-committed-shape`, `worker-must-be-one-container-with-no-args`, `worker-volumes-must-be-workspace-or-tmp-emptydirs`, `worker-may-not-declare-init-or-ephemeral-containers`, `worker-env-names-must-be-the-committed-set`, `worker-may-not-use-envfrom`). This is the check that mattered most: a ClusterPolicy added to the wrong kustomization list ships green and never reaches the cluster, which is what happened to the conformance boundary on 2026-09-29. - Namespace contents: both ServiceAccounts, `pm-agent-config` (17 keys), the `pm-agent-credentials` Secret, the `default-deny` NetworkPolicy, and both CiliumNetworkPolicies with `VALID=True`. - **SOPS decryption works through the new overlay patch.** Checked by KEY NAMES and VALUE LENGTHS only, never values (CLAUDE.md §6): keys `gitea-token`, `claude-code-oauth-token`, lengths 76 and 52 bytes — which are the base64 lengths of the committed placeholder strings, so the Secret holds the decrypted placeholders and not ciphertext. A ciphertext apply succeeds silently and would have left `ENC[AES256_GCM,…]` values an order of magnitude longer. - **`pm-agent-worker` really is bound to nothing.** The namespace has exactly one RoleBinding (`pm-agent` → Role `pm-agent`, subject `ServiceAccount/pm-agent`), and no ClusterRoleBinding anywhere names either pm-agent ServiceAccount. - `gitea-allow-pm-agent-ingress` is applied in the `gitea` namespace, matching `io.kubernetes.pod.namespace: pm-agent`. - The dispatcher is `ImagePullBackOff` on the all-zero placeholder digest — the designed, visible failure. **The two prerequisites, measured.** 1. **The nodes cannot verify the registry's TLS certificate.** containerd's error on the pull is not about the digest: failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:000…": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev "not valid for **any** names" means the certificate presented to the node carries no matching SAN at all. From the operator's workstation the same name is healthy: it resolves to `192.168.2.20` and serves a Let's Encrypt certificate `CN=*.home.keithjackson.dev` with SANs `*.home.keithjackson.dev, home.keithjackson.dev` — and `Certificate gateway/wildcard-home` is `Ready=True` for exactly those names. So the certificate that *should* be served is fine, and what the NODE reaches is not what the workstation reaches. **Correcting the digest will not clear this.** 2. **Anonymous pulls from this registry are refused.** Not inferred from a bare 401 — that would have been wrong, because this Gitea answers 401 for *everything* anonymously: `/v2/home-network/agent-runner/manifests/latest`, `/v2/home-network/conformance-runner/…`, a randomly-named nonexistent repository, and `nosuchowner/nosuchimage` all return 401, and `/api/v1/packages/Home-Network` returns `{"message":"token is required"}`. The registry *does* issue a token to an anonymous client, but its `access` claim is `null` — no pull scope. So the conclusion that holds is the simple one: **an unauthenticated client gets nothing from this registry**, so a pull with no credential cannot succeed. And `grep` over the tree finds **no `imagePullSecrets` anywhere** — not in `platform/pm-agent/deployment.yaml`, not in `infra/agent/worker-job-template.yaml`, not in `conformance/incluster/`. ## What I did NOT verify - **Why the node sees a nameless certificate.** I measured that it does, and that the LAN does not. I did not diagnose it: I did not read the nodes' resolver or `/etc/rancher/k3s/registries.yaml` (node files, not reachable read-only through the API), and I did not check whether the node resolves the name to the same `192.168.2.20` the workstation does or to something else. **A correction to a remark I first wrote here:** I cited `cluster-vars`'s `GATEWAY_VIP: 192.168.1.53` as suspicious next to the nodes' `192.168.2.x`. That was noise — the operator confirms those base `cluster-vars` values are **dev-era leftovers for a cluster that is decommissioned**, so the 192.168.1.x addresses in that file say nothing about prod and should not be read as evidence about anything. The measured facts stand on their own and none of them depended on that knob. - **Whether the package would be pullable once authenticated.** Nothing has ever been pushed to that path, so "a credential is sufficient" is untested. The only thing established is that no credential is insufficient. - **Which fix the operator wants.** Two routes, and they are not equivalent: (a) an `imagePullSecret` — a `dockerconfigjson` holding a Gitea token, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment *and* from the committed worker Job template; note the Job template's shape is pinned by admission, and `imagePullSecrets` is a field that policy does not currently constrain, so adding one is a template change that needs a deliberate look at whether the policy should pin it too. (b) allowing anonymous reads of that package in Gitea — no manifest change at all, but it makes the image world-readable to anything that can reach the host. I have implemented neither: both need an operator action I must not take, and (b) is a posture decision. The docs now present both. - **Everything the #57 entry listed as unverified about RUNTIME remains unverified.** The image has still never been built or pushed. No dispatcher has authenticated to Gitea, no issue has been triaged, no worker Job has been created — so the admission policy, though now provably applied and Enforce, has still never refused or admitted a real Job. The network policies have not carried traffic. - **I did not re-run the merge gate against `main` after the merge.** This branch's own gate run is the evidence for this change; the merged state's is CI run on PR #57.