home-network-k3s-impl/decisions/2026-10-05-pm-agent-registry-pull-prereq.md

date, branch, title

date	branch	title
2026-10-05	docs/pm-agent-registry-pull-prereq	pm-agent reconciled exactly as designed — and is the first workload ever to pull from the in-cluster registry, which does not work yet

What changed

Documentation only. No manifest, no policy, no code.

PR #57 merged and pm-agent reconciled. I measured what landed, and the measurement turned up two prerequisites for the next operator step that nothing in this repository declares — and that the handoff entry for #57 could not have known, because it listed the image pull as unverified and that is exactly where it bit.

So this entry records the post-merge measurement, and platform/pm-agent/README.md, docs/plans/pm-agent-issue-to-pr.md and runbooks/pm-agent.md now name both prerequisites where they say "build, push, pin the digest". Before this, that step read as one action; it is three, and the two new ones are not an agent's to perform.

The finding is not specific to pm-agent. conformance-runner has the identical registry path (gitea.home.keithjackson.dev/home-network/...), the identical "operator pushes, a digest goes in a PR" model, and has never been pulled either — so the same two blockers sit in front of the L2/L3 conformance wiring. Measured: no pod anywhere in the cluster references that registry except the pm-agent dispatcher, which is therefore the first workload ever to try it.

What I verified

Read-only, through the pinned toolbox, against the live prod cluster.

The merge reconciled, completely and exactly as designed.

• GitRepository/flux-system artifact revision: main@sha1:95597bd6883637… — the merge commit. platform-config, pm-agent and apps-wave-0 all report that same lastAppliedRevision with Ready=True. Compared field to field, per CLAUDE.md §2b, rather than assumed from the merge.
• The security boundary is live. ClusterPolicy/pm-agent-job-shape: validationFailureAction=Enforce, failurePolicy=Fail, Ready=True, and all six rules present (worker-must-be-the-committed-shape, worker-must-be-one-container-with-no-args, worker-volumes-must-be-workspace-or-tmp-emptydirs, worker-may-not-declare-init-or-ephemeral-containers, worker-env-names-must-be-the-committed-set, worker-may-not-use-envfrom). This is the check that mattered most: a ClusterPolicy added to the wrong kustomization list ships green and never reaches the cluster, which is what happened to the conformance boundary on 2026-09-29.
• Namespace contents: both ServiceAccounts, pm-agent-config (17 keys), the pm-agent-credentials Secret, the default-deny NetworkPolicy, and both CiliumNetworkPolicies with VALID=True.
• SOPS decryption works through the new overlay patch. Checked by KEY NAMES and VALUE LENGTHS only, never values (CLAUDE.md §6): keys gitea-token, claude-code-oauth-token, lengths 76 and 52 bytes — which are the base64 lengths of the committed placeholder strings, so the Secret holds the decrypted placeholders and not ciphertext. A ciphertext apply succeeds silently and would have left ENC[AES256_GCM,…] values an order of magnitude longer.
• pm-agent-worker really is bound to nothing. The namespace has exactly one RoleBinding (pm-agent → Role pm-agent, subject ServiceAccount/pm-agent), and no ClusterRoleBinding anywhere names either pm-agent ServiceAccount.
• gitea-allow-pm-agent-ingress is applied in the gitea namespace, matching io.kubernetes.pod.namespace: pm-agent.
• The dispatcher is ImagePullBackOff on the all-zero placeholder digest — the designed, visible failure.

The two prerequisites, measured.

1. The nodes cannot verify the registry's TLS certificate. containerd's error on the pull is not about the digest:

   failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:000…":
   tls: failed to verify certificate: x509: certificate is not valid for any names,
   but wanted to match gitea.home.keithjackson.dev
   

   "not valid for any names" means the certificate presented to the node carries no matching SAN at all. From the operator's workstation the same name is healthy: it resolves to 192.168.2.20 and serves a Let's Encrypt certificate CN=*.home.keithjackson.dev with SANs *.home.keithjackson.dev, home.keithjackson.dev — and Certificate gateway/wildcard-home is Ready=True for exactly those names. So the certificate that should be served is fine, and what the NODE reaches is not what the workstation reaches. Correcting the digest will not clear this.

2. Anonymous pulls from this registry are refused. Not inferred from a bare 401 — that would have been wrong, because this Gitea answers 401 for everything anonymously: /v2/home-network/agent-runner/manifests/latest, /v2/home-network/conformance-runner/…, a randomly-named nonexistent repository, and nosuchowner/nosuchimage all return 401, and /api/v1/packages/Home-Network returns {"message":"token is required"}. The registry does issue a token to an anonymous client, but its access claim is null — no pull scope. So the conclusion that holds is the simple one: an unauthenticated client gets nothing from this registry, so a pull with no credential cannot succeed. And grep over the tree finds no imagePullSecrets anywhere — not in platform/pm-agent/deployment.yaml, not in infra/agent/worker-job-template.yaml, not in conformance/incluster/.

What I did NOT verify

• Why the node sees a nameless certificate. I measured that it does, and that the LAN does not. I did not diagnose it: I did not read the nodes' resolver or /etc/rancher/k3s/registries.yaml (node files, not reachable read-only through the API), and I did not check whether the node resolves the name to the same 192.168.2.20 the workstation does or to something else.

  A correction to a remark I first wrote here: I cited cluster-vars's GATEWAY_VIP: 192.168.1.53 as suspicious next to the nodes' 192.168.2.x. That was noise — the operator confirms those base cluster-vars values are dev-era leftovers for a cluster that is decommissioned, so the 192.168.1.x addresses in that file say nothing about prod and should not be read as evidence about anything. The measured facts stand on their own and none of them depended on that knob.

• Whether the package would be pullable once authenticated. Nothing has ever been pushed to that path, so "a credential is sufficient" is untested. The only thing established is that no credential is insufficient.

• Which fix the operator wants. Two routes, and they are not equivalent: (a) an imagePullSecret — a dockerconfigjson holding a Gitea token, SOPS-encrypted like pm-agent-credentials, referenced from the Deployment and from the committed worker Job template; note the Job template's shape is pinned by admission, and imagePullSecrets is a field that policy does not currently constrain, so adding one is a template change that needs a deliberate look at whether the policy should pin it too. (b) allowing anonymous reads of that package in Gitea — no manifest change at all, but it makes the image world-readable to anything that can reach the host. I have implemented neither: both need an operator action I must not take, and (b) is a posture decision. The docs now present both.

• Everything the #57 entry listed as unverified about RUNTIME remains unverified. The image has still never been built or pushed. No dispatcher has authenticated to Gitea, no issue has been triaged, no worker Job has been created — so the admission policy, though now provably applied and Enforce, has still never refused or admitted a real Job. The network policies have not carried traffic.

• I did not re-run the merge gate against main after the merge. This branch's own gate run is the evidence for this change; the merged state's is CI run on PR #57.