home-network-k3s-impl/decisions/2026-10-06-pm-agent-registry-pull-secret.md

date, branch, title

date	branch	title
2026-10-06	feat/pm-agent-registry-pull-secret	The kubelet's pull credential — the second half of making agent-runner pullable, and the seventh field the admission policy had not read

What changed

The registry had two blockers. The first — the nodes resolving the registry hostname to the non-hairpinned WAN and failing TLS against the router's CN=%D certificate — was diagnosed and fixed by an /etc/hosts pin, merged in #59 and applied by the operator. This is the second: the pull is unauthenticated, and this registry refuses unauthenticated clients outright.

pm-agent-registry is a new kubernetes.io/dockerconfigjson Secret, SOPS-encrypted with placeholder values, referenced by imagePullSecrets on both the dispatcher Deployment and the committed worker Job template. It is a second Secret holding the same Gitea token as pm-agent-credentials, because the two consumers need two formats: the application reads a raw token from env, the kubelet reads a dockerconfigjson before any container starts. That duplication is the established pattern here (the Keycloak client secret is mirrored the same way, for the same reason) and it is documented in both .example files.

The operator's part is the value, and one scope: the Gitea token must now carry read:package alongside write:repository and write:issue. Steps are in platform/pm-agent/pm-agent-registry.sops.yaml.example, including the trap that .dockerconfigjson's registry key must be the bare host as the image reference spells it — with a scheme or a port it matches nothing, the kubelet pulls anonymously, and the failure is indistinguishable from the Secret being absent.

The admission policy gained a seventh rule. imagePullSecrets is a pod-spec field that no container references, which is exactly why it is easy to leave unread — the same shape of omission as initContainers, ephemeralContainers, env and envFrom before it. The kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now REQUIRES the committed entry and rule 7 forbids any other or any second one; both halves are needed because a foreach over an absent list yields no result.

The leverage is small and worth saying so: the image is digest-pinned, so a different pull credential cannot change what runs, and the only Secrets in this namespace are the agent's own. It is pinned anyway, because "only the committed shape" is this policy's whole premise and the rule costs four lines.

What I verified

• kyverno test platform/kyverno/tests/pm-agent → 33 passed, 0 failed under the pinned CLI (was 28 before this change). Three new fixtures, each a real path rather than a style violation: extra-pull-secret (the committed entry PLUS a second one — rule 1's array pattern is satisfied by the first element, which is why the count is asserted separately), wrong-pull-secret (a different Secret in this namespace), and no-pull-secret (absent entirely, refused by rule 1's pattern because rule 7's foreach cannot see it).
• PM_SELFTEST=1 pm-dispatch → exit 0, now also asserting that the committed Job template names exactly pm-agent-registry — so the template and the policy cannot drift apart without the image build failing.
• ./ci/gate.sh in the pinned toolbox, from a §0 worktree → exit 3 (UNDETERMINED), with gitea-settings the only stage that did not run (no SETTINGS_API_TOKEN on this workstation). Every other stage green, spec-immutability included. rc 3 is not a pass.
• The gate caught me shipping the Secret in plaintext. I wrote the placeholder pm-agent-registry.sops.yaml and never ran sops -e -i on it. no-plaintext-secrets refused it with both halves named — "has no top-level sops: block with a mac" and "Secret value '.dockerconfigjson' is not SOPS ciphertext" — which is exactly the check earning its place: the file looked finished, and the value was a placeholder, so nothing about reading the diff would have made it obvious. Encrypted, and the stage is green.
• The registry really does refuse anonymous clients, measured before building this (2026-10-05): the anonymous registry token's access claim is null, every /v2/… path answers 401 whether it exists or not, and /api/v1/packages/Home-Network replies {"message":"token is required"}. That is why this is a credential and not a posture tweak.

What I did NOT verify

• That the pull now works. Nothing has been pushed to that registry and the committed digest is still the all-zero placeholder, so neither half of the registry fix has been exercised end to end. The first real pull is what proves both at once — and if it fails, the error names which half: x509 … not valid for any names is the resolution pin, 401 / unauthorized is this credential.
• That the /etc/hosts pin from #59 took effect. The operator applied it; I could not confirm the effect, because the dispatcher pod's retained events are all repeating BackOff with no fresh Pulling, so the x509 message still visible is the one from before the play ran. Not evidence either way.
• Whether read:package is sufficient for a pull from a Gitea package of this type. It is the documented read scope for packages; nothing here has exercised it, because no package exists yet.
• The dockerconfigjson placeholder is not a valid credential and is not meant to be. Its auth field is a REPLACE-ME string rather than base64, so a pull with it will fail at authentication rather than be mistaken for working.
• Anything about runtime. Unchanged from the previous entries: the image has never been built or pushed, no dispatcher has authenticated to Gitea, no issue has been triaged, no worker Job has ever been created, and the admission policy — live, Enforce, now seven rules — has still never admitted or refused a real Job.

Addendum 2026-10-30 (rebased onto #81/#82/#83; see decisions/2026-10-30-pm-agent-live.md). Three statements above are now out of date, and one was wrong when written:

• "Nothing has been pushed to that registry" — no longer true. CI published agent-runner in images run 340 (sha-523cd9118f95 = main, digest sha256:c4eba5b3…6eb42), and this branch now pins it.
• "That the /etc/hosts pin from #59 took effect" — now measured: all three prod nodes carry the line, resolve the name to 192.168.2.20, and get 401 with ssl_verify_result=0 from https://gitea.home.keithjackson.dev/v2/ (see the new entry).
• "Whether read:package is sufficient" — the scope is necessary but not the whole answer: org-package access is decided by account permission (team Packages unit, or org visibility). Handled in the registry .example.
• Wrong when written: the scope list. Gitea 1.24.3 puts GET /api/v1/user — the dispatcher's first call — behind the user scope category, so a token with only write:repository, write:issue and read:package CrashLoops the dispatcher on startup. The token needs read:user too.