home-network-k3s-impl/decisions/2026-10-05-pm-agent-registry-pull-prereq.md

---
date: 2026-10-05
branch: docs/pm-agent-registry-pull-prereq
title: pm-agent reconciled exactly as designed — and is the first workload ever to pull from the in-cluster registry, which does not work yet
---

## What changed

Documentation only. No manifest, no policy, no code.

PR #57 merged and `pm-agent` reconciled. I measured what landed, and the measurement turned
up **two prerequisites for the next operator step that nothing in this repository
declares** — and that the handoff entry for #57 could not have known, because it listed the
image pull as unverified and that is exactly where it bit.

So this entry records the post-merge measurement, and
`platform/pm-agent/README.md`, `docs/plans/pm-agent-issue-to-pr.md` and
`runbooks/pm-agent.md` now name both prerequisites where they say "build, push, pin the
digest". Before this, that step read as one action; it is three, and the two new ones are
not an agent's to perform.

**The finding is not specific to pm-agent.** `conformance-runner` has the identical
registry path (`gitea.home.keithjackson.dev/home-network/...`), the identical "operator
pushes, a digest goes in a PR" model, and has never been pulled either — so the same two
blockers sit in front of the L2/L3 conformance wiring. Measured: **no pod anywhere in the
cluster references that registry** except the pm-agent dispatcher, which is therefore the
first workload ever to try it.

## What I verified

Read-only, through the pinned toolbox, against the live prod cluster.

**The merge reconciled, completely and exactly as designed.**

- `GitRepository/flux-system` artifact revision: `main@sha1:95597bd6883637…` — the merge
  commit. `platform-config`, `pm-agent` and `apps-wave-0` all report that same
  `lastAppliedRevision` with `Ready=True`. Compared field to field, per CLAUDE.md §2b,
  rather than assumed from the merge.
- **The security boundary is live.** `ClusterPolicy/pm-agent-job-shape`:
  `validationFailureAction=Enforce`, `failurePolicy=Fail`, `Ready=True`, and all six rules
  present (`worker-must-be-the-committed-shape`,
  `worker-must-be-one-container-with-no-args`,
  `worker-volumes-must-be-workspace-or-tmp-emptydirs`,
  `worker-may-not-declare-init-or-ephemeral-containers`,
  `worker-env-names-must-be-the-committed-set`, `worker-may-not-use-envfrom`). This is the
  check that mattered most: a ClusterPolicy added to the wrong kustomization list ships
  green and never reaches the cluster, which is what happened to the conformance boundary
  on 2026-09-29.
- Namespace contents: both ServiceAccounts, `pm-agent-config` (17 keys), the
  `pm-agent-credentials` Secret, the `default-deny` NetworkPolicy, and both
  CiliumNetworkPolicies with `VALID=True`.
- **SOPS decryption works through the new overlay patch.** Checked by KEY NAMES and VALUE
  LENGTHS only, never values (CLAUDE.md §6): keys `gitea-token`, `claude-code-oauth-token`,
  lengths 76 and 52 bytes — which are the base64 lengths of the committed placeholder
  strings, so the Secret holds the decrypted placeholders and not ciphertext. A
  ciphertext apply succeeds silently and would have left `ENC[AES256_GCM,…]` values an
  order of magnitude longer.
- **`pm-agent-worker` really is bound to nothing.** The namespace has exactly one
  RoleBinding (`pm-agent` → Role `pm-agent`, subject `ServiceAccount/pm-agent`), and no
  ClusterRoleBinding anywhere names either pm-agent ServiceAccount.
- `gitea-allow-pm-agent-ingress` is applied in the `gitea` namespace, matching
  `io.kubernetes.pod.namespace: pm-agent`.
- The dispatcher is `ImagePullBackOff` on the all-zero placeholder digest — the designed,
  visible failure.

**The two prerequisites, measured.**

1. **The nodes cannot verify the registry's TLS certificate.** containerd's error on the
   pull is not about the digest:

       failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:000…":
       tls: failed to verify certificate: x509: certificate is not valid for any names,
       but wanted to match gitea.home.keithjackson.dev

   "not valid for **any** names" means the certificate presented to the node carries no
   matching SAN at all. From the operator's workstation the same name is healthy: it
   resolves to `192.168.2.20` and serves a Let's Encrypt certificate
   `CN=*.home.keithjackson.dev` with SANs `*.home.keithjackson.dev, home.keithjackson.dev`
   — and `Certificate gateway/wildcard-home` is `Ready=True` for exactly those names. So
   the certificate that *should* be served is fine, and what the NODE reaches is not what
   the workstation reaches. **Correcting the digest will not clear this.**
2. **Anonymous pulls from this registry are refused.** Not inferred from a bare 401 — that
   would have been wrong, because this Gitea answers 401 for *everything* anonymously:
   `/v2/home-network/agent-runner/manifests/latest`, `/v2/home-network/conformance-runner/…`,
   a randomly-named nonexistent repository, and `nosuchowner/nosuchimage` all return 401,
   and `/api/v1/packages/Home-Network` returns `{"message":"token is required"}`. The
   registry *does* issue a token to an anonymous client, but its `access` claim is `null` —
   no pull scope. So the conclusion that holds is the simple one: **an unauthenticated
   client gets nothing from this registry**, so a pull with no credential cannot succeed.
   And `grep` over the tree finds **no `imagePullSecrets` anywhere** — not in
   `platform/pm-agent/deployment.yaml`, not in `infra/agent/worker-job-template.yaml`, not
   in `conformance/incluster/`.

## What I did NOT verify

- **Why the node sees a nameless certificate.** I measured that it does, and that the LAN
  does not. I did not diagnose it: I did not read the nodes' resolver or
  `/etc/rancher/k3s/registries.yaml` (node files, not reachable read-only through the API),
  and I did not check whether the node resolves the name to the same `192.168.2.20` the
  workstation does or to something else. Note `cluster-vars` has
  `GATEWAY_VIP: 192.168.1.53` while the nodes are `192.168.2.10-12` and the name resolves
  to `192.168.2.20` — three different subnets in one sentence is a reason to measure rather
  than to theorise, and I have not.
- **Whether the package would be pullable once authenticated.** Nothing has ever been
  pushed to that path, so "a credential is sufficient" is untested. The only thing
  established is that no credential is insufficient.
- **Which fix the operator wants.** Two routes, and they are not equivalent:
  (a) an `imagePullSecret` — a `dockerconfigjson` holding a Gitea token, SOPS-encrypted
  like `pm-agent-credentials`, referenced from the Deployment *and* from the committed
  worker Job template; note the Job template's shape is pinned by admission, and
  `imagePullSecrets` is a field that policy does not currently constrain, so adding one is
  a template change that needs a deliberate look at whether the policy should pin it too.
  (b) allowing anonymous reads of that package in Gitea — no manifest change at all, but it
  makes the image world-readable to anything that can reach the host.
  I have implemented neither: both need an operator action I must not take, and (b) is a
  posture decision. The docs now present both.
- **Everything the #57 entry listed as unverified about RUNTIME remains unverified.** The
  image has still never been built or pushed. No dispatcher has authenticated to Gitea, no
  issue has been triaged, no worker Job has been created — so the admission policy, though
  now provably applied and Enforce, has still never refused or admitted a real Job. The
  network policies have not carried traffic.
- **I did not re-run the merge gate against `main` after the merge.** This branch's own gate
  run is the evidence for this change; the merged state's is CI run on PR #57.