home-network-k3s-impl/decisions/2026-10-05-pm-agent-registry-pull-prereq.md

---
date: 2026-10-05
branch: docs/pm-agent-registry-pull-prereq
title: pm-agent reconciled exactly as designed — and is the first workload ever to pull from the in-cluster registry, which does not work yet
---

## What changed

Documentation only. No manifest, no policy, no code.

PR #57 merged and `pm-agent` reconciled. I measured what landed, and the measurement turned
up **two prerequisites for the next operator step that nothing in this repository
declares** — and that the handoff entry for #57 could not have known, because it listed the
image pull as unverified and that is exactly where it bit.

So this entry records the post-merge measurement, and
`platform/pm-agent/README.md`, `docs/plans/pm-agent-issue-to-pr.md` and
`runbooks/pm-agent.md` now name both prerequisites where they say "build, push, pin the
digest". Before this, that step read as one action; it is three, and the two new ones are
not an agent's to perform.

**The finding is not specific to pm-agent.** `conformance-runner` has the identical
registry path (`gitea.home.keithjackson.dev/home-network/...`), the identical "operator
pushes, a digest goes in a PR" model, and has never been pulled either — so the same two
blockers sit in front of the L2/L3 conformance wiring. Measured: **no pod anywhere in the
cluster references that registry** except the pm-agent dispatcher, which is therefore the
first workload ever to try it.

## What I verified

Read-only, through the pinned toolbox, against the live prod cluster.

**The merge reconciled, completely and exactly as designed.**

- `GitRepository/flux-system` artifact revision: `main@sha1:95597bd6883637…` — the merge
  commit. `platform-config`, `pm-agent` and `apps-wave-0` all report that same
  `lastAppliedRevision` with `Ready=True`. Compared field to field, per CLAUDE.md §2b,
  rather than assumed from the merge.
- **The security boundary is live.** `ClusterPolicy/pm-agent-job-shape`:
  `validationFailureAction=Enforce`, `failurePolicy=Fail`, `Ready=True`, and all six rules
  present (`worker-must-be-the-committed-shape`,
  `worker-must-be-one-container-with-no-args`,
  `worker-volumes-must-be-workspace-or-tmp-emptydirs`,
  `worker-may-not-declare-init-or-ephemeral-containers`,
  `worker-env-names-must-be-the-committed-set`, `worker-may-not-use-envfrom`). This is the
  check that mattered most: a ClusterPolicy added to the wrong kustomization list ships
  green and never reaches the cluster, which is what happened to the conformance boundary
  on 2026-09-29.
- Namespace contents: both ServiceAccounts, `pm-agent-config` (17 keys), the
  `pm-agent-credentials` Secret, the `default-deny` NetworkPolicy, and both
  CiliumNetworkPolicies with `VALID=True`.
- **SOPS decryption works through the new overlay patch.** Checked by KEY NAMES and VALUE
  LENGTHS only, never values (CLAUDE.md §6): keys `gitea-token`, `claude-code-oauth-token`,
  lengths 76 and 52 bytes — which are the base64 lengths of the committed placeholder
  strings, so the Secret holds the decrypted placeholders and not ciphertext. A
  ciphertext apply succeeds silently and would have left `ENC[AES256_GCM,…]` values an
  order of magnitude longer.
- **`pm-agent-worker` really is bound to nothing.** The namespace has exactly one
  RoleBinding (`pm-agent` → Role `pm-agent`, subject `ServiceAccount/pm-agent`), and no
  ClusterRoleBinding anywhere names either pm-agent ServiceAccount.
- `gitea-allow-pm-agent-ingress` is applied in the `gitea` namespace, matching
  `io.kubernetes.pod.namespace: pm-agent`.
- The dispatcher is `ImagePullBackOff` on the all-zero placeholder digest — the designed,
  visible failure.

**The two prerequisites, measured.**

1. **The nodes cannot verify the registry's TLS certificate.** containerd's error on the
   pull is not about the digest:

       failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:000…":
       tls: failed to verify certificate: x509: certificate is not valid for any names,
       but wanted to match gitea.home.keithjackson.dev

   "not valid for **any** names" means the certificate presented to the node carries no
   matching SAN at all. From the operator's workstation the same name is healthy: it
   resolves to `192.168.2.20` and serves a Let's Encrypt certificate
   `CN=*.home.keithjackson.dev` with SANs `*.home.keithjackson.dev, home.keithjackson.dev`
   — and `Certificate gateway/wildcard-home` is `Ready=True` for exactly those names. So
   the certificate that *should* be served is fine, and what the NODE reaches is not what
   the workstation reaches. **Correcting the digest will not clear this.**
2. **Anonymous pulls from this registry are refused.** Not inferred from a bare 401 — that
   would have been wrong, because this Gitea answers 401 for *everything* anonymously:
   `/v2/home-network/agent-runner/manifests/latest`, `/v2/home-network/conformance-runner/…`,
   a randomly-named nonexistent repository, and `nosuchowner/nosuchimage` all return 401,
   and `/api/v1/packages/Home-Network` returns `{"message":"token is required"}`. The
   registry *does* issue a token to an anonymous client, but its `access` claim is `null` —
   no pull scope. So the conclusion that holds is the simple one: **an unauthenticated
   client gets nothing from this registry**, so a pull with no credential cannot succeed.
   And `grep` over the tree finds **no `imagePullSecrets` anywhere** — not in
   `platform/pm-agent/deployment.yaml`, not in `infra/agent/worker-job-template.yaml`, not
   in `conformance/incluster/`.

## What I did NOT verify

- **Why the node sees a nameless certificate.** I measured that it does, and that the LAN
  does not. I did not diagnose it: I did not read the nodes' resolver or
  `/etc/rancher/k3s/registries.yaml` (node files, not reachable read-only through the API),
  and I did not check whether the node resolves the name to the same `192.168.2.20` the
  workstation does or to something else.

  **A correction to a remark I first wrote here:** I cited
  `cluster-vars`'s `GATEWAY_VIP: 192.168.1.53` as suspicious next to the nodes'
  `192.168.2.x`. That was noise — the operator confirms those base `cluster-vars` values are
  **dev-era leftovers for a cluster that is decommissioned**, so the 192.168.1.x addresses in
  that file say nothing about prod and should not be read as evidence about anything. The
  measured facts stand on their own and none of them depended on that knob.
- **Whether the package would be pullable once authenticated.** Nothing has ever been
  pushed to that path, so "a credential is sufficient" is untested. The only thing
  established is that no credential is insufficient.
- **Which fix the operator wants.** Two routes, and they are not equivalent:
  (a) an `imagePullSecret` — a `dockerconfigjson` holding a Gitea token, SOPS-encrypted
  like `pm-agent-credentials`, referenced from the Deployment *and* from the committed
  worker Job template; note the Job template's shape is pinned by admission, and
  `imagePullSecrets` is a field that policy does not currently constrain, so adding one is
  a template change that needs a deliberate look at whether the policy should pin it too.
  (b) allowing anonymous reads of that package in Gitea — no manifest change at all, but it
  makes the image world-readable to anything that can reach the host.
  I have implemented neither: both need an operator action I must not take, and (b) is a
  posture decision. The docs now present both.
- **Everything the #57 entry listed as unverified about RUNTIME remains unverified.** The
  image has still never been built or pushed. No dispatcher has authenticated to Gitea, no
  issue has been triaged, no worker Job has been created — so the admission policy, though
  now provably applied and Enforce, has still never refused or admitted a real Job. The
  network policies have not carried traffic.
- **I did not re-run the merge gate against `main` after the merge.** This branch's own gate
  run is the evidence for this change; the merged state's is CI run on PR #57.