feat(pm-agent): the kubelet's pull credential, and the seventh field admission had not read

The registry had two blockers. #59 fixed the first (nodes resolved the hostname to the
non-hairpinned WAN and failed TLS against the router's CN=%D cert). This is the second:
the pull is unauthenticated, and this registry refuses unauthenticated clients outright —
measured 2026-10-05, its anonymous token carries access: null and every path 401s.

pm-agent-registry is a dockerconfigjson Secret (SOPS, placeholders) referenced by
imagePullSecrets on the dispatcher Deployment and on the committed worker Job template. It
holds the SAME Gitea token as pm-agent-credentials, because the two consumers need two
formats: the application reads a raw token from env, the kubelet reads a dockerconfigjson
before any container starts. Same one-value-in-two-files pattern as the Keycloak client
secret, documented in both .example files. The operator's part is the value plus one new
scope: read:package.

ADMISSION GAINED A SEVENTH RULE. imagePullSecrets is a pod-spec field no container
references, which is exactly why it is easy to leave unread — the same shape of omission
as initContainers, ephemeralContainers, env and envFrom before it. The kubelet reads the
named Secret, in this namespace, before anything runs. Rule 1's pattern now REQUIRES the
committed entry; rule 7 forbids any other and any second one. Both halves are needed: a
foreach over an absent list yields no result.

The leverage is small and the policy says so: the image is digest-pinned, so a different
pull credential cannot change WHAT runs. Pinned anyway — 'only the committed shape' is the
premise, and the rule costs four lines.

kyverno test: 33 passed, 0 failed (was 28). Three new fixtures, each a real path: the
committed entry plus a second one, a different Secret, and none at all.

The gate also caught me shipping the Secret in plaintext — I never ran sops -e -i on the
placeholder. Recorded in the handoff entry, because the file looked finished.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

decisions/2026-10-06-pm-agent-registry-pull-secret.md

@@ -0,0 +1,88 @@
+---
+date: 2026-10-06
+branch: feat/pm-agent-registry-pull-secret
+title: The kubelet's pull credential — the second half of making agent-runner pullable, and the seventh field the admission policy had not read
+---
+
+## What changed
+
+The registry had two blockers. The first — the nodes resolving the registry hostname to the
+non-hairpinned WAN and failing TLS against the router's `CN=%D` certificate — was diagnosed
+and fixed by an `/etc/hosts` pin, merged in #59 and applied by the operator. This is the
+second: **the pull is unauthenticated, and this registry refuses unauthenticated clients
+outright.**
+
+`pm-agent-registry` is a new `kubernetes.io/dockerconfigjson` Secret, SOPS-encrypted with
+placeholder values, referenced by `imagePullSecrets` on both the dispatcher Deployment and
+the committed worker Job template. It is a **second Secret holding the same Gitea token** as
+`pm-agent-credentials`, because the two consumers need two formats: the application reads a
+raw token from env, the kubelet reads a dockerconfigjson before any container starts. That
+duplication is the established pattern here (the Keycloak client secret is mirrored the same
+way, for the same reason) and it is documented in both `.example` files.
+
+The operator's part is the value, and one scope: the Gitea token must now carry
+**`read:package`** alongside `write:repository` and `write:issue`. Steps are in
+`platform/pm-agent/pm-agent-registry.sops.yaml.example`, including the trap that
+`.dockerconfigjson`'s registry key must be the bare host as the image reference spells it —
+with a scheme or a port it matches nothing, the kubelet pulls anonymously, and the failure is
+indistinguishable from the Secret being absent.
+
+**The admission policy gained a seventh rule.** `imagePullSecrets` is a pod-spec field that
+no container references, which is exactly why it is easy to leave unread — the same shape of
+omission as `initContainers`, `ephemeralContainers`, `env` and `envFrom` before it. The
+kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now
+REQUIRES the committed entry and rule 7 forbids any other or any second one; both halves are
+needed because a `foreach` over an absent list yields no result.
+
+The leverage is small and worth saying so: the image is digest-pinned, so a different pull
+credential cannot change *what* runs, and the only Secrets in this namespace are the agent's
+own. It is pinned anyway, because "only the committed shape" is this policy's whole premise
+and the rule costs four lines.
+
+## What I verified
+
+- **`kyverno test platform/kyverno/tests/pm-agent` → 33 passed, 0 failed** under the pinned
+  CLI (was 28 before this change). Three new fixtures, each a real path rather than a style
+  violation: `extra-pull-secret` (the committed entry PLUS a second one — rule 1's array
+  pattern is satisfied by the first element, which is why the count is asserted separately),
+  `wrong-pull-secret` (a different Secret in this namespace), and `no-pull-secret` (absent
+  entirely, refused by rule 1's pattern because rule 7's foreach cannot see it).
+- **`PM_SELFTEST=1 pm-dispatch` → exit 0**, now also asserting that the committed Job
+  template names exactly `pm-agent-registry` — so the template and the policy cannot drift
+  apart without the image build failing.
+- **`./ci/gate.sh` in the pinned toolbox, from a §0 worktree → exit 3 (UNDETERMINED)**, with
+  `gitea-settings` the only stage that did not run (no `SETTINGS_API_TOKEN` on this
+  workstation). Every other stage green, `spec-immutability` included. rc 3 is not a pass.
+- **The gate caught me shipping the Secret in plaintext.** I wrote the placeholder
+  `pm-agent-registry.sops.yaml` and never ran `sops -e -i` on it. `no-plaintext-secrets`
+  refused it with both halves named — "has no top-level `sops:` block with a mac" and
+  "Secret value '.dockerconfigjson' is not SOPS ciphertext" — which is exactly the check
+  earning its place: the file looked finished, and the value was a placeholder, so nothing
+  about reading the diff would have made it obvious. Encrypted, and the stage is green.
+- **The registry really does refuse anonymous clients**, measured before building this
+  (2026-10-05): the anonymous registry token's `access` claim is `null`, every `/v2/…`
+  path answers 401 whether it exists or not, and `/api/v1/packages/Home-Network` replies
+  `{"message":"token is required"}`. That is why this is a credential and not a posture
+  tweak.
+
+## What I did NOT verify
+
+- **That the pull now works.** Nothing has been pushed to that registry and the committed
+  digest is still the all-zero placeholder, so neither half of the registry fix has been
+  exercised end to end. The first real pull is what proves both at once — and if it fails, the
+  error names which half: `x509 … not valid for any names` is the resolution pin, `401` /
+  `unauthorized` is this credential.
+- **That the `/etc/hosts` pin from #59 took effect.** The operator applied it; I could not
+  confirm the effect, because the dispatcher pod's retained events are all repeating
+  `BackOff` with no fresh `Pulling`, so the `x509` message still visible is the one from
+  before the play ran. Not evidence either way.
+- **Whether `read:package` is sufficient** for a pull from a Gitea package of this type. It is
+  the documented read scope for packages; nothing here has exercised it, because no package
+  exists yet.
+- **The dockerconfigjson placeholder is not a valid credential and is not meant to be.** Its
+  `auth` field is a `REPLACE-ME` string rather than base64, so a pull with it will fail at
+  authentication rather than be mistaken for working.
+- **Anything about runtime.** Unchanged from the previous entries: the image has never been
+  built or pushed, no dispatcher has authenticated to Gitea, no issue has been triaged, no
+  worker Job has ever been created, and the admission policy — live, `Enforce`, now seven
+  rules — has still never admitted or refused a real Job.

docs/plans/pm-agent-issue-to-pr.md

@@ -241,10 +241,11 @@ about runtime behaviour in this document is a design claim, not a measurement.
    `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
    pin the Keycloak issuer already carries, applied by
    `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`;
-   (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
-   the credential decision now being local Gitea accounts with PATs, that means a
-   `dockerconfigjson` for an account whose token carries `read:package`. Only then does the
-   digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
+   (ii) the pull must be authenticated — **now wired**: `pm-agent-registry`
+   (`dockerconfigjson`) is referenced by the Deployment and the committed worker Job
+   template, and the admission policy pins `imagePullSecrets` to exactly that name. The
+   operator supplies the value, from a Gitea token that also carries `read:package`. Only
+   then does the digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
    placeholder digests. Expect the build itself to need iteration too: the nodejs module
    stream and the npm install have never run against this base. The same two blockers sit in
    front of `conformance-runner`, which shares the registry path and has never been pulled.

infra/agent/pm-dispatch.py

@@ -922,6 +922,8 @@ def selftest_template(path: str) -> None:
     assert pod.get("automountServiceAccountToken") is False, \
         "template must not mount a ServiceAccount token"
     assert pod.get("restartPolicy") == "Never", "a retried agent would double-push"
+    assert [x.get("name") for x in pod.get("imagePullSecrets") or []] == ["pm-agent-registry"], \
+        "the template must name exactly the committed pull secret (admission pins it)"
     assert len(pod.get("containers") or []) == 1, "exactly one container"
     c = pod["containers"][0]
     assert c.get("command") == ["/usr/local/bin/pm-worker"], "command must be the pinned entrypoint"

infra/agent/worker-job-template.yaml

@@ -68,6 +68,13 @@ spec:
       # guarantee it cannot use one is for it not to have one.
       automountServiceAccountToken: false
       serviceAccountName: pm-agent-worker
+      # The kubelet's pull credential, the same one the dispatcher uses. Pinned by the
+      # admission policy to exactly this name: `imagePullSecrets` is a pod-spec field, so a
+      # Job could otherwise name any Secret in the namespace and have the kubelet read it
+      # before a container ever starts. The image is digest-pinned, so the leverage is small
+      # — but "only the committed shape" is the rule this template exists to express.
+      imagePullSecrets:
+        - name: pm-agent-registry
       securityContext:
         runAsNonRoot: true
         runAsUser: 65532

platform/kyverno/admission/pm-agent-job-shape.yaml

@@ -111,6 +111,12 @@ spec:
                 =(hostNetwork): "false"
                 =(hostPID): "false"
                 =(hostIPC): "false"
+                # The kubelet reads this Secret BEFORE any container starts, so it is part
+                # of the shape even though no container references it. A pattern on an array
+                # asserts "at least one element matches", which is enough to REQUIRE the
+                # committed entry; rule 7 is what forbids a second one.
+                imagePullSecrets:
+                  - name: pm-agent-registry
                 containers:
                   - image: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:*"
                     command:
@@ -336,6 +342,54 @@ spec:
     # dispatcher itself uses envFrom (it is a Deployment, not a Job, and not matched by this
     # policy); a worker must not.
     # =====================================================================================
+    # =====================================================================================
+    # Rule 7 — exactly one imagePullSecret, and it is the committed one.
+    #
+    # `imagePullSecrets` is a pod-spec field that no container references, which is precisely
+    # why it is easy to leave unread — the same shape of omission as `initContainers` and
+    # `env` before it. The kubelet reads the named Secret, in this namespace, before any
+    # container starts. A Job naming a different one would have the kubelet read a Secret the
+    # worker was never meant to touch.
+    #
+    # The leverage is genuinely small: the image is digest-pinned, so a different credential
+    # cannot change WHAT runs, and the only Secrets in this namespace are the agent's own.
+    # It is pinned anyway, because "only the committed shape" is the whole premise of this
+    # policy, and because the cost of the rule is four lines while the cost of rediscovering
+    # the omission is an incident.
+    #
+    # Rule 1's pattern REQUIRES the committed entry; this forbids any OTHER entry and any
+    # second entry. A foreach over an absent list yields no result, which is why both halves
+    # are needed.
+    # =====================================================================================
+    - name: worker-pull-secret-must-be-the-committed-one
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - pm-agent
+            subjects:
+              - kind: ServiceAccount
+                name: pm-agent
+                namespace: pm-agent
+      validate:
+        message: >-
+          A worker Job may declare exactly ONE imagePullSecret, named `pm-agent-registry`.
+          The kubelet reads that Secret before any container starts, so naming another one
+          reaches a credential the worker was never meant to touch.
+        foreach:
+          - list: "request.object.spec.template.spec.imagePullSecrets"
+            deny:
+              conditions:
+                any:
+                  - key: "{{ length(request.object.spec.template.spec.imagePullSecrets) }}"
+                    operator: NotEquals
+                    value: 1
+                  - key: "{{ element.name }}"
+                    operator: AnyNotIn
+                    value: ["pm-agent-registry"]
+
     - name: worker-may-not-use-envfrom
       match:
         all:

platform/kyverno/tests/pm-agent/kyverno-test.yaml

@@ -56,6 +56,11 @@ results:
     kind: Job
     resources: [good-run, real-template]
     result: pass
+  - policy: pm-agent-job-shape
+    rule: worker-pull-secret-must-be-the-committed-one
+    kind: Job
+    resources: [good-run, real-template]
+    result: pass
 
   # ---------------- THE escalation, and the shape rule's other refusals ----------------
   # `evil-command` is the case the whole model exists for: right namespace, right creator,
@@ -73,6 +78,14 @@ results:
       - host-namespaces
       - restart-onfailure
       - second-container
+      - no-pull-secret
+    result: fail
+
+  # ---------------- the pull secret: a second one, or a different one ----------------
+  - policy: pm-agent-job-shape
+    rule: worker-pull-secret-must-be-the-committed-one
+    kind: Job
+    resources: [extra-pull-secret, wrong-pull-secret]
     result: fail
 
   # ---------------- args, and the second container ----------------

platform/kyverno/tests/pm-agent/resources.yaml

@@ -31,6 +31,8 @@ spec:
       restartPolicy: Never
       automountServiceAccountToken: false
       serviceAccountName: pm-agent-worker
+      imagePullSecrets:
+        - name: pm-agent-registry
       securityContext:
         runAsNonRoot: true
         runAsUser: 65532
@@ -95,6 +97,8 @@ spec:
       restartPolicy: Never
       automountServiceAccountToken: false
       serviceAccountName: pm-agent-worker
+      imagePullSecrets:
+        - name: pm-agent-registry
       containers:
         - name: worker
           image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
@@ -492,3 +496,75 @@ spec:
             readOnlyRootFilesystem: true
           envFrom:
             - secretRef: { name: pm-agent-credentials }
+---
+# A SECOND pull secret beside the committed one. containers[0] is perfectly compliant, and
+# rule 1's array pattern is satisfied by the first element — which is why the count is
+# asserted separately.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: extra-pull-secret
+  namespace: pm-agent
+spec:
+  template:
+    spec:
+      restartPolicy: Never
+      automountServiceAccountToken: false
+      serviceAccountName: pm-agent-worker
+      imagePullSecrets:
+        - name: pm-agent-registry
+        - name: pm-agent-credentials
+      containers:
+        - name: worker
+          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/pm-worker"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+---
+# A DIFFERENT pull secret — the kubelet would read a Secret the worker was never meant to
+# touch, before any container starts.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: wrong-pull-secret
+  namespace: pm-agent
+spec:
+  template:
+    spec:
+      restartPolicy: Never
+      automountServiceAccountToken: false
+      serviceAccountName: pm-agent-worker
+      imagePullSecrets:
+        - name: pm-agent-credentials
+      containers:
+        - name: worker
+          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/pm-worker"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+---
+# NO pull secret at all. Rule 7's foreach yields nothing over an absent list, so this is
+# rule 1's pattern to refuse — which is why the requirement lives in both.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: no-pull-secret
+  namespace: pm-agent
+spec:
+  template:
+    spec:
+      restartPolicy: Never
+      automountServiceAccountToken: false
+      serviceAccountName: pm-agent-worker
+      containers:
+        - name: worker
+          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/usr/local/bin/pm-worker"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true

platform/pm-agent/README.md

@@ -71,21 +71,33 @@ Both are deliberate, and both fail **visibly** rather than silently:
      inventory, never typed on a node. Apply it scoped to that one task:
      `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
      --tags hosts`.
-   - **The pull is still unauthenticated** once the node reaches the right endpoint. An
-     unauthenticated client gets nothing from this Gitea's registry, and there is no
-     `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
-     (local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea
-     account whose token carries `read:package`, referenced from the Deployment and the
-     committed worker Job template — not built yet, because minting that token is yours.
+   - **The pull must be authenticated** once the node reaches the right endpoint: an
+     unauthenticated client gets nothing from this registry. **Now wired** —
+     `pm-agent-registry` is a `dockerconfigjson` Secret referenced by both the Deployment and
+     the committed worker Job template, and the admission policy pins it to exactly that
+     name. What is left to you is the value: the Gitea token must carry **`read:package`** in
+     addition to `write:repository` and `write:issue`, and the dockerconfigjson is built from
+     it. Steps:
+     [`pm-agent-registry.sops.yaml.example`](./pm-agent-registry.sops.yaml.example).
 
    The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
    never pulled either.
-2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
-   repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
-   which Gitea account to create, which scopes, and why it must *not* be an administrator —
-   is in [`pm-agent-credentials.sops.yaml.example`](./pm-agent-credentials.sops.yaml.example).
-   Until then the dispatcher CrashLoops with `GET /user returned no login — is GITEA_TOKEN
-   valid?`.
+2. **The credentials are placeholders — two Secrets, one token.** Both are encrypted to the
+   repository's age recipient and both hold `REPLACE-ME-…` strings:
+
+   | Secret | type | read by | hand-off |
+   |---|---|---|---|
+   | `pm-agent-credentials` | Opaque | the application, as env | [`.example`](./pm-agent-credentials.sops.yaml.example) |
+   | `pm-agent-registry` | dockerconfigjson | the **kubelet**, before any container starts | [`.example`](./pm-agent-registry.sops.yaml.example) |
+
+   They hold the **same** Gitea token in two formats, because a Secret cannot be both — the
+   same one-value-in-two-files pattern the Keycloak client secret already uses here. So mint
+   it once, with all three scopes: `read:package` (the pull), `write:repository` (clone,
+   branch push, pull request) and `write:issue` (issue comments). The first `.example`
+   explains which account to create and why it must *not* be an administrator.
+
+   Until they are filled in: the pull fails `401`, and if you fix only the pull, the
+   dispatcher CrashLoops with `GET /user returned no login — is GITEA_TOKEN valid?`.
 
 Its Flux Kustomization is `wait: false` precisely because of those two states: a workload
 that is *expected* to be unhealthy for a while must never be able to gate anything.
@@ -148,6 +160,7 @@ Job from a committed template. **The model's output is data, never a command.**
 | **cannot** | clone from or push to anywhere but `gitea.gitea.svc.cluster.local:3000`: `pm-worker` validates `PM_GITEA_URL` against a one-entry allowlist baked into the image. In-cluster consumers reach a service by its cluster-local name — a different path by design from how a client outside the cluster reaches it — so the external hostname is not an accepted value |
 | **cannot** | authenticate a `git push` **during the model session**: `GIT_ASKPASS` is unset for its duration, so the restriction is structural rather than a request in a prompt |
 | **cannot** | be created in any shape but the committed one — see the admission policy |
+| **cannot** | have the kubelet read any Secret but `pm-agent-registry` on its behalf: `imagePullSecrets` is pinned to exactly that one name |
 
 ### What is *not* defended, stated plainly
 
@@ -193,14 +206,15 @@ Two things answer it, and neither is sufficient alone:
    `command` with no args, that powerless SA, no mounted token, `restartPolicy: Never`,
    non-root with no privilege escalation and a read-only root filesystem, only the
    `workspace`/`tmp` **emptyDir** volumes, no init or ephemeral containers, only the
-   committed env **names**, and no `envFrom`.
+   committed env **names**, no `envFrom`, and exactly one `imagePullSecret` — the committed
+   one.
 
-Six rules, because pinning `command` only moves the injection to whatever adjacent field
+Seven rules, because pinning `command` only moves the injection to whatever adjacent field
 nobody enumerated — on the sibling policy that was `initContainers` (measured: a Job with a
 compliant `containers[0]` and a privileged init container was **admitted**, pass 3 fail 0),
 then `ephemeralContainers`, then `env`, then `envFrom`. All of them are closed here from the
-start, and `platform/kyverno/tests/pm-agent/` proves each refusal offline: **28 assertions,
-28 passing** under the pinned `kyverno` CLI. The worker's own input guards have their own
+start, and `platform/kyverno/tests/pm-agent/` proves each refusal offline: **33 assertions,
+33 passing** under the pinned `kyverno` CLI. The worker's own input guards have their own
 offline probe — `infra/agent/probe-validation.sh`, **17 refusals** — because a guard nothing
 exercises is a claim rather than a control.
 

platform/pm-agent/deployment.yaml

@@ -51,6 +51,10 @@ spec:
         pm-agent.homelab.keithjackson.dev/config-generation: "1"
     spec:
       serviceAccountName: pm-agent
+      # The kubelet's credential for pulling this image. Not the agent's Gitea credential —
+      # see pm-agent-registry.sops.yaml.example for why they are two Secrets of one token.
+      imagePullSecrets:
+        - name: pm-agent-registry
       # The dispatcher DOES need its token: `kubectl create job` is its one privileged act,
       # and platform/pm-agent/rbac.yaml is what bounds it to that. Contrast the worker, whose
       # Job template sets this false because it needs no API access at all.

platform/pm-agent/kustomization.yaml

@@ -35,6 +35,12 @@ resources:
   # the distance between naming a file in a comment and listing it under `resources:` is one
   # careless edit.
   - pm-agent-credentials.sops.yaml
+  # The KUBELET's pull credential for agent-runner — a separate Secret from the one above
+  # because it is a different consumer in a different format: dockerconfigjson, read before
+  # any container starts. Same underlying Gitea token, which must also carry `read:package`.
+  # Measured 2026-10-05: this registry refuses unauthenticated clients outright, so without
+  # this the pull 401s. Also placeholders until the operator fills it in.
+  - pm-agent-registry.sops.yaml
   - deployment.yaml
   # default-deny baseline + the two egress allow policies (gitea, kube-apiserver for the
   # dispatcher only, tcp/443 for the model). NO ingress allow exists, by design.

platform/pm-agent/pm-agent-registry.sops.yaml

@@ -0,0 +1,39 @@
+# PLACEHOLDER VALUES — the kubelet cannot pull agent-runner until the operator replaces them.
+# Full hand-off steps: pm-agent-registry.sops.yaml.example (same directory).
+# SOPS-encrypted to the repository's age recipient; keys stay cleartext for reviewable diffs.
+#
+# This is the KUBELET's pull credential, not the agent's Gitea credential. The agent's is
+# pm-agent-credentials (Opaque, read as env by the application); this one is
+# dockerconfigjson and is read before any container starts. Same underlying Gitea token,
+# which must carry read:package as well as write:repository and write:issue.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: pm-agent-registry
+    namespace: pm-agent
+    labels:
+        app.kubernetes.io/name: pm-agent
+        app.kubernetes.io/part-of: pm-agent
+type: kubernetes.io/dockerconfigjson
+stringData:
+    .dockerconfigjson: ENC[AES256_GCM,data:4OJS0P/oTcaXYuqlrc/936l+5/ZSqA9dlNMPx8HkDX9zmQ4q1qIumYmlXYMbcVm2c2byeYt0II0mGVLVVNOBjb5jgDB4vjyhSLN4HhoYBxgB+//hNcYe+uobmrdJTBJLEbd+lXamf1vfsEUTmLL7I4VQ2l/mjJbvqu8p2YNw25Ska9iO9mkxCzjOTfRdzMD0Pn1Z09fjo8GpYVo=,iv:AEysZLEqrybR+3l9kReHEeTbVFApzKPb8kAkibEYerE=,tag:TgE7T1zasye1bsTZ2BTFrA==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvdG1lQm16V0tJdWZXRXlU
+            Z2R4RXVyTXd6VTVJUWtkbUVjZldnb1JFOWhVCjVMMDdDb3AwVkt3TzJzdjR0VnNs
+            SFlncWg2cWlFTy9IZ1YxSC9Zd3ZMaEkKLS0tIEZFQWhIMW5zMURWYmlXaFg1ZjhD
+            cG1Fb0V3VncrTm9SQ091a0J0Z3RURWMKpdzNVub0zH5S8nnCSCt2H6NcCqDbV24L
+            Mf8K2eIfu/ixn2NfwoVVj2QO+n+3/P6PLphdoiKMrL+ms0+JqWFpkQ==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-06T06:01:43Z"
+    mac: ENC[AES256_GCM,data:zwmmMbM9Me7wE8EOvrKD6hEMLVHNCxI85FHvMjtRcHHm+mtdACjofDNfpynFVc1xFMMchcoOfipxEfAAJcwbH0FtfLeZq6M+XNHAf81qIk3NJxrS5Kgi/9FXZJAAwhqo/eced89Uozq8ntwt+mgxOuyjMds5z4h/VfN7/RmsWHg=,iv:VlvAEqozM9VV50vVx7r5aq3BdaaLLubhHJyHt8SSgS0=,tag:4ji7UqfRfm9P+qJrezFpmw==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

platform/pm-agent/pm-agent-registry.sops.yaml.example

@@ -0,0 +1,88 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (`.yaml.example` is ignored by kustomize and is
+# never auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the credential the KUBELET uses to pull `agent-runner` from the in-cluster
+# Gitea registry. It is NOT the credential the agent uses to talk to Gitea — that one is
+# `pm-agent-credentials` and it is read by the application. Two Secrets, because they are
+# consumed by two different things in two different formats:
+#
+#   pm-agent-credentials   Opaque        read by pm-dispatch / pm-worker as env vars
+#   pm-agent-registry      dockerconfigjson  read by the KUBELET, before any container runs
+#
+# WHY A PULL CREDENTIAL IS NEEDED AT ALL. Measured 2026-10-05: an unauthenticated client
+# gets nothing from this Gitea's registry — it issues an anonymous token whose `access`
+# claim is `null`, and every path answers 401 whether it exists or not
+# (`/api/v1/packages/Home-Network` replies `{"message":"token is required"}`). So a pull
+# with no credential cannot succeed, and before this Secret there was no `imagePullSecret`
+# anywhere in the tree. See decisions/2026-10-05-pm-agent-registry-pull-prereq.md.
+#
+# THE OTHER HALF, ALREADY FIXED: the nodes used to resolve this registry's hostname to the
+# non-hairpinned WAN address and fail TLS against the router's `CN=%D` placeholder
+# certificate. That is the `/etc/hosts` pin in
+# infra/ansible/inventory/group_vars/k3s_prod/vars.yml
+# (decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md). Both halves are needed; a
+# pull fails on whichever is missing.
+#
+# =============================================================================
+# HAND-OFF STEPS (operator)
+# =============================================================================
+#
+# (1) USE THE SAME `pm-agent` ACCOUNT the agent already has (created per
+#     pm-agent-credentials.sops.yaml.example — the dedicated, NON-ADMIN one). A second
+#     account would be a second thing to keep non-admin for no gain: this token reads one
+#     package and nothing else.
+#
+# (2) ADD THE PACKAGE SCOPE. Gitea token scopes are fixed at creation, so this means
+#     generating a token with all three scopes and using it in BOTH Secrets:
+#
+#       read:package        <- NEW, what the kubelet needs to pull
+#       write:repository    <- the agent's clone / branch push / pull-request create
+#       write:issue         <- the agent's issue comments
+#
+#     Gitea -> (as pm-agent) Settings -> Applications -> Generate New Token.
+#     Put the SAME value in `pm-agent-credentials` (`gitea-token`) and in the
+#     `.dockerconfigjson` below. One token, two consumers, two formats — the same
+#     one-value-in-two-files pattern the Keycloak client secret already uses here, and for
+#     the same reason: a Secret cannot be shared across formats.
+#
+# (3) BUILD THE DOCKERCONFIGJSON. `auth` is base64 of `<username>:<token>` — NOT a hash,
+#     just encoding, so treat this file exactly as secret as the token itself:
+#
+#       USER=pm-agent
+#       TOKEN='<the token from step 2>'
+#       AUTH=$(printf '%s:%s' "$USER" "$TOKEN" | base64 -w0)
+#       cat > /tmp/dockerconfig.json <<JSON
+#       {"auths":{"gitea.home.keithjackson.dev":{"username":"$USER","password":"$TOKEN","auth":"$AUTH"}}}
+#       JSON
+#
+#     THE REGISTRY KEY MUST BE THE HOST AS THE IMAGE REFERENCE SPELLS IT —
+#     `gitea.home.keithjackson.dev`, no scheme, no port, no path. The kubelet matches the
+#     pull host against these keys literally; `https://…` or a trailing `/v2/` silently
+#     matches nothing and the pull is attempted anonymously, which 401s exactly as if this
+#     Secret were absent.
+#
+# (4) COPY, FILL, ENCRYPT, COMMIT:
+#
+#       cp platform/pm-agent/pm-agent-registry.sops.yaml.example \
+#          platform/pm-agent/pm-agent-registry.sops.yaml
+#       # paste the one-line JSON from step 3 as the .dockerconfigjson value
+#       sops -e -i platform/pm-agent/pm-agent-registry.sops.yaml
+#
+#     The root `.sops.yaml` rule matches this path and encrypts `stringData`, so the diff
+#     stays reviewable. NEVER `sops -d` it into a terminal or a transcript (CLAUDE.md §6).
+#     It is already listed under `resources:` in platform/pm-agent/kustomization.yaml.
+#
+# (5) ROTATION. Rotating the Gitea token means updating BOTH Secrets, because it is one
+#     token. The dispatcher must then be restarted to re-read its env; worker Jobs and the
+#     kubelet pick the new value up on their own.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: pm-agent-registry
+  namespace: pm-agent
+type: kubernetes.io/dockerconfigjson
+stringData:
+  # One line. Registry key = the host exactly as the image reference spells it.
+  .dockerconfigjson: '{"auths":{"gitea.home.keithjackson.dev":{"username":"pm-agent","password":"REPLACE-ME-the-same-gitea-token","auth":"REPLACE-ME-base64-of-username:token"}}}'

runbooks/pm-agent.md

@@ -69,7 +69,7 @@ Work through these in order; the first three are by far the most likely.
 | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
 | `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
 | `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
-| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
+| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o jsonpath='{.data}' \| tr ',' '\n' \| cut -d'"' -f2` should list `.dockerconfigjson`. Then confirm the token carries `read:package` and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
 | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
 | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
 | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |