feat(pm-agent): pin the first published agent-runner digest, start dry, and fix the token scopes

Rebased #61 onto the CI image pipeline (#81). This commit makes the remaining in-git
blockers to going live go away:

- deployment.yaml and PM_WORKER_IMAGE pin images run 340's digest (sha256:c4eba5b3...,
  tag sha-523cd9118f95 = main), read from the registry. Renovate markers unchanged.
- PM_DRY_RUN=true and PM_REQUIRE_LABEL=agent for the first run. Dry run is required, not
  just cautious: the run-340 image's baked worker template predates imagePullSecrets, so
  admission would deny its Jobs until CI re-publishes after this merge and Renovate bumps.
- Token scopes: Gitea 1.24.3 scopes GET /api/v1/user (the dispatcher's first call) under
  read:user. The documented list was missing it. Now read:user, write:repository,
  write:issue, read:package everywhere.
- Org-package read documented as a team (Packages=Read, no repos), not just a scope.
- sops is toolbox-only; the .examples say so and give the in-place edit.
- runbook: key-name check no longer emits Secret values before the pipe.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

decisions/2026-10-06-pm-agent-registry-pull-secret.md

@@ -86,3 +86,19 @@ and the rule costs four lines.
   built or pushed, no dispatcher has authenticated to Gitea, no issue has been triaged, no
   worker Job has ever been created, and the admission policy — live, `Enforce`, now seven
   rules — has still never admitted or refused a real Job.
+
+**Addendum 2026-10-30 (rebased onto #81/#82/#83; see `decisions/2026-10-30-pm-agent-live.md`).**
+Three statements above are now out of date, and one was wrong when written:
+- *"Nothing has been pushed to that registry"* — no longer true. CI published agent-runner in
+  images run 340 (`sha-523cd9118f95` = `main`, digest `sha256:c4eba5b3…6eb42`), and this
+  branch now pins it.
+- *"That the `/etc/hosts` pin from #59 took effect"* — now measured: all three prod nodes
+  carry the line, resolve the name to `192.168.2.20`, and get `401` with `ssl_verify_result=0`
+  from `https://gitea.home.keithjackson.dev/v2/` (see the new entry).
+- *"Whether `read:package` is sufficient"* — the scope is necessary but not the whole answer:
+  org-package access is decided by account permission (team Packages unit, or org
+  visibility). Handled in the registry `.example`.
+- **Wrong when written:** the scope list. Gitea 1.24.3 puts `GET /api/v1/user` — the
+  dispatcher's first call — behind the `user` scope category, so a token with only
+  `write:repository`, `write:issue` and `read:package` CrashLoops the dispatcher on startup.
+  The token needs `read:user` too.

decisions/2026-10-30-pm-agent-live.md

@@ -0,0 +1,101 @@
+---
+date: 2026-10-30
+branch: feat/pm-agent-registry-pull-secret
+title: PM agent — #61 rebased onto the CI image pipeline, first published digest pinned, and the remaining path to live audited
+---
+
+## What changed
+
+PR #61 (`feat/pm-agent-registry-pull-secret`, the kubelet pull credential and admission rule
+7) was rebased onto `origin/main` after #81 (CI image pipeline), #82 and #83. Conflicts in
+`docs/plans/pm-agent-issue-to-pr.md`, `platform/pm-agent/README.md` and
+`runbooks/pm-agent.md` were resolved keeping both sides: #61's "pull credential now wired"
+and #81's "CI publishes, Renovate bumps". No manual build-and-push text came back.
+
+On top of the rebase, as a second commit:
+
+- **The first published digest is pinned**: `deployment.yaml` and `PM_WORKER_IMAGE` in
+  `configmap.yaml` now name
+  `gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`
+  (images run 340, tag `sha-523cd9118f95`, which is also `main`). The `# renovate-digest:`
+  markers are unchanged and still sit on the line directly above each value, so later bumps
+  stay Renovate's. Pinning by hand once removes the dependency on Renovate's first-ever run.
+- **The first run is dry, and scoped to a label**: `PM_DRY_RUN: "true"`,
+  `PM_REQUIRE_LABEL: "agent"` (the plan's follow-up 3); `config-generation` bumped to `2`.
+  Dry run is also a *correctness* requirement, not only caution: the pinned digest was
+  built before this PR added `imagePullSecrets` to the worker Job template, which is baked
+  into the image. Admission now requires that entry, so a worker Job built from the run-340
+  image would be denied. Merging this PR touches `infra/agent/**`, so `images.yaml`
+  re-publishes agent-runner; Renovate's bump to that digest is what makes workers
+  admissible. Only then flip `PM_DRY_RUN` to `false`.
+- **The token scope list was wrong; it is fixed.** Gitea 1.24.3's `routers/api/v1/api.go`
+  closes the `/user` route group with `tokenRequiresScopes(AccessTokenScopeCategoryUser)`,
+  and the dispatcher's first call is `GET /user`. The documented scopes are now
+  `read:user`, `write:repository`, `write:issue`, `read:package` everywhere (both
+  `.example`s, README, runbook row).
+- **Org-package permission is documented separately from the scope.** Per
+  `services/context/package.go` `determineAccessMode` (v1.24.3), a non-admin's access to an
+  org-owned package is the highest Packages-unit mode across its org teams, else read if the
+  org is visible to it. Home-Network is `public`, so it works today by visibility alone; the
+  registry `.example` now has the operator create a `pm-agent-pull` team (no repositories,
+  Packages = Read) so it does not depend on that.
+- **`sops` is toolbox-only**: both `.example`s now say so and give the in-place
+  `./infra/devcontainer/dev.sh run sops <file>` edit (the same procedure
+  `runbooks/vpn-remote-access.md` documents), which writes no plaintext file.
+- The runbook's Secret-key check used `-o jsonpath='{.data}'`, which prints values before the
+  pipe trims them; it is now a `go-template` that emits key names only. Two new runbook
+  rows: the `read:user` 403, and the admission denial from a pre-#61 digest.
+
+## What I verified
+
+- **Registry digest**, read-only from the registry API (bearer token from `/v2/token`,
+  `HEAD /v2/home-network/agent-runner/manifests/<tag>` with OCI and Docker accept headers):
+  `main` and `sha-523cd9118f95` both answer `200`,
+  `Docker-Content-Digest: sha256:c4eba5b3…6eb42`, `application/vnd.oci.image.manifest.v1+json`.
+  Tags list: `[main, sha-523cd9118f95]`. The admission pattern
+  `gitea.home.keithjackson.dev/home-network/agent-runner@sha256:*` matches the pinned value.
+- **#81's claim that the template's digest needs no tracking holds**: `pm-dispatch.build_job`
+  sets `containers[0].image = cfg.worker_image` (from `PM_WORKER_IMAGE`) on every Job.
+  Renovate's `renovate-digest` custom manager only matches `platform/pm-agent/*.yaml` and
+  `versions.env`, so the template is not touched by it.
+- **Node path to the registry, read-only over SSH, all three prod nodes**: `/etc/hosts` line 9
+  is `192.168.2.20 keycloak.home.keithjackson.dev gitea.home.keithjackson.dev`, nsswitch is
+  `hosts: files dns`, and `curl https://gitea.home.keithjackson.dev/v2/` from the node answers
+  `401` with `ssl_verify_result=0` and `remote_ip=192.168.2.20`. That is the host resolver,
+  which is what containerd (a node process) uses; a pod's view would not have shown it.
+- **Network path from the namespace**: dispatcher and worker both use
+  `PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000` (ConfigMap, Job template, and
+  `pm-worker`'s hard allowlist, which refuses anything else); the git remote and the API base
+  are both derived from it. So nothing goes through the Gateway and #83's
+  `toEndpoints`-through-Gateway issue does not apply. Egress: `toEndpoints` gitea pods `:3000`
+  (both CNPs), `kube-apiserver` (dispatcher only), `world:443` (the model). Ingress on the
+  Gitea side: `gitea-allow-pm-agent-ingress` exists live and is `VALID True`.
+- **Live state (read-only, `--context k3s-prod`)**: namespace `pm-agent` exists; Deployment
+  `0/1`, pod `ImagePullBackOff` for 24d on the all-zero digest; Secret `pm-agent-credentials`
+  (2 keys) exists, `pm-agent-registry` does not yet (it ships in this PR); Flux Kustomization
+  `pm-agent` Ready at `main@dfc432c`; ClusterPolicy `pm-agent-job-shape` Ready.
+- **Gitea state (read-only API)**: 1.24.3; org Home-Network `public`; no user `pm-agent`
+  exists; teams are `Owners` and `registry-push`; collaborators are `jakman85` (owner) and
+  `renovate-bot` (write); the repository has **no labels** and **no open issues**.
+- **Placeholders, without decrypting**: the registry ciphertext decodes to 155 bytes, exactly
+  the length of the `.example`'s placeholder JSON. The credentials values decode to 57 and 39
+  bytes — neither a 40-char Gitea token nor a ~100-char Claude OAuth token. Both files'
+  `lastmodified` is the commit that created them as placeholders.
+- `kyverno test`, `PM_SELFTEST=1 pm-dispatch` and the gate: numbers in the PR description and
+  the final handoff report for this run.
+
+## What I did NOT verify
+
+- **Any authenticated pull, poll or worker.** No `pm-agent` account or token exists, so the
+  scope set has not been exercised end to end. The `read:user` requirement is read from
+  Gitea's v1.24.3 source, not from a live 403.
+- **That the credentials ciphertext is a placeholder.** Lengths are strong evidence, not
+  proof; I did not decrypt.
+- **That Renovate will open the post-merge bump.** Its token was being re-minted at the time
+  of writing and it has never completed a run here. Renovate's `kubernetes` manager also
+  scans `infra/agent/*.yaml` and `platform/**`; what it does with a digest-only reference
+  with no tag (the template) is unmeasured.
+- **The dispatcher's `claude --print` call to `api.anthropic.com`** through `world:443`, and
+  whether the run-340 image's dispatcher behaves as this tree's code does (it predates #61's
+  selftest line, nothing else in `pm-dispatch.py`).
+- **The admission policy against a real Job.** Still never admitted or refused one live.

platform/pm-agent/README.md

@@ -46,15 +46,17 @@ waive the sensitive-path human review on its own pull request.
 Both are deliberate, and both fail **visibly** rather than silently:
 
 1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
-   `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
-   publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
+   `PM_WORKER_IMAGE` in `configmap.yaml` carry the digest of the **first** publish (images
+   run 340, `sha-523cd9118f95`), pinned once on 2026-10-30 so going live did not wait on
+   Renovate's first run; every later bump is Renovate's. That digest predates the worker
+   template's `imagePullSecrets` entry, so worker Jobs built from it are refused by
+   admission — which is why `PM_DRY_RUN` starts `true` (see `configmap.yaml`). `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
    (`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
    `infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
    `gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
    Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
    lines and moves them together in one pull request. Nobody builds-and-pushes this image from
-   a workstation, and nobody pastes a digest. Until the first publish, and until that PR
-   merges: `ImagePullBackOff`. The one operator prerequisite on the publishing side is the
+   a workstation. The one operator prerequisite on the publishing side is the
    `REGISTRY_PUSH_TOKEN` Actions secret (see the header of `.gitea/workflows/images.yaml`);
    without it the publish job fails loudly rather than skipping.
 
@@ -83,7 +85,7 @@ Both are deliberate, and both fail **visibly** rather than silently:
      `pm-agent-registry` is a `dockerconfigjson` Secret referenced by both the Deployment and
      the committed worker Job template, and the admission policy pins it to exactly that
      name. What is left to you is the value: the Gitea token must carry **`read:package`** in
-     addition to `write:repository` and `write:issue`, and the dockerconfigjson is built from
+     addition to `read:user`, `write:repository` and `write:issue`, and the dockerconfigjson is built from
      it. Steps:
      [`pm-agent-registry.sops.yaml.example`](./pm-agent-registry.sops.yaml.example).
 
@@ -99,8 +101,11 @@ Both are deliberate, and both fail **visibly** rather than silently:
 
    They hold the **same** Gitea token in two formats, because a Secret cannot be both — the
    same one-value-in-two-files pattern the Keycloak client secret already uses here. So mint
-   it once, with all three scopes: `read:package` (the pull), `write:repository` (clone,
-   branch push, pull request) and `write:issue` (issue comments). The first `.example`
+   it once, with all four scopes: `read:package` (the pull), `read:user` (the dispatcher's
+   startup `GET /user` — Gitea 1.24.3 scopes the whole `/user` group), `write:repository`
+   (clone, branch push, pull request) and `write:issue` (issue comments). The account also
+   needs package *permission*, not just the scope: a `pm-agent-pull` org team with Packages
+   = Read and no repositories (steps in the registry `.example`). The first `.example`
    explains which account to create and why it must *not* be an administrator.
 
    Until they are filled in: the pull fails `401`, and if you fix only the pull, the
@@ -174,7 +179,7 @@ Job from a committed template. **The model's output is data, never a command.**
 - **The worker's own credentials are inside the model's reach.** The session runs as the same
   uid in the same container as the wrapper, so it can read the environment, including its
   `GITEA_TOKEN` and `CLAUDE_CODE_OAUTH_TOKEN`. There is no isolation between them and none is
-  claimed. What bounds it: the Gitea token is scoped `write:repository, write:issue` on **one
+  claimed. What bounds it: the Gitea token is scoped `read:user, write:repository, write:issue, read:package` on **one
   repository** with no admin permission, and egress reaches only this Gitea and tcp/443.
 - **`world:443` is wider than `api.anthropic.com`.** Cilium can express the tighter rule
   (`toFQDNs`) but nothing in this repository uses FQDN policy and its prerequisite (Cilium's

platform/pm-agent/configmap.yaml

@@ -51,15 +51,20 @@ data:
   # the dispatcher and its workers share the committed Job template and the env-name
   # contract. A tag here is refused at startup (and would be DENIED by admission anyway).
   #
-  # PLACEHOLDER UNTIL THE IMAGE IS PUBLISHED. The all-zero digest below is deliberately
-  # impossible. CI builds, verifies and — on main — publishes agent-runner
-  # (.gitea/workflows/images.yaml), and Renovate replaces this value with the digest the
-  # REGISTRY returned for the image's `main` tag, in a pull request that also moves the
-  # Deployment's image (same marker, same branch). Nobody pushes this image from a workstation.
-  # Until the first publish the dispatcher starts, fails to pull, and says so — which is the
-  # visible failure, not a silent one.
+  # THE FIRST PUBLISHED DIGEST. CI builds, verifies and — on main — publishes agent-runner
+  # (.gitea/workflows/images.yaml). This value is images run 340's publish (tag
+  # sha-523cd9118f95, which `main` pointed at), read from the registry on 2026-10-30. From here
+  # on Renovate replaces it with the digest the REGISTRY returns for the `main` tag, in a pull
+  # request that also moves the Deployment's image (same marker, same branch). Nobody pushes
+  # this image from a workstation.
+  #
+  # READ BEFORE TURNING PM_DRY_RUN OFF: this digest PREDATES the worker template's
+  # `imagePullSecrets` entry (PR #61). Its baked template has none, and admission now REQUIRES
+  # it — so a worker Job built from this image is DENIED. The merge that adds the entry
+  # changes infra/agent/**, CI re-publishes agent-runner, and Renovate's bump to that digest
+  # is what makes worker Jobs admissible. Until then, dry run is the only mode that works.
   # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
-  PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000"
+  PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42"
 
   # --- the model -------------------------------------------------------------
   # The PM only reads a thread and answers with one JSON object, so it holds no tools and
@@ -104,13 +109,21 @@ data:
   # work somebody deliberately handed it while trust is being established, with no code
   # change and no redeploy of anything but this ConfigMap. The operator's call; the default
   # here honours the request as made.
-  PM_REQUIRE_LABEL: ""
+  #
+  # SET TO `agent` FOR THE FIRST RUN (2026-10-30, docs/plans/pm-agent-issue-to-pr.md
+  # follow-up 3). The label does not exist in the repository yet: create it in Gitea
+  # (Issues -> Labels -> New Label, name `agent`) before labelling a test issue.
+  PM_REQUIRE_LABEL: "agent"
 
   # Dry run: the PM still reads threads, still asks the model and still posts its decision as
   # an issue comment — so the operator can read exactly what it WOULD have done — but creates
   # no worker Job and so changes no code. The honest way to watch this service for a day
   # before letting it write anything.
-  PM_DRY_RUN: "false"
+  #
+  # "true" FOR THE FIRST RUN, and not only for caution: the pinned digest above cannot yet
+  # create an admissible worker Job (see the PM_WORKER_IMAGE comment). Flip to "false" in the
+  # same pull request as — or after — Renovate's first agent-runner digest bump.
+  PM_DRY_RUN: "true"
 
   # If the gate refuses a worker's change, the branch is NOT pushed by default and the issue
   # gets the gate log instead. Set to "true" to push anyway and open a red pull request —

platform/pm-agent/deployment.yaml

@@ -48,7 +48,7 @@ spec:
         # It is a hand-maintained marker rather than a computed hash because Flux's
         # Kustomization here does not run a configmap generator; bump it in the same commit
         # as a change to configmap.yaml.
-        pm-agent.homelab.keithjackson.dev/config-generation: "1"
+        pm-agent.homelab.keithjackson.dev/config-generation: "2"
     spec:
       serviceAccountName: pm-agent
       # The kubelet's credential for pulling this image. Not the agent's Gitea credential —
@@ -73,10 +73,12 @@ spec:
           # disagree about the shape the admission policy enforces. Replaced together, in ONE
           # Renovate pull request: both lines carry the same marker, so Renovate treats them as
           # one dependency and bumps them in the same branch. The digest comes from CI's publish
-          # of agent-runner's `main` tag (.gitea/workflows/images.yaml) — the all-zero
-          # placeholder below is impossible on purpose (see configmap.yaml PM_WORKER_IMAGE).
+          # of agent-runner's `main` tag (.gitea/workflows/images.yaml). The value below is the
+          # FIRST publish (images run 340, tag sha-523cd9118f95 = main at the time), read from
+          # the registry's manifest endpoint on 2026-10-30 and pinned by hand once so the
+          # dispatcher does not wait on Renovate's first run; every later bump is Renovate's.
           # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
-          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
+          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42
           imagePullPolicy: IfNotPresent
           command:
             - /usr/local/bin/pm-dispatch

platform/pm-agent/pm-agent-credentials.sops.yaml.example

@@ -60,11 +60,21 @@
 #     Keycloak service-account token therefore cannot authenticate to Gitea's API or to
 #     git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that.
 #
-#       Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT
-#       `write:admin`, NOT `write:organization`, NOT `write:user`.
+#       Scopes, exactly these four (one token serves both Secrets — see
+#       pm-agent-registry.sops.yaml.example):
+#         read:user         GET /user, the dispatcher's startup identity check. Gitea 1.24.3
+#                           puts the whole /api/v1/user group behind the `user` scope
+#                           category (routers/api/v1/api.go), so without it the dispatcher
+#                           CrashLoops on its first call even with an otherwise-valid token.
+#         write:repository  the clone, the branch push, the pull-request create, and the
+#                           dispatcher's reads of pulls/branches/commits.
+#         write:issue       reading open issues and posting issue comments.
+#         read:package      the kubelet's image pull (pm-agent-registry).
+#       Nothing else — in particular NOT `write:admin`, NOT `write:organization`, NOT
+#       `write:user`. (Until 2026-10-30 this list said two scopes; `read:user` was missing
+#       and `read:package` lived only in the registry template.)
 #
-#     `write:repository` covers the clone, the branch push and the pull-request create;
-#     `write:issue` covers reading and posting issue comments. (The repository's OTHER CI
+#     (The repository's OTHER CI
 #     token, `SETTINGS_API_TOKEN`, is a separate credential with `read:admin` — do not
 #     conflate them, and do not give this service that one. The 2026-10-02 measurement on
 #     PR #38 recorded in infra/gitea/approval-policy.json is about that token, not this one.)
@@ -94,6 +104,17 @@
 #
 # (4) COPY, FILL, ENCRYPT, COMMIT:
 #
+#     `sops` IS IN THE PINNED TOOLBOX, NOT ON THE HOST (measured 2026-10-30: absent from the
+#     workstation's PATH; sops 3.9.4 and vi in the toolbox). The simplest path needs no
+#     plaintext file at all — edit the committed, already-encrypted file in place, from a
+#     worktree, the way runbooks/vpn-remote-access.md does for its key:
+#
+#       ./infra/devcontainer/dev.sh run sops platform/pm-agent/pm-agent-credentials.sops.yaml
+#
+#     It opens vi on the decrypted document; replace the REPLACE-ME value(s), save, quit, and
+#     sops re-encrypts on write. If you take the cp route below instead, run the `sops -e -i`
+#     line through `./infra/devcontainer/dev.sh run` as well, and delete any plaintext temp file.
+#
 #       cp platform/pm-agent/pm-agent-credentials.sops.yaml.example \
 #          platform/pm-agent/pm-agent-credentials.sops.yaml
 #       $EDITOR platform/pm-agent/pm-agent-credentials.sops.yaml      # the two real values
@@ -120,7 +141,7 @@ metadata:
 type: Opaque
 stringData:
   # Gitea token for the dedicated, NON-ADMIN `pm-agent` account.
-  # Scopes: write:repository, write:issue.
+  # Scopes: read:user, write:repository, write:issue, read:package.
   gitea-token: REPLACE-ME-gitea-token
   # Output of `claude setup-token` on a subscription account. One year's validity.
   claude-code-oauth-token: REPLACE-ME-claude-code-oauth-token

platform/pm-agent/pm-agent-registry.sops.yaml.example

@@ -34,12 +34,31 @@
 #     package and nothing else.
 #
 # (2) ADD THE PACKAGE SCOPE. Gitea token scopes are fixed at creation, so this means
-#     generating a token with all three scopes and using it in BOTH Secrets:
+#     generating a token with all four scopes and using it in BOTH Secrets:
 #
 #       read:package        <- NEW, what the kubelet needs to pull
+#       read:user           <- the dispatcher's startup GET /user (Gitea 1.24.3 scopes it)
 #       write:repository    <- the agent's clone / branch push / pull-request create
 #       write:issue         <- the agent's issue comments
 #
+#     THE SCOPE IS NOT THE PERMISSION. `read:package` only lets the token use whatever
+#     package access the ACCOUNT has, and these packages belong to the ORGANISATION
+#     (`home-network/…`), not to a repository. Gitea 1.24.3 decides org-package access in
+#     services/context/package.go `determineAccessMode`: the highest Packages-unit mode of
+#     any org team the user is in, else READ if the org is visible to that user. Home-Network
+#     is `public` today (measured 2026-10-30), so a signed-in non-member already gets read —
+#     but that is a side effect of a visibility setting nobody chose for this. Make it
+#     explicit, and independent of visibility:
+#
+#       Organisation Home-Network -> Teams -> New Team
+#         name: pm-agent-pull     permission: none (unit-level)
+#         repositories: NONE (do not tick "all repositories")
+#         units: Packages = Read; every other unit = No Access
+#       then add `pm-agent` as its only member.
+#
+#     That grants package read on the org and nothing on any repository; the repo `Write`
+#     comes only from the collaborator entry in the credentials template.
+#
 #     Gitea -> (as pm-agent) Settings -> Applications -> Generate New Token.
 #     Put the SAME value in `pm-agent-credentials` (`gitea-token`) and in the
 #     `.dockerconfigjson` below. One token, two consumers, two formats — the same
@@ -64,6 +83,17 @@
 #
 # (4) COPY, FILL, ENCRYPT, COMMIT:
 #
+#     `sops` IS IN THE PINNED TOOLBOX, NOT ON THE HOST (measured 2026-10-30: absent from the
+#     workstation's PATH; sops 3.9.4 and vi in the toolbox). The simplest path needs no
+#     plaintext file at all — edit the committed, already-encrypted file in place, from a
+#     worktree, the way runbooks/vpn-remote-access.md does for its key:
+#
+#       ./infra/devcontainer/dev.sh run sops platform/pm-agent/pm-agent-registry.sops.yaml
+#
+#     It opens vi on the decrypted document; replace the REPLACE-ME value(s), save, quit, and
+#     sops re-encrypts on write. If you take the cp route below instead, run the `sops -e -i`
+#     line through `./infra/devcontainer/dev.sh run` as well, and delete any plaintext temp file.
+#
 #       cp platform/pm-agent/pm-agent-registry.sops.yaml.example \
 #          platform/pm-agent/pm-agent-registry.sops.yaml
 #       # paste the one-line JSON from step 3 as the .dockerconfigjson value

runbooks/pm-agent.md

@@ -67,9 +67,11 @@ Work through these in order; the first three are by far the most likely.
 | Symptom in the log | Cause | Fix |
 |---|---|---|
 | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
-| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been published; the committed digest is the deliberate placeholder | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
+| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
 | `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
 | `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
+| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets |
+| a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
 | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push |
 | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
 | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |