docs: Massive amounts of historical data

From Gitea issue #97. Authored by the PM agent's worker; the merge gate is the approver. Refs #97. Agent: pm-agent (issue #97, mode implement)
Merge pull request 'fix(pm-agent): log the CLI's stdout envelope when triage fails; add PM_DECISION_MAX_TURNS' (#99 ) from fix/pm-agent-triage-exit into main
2026-10-31 17:24:23 +00:00 · 2026-10-31 17:12:40 +00:00 · 2026-10-31 12:04:09 -05:00 · 2026-10-31 16:52:16 +00:00 · 2026-10-31 11:45:52 -05:00 · 2026-10-31 16:34:49 +00:00
21 changed files with 1395 additions and 59 deletions
@@ -0,0 +1,104 @@
 ---
 date: 2026-10-06
 branch: feat/pm-agent-registry-pull-secret
 title: The kubelet's pull credential — the second half of making agent-runner pullable, and the seventh field the admission policy had not read
 ---
 ## What changed
 The registry had two blockers. The first — the nodes resolving the registry hostname to the
 non-hairpinned WAN and failing TLS against the router's `CN=%D` certificate — was diagnosed
 and fixed by an `/etc/hosts` pin, merged in #59 and applied by the operator. This is the
 second: **the pull is unauthenticated, and this registry refuses unauthenticated clients
 outright.**
 `pm-agent-registry` is a new `kubernetes.io/dockerconfigjson` Secret, SOPS-encrypted with
 placeholder values, referenced by `imagePullSecrets` on both the dispatcher Deployment and
 the committed worker Job template. It is a **second Secret holding the same Gitea token** as
 `pm-agent-credentials`, because the two consumers need two formats: the application reads a
 raw token from env, the kubelet reads a dockerconfigjson before any container starts. That
 duplication is the established pattern here (the Keycloak client secret is mirrored the same
 way, for the same reason) and it is documented in both `.example` files.
 The operator's part is the value, and one scope: the Gitea token must now carry
 **`read:package`** alongside `write:repository` and `write:issue`. Steps are in
 `platform/pm-agent/pm-agent-registry.sops.yaml.example`, including the trap that
 `.dockerconfigjson`'s registry key must be the bare host as the image reference spells it —
 with a scheme or a port it matches nothing, the kubelet pulls anonymously, and the failure is
 indistinguishable from the Secret being absent.
 **The admission policy gained a seventh rule.** `imagePullSecrets` is a pod-spec field that
 no container references, which is exactly why it is easy to leave unread — the same shape of
 omission as `initContainers`, `ephemeralContainers`, `env` and `envFrom` before it. The
 kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now
 REQUIRES the committed entry and rule 7 forbids any other or any second one; both halves are
 needed because a `foreach` over an absent list yields no result.
 The leverage is small and worth saying so: the image is digest-pinned, so a different pull
 credential cannot change *what* runs, and the only Secrets in this namespace are the agent's
 own. It is pinned anyway, because "only the committed shape" is this policy's whole premise
 and the rule costs four lines.
 ## What I verified
 - **`kyverno test platform/kyverno/tests/pm-agent` → 33 passed, 0 failed** under the pinned
  CLI (was 28 before this change). Three new fixtures, each a real path rather than a style
  violation: `extra-pull-secret` (the committed entry PLUS a second one — rule 1's array
  pattern is satisfied by the first element, which is why the count is asserted separately),
  `wrong-pull-secret` (a different Secret in this namespace), and `no-pull-secret` (absent
  entirely, refused by rule 1's pattern because rule 7's foreach cannot see it).
 - **`PM_SELFTEST=1 pm-dispatch` → exit 0**, now also asserting that the committed Job
  template names exactly `pm-agent-registry` — so the template and the policy cannot drift
  apart without the image build failing.
 - **`./ci/gate.sh` in the pinned toolbox, from a §0 worktree → exit 3 (UNDETERMINED)**, with
  `gitea-settings` the only stage that did not run (no `SETTINGS_API_TOKEN` on this
  workstation). Every other stage green, `spec-immutability` included. rc 3 is not a pass.
 - **The gate caught me shipping the Secret in plaintext.** I wrote the placeholder
  `pm-agent-registry.sops.yaml` and never ran `sops -e -i` on it. `no-plaintext-secrets`
  refused it with both halves named — "has no top-level `sops:` block with a mac" and
  "Secret value '.dockerconfigjson' is not SOPS ciphertext" — which is exactly the check
  earning its place: the file looked finished, and the value was a placeholder, so nothing
  about reading the diff would have made it obvious. Encrypted, and the stage is green.
 - **The registry really does refuse anonymous clients**, measured before building this
  (2026-10-05): the anonymous registry token's `access` claim is `null`, every `/v2/…`
  path answers 401 whether it exists or not, and `/api/v1/packages/Home-Network` replies
  `{"message":"token is required"}`. That is why this is a credential and not a posture
  tweak.
 ## What I did NOT verify
 - **That the pull now works.** Nothing has been pushed to that registry and the committed
  digest is still the all-zero placeholder, so neither half of the registry fix has been
  exercised end to end. The first real pull is what proves both at once — and if it fails, the
  error names which half: `x509 … not valid for any names` is the resolution pin, `401` /
  `unauthorized` is this credential.
 - **That the `/etc/hosts` pin from #59 took effect.** The operator applied it; I could not
  confirm the effect, because the dispatcher pod's retained events are all repeating
  `BackOff` with no fresh `Pulling`, so the `x509` message still visible is the one from
  before the play ran. Not evidence either way.
 - **Whether `read:package` is sufficient** for a pull from a Gitea package of this type. It is
  the documented read scope for packages; nothing here has exercised it, because no package
  exists yet.
 - **The dockerconfigjson placeholder is not a valid credential and is not meant to be.** Its
  `auth` field is a `REPLACE-ME` string rather than base64, so a pull with it will fail at
  authentication rather than be mistaken for working.
 - **Anything about runtime.** Unchanged from the previous entries: the image has never been
  built or pushed, no dispatcher has authenticated to Gitea, no issue has been triaged, no
  worker Job has ever been created, and the admission policy — live, `Enforce`, now seven
  rules — has still never admitted or refused a real Job.
 **Addendum 2026-10-30 (rebased onto #81/#82/#83; see `decisions/2026-10-30-pm-agent-live.md`).**
 Three statements above are now out of date, and one was wrong when written:
 - *"Nothing has been pushed to that registry"* — no longer true. CI published agent-runner in
  images run 340 (`sha-523cd9118f95` = `main`, digest `sha256:c4eba5b3…6eb42`), and this
  branch now pins it.
 - *"That the `/etc/hosts` pin from #59 took effect"* — now measured: all three prod nodes
  carry the line, resolve the name to `192.168.2.20`, and get `401` with `ssl_verify_result=0`
  from `https://gitea.home.keithjackson.dev/v2/` (see the new entry).
 - *"Whether `read:package` is sufficient"* — the scope is necessary but not the whole answer:
  org-package access is decided by account permission (team Packages unit, or org
  visibility). Handled in the registry `.example`.
 - **Wrong when written:** the scope list. Gitea 1.24.3 puts `GET /api/v1/user` — the
  dispatcher's first call — behind the `user` scope category, so a token with only
  `write:repository`, `write:issue` and `read:package` CrashLoops the dispatcher on startup.
  The token needs `read:user` too.
@@ -0,0 +1,101 @@
 ---
 date: 2026-10-30
 branch: feat/pm-agent-registry-pull-secret
 title: PM agent — #61 rebased onto the CI image pipeline, first published digest pinned, and the remaining path to live audited
 ---
 ## What changed
 PR #61 (`feat/pm-agent-registry-pull-secret`, the kubelet pull credential and admission rule
 7) was rebased onto `origin/main` after #81 (CI image pipeline), #82 and #83. Conflicts in
 `docs/plans/pm-agent-issue-to-pr.md`, `platform/pm-agent/README.md` and
 `runbooks/pm-agent.md` were resolved keeping both sides: #61's "pull credential now wired"
 and #81's "CI publishes, Renovate bumps". No manual build-and-push text came back.
 On top of the rebase, as a second commit:
 - **The first published digest is pinned**: `deployment.yaml` and `PM_WORKER_IMAGE` in
  `configmap.yaml` now name
  `gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`
  (images run 340, tag `sha-523cd9118f95`, which is also `main`). The `# renovate-digest:`
  markers are unchanged and still sit on the line directly above each value, so later bumps
  stay Renovate's. Pinning by hand once removes the dependency on Renovate's first-ever run.
 - **The first run is dry, and scoped to a label**: `PM_DRY_RUN: "true"`,
  `PM_REQUIRE_LABEL: "agent"` (the plan's follow-up 3); `config-generation` bumped to `2`.
  Dry run is also a *correctness* requirement, not only caution: the pinned digest was
  built before this PR added `imagePullSecrets` to the worker Job template, which is baked
  into the image. Admission now requires that entry, so a worker Job built from the run-340
  image would be denied. Merging this PR touches `infra/agent/**`, so `images.yaml`
  re-publishes agent-runner; Renovate's bump to that digest is what makes workers
  admissible. Only then flip `PM_DRY_RUN` to `false`.
 - **The token scope list was wrong; it is fixed.** Gitea 1.24.3's `routers/api/v1/api.go`
  closes the `/user` route group with `tokenRequiresScopes(AccessTokenScopeCategoryUser)`,
  and the dispatcher's first call is `GET /user`. The documented scopes are now
  `read:user`, `write:repository`, `write:issue`, `read:package` everywhere (both
  `.example`s, README, runbook row).
 - **Org-package permission is documented separately from the scope.** Per
  `services/context/package.go` `determineAccessMode` (v1.24.3), a non-admin's access to an
  org-owned package is the highest Packages-unit mode across its org teams, else read if the
  org is visible to it. Home-Network is `public`, so it works today by visibility alone; the
  registry `.example` now has the operator create a `pm-agent-pull` team (no repositories,
  Packages = Read) so it does not depend on that.
 - **`sops` is toolbox-only**: both `.example`s now say so and give the in-place
  `./infra/devcontainer/dev.sh run sops <file>` edit (the same procedure
  `runbooks/vpn-remote-access.md` documents), which writes no plaintext file.
 - The runbook's Secret-key check used `-o jsonpath='{.data}'`, which prints values before the
  pipe trims them; it is now a `go-template` that emits key names only. Two new runbook
  rows: the `read:user` 403, and the admission denial from a pre-#61 digest.
 ## What I verified
 - **Registry digest**, read-only from the registry API (bearer token from `/v2/token`,
  `HEAD /v2/home-network/agent-runner/manifests/<tag>` with OCI and Docker accept headers):
  `main` and `sha-523cd9118f95` both answer `200`,
  `Docker-Content-Digest: sha256:c4eba5b3…6eb42`, `application/vnd.oci.image.manifest.v1+json`.
  Tags list: `[main, sha-523cd9118f95]`. The admission pattern
  `gitea.home.keithjackson.dev/home-network/agent-runner@sha256:*` matches the pinned value.
 - **#81's claim that the template's digest needs no tracking holds**: `pm-dispatch.build_job`
  sets `containers[0].image = cfg.worker_image` (from `PM_WORKER_IMAGE`) on every Job.
  Renovate's `renovate-digest` custom manager only matches `platform/pm-agent/*.yaml` and
  `versions.env`, so the template is not touched by it.
 - **Node path to the registry, read-only over SSH, all three prod nodes**: `/etc/hosts` line 9
  is `192.168.2.20 keycloak.home.keithjackson.dev gitea.home.keithjackson.dev`, nsswitch is
  `hosts: files dns`, and `curl https://gitea.home.keithjackson.dev/v2/` from the node answers
  `401` with `ssl_verify_result=0` and `remote_ip=192.168.2.20`. That is the host resolver,
  which is what containerd (a node process) uses; a pod's view would not have shown it.
 - **Network path from the namespace**: dispatcher and worker both use
  `PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000` (ConfigMap, Job template, and
  `pm-worker`'s hard allowlist, which refuses anything else); the git remote and the API base
  are both derived from it. So nothing goes through the Gateway and #83's
  `toEndpoints`-through-Gateway issue does not apply. Egress: `toEndpoints` gitea pods `:3000`
  (both CNPs), `kube-apiserver` (dispatcher only), `world:443` (the model). Ingress on the
  Gitea side: `gitea-allow-pm-agent-ingress` exists live and is `VALID True`.
 - **Live state (read-only, `--context k3s-prod`)**: namespace `pm-agent` exists; Deployment
  `0/1`, pod `ImagePullBackOff` for 24d on the all-zero digest; Secret `pm-agent-credentials`
  (2 keys) exists, `pm-agent-registry` does not yet (it ships in this PR); Flux Kustomization
  `pm-agent` Ready at `main@dfc432c`; ClusterPolicy `pm-agent-job-shape` Ready.
 - **Gitea state (read-only API)**: 1.24.3; org Home-Network `public`; no user `pm-agent`
  exists; teams are `Owners` and `registry-push`; collaborators are `jakman85` (owner) and
  `renovate-bot` (write); the repository has **no labels** and **no open issues**.
 - **Placeholders, without decrypting**: the registry ciphertext decodes to 155 bytes, exactly
  the length of the `.example`'s placeholder JSON. The credentials values decode to 57 and 39
  bytes — neither a 40-char Gitea token nor a ~100-char Claude OAuth token. Both files'
  `lastmodified` is the commit that created them as placeholders.
 - `kyverno test`, `PM_SELFTEST=1 pm-dispatch` and the gate: numbers in the PR description and
  the final handoff report for this run.
 ## What I did NOT verify
 - **Any authenticated pull, poll or worker.** No `pm-agent` account or token exists, so the
  scope set has not been exercised end to end. The `read:user` requirement is read from
  Gitea's v1.24.3 source, not from a live 403.
 - **That the credentials ciphertext is a placeholder.** Lengths are strong evidence, not
  proof; I did not decrypt.
 - **That Renovate will open the post-merge bump.** Its token was being re-minted at the time
  of writing and it has never completed a run here. Renovate's `kubernetes` manager also
  scans `infra/agent/*.yaml` and `platform/**`; what it does with a digest-only reference
  with no tag (the template) is unmeasured.
 - **The dispatcher's `claude --print` call to `api.anthropic.com`** through `world:443`, and
  whether the run-340 image's dispatcher behaves as this tree's code does (it predates #61's
  selftest line, nothing else in `pm-dispatch.py`).
 - **The admission policy against a real Job.** Still never admitted or refused one live.
@@ -0,0 +1,55 @@
 ---
 date: 2026-10-31
 branch: chore/pm-agent-go-live
 title: Take pm-agent out of dry run (operator-approved); label scope and worker caps unchanged
 ---
 ## What changed
 - `platform/pm-agent/configmap.yaml`: `PM_DRY_RUN` `"true"` -> `"false"`. The operator approved
  this after reading the dry-run triage of issue #97. `PM_REQUIRE_LABEL: agent`,
  `PM_MAX_ACTIVE_WORKERS: 2`, `PM_MAX_NEW_WORKERS_PER_CYCLE: 1`, the worker image digest and every
  other knob are unchanged. The two comments that said dry run was the only mode that worked
  (because the run-340 digest lacked `imagePullSecrets`) now describe the run-383 digest that
  PR #96 pinned.
 - `platform/pm-agent/deployment.yaml`: `config-generation` `"3"` -> `"4"`. Without the bump the
  dispatcher keeps its startup config and stays in dry run.
 - `platform/pm-agent/README.md` and `docs/plans/pm-agent-issue-to-pr.md` (follow-up 3): the
  "currently dry run" status text now says dry run is off. `runbooks/pm-agent.md` is unchanged.
  Its "stop it: set `PM_DRY_RUN: "true"`" step and its admission-refusal troubleshooting row
  are still accurate.
 ## What I verified
 Read-only, `kubectl --context k3s-prod` through the toolbox, 2026-10-31 ~11:45 local, with Flux
 `pm-agent` and `platform-config` both at `main@1126740`:
 - ClusterPolicy `pm-agent-job-shape`: `Ready=True`, `validationFailureAction=Enforce`,
  `background=false`. It has seven rules, including `worker-pull-secret-must-be-the-committed-one`.
 - RBAC: ServiceAccounts `pm-agent` and `pm-agent-worker`, Role `pm-agent` and RoleBinding
  `pm-agent` (roleRef `pm-agent`, subject `ServiceAccount/pm-agent`) all exist. `auth can-i`
  as `system:serviceaccount:pm-agent:pm-agent` returned `yes` for `create jobs.batch` and `no`
  for `delete jobs.batch`.
 - Secrets (key names only): `pm-agent-registry` is `kubernetes.io/dockerconfigjson` with key
  `.dockerconfigjson`. `pm-agent-credentials` is `Opaque` with keys `claude-code-oauth-token`
  and `gitea-token`.
 - Egress: `default-deny` NetworkPolicy exists. CiliumNetworkPolicies `pm-agent-allow-egress`
  and `pm-agent-worker-allow-egress` both report `Valid=True`.
 - Deployment `pm-agent` is 1/1 on `agent-runner@sha256:df36d9b7…` with
  `config-generation: "3"` live. No Jobs exist in the namespace.
 - `platform/pm-agent/{configmap,deployment}.yaml`, README and docs do not match any
  `sensitive_paths` glob in `infra/gitea/approval-policy.json`, so no review or override applies.
 ## What I did NOT verify
 - **That a real worker Job is admitted and completes.** Nothing can show this until a labelled
  issue gets a `start` decision after the merge. The L0 Kyverno suite shows the policy accepts
  the committed template. Whether the API server admits the Job built from the df36d9b image's
  baked template has not been tested live.
 - That the worker pod can pull its image with `pm-agent-registry`, reach Gitea and the model
  through its egress policy, and push a branch and open a PR. None of this has run before.
 - The pod list also shows a stale `pm-agent-65db599649-v678g` (25d, `ContainerStatusUnknown`,
  from an old ReplicaSet). It does not block anything, but I did not investigate it. Clearing
  it is an operator action.
 - CI: see the PR for the gate result. Locally the best possible result is rc 3, with only
  `gitea-settings` not run.
@@ -0,0 +1,78 @@
 ---
 date: 2026-10-31
 branch: chore/pm-agent-runner-digest
 title: Bump pm-agent to the run-383 agent-runner digest by hand; Renovate saw it but the schedule withheld it
 ---
 ## What changed
 `platform/pm-agent/deployment.yaml` (the dispatcher `image:`) and `platform/pm-agent/configmap.yaml`
 (`PM_WORKER_IMAGE`) move:
 - from `agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`, the
  run-340 publish;
 - to `agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d`, the
  images run 383 publish of ddd04d4.
 Both `# renovate-digest:` markers are unchanged, directly above their values.
 `pm-agent.homelab.keithjackson.dev/config-generation` goes `"2"` → `"3"`. That is the convention in
 `deployment.yaml`'s own comment: "bump it in the same commit as a change to configmap.yaml". The
 image change alone would roll the pod, but the convention is followed regardless.
 No `*.sops.yaml` file is touched. Branched from 06d4a9a, which already contains the operator's #95.
 ### Why Renovate did not do this: the schedule, not a config defect
 - **Run 385** (dispatch, 2026-10-31 16:14 UTC, on ddd04d4) was a **real** run. It logged
  `Deleting orphan branch (… renovate/pin-dependencies)`, not a `DRY-RUN:` line. It ran at **12:14
  America/New_York**, outside `"schedule": ["after 3am and before 6am"]`, so it created and updated
  no branch.
 - **The lookup DID see the new digest.** Dependency Dashboard issue #91 lists, under **"Awaiting
  Schedule"**:
  `chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to df36d9b`.
 - **Nothing else blocked it.** There are no open Renovate PRs and no `renovate/*` branch on Gitea
  (111 branches listed across all pages). The previous agent-runner PR, #90 (to c4eba5b), was
  auto-merged by renovate-bot at 08:22 on 10-31. No PR limit was hit, and digests are not
  dashboard-approval gated.
 - **The registry, read-only, with Docker/OCI `Accept` headers:**
  - `HEAD …/agent-runner/manifests/main` → `docker-content-digest: sha256:df36d9b7…771d`
  - `HEAD …/agent-runner/manifests/sha-ddd04d460ef6` → the same digest.
 **Without this PR** the next scheduled run (08:15 UTC, 04:15 EDT) would open it, and it would
 auto-merge as a `digest` update. Ticking that "Awaiting Schedule" checkbox on #91 makes the next
 run of any kind create it out of schedule.
 ### Found on the way: Renovate's labels have never been applied
 - The repo has exactly one label, `agent`. The org has none.
 - Every Renovate PR so far (#89, #90, #92, #93) was created with `"labels": []`.
 - On Gitea, Renovate drops a label that does not exist in the repo. So `needs-review`,
  `blast-radius`, `range-widen`, `own-image`, `dependencies`, `automated-update`, `breaking` and
  `ci-tooling` are never attached.
 - The **automerge flags still work.** Only the labels, the visible review signal, are missing.
 - **Operator fix:** create those labels on the repo. This was not changed here.
 ## What I verified
 - **The new image carries the fix.** I fetched both images' layers read-only from the registry and
  read `/etc/pm-agent/worker-job-template.yaml` from each (`infra/agent/Containerfile:116` copies
  it there):
  - df36d9b's has `imagePullSecrets:` / `- name: pm-agent-registry`;
  - c4eba5b's has no `imagePullSecrets` at all.
  This matches `infra/agent/worker-job-template.yaml:76-77` at ddd04d4.
 - **`main` = `sha-ddd04d460ef6` = df36d9b**, both measured.
 - **No other file references c4eba5b.** No admission policy pins it.
 - **No override is needed.** `infra/gitea/approval-policy.json` lists only `platform/kyverno/**`
  under `platform/`, so `platform/pm-agent/**` is not sensitive.
 - I read the whole job logs of runs 362, 367 and 385.
 ## What I did NOT verify
 - **That the rolled dispatcher starts on df36d9b and that a worker Job is admitted.** That needs
  Flux to apply this and a real dispatch. Both are operator actions.
 - **That the `pm-agent-registry` Secret exists and decrypts in-cluster** (#95's sops file). I did not
  read it.
 - **That Renovate will treat this hand bump as current.** It should: its next lookup finds
  `main` == the pinned digest, so it proposes nothing and drops the dashboard entry. Not observed.
@@ -0,0 +1,77 @@
 ---
 date: 2026-10-31
 branch: fix/pm-agent-triage-exit
 title: pm-agent triage fails with "claude exited 1:" and no message; log the CLI's stdout envelope so the cause is visible
 ---
 ## What changed
 - `infra/agent/pm-dispatch.py`, `ask_pm()`: on a non-zero exit the error now comes from a new
  `describe_cli_failure()`, not from `proc.stderr` alone. With `--output-format json` the CLI
  reports its own failures as the result envelope on **stdout** and writes nothing to stderr,
  which is why both failures today logged `claude exited 1: ` with an empty message. The new
  line carries the envelope's `subtype`, `is_error`, `api_error_status`, `num_turns`,
  `duration_ms`, `total_cost_usd`, `stop_reason`, its `result`/`error` text, and the tails of
  stdout and stderr. It names silence explicitly when both streams are empty.
 - `redact()`: everything in that message, and in `extract_decision()`'s "no decision object"
  error, passes through it. It covers `sk-ant-…`, `Bearer`/`token` values, and `user:pass@` in
  both `http://` and `https://` URLs.
 - `PM_DECISION_MAX_TURNS`: a new knob. It defaults to 1, today's value, so behaviour is
  unchanged, and it is bounded to 1–5. If the newly visible envelope says `error_max_turns`,
  the remedy is a ConfigMap change rather than another image build. Documented in
  `platform/pm-agent/README.md`.
 - Selftest: four new assertions. They cover an empty stderr with the cause on stdout, a
  stderr-only failure, total silence, and redaction.
 ## What I verified
 - **Timeline (Loki, `{namespace="pm-agent"}`, and `kubectl logs`):**
  - **16:37:38 to 16:38:00**: pod `pm-agent-898fd8f5f-dp5hs`, image **c4eba5b3**,
    config-generation 2, dry run. It triaged #97 in **22 s**, logged `PM says comment`, and
    posted the 16:38:00Z comment. The thread then held only the issue body.
  - **16:43:09 to 16:44:00**: pod `pm-agent-f7c9b7fcf-j8mzn`, image **df36d9b7**,
    config-generation 3, **still dry run**. It logged `claude exited 1: ` with an empty
    message after **51 s**.
  - **16:55:42 to 16:56:36**: pod `pm-agent-7796db8bcb-p5dsm`, df36d9b7, config-generation
    4, live. The same empty failure, after **54 s**.
  - The only successful triage ever was the 16:38 one. Every earlier `898fd8f5f`/`5db55d558c`
    log line is a startup crash (Gitea `GET /user` 401), not a triage.
 - **Going live is NOT the cause.** The first failure happened in dry run. `ask_pm()` has no
  dry-run branch: the prompt, flags, tools (`--tools ""`) and cwd are identical in both
  modes. `PM_DRY_RUN` gates only `create_job()`, which runs after the model has answered.
 - **Ruled out, with evidence:**
  - **Timeout**: 51 to 54 s against a 900 s limit, and a timeout raises `TimeoutExpired`,
    not rc 1.
  - **Budget cap**: the PM call passes no `--max-budget-usd`.
  - **Prompt size**: the thread is the issue body (293 chars) plus three comments
    (1871 + 404 + 77 chars), and the persona is 8 KB.
  - **Writable HOME**: HOME=`/workspace` (emptyDir), and `CLAUDE_CONFIG_DIR=/workspace/.claude`
    was written at 16:56.
  - **Egress**: a TLS 1.3 handshake to `api.anthropic.com:443` from the dispatcher pod
    succeeds in 0.06 s.
  - **Model name and CLI version**: the same `claude-opus-5` and CLI 2.1.285 succeeded at
    16:38. Between c4eba5b and df36d9b, `infra/agent/` differs only by a template selftest
    assertion and the toolbox base digest.
 - **A hint, not proof:** `/workspace/.claude/.claude.json` in the live pod has
  `cachedExtraUsageDisabledReason: 'out_of_credits'`. No token was read or printed.
 - Offline selftest: `PM_SELFTEST=1` against the worktree's template and persona prints
  `selftest OK`, including the 4 new assertions.
 - Toolbox gate: see the PR description for the rc and the stages that were skipped.
 ## What I did NOT verify
 - **The root cause. UNVERIFIED.** The CLI's own message was discarded, `--no-session-persistence`
  leaves no transcript, and I did not run `claude` in the pod. The candidates still open:
  - **Usage or rate limit on the subscription seat.** The seat is shared with the operator's
    interactive use. The `out_of_credits` extra-usage flag fits this. So does a ~50 s run,
    which matches the CLI exhausting its 429/529 retries.
  - **`error_max_turns`.** With `--max-turns 1` and `--json-schema`, an answer that fails the
    schema cannot be retried. The failing thread now invites `implement` (a branch plus a
    long brief), where the 16:38 one produced a short `comment`. ~50 s is about two
    generations of the 22 s success.
 - **The 17:11 retry will NOT reveal the cause.** It runs the old image, which still discards
  stdout. The cause becomes visible only after this PR merges, CI re-publishes agent-runner,
  and a follow-up PR bumps the dispatcher digest (`deployment.yaml` and the `PM_WORKER_IMAGE`
  in `configmap.yaml`, plus a `config-generation` bump).
 - That the CLI's error envelope carries exactly these key names. The parser takes whichever
  keys are present and falls back to raw tails.
@@ -0,0 +1,441 @@
 ---
 date: 2026-10-31
 branch: docs/source-comment-audit
 title: Audit — historical narrative in source comments (read-only inventory, sample not exhaustive)
 ---
 ## What changed
 **Nothing but this file.** This is the read-only audit asked for in issue #97. No source
 file was edited; the pruning is to be briefed afterwards, per directory, from this list.
 ### What was asked, and the calibration applied
 From the reporter (issue #97, 2026-10-31T16:42:30Z): *"'historical' talks about previous
 issues or intent that have changed and why/how it is used today within comments of source
 files rather than keeping them in the decisions and documentation. Source file comments
 should only talk about the current files and sections of code/yaml purposes, uses,
 expectations."*
 Each row below is classified:
 - **`historical`** — narrative about a superseded state, a past incident, a corrected
  earlier belief, or a closed issue. Says nothing about what the line does today.
 - **`load-bearing`** — phrased as history, but the sentence is the only record of a
  **current** non-obvious constraint. Per the brief these get **rewrite**, never delete.
 - **`ambiguous`** — I could not rule without the operator. Left for a human, deliberately
  not resolved here.
 ### THIS IS A SAMPLE, NOT AN EXHAUSTIVE INVENTORY
 The brief caps the audit at roughly 40–50 findings and asks for honesty over padding.
 Measured candidate volume, tree-wide, with the keyword sweep below:
 ```
 git ls-files -z | xargs -0 grep -nEi '#.*(used to|previously|formerly|no longer|we had|
  before we|historically|in the past|originally|superseded|turned out|was wrong|predates|
  legacy|issue #[0-9]+|PR #[0-9]+|measured 20|as of 20[0-9][0-9]-|see decisions/|DECISIONS-LOG)'
 ```
 | Directory | candidate hits | files | coverage here | findings listed |
 |---|---|---|---|---|
 | `platform/` | 107 | 56 | **sampled** | 9 |
 | `infra/` | 75 | 44 | **sampled** | 8 |
 | `conformance/` (excl. `runs/`) | 72 | ~31 | **sampled** | 5 |
 | `clusters/` | 43 | 19 | **sampled** | 7 |
 | `ci/` | 42 | 15 | **sampled** | 6 |
 | `catalog/` | 39 | 9 | **sampled** | 7 |
 | `.gitea/` | 10 | 5 | **full** | 5 |
 | `bootstrap/` | 4 | 3 | **full** | 3 |
 | `platform-config/` | 3 | 1 | **full** | 3 |
 | `apps/vms/` (hand-authored) | 3 | 2 | **full** | 2 |
 | `apps/wave-*` (generated) | 15 | — | **excluded — see below** | 0 |
 | `conformance/runs/` (archived logs) | 8 | 18 files | **excluded — see below** | 0 |
 | `runbooks/` | 11 | — | **not audited** (prose docs, not source comments) | 0 |
 | **total** | **~420** | **~190** | — | **55** |
 Covered **fully**: `.gitea/`, `bootstrap/`, `platform-config/`, `apps/vms/`.
 Covered **by sample only**: `platform/`, `infra/`, `conformance/`, `clusters/`, `ci/`,
 `catalog/`. Roughly **365 candidate hits remain unexamined**, and candidate ≠ finding —
 the keyword sweep's precision on the parts I did read was well under half, so the true
 residual is smaller than 365 but is certainly in the low hundreds.
 **The issue's premise holds**, with one qualification the operator should price in: of the
 55 rows below, **22 are `load-bearing`** and get a rewrite rather than a deletion. The
 narrative is genuinely excessive, but a mechanical "delete the history" sweep would destroy
 real constraints. That ratio is the single most important number in this report.
 ### Two structural warnings that change how the follow-up PRs must be written
 **1. `apps/wave-0/`, `apps/wave-1/`, `apps/wave-2/` are GENERATED. Do not edit comments there.**
 They are goldens produced by one canonical invocation of `catalog/render.py`
 (`ci/render-golden.sh:21`), and `ci/render-golden.sh:136-150` byte-compares every rendered
 file against its committed golden — **any diff is a hard `exit 1` that blocks the merge**.
 So the 15 candidate hits under `apps/wave-*` are *copies* of comments whose source is
 `catalog/*.yaml` or `catalog/render.py`; e.g. `apps/wave-0/gitea.yaml:130` is the rendered
 form of `catalog/gitea.yaml:100`. Edit the catalog source, then re-render with the canonical
 command and commit both. A pruning PR that hand-edits `apps/wave-*` will sit red.
 **2. `conformance/runs/*/run.log` are archived evidence artifacts, not source.** 8 candidate
 hits sit in 18 committed run logs. They are the recorded output of past conformance runs —
 the comments inside them are copies of the test bodies as they stood at run time. Rewriting
 them would falsify evidence. Excluded from scope entirely; recommend the operator confirm.
 ### A dangling pointer, replicated across 18 files — fix this before any "replace with a pointer" work
 **18 files cite `DECISIONS-LOG: "Headlamp gating stall"` as their rationale pointer, and no
 such entry exists.** `grep -ci "gating stall" DECISIONS-LOG.md` → **0**; the phrase appears
 nowhere in `decisions/` either. Citers:
 ```
 clusters/base/crafty.yaml           clusters/base/headlamp.yaml
 clusters/base/homepage.yaml         clusters/base/kustomization.yaml
 clusters/base/runbooks.yaml         clusters/base/vpn-ingress.yaml
 platform-config/kustomization.yaml  platform/crafty/deployment.yaml
 platform/crafty/kustomization.yaml  platform/headlamp/kustomization.yaml
 platform/homepage/kustomization.yaml platform/homepage/README.md
 runbooks/kustomization.yaml         runbooks/flux-reconcile.md
 conformance/tests/l0-static/alert-runbooks-resolvable/check.sh   (+3 more)
 ```
 This is the audit's clearest warning about its own remedy: "replace the narrative with a
 pointer" is only an improvement if the target exists and keeps existing. The archive is
 frozen and has no index, so a title-based citation into it cannot be checked by anything.
 **Recommendation:** when the operator takes the pruning PRs, either write a real
 `decisions/` entry for the leaf-gating lesson and repoint all 18 citations at its filename,
 or drop the citation. Pointer targets of the form `decisions/<file>.md` are checkable;
 `DECISIONS-LOG: "<title>"` is not. Nothing below recommends a pointer target I did not
 confirm exists.
 ---
 ### `platform/` — 107 candidates, 9 listed (sampled). NOT sensitive, except `platform/kyverno/**`
 | # | path:lines | comment (gist) | class | action | sensitive |
 |---|---|---|---|---|---|
 | 1 | `platform/crafty/service-l4.yaml:33-91` | "CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE … This file used to say `Local`" — 58 lines of outage narrative | load-bearing | **rewrite.** Keep the current constraints (Cilium's L2 announcer is not backend-aware; SNAT mode loses client IP; `/ban-ip` degrades so use `/ban`; `remote-node` is subsumed by `cluster` so do NOT add it; no `loadBalancerSourceRanges` deliberately). Drop the outage story, the lease-age detail and the per-protocol tell | no |
 | 2 | `platform/longhorn/helmrelease.yaml:191-200` | "`orphanAutoDeletion: true` USED TO BE HERE AND WAS INERT. Removed 2026-09-13" | load-bearing | **rewrite** to the forward-looking rule: Longhorn ≥1.9 replaced it with `orphanResourceAutoDeletion` and the chart silently drops unknown `defaultSettings` keys — so do not set it | no |
 | 3 | `platform/longhorn/helmrelease.yaml:249-254` | "CORRECTION, re-measured 2026-09-13 later the same day: the line originally recorded here read …" — corrects a superseded measurement | historical | **delete.** The correct reading is already stated at `:246-248` | no |
 | 4 | `platform/monitoring/alerts.yaml:625-636` | "Both rules below used to read `== 2` / `== 3`, with the comment 0=unknown 1=healthy…" | load-bearing | **rewrite.** Keep "`longhorn_volume_robustness` is a one-hot `state`-labelled gauge, value 0 or 1" — that is why the expressions look as they do. Drop the old-enum story | no |
 | 5 | `platform/monitoring/alerts.yaml:869-875` | "This alert originally used `severity: info`. That was WRONG, and wrong in the silent direction" | load-bearing | **rewrite** to the current rule: carries no `severity` deliberately, because `general.rules`' `InfoInhibitor` would swallow an info-severity deploy notification | no |
 | 6 | `platform/monitoring/pvc-capacity-alerts.yaml:12-27` | "`LonghornVolumeCapacityHigh` … never did, for two different reasons in sequence, and it is now GONE" — dated 2026-09-06/2026-09-13 timeline | load-bearing | **rewrite.** Keep only "these rules key on `kubelet_volume_stats_*` because `longhorn_volume_actual_size_bytes` measures allocation, not filesystem use". Delete the sequence | no |
 | 7 | `platform/headlamp/helmrelease.yaml:77-85` | "These two used to be keys INSIDE headlamp-oidc-credentials.sops.yaml. That was the bug…" + the dev-Keycloak `i/o timeout` story | load-bearing | **rewrite** to the current rule: issuer and callback URLs are public and MUST be cleartext, because Flux's `${BASE_DOMAIN}` postBuild substitution cannot reach inside an encrypted Secret. Keep the `[parity]` note at `:87-92` as-is | no |
 | 8 | `platform/vpn-egress/egress-steering.yaml:79` | "CORRECTION 2026-09-12 — this comment used to say the host nftables kill-switch 'is the…'" | load-bearing | **rewrite** to state the actual mechanism (Cilium's eBPF egress-gateway drop) without the correction framing | no |
 | 9 | `platform/keycloak/passkey-reconciler.yaml:18` | "It used to be duplicated here almost verbatim" | historical | **delete**, or replace with pointer `decisions/2026-10-18-keycloak-passkey-reconciler.md` (confirmed present) | no |
 Also seen in `platform/` and left unlisted for the cap, same shapes: `platform/cilium/helmrelease.yaml:57-65`
 (load-bearing — the `k3s_api_host`/`k3s_apiserver_endpoint` split is the current parity contract),
 `platform/kubevirt/kubevirt-cr.yaml:7` (`useEmulation was previously true`), `platform/external-snapshotter/kustomization.yaml:35`
 (`Previously these were REMOTE`), `platform/kustomization.yaml:7` (`the same pattern by which kyverno was previously deferred`),
 `platform/pm-agent/configmap.yaml:22`, `platform/monitoring/tests/run.sh:61` and `:129`,
 `platform/cnpg/alerts.promtool-test.yaml:20`, `platform/oauth2-proxy/auth-visibility-alerts.yaml:36-42`
 and `:98-101`, `platform/vpn-ingress/daemonset.yaml:165`.
 ### `infra/` — 75 candidates, 8 listed (sampled). NOT sensitive, except `infra/gitea/**`
 | # | path:lines | comment (gist) | class | action | sensitive |
 |---|---|---|---|---|---|
 | 10 | `infra/plex/claim-ship.sh:29-32` | "Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered feature/** … Those per-branch push triggers were deleted" | historical | **delete.** The live trigger set is already documented at `.gitea/workflows/gate.yml:48-56` and `CLAUDE.md` §4 | no |
 | 11 | `infra/ansible/inventory/hosts.ini:105` | "GROUP-NAME GAP: CLOSED 2026-08-31. The roles and playbooks used to hardcode the…" | historical | **delete** — the gap is closed and the current variable is named at `playbooks/node-health.yml:12` | no |
 | 12 | `infra/ansible/inventory/hosts.ini:147-153` | "NOTE (CORRECTED 2026-09-12): this used to read 'node 3 also has wlp1s0 up on 192.168.2.195 (wifi)'. It does not" | load-bearing | **rewrite** as a positive present-tense assertion: node 3 has no `wl*` netdev and a single IPv4 uplink on `eno1`. That fact is load-bearing for the kill-switch datapath; only the correction framing should go | no |
 | 13 | `infra/ansible/roles/node_health/defaults/main.yml:110-121` | "The first real run of this role FAILED k3s-prod-1's Crucial/Micron C400 on `reallocated=61440` … That verdict was WRONG" | load-bearing | **rewrite.** Keep the current rule (SMART raw values are vendor-defined; Micron TN-FD-20 scales ID 5 by `Block_Sector_Count`, so ID 170/196 are the trustworthy corroborators). Drop "the first real run failed" | no |
 | 14 | `infra/ansible/roles/unattended_upgrades/tasks/main.yml:304-309` | "This was originally a `find` over /etc/kernel/postinst.d … That theory was wrong" | load-bearing | **rewrite** to the current justification: the sentinel writer is ours at two known paths, and two exact `stat`s are strictly stronger than a glob that would match Debian's own conffile | no |
 | 15 | `infra/ansible/roles/node_health/tasks/assess.yml:398` | "measurement was right; the inference from the log line was wrong — the memattr block below…" | historical | **delete** | no |
 | 16 | `infra/devcontainer/dev.sh:103-108` | "SUPERSEDED but still honoured: `DEV_KUBECONFIG` used to be the only way to reach a cluster other…" | load-bearing | **rewrite.** "Still honoured" is current behaviour and `CLAUDE.md` §3 tells every agent to set `DEV_KUBECONFIG=/dev/null`, so this must keep documenting the variable — just not as history | no |
 | 17 | `infra/vpn-ingress/new-peer.sh:136` | "COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the…" | load-bearing | **rewrite**, replacing the history with pointer `decisions/2026-10-06-new-peer-ignores-template-comments.md` (confirmed present) | no |
 Unlisted, same shapes: `infra/ansible/inventory/group_vars/all.yml:31`,
 `infra/ansible/inventory/group_vars/k3s_prod/vars.yml:86` ("CORRECTION 2026-09-06: an earlier
 version of this comment claimed…"), `infra/ansible/roles/node_baseline/tasks/disks.yml:559`,
 `infra/ansible/roles/node_baseline/tasks/main.yml:801`, `infra/ansible/roles/kube_vip/tasks/main.yml:68`,
 `infra/ansible/roles/cilium/defaults/main.yml:30`, `infra/ansible/playbooks/node-health.yml:12`,
 `infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2:39` and
 `vpn-egress-down.sh.j2:15`, `infra/devcontainer/kubeconfig.sh:26`, `infra/agent/Containerfile:216`,
 `infra/agent/build-image.sh:30`, `infra/agent/pm-worker:97`, `infra/agent/probe-validation.sh:98`.
 ### `catalog/` — 39 candidates, 7 listed (sampled). NOT sensitive — but every edit here re-renders `apps/wave-*`
 | # | path:lines | comment (gist) | class | action | sensitive |
 |---|---|---|---|---|---|
 | 18 | `catalog/qbittorrent.yaml:7-12` | "BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record). This entry used to ship config-only, with a comment promising … 'once OQ-2 is resolved'" | historical | **delete.** Self-describes as "kept for the record", which is precisely the issue's complaint | no |
 | 19 | `catalog/qbittorrent.yaml:141-151` | "It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var … That MISDIAGNOSIS is preserved in this service's own marker log at 2026-09-13T05:35Z" | load-bearing | **rewrite.** Keep "PORT is a literal matching `ingressRoutes[0].port`; a literal cannot be mangled by envsubst". Drop the misdiagnosis story (it is already preserved in the marker log the comment itself cites) | no |
 | 20 | `catalog/plex.yaml:43-46` | "(This comment previously asserted '~4 minutes' as established fact. It was … Measured 2026-09-29: the full loop is ~2m45s and fits.)" | historical | **rewrite** to the measured figure alone; delete the self-correction parenthesis | no |
 | 21 | `catalog/plex.yaml:65` | "So the earlier 'a placeholder claim Secret would crashloop the pod' note was WRONG about the…" | historical | **delete** | no |
 | 22 | `catalog/plex.yaml:318` | "actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep" | historical | **rewrite** with pointer `decisions/2026-10-01-quicksync-via-device-plugin.md` and/or `decisions/2026-10-03-cilium-cgroup-hostroot.md` (both confirmed present) | no |
 | 23 | `catalog/gitea.yaml:14` | "Headless-install bootstrap (formerly `apps/gitea-bootstrap-patch.yaml`; now first-class via the…)" | historical | **delete** the "formerly" clause; keep the description of what the block does | no |
 | 24 | `catalog/render.py:159-166` | "THE SCRAPE WORKS AS OF 2026-10-05 — but it took THREE independent pieces … (PR #28) … (PR #49)" | load-bearing | **rewrite.** The three-part requirement and "do not read this flag alone as 'metrics are working'" are current and genuinely non-obvious; drop the PR numbers and the "63.7h" incident, pointing at `decisions/2026-10-05-oauth2-sidecar-scrape-verified.md` (confirmed present) | no |
 `catalog/pihole.yaml:210` and `:218` are the **exemplar to copy**: narrative already reduced to
 `see decisions/2026-10-04-pihole-multi-replica.md` / `-pihole-declare-state.md`, both of which
 exist. Unlisted: `catalog/gitea.yaml:100-130`, `catalog/nextcloud.yaml:211` and `:226-240`,
 `catalog/semaphore.yaml:84`, `catalog/pihole.yaml:179`, `catalog/qbittorrent.yaml:389-417`,
 `catalog/sonarr.yaml:120`, `catalog/render.py:672`, `:1371`, `:2161`.
 `catalog/jackett.yaml:73` ("floor only, no ceiling; 0.24.x runs as of 2026-10-30") is
 **load-bearing, leave alone** — a dated version constraint, exactly the carve-out the brief names.
 ### `.gitea/` — 10 candidates, 5 listed (FULL coverage). **SENSITIVE** (`.gitea/**`)
 | # | path:lines | comment (gist) | class | action | sensitive |
 |---|---|---|---|---|---|
 | 25 | `.gitea/workflows/gate.yml:14-16` | "(it used to print '— not green' and then 'MERGE GATE: PASS', exit 0 — fail-open in the gate itself)" | historical | **delete** the parenthesis; the surrounding `GATE_STRICT_SKIPS` rationale is current and stays | **yes** |
 | 26 | `.gitea/workflows/gate.yml:48-56` | "There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/** …) and it was a net negative here" + the 2026-09-29 plex-claim cost | load-bearing | **rewrite.** "Do not re-add a push trigger; `act_runner` is capacity 1 and the duplicate run serialises ahead of the PR run" is a live constraint a future editor needs. Drop the plex-claim anecdote | **yes** |
 | 27 | `.gitea/workflows/gate.yml:118-123` | "WIDENED TOLERANCE, DECLARED: `\|\| [ "$?" -eq 3 ]` used to cover exactly one skip (spec-immutability)" | load-bearing | **rewrite.** This documents a deliberate *current* weakening plus its exit condition (`REQUIRE_GITEA_SETTINGS=true` once the secret is seen green). Keep the instruction, drop "used to" | **yes** |
 | 28 | `.gitea/workflows/renovate.yaml:72-76` | "CORRECTED 2026-10-30. This list used to name only write:repository + read:user … Run 350 then failed at `GET /api/v1/orgs/… 403`" | load-bearing | **rewrite** to the required scope list as a plain present-tense requirement; point at `decisions/2026-10-30-renovate-auth.md` (confirmed present) for the run-350 story | **yes** |
 | 29 | `.gitea/workflows/merge-gate.yaml:15` | "The text that used to sit here…" | historical | **delete** | **yes** |
 Remaining `.gitea/` candidates examined and judged **not findings** (current-state
 measurements, correctly phrased): `renovate.yaml:6`, `:34`, `:259`, `images.yaml:44`,
 `scripts/images.sh:11`.
 ### `ci/` — 42 candidates, 6 listed (sampled). **SENSITIVE** (`ci/**`)
 | # | path:lines | comment (gist) | class | action | sensitive |
 |---|---|---|---|---|---|
 | 30 | `ci/decisions-log.sh:5-23` | 19-line "WHY THIS STAGE EXISTS" block reproducing the four-PR conflict table verbatim | historical | **replace with a pointer.** It is a near-verbatim duplicate of `decisions/README.md:9-30`, which the comment itself already cites at `:23`. Cut to the assertion list + that pointer (and `decisions/2026-09-29-decisions-log-conflict-free.md`, confirmed present) | **yes** |
 | 31 | `ci/gitea-settings.sh:7-23` | "On 2026-09-29 that blind spot was measured … PR #1 … sat `pending` for 30.6 minutes … PRs #5-#9 merged with zero approvals" | load-bearing | **rewrite.** "No other check can see Gitea's own database; assert-scripts do not detect drift from their own target" is the current reason the stage exists. Drop the PR-by-PR history | **yes** |
 | 32 | `ci/l0.sh:15-23` | "That union replaces what used to be a hand-written 'UNCONDITIONAL' stage per whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e bulk-tier)" | historical | **rewrite** to describe the present selection rule only (touched-requirement subset ∪ every test declaring `scope: whole-repo`) | **yes** |
 | 33 | `ci/lib.sh:138-142` | "WHY THIS EXISTS: `ci/l0.sh` used to carry a hand-written stage per whole-repo check, each with its own `summary …` line" | historical | **rewrite** to "names every offline check that ran and its verdict" | **yes** |
 | 34 | `ci/approval-policy.sh:107-112` | "Blocking on that turned 'CI cannot read your override' into a message that said 'you did not post one', and cost a real debugging session on PR #38" | load-bearing | **rewrite.** The token-scope fact and the fail-direction guarantee ("an unreadable source can only cause a false BLOCK, never a false pass") are current; the debugging-session cost is not | **yes** |
 | 35 | `ci/touched-requirements.sh:288-293` | "R-TEST-06 added 2026-09-12. The comment in `changed_diff()`'s rule 3 above records why it was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE" | load-bearing | **rewrite.** Keep why a `ci/` change must attribute R-TEST-06 today; drop the absence history | **yes** |
 Unlisted: `ci/approval-policy.sh:30-38` and `:182`, `ci/conformance.sh:73`, `ci/gate.sh:133`,
 `:233-234`, `ci/l0.sh:86`, `ci/l1.sh:33`, `ci/lib.sh:65`, `ci/render-golden.sh:5-6` and `:24`,
 `ci/touched-requirements.sh:39`, `:316`, plus `ci/tests/*.test.sh` (5 files — note these are
 test *rationale* headers, where a "this used to fail on main" line arguably earns its keep;
 I would send the whole `ci/tests/` group to `ambiguous`).
 ### `clusters/` — 43 candidates, 7 listed (sampled). **SENSITIVE** (`clusters/**`)
 | # | path:lines | comment (gist) | class | action | sensitive |
 |---|---|---|---|---|---|
 | 36 | `clusters/base/cluster-vars.yaml:244-251` | "IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read 'the two clusters are on OPPOSITE SIDES of pfSense'…" | load-bearing | **rewrite.** Keep "exactly one storage host, 192.168.2.30; still a knob only because the renderer reads it per-environment" and the immutability warning at `:253+`. Replace the pfSense/DNAT history with pointer `decisions/2026-10-07-bulk-nfs-truenas.md` (confirmed present) | **yes** |
 | 37 | `clusters/prod/kustomization.yaml:335-351` | "APEX AS OF 2026-09-13 … Prerequisites completed and VERIFIED before this flip, in this order: 1. dev's 18 A records released … 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted by hand … 3. dev's Certificate deleted" | historical | **rewrite.** The three completed prerequisites are done and unrepeatable. Keep only the still-true `NOTE` at `:348-351` (the zone's two wildcards point at the legacy stack's public IP and prod's specific records shadow them) | **yes** |
 | 38 | `clusters/prod/kustomization.yaml:472-479` | "*** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously read 'Not yet decided' while shipping .10 …)" | load-bearing | **rewrite.** Keep "this is the control-plane VIP, NOT .20 the ingress VIP and NOT .10 the single node address" and why a node-pinned CNI endpoint fails silently. Drop the previous-wording parenthesis | **yes** |
 | 39 | `clusters/base/apps-vms.yaml:51-55` | "postBuild.substituteFrom ENABLED 2026-08-31 (R-ENV-02). It was previously omitted because Flux's envsubst runs over the ENTIRE manifest…" | load-bearing | **rewrite.** The envsubst hazard over VM cloud-init and the `$${...}` escape obligation are live constraints for anyone adding a VM manifest; the enablement date and prior omission are not | **yes** |
 | 40 | `clusters/dev/kustomization.yaml:282-287` | "THIS VALUE IS INERT — dev is decommissioned, its nodes no longer exist, and the Proxmox host this used to name (192.168.1.9) went with the pools it held" | load-bearing | **keep, trim.** "Inert, restated per this overlay's supply-every-knob convention, unproven because there is no dev node" is honest current state and the kind of comment the issue wants | **yes** |
 | 41 | `clusters/base/apps-wave-0.yaml:4`, `apps-wave-1.yaml:4`, `apps-wave-2.yaml:4` | "wait:true health-gates ONLY this wave's resources, so a later wave can no longer stall an earlier one" (×3, and the same sentence is emitted by `catalog/render.py:2385`) | load-bearing | **rewrite** the "no longer" to plain present tense. Note the renderer emits this string, so fix `render.py:2385` and re-render rather than editing the three files by hand | **yes** |
 | 42 | `clusters/base/headlamp.yaml:9-12`, `homepage.yaml:10`, `crafty.yaml:11`, `kustomization.yaml:53`, `runbooks.yaml:15`, `vpn-ingress.yaml:9` | six files citing `DECISIONS-LOG: "Headlamp gating stall"` — **target does not exist** (see above) | historical | **rewrite** to the present-tense rule (a leaf UI belongs in its own post-`platform-config` Kustomization so its crashloop cannot stall the chain) and drop or repoint the dead citation | **yes** |
 Unlisted: `clusters/base/cilium.yaml:52`, `cluster-vars.yaml:29`, `:51`, `:264`,
 `platform.yaml:17`, `:56`, `platform-config.yaml:6`, `platform-config-cdi.yaml:13`,
 `oauth2-proxy.yaml:10`, `clusters/dev/kustomization.yaml:67`, `:81`, `:222`,
 `clusters/dev/sync.yaml:8`, `clusters/prod/kustomization.yaml:277-283`, `:317`, `:363`, `:546`, `:554`.
 ### `conformance/` — 72 candidates (excl. `runs/`), 5 listed (sampled). **SENSITIVE** (`conformance/**`)
 | # | path:lines | comment (gist) | class | action | sensitive |
 |---|---|---|---|---|---|
 | 43 | `conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml:16-24` | "MECHANISM CORRECTED 2026-09-12: this comment used to credit 'the host nftables kill-switch' … The ASSERTION below is unchanged and still the right one — only the explanation was wrong" | load-bearing | **rewrite** to state the real mechanism (Cilium's eBPF egress-gateway drop, netfilter never invoked) in the present tense. Pairs with finding #8 — same correction duplicated in `platform/vpn-egress/` | **yes** |
 | 44 | `conformance/tests/l3-fault-injection/vm-data-survival/chainsaw-test.yaml:18-27` | "(Feature-gate note, revalidated 2026-08: an earlier revision of this comment also claimed the containerDisk BOOT path was broken … That premise is now STALE: the cluster is k3s v1.36.4, where ImageVolume is ENABLED)" | historical | **rewrite** to the one current sentence that matters — DataVolume is used because only it survives a rebuild — and drop the whole stale-premise parenthesis | **yes** |
 | 45 | `conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml:51-60` | "TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible)" | historical | **rewrite.** Both gaps are closed and now asserted; state the assertions, drop the gap history. Same in `check.sh:98` and `:500` ("This section used to be an ESCALATION in the header's gap list") | **yes** |
 | 46 | `conformance/lib/status.py:303-310` | "It used to be, and that mis-reported a SAFETY-CRITICAL requirement … A previous engineer worked around it by rewording prose" | load-bearing | **rewrite.** "Scaffold detection must name the idioms it hunts for, because a placeholder-detection test legitimately contains the words" is the current design constraint; the R-STATE-03 misreport and the previous engineer are not | **yes** |
 | 47 | `conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh:447-455` | "It used to assert `… in L0`: ci/l0.sh carried a hand-written 'UNCONDITIONAL' stage per whole-repo check" — **the same paragraph is duplicated verbatim in 3 files** (also `flux-postbuild-substitution/check.sh:604`, `keycloak-client-secret-reconciler/check.sh:678`) | historical | **rewrite** all three to the present mechanism (the test declares `scope: whole-repo`; `ci/l0.sh` unions declaring tests via `--include-scope`; both halves checked). Good candidate for one small PR — identical edit ×3 | **yes** |
 Unlisted: `conformance/incluster/chainsaw-job.yaml:44`, `conformance/incluster/rbac.yaml:105`
 (**also sensitive as `**/rbac.yaml`** — "superseded design" for the gluetun Deployment;
 load-bearing, since it explains why the fault Job needs no RBAC), `conformance/lib/common.sh:33`,
 `conformance/lib/status.py:114`, and ~10 further `tests/l0-static/*/check.sh` headers of the
 same shape (`alert-metrics-have-scrape-targets`, `alert-rules-unit-tested`,
 `alert-runbooks-resolvable`, `auditability-proofs-producible`, `keycloak-passkey-reconciler`).
 ### `bootstrap/` — 4 candidates, 3 listed (FULL coverage). **SENSITIVE** (`bootstrap/**`)
 | # | path:lines | comment (gist) | class | action | sensitive |
 |---|---|---|---|---|---|
 | 48 | `bootstrap/flux/components/kustomization.yaml:36-38` | "On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry)" | load-bearing | **rewrite.** The fail-open default and what the feature gate changes are the current justification; the incident is one clause. Pointer is vague ("INCIDENT entry") — `grep` finds 2026-09-01 material in the archive, so if a citation is kept it should name a locatable heading, not "INCIDENT entry" | **yes** |
 | 49 | `bootstrap/flux/bootstrap.sh:84-86` | "see ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md" | historical | **rewrite** to keep only the cross-reference to `./components/kustomization.yaml`, which is checkable; drop the archive citation (duplicate of #48) | **yes** |
 | 50 | `bootstrap/sops/secret-templates/pia-custom-ovpn.yaml.example:6` | `See DECISIONS-LOG "Phase 3 VPN — custom .ovpn"` | ambiguous | **operator call.** The target *does* exist (10 hits for "Phase 3 VPN" in the archive), so this is a working pointer of the discouraged form. Keep, or repoint if a `decisions/` entry is written. Cited by path+line only; no content quoted (secret-adjacent `.example` template) | no (`*.sops.yaml.example` is not matched by `**/*.sops.yaml`) |
 Remaining candidate `bootstrap/flux/bootstrap.sh:122` ("a patch target that no longer exists
 after a Flux…") examined — **not a finding**, it describes current failure behaviour.
 ### `platform-config/` — 3 candidates, 3 listed (FULL coverage). **SENSITIVE** (`platform-config/kustomization.yaml`)
 | # | path:lines | comment (gist) | class | action | sensitive |
 |---|---|---|---|---|---|
 | 51 | `platform-config/kustomization.yaml:8` | "See `clusters/base/platform-config.yaml` + DECISIONS-LOG." | historical | **rewrite** to keep the checkable half (`clusters/base/platform-config.yaml`) and drop the bare, untargeted `DECISIONS-LOG` reference — it names no entry and cannot be resolved | **yes** |
 | 52 | `platform-config/kustomization.yaml:93` | "Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05." | load-bearing | **rewrite.** "THE RULE" at `:95-96` is the keeper; the dev-invisibility observation is stale (dev is decommissioned — see finding #40) and should go | **yes** |
 | 53 | `platform-config/kustomization.yaml:167` | "headlamp: httproute MOVED OUT … Keeping it here … meant a headlamp crashloop stalled platform-config. See DECISIONS-LOG 'Headlamp gating stall'." | historical | **rewrite** to the present-tense statement of where the HTTPRoute lives now; **dead pointer**, see the dangling-pointer section | **yes** |
 ### `apps/vms/` — 3 candidates, 2 listed (FULL coverage). Hand-authored, NOT generated
 | # | path:lines | comment (gist) | class | action | sensitive |
 |---|---|---|---|---|---|
 | 54 | `apps/vms/unifi/50-service-l4.yaml:27-78` | "CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE … This file used to say `Local` … BOTH HALVES OF THAT WERE FALSE" — ~52 lines | load-bearing | **rewrite**, same treatment as #1 (its near-twin). Keep: L2 announcer is not backend-aware; KubeVirt `masquerade` already destroyed the client IP so `Cluster` costs nothing here; `cluster` subsumes `remote-node` so do NOT "fix" the CNP; no `loadBalancerSourceRanges` deliberately. Drop the outage story | no |
 | 55 | `apps/vms/unifi/30-virtualmachine.yaml:119`, `40-networkpolicy.yaml:36` | `# legacy Network UI (still served)`, `# legacy UI` on port 8443 | load-bearing | **keep.** "legacy" here is the upstream product's own name for a port that is still served — it describes current behaviour, not history. Listed so a keyword-driven sweep does not delete it | no |
 ---
 ### Per-directory counts (findings listed above)
 | Directory | historical | load-bearing | ambiguous | total | sensitive-path? |
 |---|---|---|---|---|---|
 | `platform/` | 2 | 7 | 0 | 9 | no (except `kyverno/**`) |
 | `infra/` | 3 | 5 | 0 | 8 | no (except `gitea/**`) |
 | `catalog/` | 4 | 3 | 0 | 7 | no |
 | `clusters/` | 2 | 5 | 0 | 7 | **yes** |
 | `ci/` | 3 | 3 | 0 | 6 | **yes** |
 | `conformance/` | 3 | 2 | 0 | 5 | **yes** |
 | `.gitea/` | 2 | 3 | 0 | 5 | **yes** |
 | `bootstrap/` | 1 | 1 | 1 | 3 | **yes** (×2) |
 | `platform-config/` | 2 | 1 | 0 | 3 | **yes** |
 | `apps/vms/` | 0 | 2 | 0 | 2 | no |
 | **total** | **22** | **32** | **1** | **55** | 29 of 55 sensitive |
 The `ambiguous` bucket is thin because I could rule on most rows against the reporter's
 calibration. Two groups I am **deliberately not ruling on** and want the operator's call:
 1. **`ci/tests/*.test.sh` headers** (5 files). These say things like "this suite exists
   because `ci/gate.sh` used to set BLOCK=1 for a FAILED stage but…" and "a stage was added
   to gate.sh by PR #17, never added here, so 15 of 50 assertions failed on `main`". For a
   *regression test*, the regression it guards is arguably the test's subject rather than
   fluff. Ruling these `historical` would strip the only record of what each assertion is
   for. **Operator to decide.**
 2. **In-repo `README.md` prose** (`platform/vpn-egress/README.md:91`,
   `platform/crafty/README.md:52`, `infra/devcontainer/README.md:126`,
   `infra/gitea/README.md:253`). These are documentation, not source comments, so arguably
   already in the place the issue wants narrative to live. I did not audit them as findings.
 ### Suggested order for the follow-up PRs — cheapest and least sensitive first
 1. **`catalog/` + re-render `apps/wave-*`** (7 findings, not sensitive). Highest value per
   unit of review: it also fixes the generated copies under `apps/wave-*`. Must re-run
   `python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`,
   drop the not-onboarded outputs, and commit the goldens, or `render-golden` fails.
 2. **`platform/` excluding `kyverno/**`** (9 findings, not sensitive). Biggest residual too
   (107 candidates) — worth splitting by component (`monitoring/`, `longhorn/`, `crafty/`,
   `headlamp/`, `vpn-egress/`) into 2–3 PRs rather than one sweep.
 3. **`infra/` excluding `gitea/**`** (8 findings, not sensitive). `ansible/roles/*` is most
   of it; `infra/plex/claim-ship.sh:29-32` (#10) is a clean one-paragraph deletion and a
   good first commit.
 4. **`apps/vms/`** (2 findings, not sensitive). Self-contained; #54 is the twin of #1, so do
   it right after `platform/crafty/service-l4.yaml` while the reasoning is fresh.
 5. **The dangling-pointer fix** (18 files, crosses sensitive and non-sensitive). Either
   write one `decisions/` entry for the leaf-gating lesson and repoint all 18, or drop the
   citations. Doing this *before* steps 6–9 means those PRs do not have to re-litigate it.
 6. **`platform-config/kustomization.yaml`** (3 findings, **sensitive**). One file, three
   edits — the smallest possible sensitive-path PR, so it is the cheapest review.
 7. **`bootstrap/`** (3 findings, **sensitive**). Two files.
 8. **`.gitea/`** (5 findings, **sensitive**). Keep #27 (`gate.yml:118-123`) in its own commit
   — it documents a live, deliberate tolerance-widening and is the one most likely to be
   misread as a behaviour change.
 9. **`ci/`** then **`clusters/`** then **`conformance/`** (6 + 7 + 5, all **sensitive**,
   largest residuals). Last, because each needs a non-author approving review or an operator
   `/approve-override`, and because `ci/` edits touch the thing doing the approving.
 A note on review cost: **29 of the 55 findings are on sensitive paths**, so roughly half
 this work cannot merge on a green gate alone. Steps 1–4 (26 findings) are the portion that
 can. That is the argument for the ordering above.
 ## What I verified
 Everything below was run in this container, offline, against the tree at
 `466f63e` on branch `docs/source-comment-audit`.
 - **The tree layout, against the brief's believed list.** `git ls-files` → **855** tracked
  files. The brief's list was broadly right but differs in three ways worth recording:
  (a) there is **no `scripts/` and no `tools/` directory**; (b) `docs/` **does exist in this
  repo** (9 files: `docs/BULK-NFS-STORAGE.md` + `docs/plans/*.md`) and is *not* the sibling
  spec repo's `docs/` — I audited neither, since both are prose, not source comments;
  (c) `platform-config-cdi/` exists as a separate top-level directory (1 file, 0 candidate
  hits).
 - **Candidate volume, per directory**, with the keyword sweep quoted verbatim above:
  `platform/` 107 hits / 56 files, `infra/` 75 / 44, `conformance/` 80 / 40 (**72 excluding
  the 8 in `conformance/runs/`**), `clusters/` 43 / 19, `ci/` 42 / 15, `catalog/` 39 / 9,
  `apps/` 18 (**15 under the generated `apps/wave-*`, 3 under hand-authored `apps/vms/`**),
  `runbooks/` 11, `.gitea/` 10 / 5, `bootstrap/` 4 / 3, `platform-config/` 3 / 1,
  `platform-config-cdi/` 0. Total ≈ **420 candidate hits in ≈190 files**.
 - **`apps/wave-*` is generated and byte-enforced.** Read `ci/render-golden.sh`: the canonical
  command is at `:21`/`:120`, and the comparison loop at `:136-150` iterates the *rendered*
  output, requiring each golden to exist and to be byte-identical, `exit 1` on any diff.
  Confirmed the duplication concretely: the "HARDENED (was a silent first-boot RACE)"
  paragraph appears at both `catalog/gitea.yaml:100` and `apps/wave-0/gitea.yaml:130`.
  Confirmed `apps/vms/` is *not* produced by that loop (the renderer emits `wave-N/` paths
  only), so it is hand-authored and directly editable.
 - **The dangling pointer.** `grep -ci "gating stall" DECISIONS-LOG.md` → **0**;
  `grep -ci "Headlamp gating stall" DECISIONS-LOG.md` → **0**;
  `grep -ril "gating stall" decisions/` → **no output**; `git grep -il "gating stall"` →
  **18 files**. So 18 files cite a rationale entry that exists in neither the frozen archive
  nor `decisions/`.
 - **Every `decisions/...md` pointer I recommend exists.** Checked against `ls decisions/`
  (71 entries + `README.md` + `ARCHIVE.sha256`): `2026-10-18-keycloak-passkey-reconciler.md`,
  `2026-10-06-new-peer-ignores-template-comments.md`, `2026-10-01-quicksync-via-device-plugin.md`,
  `2026-10-03-cilium-cgroup-hostroot.md`, `2026-10-05-oauth2-sidecar-scrape-verified.md`,
  `2026-10-07-bulk-nfs-truenas.md`, `2026-10-30-renovate-auth.md`,
  `2026-09-29-decisions-log-conflict-free.md`, `2026-10-04-pihole-multi-replica.md`,
  `2026-10-04-pihole-declare-state.md` — all present. I recommend **no pointer target I did
  not list in `ls` output**.
 - **Archive-citation targets**, for the rows where I say a `DECISIONS-LOG` pointer does or
  does not resolve: `"Remote Path"` 5 hits, `"Phase 3 VPN"` 10, `"Phase 4 image-automation"` 1,
  `F-DNS1` 3, `F-OPS1` 1, `2026-09-01` 6 — all present in the 5,733-line archive.
  `"Headlamp gating stall"` 0, as above.
 - **The sensitive-path column is from the policy data, not the brief.** Read
  `infra/gitea/approval-policy.json`: `sensitive_paths` = `ci/**`, `.gitea/**`,
  `conformance/**`, `clusters/**`, `bootstrap/**`, `infra/gitea/**`, `platform/kyverno/**`,
  `platform-config/kustomization.yaml`, `.sops.yaml`, `**/*.sops.yaml`, `**/rbac.yaml`,
  `CLAUDE.md`. This matches the brief, and adds the root `.sops.yaml` and the explicit
  `**/` glob forms. Note `**/*.sops.yaml` does **not** match `*.sops.yaml.example`, which is
  why findings #50 and the `platform/headlamp/...sops.yaml.example` row are marked not
  sensitive.
 - **Every cited `path:line-range` was opened and read**, not matched and trusted. All 55
  rows quote text I read with `sed -n` at the line numbers given, at `466f63e`. Line numbers
  are from that commit and this PR changes no source file, so they remain valid on merge.
 - **Secret-adjacent files**: no `*.sops.yaml` content is quoted anywhere in this report, and
  no `.data`/`.stringData` value or key name appears. The two `.sops.yaml.example` rows
  (#50, and `platform/headlamp/headlamp-oidc-credentials.sops.yaml.example:15` in the
  unlisted set) are cited by path and line only. I ran no `sops -d`.
 - **The gate.** Result recorded in the PR; see "What I did NOT verify" for the stages that
  cannot run in this container.
 - **`git diff --stat` against `main`**: one added file, `decisions/2026-10-31-source-comment-audit.md`,
  and nothing else — which is the brief's "done means" condition.
 ## What I did NOT verify
 - **Anything about the live cluster. I hold no cluster credentials** — no kubeconfig and no
  ServiceAccount token is mounted in this container, so `kubectl` cannot work here. This
  audit is a pure text analysis of the tree and makes **no claim about the running
  platform**. That is appropriate (no behaviour changes), but it also means every *factual*
  assertion quoted inside the comments I audited — the measured Cilium lease holders, the
  `longhorn_volume_robustness` series counts, the 192.168.2.30 NFS reachability, the k3s
  version, the oauth2-proxy scrape targets — is **unverified by me**. I classified comments
  by what they *talk about*, not by whether their measurements are still true. A rewrite PR
  that restates one of those measurements as current fact should re-measure it first.
 - **The two gate stages that cannot run in this container**, per the environment notes and
  `CLAUDE.md` §3: `spec-immutability` (the sibling spec repo has no git remote and is not
  reachable from the cluster) and `gitea-settings` (needs `SETTINGS_API_TOKEN`, which I
  deliberately do not hold). So R-TEST-06 is **not** actually exercised on this run.
 - **~365 of the ~420 candidate hits were not examined.** The per-directory table marks which
  directories are full and which are sampled. For `platform/`, `infra/`, `conformance/`,
  `clusters/`, `ci/` and `catalog/` the listed findings are **representative, not complete**,
  and the `historical : load-bearing` ratio I report (22:32) is measured over the 55 rows I
  judged, **not** over the whole corpus — do not project it onto the residual.
 - **The keyword sweep is not a complete detector.** It finds English phrasings I thought of.
  Historical narrative that avoids all of them — a dated war story with no "used to", a
  stale rationale phrased entirely in the present tense, a comment that is simply wrong
  about today — is **invisible to it**. The most dangerous category (a confidently
  present-tense comment describing a superseded state) is precisely the one this method
  cannot find. A complete audit needs a reader per directory, not a grep.
 - **`runbooks/` (11 candidate hits), the repo-root markdown (`README.md`, `DESIGN-waves.md`),
  and the in-repo `docs/` tree were not audited.** They are prose documentation rather than
  source comments, so they fall outside the reporter's wording — but `CLAUDE.md` §1 already
  records that `runbooks/` contains stale `git push origin main` instructions, so there is
  known stale narrative there that this audit does not cover.
 - **The sibling spec repo (`R-*` requirements, ADRs) was not read or assessed at all** — it
  is not present in this container and is human-owned and immutable to agents (R-TEST-06).
  Nothing here proposes a change to it. Where a comment cites a spec document
  (e.g. `docs/06/R-STATE-01` in finding #44), I did **not** verify that the citation
  resolves.
 - **I did not verify that any recommended rewrite is correct, only that the current comment
  is historical.** Each "rewrite" row names what I believe is the load-bearing residue; the
  engineer doing the pruning must confirm that residue is still true before writing it as
  present-tense fact. I wrote no replacement text.
 - **Whether the operator agrees with my `historical` / `load-bearing` line.** I applied the
  calibration from issue #97 literally plus the brief's one carve-out. The 22:32 split is a
  judgement call, and the two groups in the `ambiguous` discussion above are explicitly
  unresolved and need a human ruling before the `ci/tests/` and README rows are touched.
 - **`conformance/runs/*/run.log` and `apps/wave-*` were excluded by my own judgement**, on
  the structural grounds given (archived evidence; gate-enforced goldens). The operator has
  not confirmed either exclusion.
 - **Nothing here traces to an `R-*` requirement**, deliberately — this is housekeeping, per
  the brief. I did not attempt to attribute one, and I did not check whether
  `ci/touched-requirements.sh` expects an attribution for a `decisions/`-only change.
@@ -241,20 +241,25 @@ about runtime behaviour in this document is a design claim, not a measurement.
   `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
   pin the Keycloak issuer already carries, applied by
   `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`;
-   (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
+   (ii) the pull must be authenticated — **now wired**: `pm-agent-registry`
-   the credential decision now being local Gitea accounts with PATs, that means a
+   (`dockerconfigjson`) is referenced by the Deployment and the committed worker Job
-   `dockerconfigjson` for an account whose token carries `read:package`. Only then does the
+   template, and the admission policy pins `imagePullSecrets` to exactly that name. The
-   digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step.
+   operator supplies the value, from a Gitea token that also carries `read:package`. Only
-   `.gitea/workflows/images.yaml` builds and verifies the image on rootless BuildKit
+   then does the digest matter. **Superseded 2026-10-29:** the build-and-push is no longer
-   (`platform/buildkit/`), publishes it on a push to `main`, and Renovate opens the one
+   an operator step. `.gitea/workflows/images.yaml` builds and verifies the image on rootless
-   PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The
+   BuildKit (`platform/buildkit/`) and publishes it on a push to `main`
-   same two blockers sit in front of `conformance-runner`, which shares the registry path and
+   (`decisions/2026-10-29-image-pipeline.md`); the first published digest is pinned in
-   has never been pulled.
+   `deployment.yaml` and `configmap.yaml`, and Renovate carries every later bump through the
   `# renovate-digest:` markers (`decisions/2026-10-30-pm-agent-live.md`). The same
   credential blocker sits in front of `conformance-runner`, which shares the registry path
   and has never been pulled.
 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the
   bot's permission is `write`, not `admin`** — that measurement is the control that keeps
   the agent out of its own approval gate.
 3. **First run with `PM_DRY_RUN=true` and `PM_REQUIRE_LABEL=agent`**, on one issue written
-   for the purpose. Read what the PM says before letting it write anything.
+   for the purpose. Read what the PM says before letting it write anything. *Done:* dry-run
   triage of issue #97 read; `PM_DRY_RUN=false` on 2026-10-31
   (`decisions/2026-10-31-pm-agent-go-live.md`), `PM_REQUIRE_LABEL=agent` kept.
 4. **Tighten the admission image pin** from the `@sha256:*` wildcard to the literal digest.
   One line, reviewed.
 5. **Narrow `world:443` to `toFQDNs: api.anthropic.com`** — after verifying Cilium's DNS
@@ -214,6 +214,12 @@ class Config:
        # ConfigMap without rebuilding the image.
        self.push_on_red = env("PM_PUSH_ON_RED_GATE", "false")
        self.pm_timeout = env_int("PM_DECISION_TIMEOUT_SECONDS", 900)
        # The PM holds no tools, so the only thing a second turn can be spent on is the CLI's
        # structured-output step re-asking the model after an answer that failed the schema.
        # 1 is the long-standing value; it is a knob so that, if the failure envelope now
        # logged by describe_cli_failure() says `error_max_turns`, the remedy is a ConfigMap
        # change rather than another image build.
        self.pm_max_turns = env_int("PM_DECISION_MAX_TURNS", 1)
        # Opt-in gate. Empty (the default) means every open issue is in scope, which is the
        # behaviour the service was asked for. Setting it to a label name — `agent` is the
        # conventional one — narrows the agent to issues a human has explicitly handed it,
@@ -231,6 +237,8 @@ class Config:
        if self.max_active_workers < 1:
            die("PM_MAX_ACTIVE_WORKERS must be >= 1")
        if not 1 <= self.pm_max_turns <= 5:
            die("PM_DECISION_MAX_TURNS must be between 1 and 5")
        if "@sha256:" not in self.worker_image:
            # The admission policy requires a digest too, so a tag here would produce a Job
            # that is created and then DENIED — a failure visible only in the dispatcher's
@@ -437,7 +445,7 @@ def ask_pm(cfg: Config, thread: str) -> dict:
        "--system-prompt-file", cfg.persona_file,
        "--tools", "",
        "--permission-prompts", "none",
-        "--max-turns", "1",
+        "--max-turns", str(cfg.pm_max_turns),
        "--output-format", "json",
        "--json-schema", json.dumps(DECISION_SCHEMA),
        "--no-session-persistence",
@@ -445,10 +453,76 @@ def ask_pm(cfg: Config, thread: str) -> dict:
    proc = subprocess.run(cmd, input=thread, capture_output=True, text=True,
                          timeout=cfg.pm_timeout)
    if proc.returncode != 0:
-        raise RuntimeError(f"claude exited {proc.returncode}: {proc.stderr.strip()[:800]}")
+        raise RuntimeError(describe_cli_failure(proc.returncode, proc.stdout, proc.stderr))
    return extract_decision(proc.stdout)
 # Anything that could be a credential in the CLI's output. The CLI does not normally echo its
 # token, but an API error body or a proxy's HTML page is not something to bet a leak on. Both
 # schemes are matched (the http one is the lesson of a scheme-specific redaction that leaked
 # ~/.git-credentials), and `token ...` is Gitea's own header scheme.
 _SECRET_PATTERNS = [
    (re.compile(r"sk-ant-[A-Za-z0-9_\-]+"), "sk-ant-[REDACTED]"),
    (re.compile(r"(?i)\b(bearer|token)(\s+|=|:\s*)[A-Za-z0-9._~+/=\-]{8,}"), r"\1\2[REDACTED]"),
    (re.compile(r"(?i)(https?://)[^/\s:@]+:[^/\s@]+@"), r"\1[REDACTED]@"),
 ]
 def redact(text: str) -> str:
    for pat, repl in _SECRET_PATTERNS:
        text = pat.sub(repl, text)
    return text
 def describe_cli_failure(rc: int, stdout: str, stderr: str, limit: int = 800) -> str:
    """Say WHY `claude --print` failed, in the dispatcher log, redacted.
    WHY THIS EXISTS: with `--output-format json` the CLI reports its own failures — an API
    error, a usage limit, `error_max_turns`, a schema the model could not satisfy — as the
    JSON result envelope on STDOUT (`is_error`, `subtype`, `result`), and writes nothing to
    stderr. Logging only stderr therefore produced `claude exited 1: ` with an empty message
    on 2026-10-31, twice, with the cause sitting unread in stdout. So: the envelope's
    diagnostic fields first, then the raw tails of both streams.
    """
    parts = [f"claude exited {rc}"]
    out = (stdout or "").strip()
    env = None
    if out:
        try:
            env = json.loads(out)
        except ValueError:
            # A stream of JSON lines, or a log line ahead of the envelope: the last line that
            # parses as an object is the result.
            for line in reversed(out.splitlines()):
                try:
                    cand = json.loads(line)
                except ValueError:
                    continue
                if isinstance(cand, dict):
                    env = cand
                    break
    if isinstance(env, dict):
        fields = []
        for key in ("subtype", "is_error", "api_error_status", "num_turns", "duration_ms",
                    "duration_api_ms", "total_cost_usd", "stop_reason"):
            if key in env:
                fields.append(f"{key}={env[key]!r}")
        if fields:
            parts.append("envelope: " + " ".join(fields))
        for key in ("result", "error", "errors", "message"):
            if env.get(key):
                val = env[key] if isinstance(env[key], str) else json.dumps(env[key])
                parts.append(f"{key}: {val[:limit]}")
    if out and not isinstance(env, dict):
        parts.append(f"stdout tail: {out[-limit:]}")
    err_text = (stderr or "").strip()
    if err_text:
        parts.append(f"stderr tail: {err_text[-limit:]}")
    if len(parts) == 1:
        parts.append("(no output on stdout or stderr)")
    return redact(" | ".join(parts))
 def extract_decision(stdout: str) -> dict:
    """Pull the decision object out of whatever `claude --print` wrapped it in.
@@ -501,7 +575,7 @@ def extract_decision(stdout: str) -> dict:
        obj = as_obj(c)
        if obj and isinstance(obj.get("action"), str):
            return obj
-    raise RuntimeError(f"no decision object in claude output: {stdout[:400]!r}")
+    raise RuntimeError(redact(f"no decision object in claude output: {stdout[-800:]!r}"))
 # --------------------------------------------------------------------------- thread
@@ -922,6 +996,8 @@ def selftest_template(path: str) -> None:
    assert pod.get("automountServiceAccountToken") is False, \
        "template must not mount a ServiceAccount token"
    assert pod.get("restartPolicy") == "Never", "a retried agent would double-push"
    assert [x.get("name") for x in pod.get("imagePullSecrets") or []] == ["pm-agent-registry"], \
        "the template must name exactly the committed pull secret (admission pins it)"
    assert len(pod.get("containers") or []) == 1, "exactly one container"
    c = pod["containers"][0]
    assert c.get("command") == ["/usr/local/bin/pm-worker"], "command must be the pinned entrypoint"
@@ -986,8 +1062,25 @@ def selftest() -> int:
        except RuntimeError:
            check(True, f"bad output refused: {label}")
    # -- a failed CLI run must say why, and must not leak ---------------------
    # The 2026-10-31 failure: rc 1, empty stderr, the cause in the stdout envelope.
    env_err = json.dumps({"type": "result", "subtype": "error_max_turns", "is_error": True,
                          "num_turns": 2, "result": "API Error: 429 rate_limit_error"})
    msg = describe_cli_failure(1, env_err, "")
    check("error_max_turns" in msg and "429" in msg,
          "CLI failure: stdout envelope surfaced when stderr is empty")
    msg = describe_cli_failure(1, "", "boom on stderr")
    check("boom on stderr" in msg, "CLI failure: stderr tail surfaced")
    msg = describe_cli_failure(1, "", "")
    check("no output" in msg, "CLI failure: silence is named, not an empty message")
    msg = describe_cli_failure(1, "plain text sk-ant-oat01-abcdefABCDEF_123 here",
                               "Authorization: Bearer abcdefghijklmnop "
                               "http://u:[email protected]/x")
    check("sk-ant-oat01" not in msg and "abcdefghijklmnop" not in msg and ":pw@" not in msg,
          "CLI failure: token, bearer and URL credential redacted")
    # -- the committed Job template -----------------------------------------
-    tmpl = os.environ.get("PM_WORKER_JOB_TEMPLATE", "/etc/pm-agent/worker-job-template.yaml")
+    tmpl =os.environ.get("PM_WORKER_JOB_TEMPLATE", "/etc/pm-agent/worker-job-template.yaml")
    try:
        selftest_template(tmpl)
        check(True, f"worker Job template satisfies its contract ({tmpl})")
@@ -68,6 +68,13 @@ spec:
      # guarantee it cannot use one is for it not to have one.
      automountServiceAccountToken: false
      serviceAccountName: pm-agent-worker
      # The kubelet's pull credential, the same one the dispatcher uses. Pinned by the
      # admission policy to exactly this name: `imagePullSecrets` is a pod-spec field, so a
      # Job could otherwise name any Secret in the namespace and have the kubelet read it
      # before a container ever starts. The image is digest-pinned, so the leverage is small
      # — but "only the committed shape" is the rule this template exists to express.
      imagePullSecrets:
        - name: pm-agent-registry
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
@@ -111,6 +111,12 @@ spec:
                =(hostNetwork): "false"
                =(hostPID): "false"
                =(hostIPC): "false"
                # The kubelet reads this Secret BEFORE any container starts, so it is part
                # of the shape even though no container references it. A pattern on an array
                # asserts "at least one element matches", which is enough to REQUIRE the
                # committed entry; rule 7 is what forbids a second one.
                imagePullSecrets:
                  - name: pm-agent-registry
                containers:
                  - image: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:*"
                    command:
@@ -336,6 +342,54 @@ spec:
    # dispatcher itself uses envFrom (it is a Deployment, not a Job, and not matched by this
    # policy); a worker must not.
    # =====================================================================================
    # =====================================================================================
    # Rule 7 — exactly one imagePullSecret, and it is the committed one.
    #
    # `imagePullSecrets` is a pod-spec field that no container references, which is precisely
    # why it is easy to leave unread — the same shape of omission as `initContainers` and
    # `env` before it. The kubelet reads the named Secret, in this namespace, before any
    # container starts. A Job naming a different one would have the kubelet read a Secret the
    # worker was never meant to touch.
    #
    # The leverage is genuinely small: the image is digest-pinned, so a different credential
    # cannot change WHAT runs, and the only Secrets in this namespace are the agent's own.
    # It is pinned anyway, because "only the committed shape" is the whole premise of this
    # policy, and because the cost of the rule is four lines while the cost of rediscovering
    # the omission is an incident.
    #
    # Rule 1's pattern REQUIRES the committed entry; this forbids any OTHER entry and any
    # second entry. A foreach over an absent list yields no result, which is why both halves
    # are needed.
    # =====================================================================================
    - name: worker-pull-secret-must-be-the-committed-one
      match:
        all:
          - resources:
              kinds:
                - Job
              namespaces:
                - pm-agent
            subjects:
              - kind: ServiceAccount
                name: pm-agent
                namespace: pm-agent
      validate:
        message: >-
          A worker Job may declare exactly ONE imagePullSecret, named `pm-agent-registry`.
          The kubelet reads that Secret before any container starts, so naming another one
          reaches a credential the worker was never meant to touch.
        foreach:
          - list: "request.object.spec.template.spec.imagePullSecrets"
            deny:
              conditions:
                any:
                  - key: "{{ length(request.object.spec.template.spec.imagePullSecrets) }}"
                    operator: NotEquals
                    value: 1
                  - key: "{{ element.name }}"
                    operator: AnyNotIn
                    value: ["pm-agent-registry"]
    - name: worker-may-not-use-envfrom
      match:
        all:
@@ -56,6 +56,11 @@ results:
    kind: Job
    resources: [good-run, real-template]
    result: pass
  - policy: pm-agent-job-shape
    rule: worker-pull-secret-must-be-the-committed-one
    kind: Job
    resources: [good-run, real-template]
    result: pass
  # ---------------- THE escalation, and the shape rule's other refusals ----------------
  # `evil-command` is the case the whole model exists for: right namespace, right creator,
@@ -73,6 +78,14 @@ results:
      - host-namespaces
      - restart-onfailure
      - second-container
      - no-pull-secret
    result: fail
  # ---------------- the pull secret: a second one, or a different one ----------------
  - policy: pm-agent-job-shape
    rule: worker-pull-secret-must-be-the-committed-one
    kind: Job
    resources: [extra-pull-secret, wrong-pull-secret]
    result: fail
  # ---------------- args, and the second container ----------------
@@ -31,6 +31,8 @@ spec:
      restartPolicy: Never
      automountServiceAccountToken: false
      serviceAccountName: pm-agent-worker
      imagePullSecrets:
        - name: pm-agent-registry
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
@@ -95,6 +97,8 @@ spec:
      restartPolicy: Never
      automountServiceAccountToken: false
      serviceAccountName: pm-agent-worker
      imagePullSecrets:
        - name: pm-agent-registry
      containers:
        - name: worker
          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
@@ -492,3 +496,75 @@ spec:
            readOnlyRootFilesystem: true
          envFrom:
            - secretRef: { name: pm-agent-credentials }
 ---
 # A SECOND pull secret beside the committed one. containers[0] is perfectly compliant, and
 # rule 1's array pattern is satisfied by the first element — which is why the count is
 # asserted separately.
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: extra-pull-secret
  namespace: pm-agent
 spec:
  template:
    spec:
      restartPolicy: Never
      automountServiceAccountToken: false
      serviceAccountName: pm-agent-worker
      imagePullSecrets:
        - name: pm-agent-registry
        - name: pm-agent-credentials
      containers:
        - name: worker
          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
          command: ["/usr/local/bin/pm-worker"]
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
 ---
 # A DIFFERENT pull secret — the kubelet would read a Secret the worker was never meant to
 # touch, before any container starts.
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: wrong-pull-secret
  namespace: pm-agent
 spec:
  template:
    spec:
      restartPolicy: Never
      automountServiceAccountToken: false
      serviceAccountName: pm-agent-worker
      imagePullSecrets:
        - name: pm-agent-credentials
      containers:
        - name: worker
          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
          command: ["/usr/local/bin/pm-worker"]
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
 ---
 # NO pull secret at all. Rule 7's foreach yields nothing over an absent list, so this is
 # rule 1's pattern to refuse — which is why the requirement lives in both.
 apiVersion: batch/v1
 kind: Job
 metadata:
  name: no-pull-secret
  namespace: pm-agent
 spec:
  template:
    spec:
      restartPolicy: Never
      automountServiceAccountToken: false
      serviceAccountName: pm-agent-worker
      containers:
        - name: worker
          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
          command: ["/usr/local/bin/pm-worker"]
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
@@ -46,15 +46,18 @@ waive the sensitive-path human review on its own pull request.
 Both are deliberate, and both fail **visibly** rather than silently:
 1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
-   `PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
+   `PM_WORKER_IMAGE` in `configmap.yaml` first carried the run-340 publish
-   publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
+   (`sha-523cd9118f95`), whose baked template predated the `imagePullSecrets` entry admission
   requires, so the service ran with `PM_DRY_RUN=true`. PR #96 pinned the run-383 digest
   (`df36d9b`, `sha-ddd04d460ef6`), whose template carries the entry, and dry run was turned
   off on 2026-10-31 (`decisions/2026-10-31-pm-agent-go-live.md`); every later bump is
   Renovate's. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
   (`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
   `infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
   `gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
   Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
   lines and moves them together in one pull request. Nobody builds-and-pushes this image from
-   a workstation, and nobody pastes a digest. Until the first publish, and until that PR
+   a workstation. The one operator prerequisite on the publishing side is the
   merges: `ImagePullBackOff`. The one operator prerequisite on the publishing side is the
   `REGISTRY_PUSH_TOKEN` Actions secret (see the header of `.gitea/workflows/images.yaml`);
   without it the publish job fails loudly rather than skipping.
@@ -78,21 +81,36 @@ Both are deliberate, and both fail **visibly** rather than silently:
     inventory, never typed on a node. Apply it scoped to that one task:
     `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
     --tags hosts`.
-   - **The pull is still unauthenticated** once the node reaches the right endpoint. An
+   - **The pull must be authenticated** once the node reaches the right endpoint: an
-     unauthenticated client gets nothing from this Gitea's registry, and there is no
+     unauthenticated client gets nothing from this registry. **Now wired** —
-     `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
+     `pm-agent-registry` is a `dockerconfigjson` Secret referenced by both the Deployment and
-     (local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea
+     the committed worker Job template, and the admission policy pins it to exactly that
-     account whose token carries `read:package`, referenced from the Deployment and the
+     name. What is left to you is the value: the Gitea token must carry **`read:package`** in
-     committed worker Job template — not built yet, because minting that token is yours.
+     addition to `read:user`, `write:repository` and `write:issue`, and the dockerconfigjson is built from
     it. Steps:
     [`pm-agent-registry.sops.yaml.example`](./pm-agent-registry.sops.yaml.example).
   The same two apply to `conformance-runner`: same registry path, same publish-then-Renovate
   model, never pulled either.
-2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
+2. **The credentials are placeholders — two Secrets, one token.** Both are encrypted to the
-   repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
+   repository's age recipient and both hold `REPLACE-ME-…` strings:
-   which Gitea account to create, which scopes, and why it must *not* be an administrator —
+
-   is in [`pm-agent-credentials.sops.yaml.example`](./pm-agent-credentials.sops.yaml.example).
+   | Secret | type | read by | hand-off |
-   Until then the dispatcher CrashLoops with `GET /user returned no login — is GITEA_TOKEN
+   |---|---|---|---|
-   valid?`.
+   | `pm-agent-credentials` | Opaque | the application, as env | [`.example`](./pm-agent-credentials.sops.yaml.example) |
   | `pm-agent-registry` | dockerconfigjson | the **kubelet**, before any container starts | [`.example`](./pm-agent-registry.sops.yaml.example) |
   They hold the **same** Gitea token in two formats, because a Secret cannot be both — the
   same one-value-in-two-files pattern the Keycloak client secret already uses here. So mint
   it once, with all four scopes: `read:package` (the pull), `read:user` (the dispatcher's
   startup `GET /user` — Gitea 1.24.3 scopes the whole `/user` group), `write:repository`
   (clone, branch push, pull request) and `write:issue` (issue comments). The account also
   needs package *permission*, not just the scope: a `pm-agent-pull` org team with Packages
   = Read and no repositories (steps in the registry `.example`). The first `.example`
   explains which account to create and why it must *not* be an administrator.
   Until they are filled in: the pull fails `401`, and if you fix only the pull, the
   dispatcher CrashLoops with `GET /user returned no login — is GITEA_TOKEN valid?`.
 Its Flux Kustomization is `wait: false` precisely because of those two states: a workload
 that is *expected* to be unhealthy for a while must never be able to gate anything.
@@ -155,13 +173,14 @@ Job from a committed template. **The model's output is data, never a command.**
 | **cannot** | clone from or push to anywhere but `gitea.gitea.svc.cluster.local:3000`: `pm-worker` validates `PM_GITEA_URL` against a one-entry allowlist baked into the image. In-cluster consumers reach a service by its cluster-local name — a different path by design from how a client outside the cluster reaches it — so the external hostname is not an accepted value |
 | **cannot** | authenticate a `git push` **during the model session**: `GIT_ASKPASS` is unset for its duration, so the restriction is structural rather than a request in a prompt |
 | **cannot** | be created in any shape but the committed one — see the admission policy |
 | **cannot** | have the kubelet read any Secret but `pm-agent-registry` on its behalf: `imagePullSecrets` is pinned to exactly that one name |
 ### What is *not* defended, stated plainly
 - **The worker's own credentials are inside the model's reach.** The session runs as the same
  uid in the same container as the wrapper, so it can read the environment, including its
  `GITEA_TOKEN` and `CLAUDE_CODE_OAUTH_TOKEN`. There is no isolation between them and none is
-  claimed. What bounds it: the Gitea token is scoped `write:repository, write:issue` on **one
+  claimed. What bounds it: the Gitea token is scoped `read:user, write:repository, write:issue, read:package` on **one
  repository** with no admin permission, and egress reaches only this Gitea and tcp/443.
 - **`world:443` is wider than `api.anthropic.com`.** Cilium can express the tighter rule
  (`toFQDNs`) but nothing in this repository uses FQDN policy and its prerequisite (Cilium's
@@ -200,14 +219,15 @@ Two things answer it, and neither is sufficient alone:
   `command` with no args, that powerless SA, no mounted token, `restartPolicy: Never`,
   non-root with no privilege escalation and a read-only root filesystem, only the
   `workspace`/`tmp` **emptyDir** volumes, no init or ephemeral containers, only the
-   committed env **names**, and no `envFrom`.
+   committed env **names**, no `envFrom`, and exactly one `imagePullSecret` — the committed
   one.
-Six rules, because pinning `command` only moves the injection to whatever adjacent field
+Seven rules, because pinning `command` only moves the injection to whatever adjacent field
 nobody enumerated — on the sibling policy that was `initContainers` (measured: a Job with a
 compliant `containers[0]` and a privileged init container was **admitted**, pass 3 fail 0),
 then `ephemeralContainers`, then `env`, then `envFrom`. All of them are closed here from the
-start, and `platform/kyverno/tests/pm-agent/` proves each refusal offline: **28 assertions,
+start, and `platform/kyverno/tests/pm-agent/` proves each refusal offline: **33 assertions,
-28 passing** under the pinned `kyverno` CLI. The worker's own input guards have their own
+33 passing** under the pinned `kyverno` CLI. The worker's own input guards have their own
 offline probe — `infra/agent/probe-validation.sh`, **17 refusals** — because a guard nothing
 exercises is a claim rather than a control.
@@ -230,6 +250,12 @@ five that matter:
 | `PM_MAX_ACTIVE_WORKERS` | `2` | concurrent workers. RAM is the binding constraint (2Gi limit each) and the subscription seat's rate limits are shared with every other use of that seat |
 | `PM_MAX_NEW_WORKERS_PER_CYCLE` | `1` | drains a backlog one issue at a time instead of in one stampede the first minute the credentials work |
 | `PM_PUSH_ON_RED_GATE` | `false` | a change the gate refused is not pushed; the issue gets the gate log instead |
 | `PM_DECISION_MAX_TURNS` | `1` (unset) | turns the triage call may take (1–5). The PM holds no tools, so a second turn is only ever the CLI re-asking after an answer that failed the decision schema. Raise it if a triage error logs `subtype='error_max_turns'` |
 **When a triage fails**, the `[error] issue #N: claude exited …` line carries the CLI's result
 envelope (`subtype`, `is_error`, `api_error_status`, `num_turns`, `result`) and the tails of
 stdout and stderr, redacted. Before 2026-10-31 it logged stderr only, which is empty under
 `--output-format json` — see `decisions/2026-10-31-pm-agent-triage-exit.md`.
 Changing a knob requires bumping the `config-generation` annotation in `deployment.yaml` in
 the same commit — the process reads its configuration once, at startup, so without the bump
@@ -51,15 +51,20 @@ data:
  # the dispatcher and its workers share the committed Job template and the env-name
  # contract. A tag here is refused at startup (and would be DENIED by admission anyway).
  #
-  # PLACEHOLDER UNTIL THE IMAGE IS PUBLISHED. The all-zero digest below is deliberately
+  # THE FIRST PUBLISHED DIGEST. CI builds, verifies and — on main — publishes agent-runner
-  # impossible. CI builds, verifies and — on main — publishes agent-runner
+  # (.gitea/workflows/images.yaml). This value is images run 340's publish (tag
-  # (.gitea/workflows/images.yaml), and Renovate replaces this value with the digest the
+  # sha-523cd9118f95, which `main` pointed at), read from the registry on 2026-10-30. From here
-  # REGISTRY returned for the image's `main` tag, in a pull request that also moves the
+  # on Renovate replaces it with the digest the REGISTRY returns for the `main` tag, in a pull
-  # Deployment's image (same marker, same branch). Nobody pushes this image from a workstation.
+  # request that also moves the Deployment's image (same marker, same branch). Nobody pushes
-  # Until the first publish the dispatcher starts, fails to pull, and says so — which is the
+  # this image from a workstation.
-  # visible failure, not a silent one.
+  #
  # The run-340 digest predated the worker template's `imagePullSecrets` entry (PR #61), which
  # admission REQUIRES, so its worker Jobs were DENIED. The digest below (df36d9b, run 383,
  # agent-runner:sha-ddd04d460ef6, pinned in PR #96) bakes a template that carries
  # `imagePullSecrets: pm-agent-registry`, which is what made turning PM_DRY_RUN off possible
  # (decisions/2026-10-31-pm-agent-runner-digest.md). A digest older than that would be DENIED.
  # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
-  PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42"
+  PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d"
  # --- the model -------------------------------------------------------------
  # The PM only reads a thread and answers with one JSON object, so it holds no tools and
@@ -104,12 +109,21 @@ data:
  # work somebody deliberately handed it while trust is being established, with no code
  # change and no redeploy of anything but this ConfigMap. The operator's call; the default
  # here honours the request as made.
-  PM_REQUIRE_LABEL: ""
+  #
  # SET TO `agent` FOR THE FIRST RUN (2026-10-30, docs/plans/pm-agent-issue-to-pr.md
  # follow-up 3). The label does not exist in the repository yet: create it in Gitea
  # (Issues -> Labels -> New Label, name `agent`) before labelling a test issue.
  PM_REQUIRE_LABEL: "agent"
  # Dry run: the PM still reads threads, still asks the model and still posts its decision as
  # an issue comment — so the operator can read exactly what it WOULD have done — but creates
  # no worker Job and so changes no code. The honest way to watch this service for a day
  # before letting it write anything.
  #
  # "true" for the first run (2026-10-30 .. 2026-10-31); dry-run triage of issue #97 was read
  # and the operator approved going live. "false" since decisions/2026-10-31-pm-agent-go-live.md.
  # Setting it back to "true" (plus a config-generation bump) is the gentlest stop — see
  # runbooks/pm-agent.md.
  PM_DRY_RUN: "false"
  # If the gate refuses a worker's change, the branch is NOT pushed by default and the issue
@@ -48,9 +48,13 @@ spec:
        # It is a hand-maintained marker rather than a computed hash because Flux's
        # Kustomization here does not run a configmap generator; bump it in the same commit
        # as a change to configmap.yaml.
-        pm-agent.homelab.keithjackson.dev/config-generation: "1"
+        pm-agent.homelab.keithjackson.dev/config-generation: "4"
    spec:
      serviceAccountName: pm-agent
      # The kubelet's credential for pulling this image. Not the agent's Gitea credential —
      # see pm-agent-registry.sops.yaml.example for why they are two Secrets of one token.
      imagePullSecrets:
        - name: pm-agent-registry
      # The dispatcher DOES need its token: `kubectl create job` is its one privileged act,
      # and platform/pm-agent/rbac.yaml is what bounds it to that. Contrast the worker, whose
      # Job template sets this false because it needs no API access at all.
@@ -69,10 +73,12 @@ spec:
          # disagree about the shape the admission policy enforces. Replaced together, in ONE
          # Renovate pull request: both lines carry the same marker, so Renovate treats them as
          # one dependency and bumps them in the same branch. The digest comes from CI's publish
-          # of agent-runner's `main` tag (.gitea/workflows/images.yaml) — the all-zero
+          # of agent-runner's `main` tag (.gitea/workflows/images.yaml). The value below is the
-          # placeholder below is impossible on purpose (see configmap.yaml PM_WORKER_IMAGE).
+          # FIRST publish (images run 340, tag sha-523cd9118f95 = main at the time), read from
          # the registry's manifest endpoint on 2026-10-30 and pinned by hand once so the
          # dispatcher does not wait on Renovate's first run; every later bump is Renovate's.
          # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
-          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42
+          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d
          imagePullPolicy: IfNotPresent
          command:
            - /usr/local/bin/pm-dispatch
@@ -35,6 +35,12 @@ resources:
  # the distance between naming a file in a comment and listing it under `resources:` is one
  # careless edit.
  - pm-agent-credentials.sops.yaml
  # The KUBELET's pull credential for agent-runner — a separate Secret from the one above
  # because it is a different consumer in a different format: dockerconfigjson, read before
  # any container starts. Same underlying Gitea token, which must also carry `read:package`.
  # Measured 2026-10-05: this registry refuses unauthenticated clients outright, so without
  # this the pull 401s. Also placeholders until the operator fills it in.
  - pm-agent-registry.sops.yaml
  - deployment.yaml
  # default-deny baseline + the two egress allow policies (gitea, kube-apiserver for the
  # dispatcher only, tcp/443 for the model). NO ingress allow exists, by design.
@@ -11,8 +11,8 @@ metadata:
        app.kubernetes.io/part-of: pm-agent
 type: Opaque
 stringData:
-    gitea-token: ENC[AES256_GCM,data:Vs6TkiNocU+qzvlv8mndXhtiqp1RyY00KhIBDuL8w0WoX72B8DxfPK8eWISy9gtx9TBlYpu2FHP7,iv:2z8+gzWuZDFACb06rnYg071h4wO7YlcTFNZDQzDxRNs=,tag:cUhj+uQUwu44HpPtFTVjVA==,type:str]
+    gitea-token: ENC[AES256_GCM,data:nkQT4m0DYykNcEUPV+1A4APuraekHN+8tY74HnRIbwU8SwYbRRqfKQ==,iv:d6Lj4Q/ys381msrrRtIIYbV9Ey3Nhua4zmX/7aCMlSk=,tag:xKKmc1i9TRuUVc/C2P1zHg==,type:str]
-    claude-code-oauth-token: ENC[AES256_GCM,data:2PjA5MOx8Enm4wxqYdJ3QY8juqPkXh4adQv3IRtw1Nb2Dw6SkOFc,iv:ZSlefJZqCN7g17D+27VkkVYu8F1umM7JOG+f/crZtsE=,tag:QyNYIbHr+ANS0Rx4USa0Iw==,type:str]
+    claude-code-oauth-token: ENC[AES256_GCM,data:Z5elUXr2wiu1S3FvSsHPNfhyrh+dGAdBf4HEmvg+Ky3QuCDpLv/XcMG5wWNywYVR4jlx/z2rm2FjjRYo+8rG0+I3SzriKG2MD7Zri4bDUf+DFksa6pI2oGDmgTjp8Zjz8oGapaMurRnuYcMm,iv:tXADrr2PD1zTpmTuaCA224wzbsq2RKzx0/ylU2k6suE=,tag:DMh31sAj0E/8ndcP1M6zAw==,type:str]
 sops:
    kms: []
    gcp_kms: []
@@ -28,8 +28,8 @@ sops:
            V3d1UGVsTlljS3lHbFRkUWE4aVdRSDAK3vlChFXHg+jOJyjzUBZ4F4NqIn8rr5xl
            WAH9bgY14yvX8qm3P4iLeFecNssqUvs2hvDW82vhMfUQlqYpTTXyjQ==
            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-10-06T03:09:21Z"
+    lastmodified: "2026-10-31T16:07:12Z"
-    mac: ENC[AES256_GCM,data:lKj3CtZ3fCvOwuJSeNb5x8dj+5O0Uhfyoir+ZfIGD52xgeTWtmQ6CmOVbtrYjp1h1dgZnWopbibfMIKE9AmjSAHA7UJOen++zcy4qrA8Yh8xGzfhb1tOJplxZP+Lp9319VYQHqXqOScs7at8Osnw2RYuibW/j6eIY0B17JzIXvo=,iv:14aKtsMd9Q6QqnWVszoi+W2+z4QPfXUd7oZfK46+gRM=,tag:NwBg43/vNeFlgHmNLB7rYw==,type:str]
+    mac: ENC[AES256_GCM,data:rK1GR6FoIsh2Y6VSQSkLhGwf1U2kBrV3VYZT/5X+TxbkURza7DOTUyMJsY0oLKApvkTvOxLd3/WfzBNW/O6Tho1S4otvi4UC7fY6DjC46LmrfSlAxoRp7ov/oIRfwLBUanpNzfT94fu7sjoNM0lRlFp9iSpBU44upcIWwclK5WY=,iv:EujucxcPU+zfN9KmOhbqfVT5anv4g/jRg41hu5BgGx8=,tag:fViHjTT6w20b2AsXZodq0g==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.9.4
@@ -60,11 +60,21 @@
 #     Keycloak service-account token therefore cannot authenticate to Gitea's API or to
 #     git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that.
 #
-#       Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT
+#       Scopes, exactly these four (one token serves both Secrets — see
-#       `write:admin`, NOT `write:organization`, NOT `write:user`.
+#       pm-agent-registry.sops.yaml.example):
 #         read:user         GET /user, the dispatcher's startup identity check. Gitea 1.24.3
 #                           puts the whole /api/v1/user group behind the `user` scope
 #                           category (routers/api/v1/api.go), so without it the dispatcher
 #                           CrashLoops on its first call even with an otherwise-valid token.
 #         write:repository  the clone, the branch push, the pull-request create, and the
 #                           dispatcher's reads of pulls/branches/commits.
 #         write:issue       reading open issues and posting issue comments.
 #         read:package      the kubelet's image pull (pm-agent-registry).
 #       Nothing else — in particular NOT `write:admin`, NOT `write:organization`, NOT
 #       `write:user`. (Until 2026-10-30 this list said two scopes; `read:user` was missing
 #       and `read:package` lived only in the registry template.)
 #
-#     `write:repository` covers the clone, the branch push and the pull-request create;
+#     (The repository's OTHER CI
 #     `write:issue` covers reading and posting issue comments. (The repository's OTHER CI
 #     token, `SETTINGS_API_TOKEN`, is a separate credential with `read:admin` — do not
 #     conflate them, and do not give this service that one. The 2026-10-02 measurement on
 #     PR #38 recorded in infra/gitea/approval-policy.json is about that token, not this one.)
@@ -94,6 +104,17 @@
 #
 # (4) COPY, FILL, ENCRYPT, COMMIT:
 #
 #     `sops` IS IN THE PINNED TOOLBOX, NOT ON THE HOST (measured 2026-10-30: absent from the
 #     workstation's PATH; sops 3.9.4 and vi in the toolbox). The simplest path needs no
 #     plaintext file at all — edit the committed, already-encrypted file in place, from a
 #     worktree, the way runbooks/vpn-remote-access.md does for its key:
 #
 #       ./infra/devcontainer/dev.sh run sops platform/pm-agent/pm-agent-credentials.sops.yaml
 #
 #     It opens vi on the decrypted document; replace the REPLACE-ME value(s), save, quit, and
 #     sops re-encrypts on write. If you take the cp route below instead, run the `sops -e -i`
 #     line through `./infra/devcontainer/dev.sh run` as well, and delete any plaintext temp file.
 #
 #       cp platform/pm-agent/pm-agent-credentials.sops.yaml.example \
 #          platform/pm-agent/pm-agent-credentials.sops.yaml
 #       $EDITOR platform/pm-agent/pm-agent-credentials.sops.yaml      # the two real values
@@ -120,7 +141,7 @@ metadata:
 type: Opaque
 stringData:
  # Gitea token for the dedicated, NON-ADMIN `pm-agent` account.
-  # Scopes: write:repository, write:issue.
+  # Scopes: read:user, write:repository, write:issue, read:package.
  gitea-token: REPLACE-ME-gitea-token
  # Output of `claude setup-token` on a subscription account. One year's validity.
  claude-code-oauth-token: REPLACE-ME-claude-code-oauth-token
@@ -0,0 +1,39 @@
 # PLACEHOLDER VALUES — the kubelet cannot pull agent-runner until the operator replaces them.
 # Full hand-off steps: pm-agent-registry.sops.yaml.example (same directory).
 # SOPS-encrypted to the repository's age recipient; keys stay cleartext for reviewable diffs.
 #
 # This is the KUBELET's pull credential, not the agent's Gitea credential. The agent's is
 # pm-agent-credentials (Opaque, read as env by the application); this one is
 # dockerconfigjson and is read before any container starts. Same underlying Gitea token,
 # which must carry read:package as well as write:repository and write:issue.
 apiVersion: v1
 kind: Secret
 metadata:
    name: pm-agent-registry
    namespace: pm-agent
    labels:
        app.kubernetes.io/name: pm-agent
        app.kubernetes.io/part-of: pm-agent
 type: kubernetes.io/dockerconfigjson
 stringData:
    .dockerconfigjson: ENC[AES256_GCM,data:NuKobTM0S0TK8U961uNTIIYiyAI2L3QCopZXvKxXHLC2tjpdyV+REKzYPmv+orCMiRsY8GNtfaHb8Hret4sQWHySAAKN0sEtL+OkzLyRH+u7irAGFYUBf8q+vIb+hnhdwejHfL1UEaHovJlZuIjqOdSItqlxqXURGtvUtBlkD7cnzDajzLyW+QTp5Gs3Fc8ps4XxRkv/589bKpn0DIEQiXUujydBZzKVnO5unxMea+4IBwcC6yaeQ+fXCoILrKD3PP2Igz0=,iv:vrQqH4dIUwDLSZGe1JKnsajtvUb7lJe3st/9K+4/w0M=,tag:P9VLwOqnTfM1uQnbousYqQ==,type:str]
 sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvdG1lQm16V0tJdWZXRXlU
            Z2R4RXVyTXd6VTVJUWtkbUVjZldnb1JFOWhVCjVMMDdDb3AwVkt3TzJzdjR0VnNs
            SFlncWg2cWlFTy9IZ1YxSC9Zd3ZMaEkKLS0tIEZFQWhIMW5zMURWYmlXaFg1ZjhD
            cG1Fb0V3VncrTm9SQ091a0J0Z3RURWMKpdzNVub0zH5S8nnCSCt2H6NcCqDbV24L
            Mf8K2eIfu/ixn2NfwoVVj2QO+n+3/P6PLphdoiKMrL+ms0+JqWFpkQ==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2026-10-31T16:11:11Z"
    mac: ENC[AES256_GCM,data:0GIYwtVcNngq2Zpz3haCx6L+j2SrgHFL5pRYeRM4Ac1T9kjKSwHik6Q4+T8FU+EJa9GP1yXrpdsjggsjzG9dI2zRyAF1C8x1znD5ls6OwT5jLak8QrHNcTavRlm6N6y1yOsQCTFMc+zeCywTnFJldTqPKH+6K/bNMNBuD14BNv8=,iv:oqGygkOmFgRw0TYbWO75/sJ/doXtorAWAg89LtaqIsg=,tag:vBO/DWV93ekZB86G+Hca2A==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.9.4
@@ -0,0 +1,118 @@
 # =============================================================================
 # TEMPLATE — NOT APPLIED, NOT ENCRYPTED (`.yaml.example` is ignored by kustomize and is
 # never auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
 #
 # WHAT THIS IS: the credential the KUBELET uses to pull `agent-runner` from the in-cluster
 # Gitea registry. It is NOT the credential the agent uses to talk to Gitea — that one is
 # `pm-agent-credentials` and it is read by the application. Two Secrets, because they are
 # consumed by two different things in two different formats:
 #
 #   pm-agent-credentials   Opaque        read by pm-dispatch / pm-worker as env vars
 #   pm-agent-registry      dockerconfigjson  read by the KUBELET, before any container runs
 #
 # WHY A PULL CREDENTIAL IS NEEDED AT ALL. Measured 2026-10-05: an unauthenticated client
 # gets nothing from this Gitea's registry — it issues an anonymous token whose `access`
 # claim is `null`, and every path answers 401 whether it exists or not
 # (`/api/v1/packages/Home-Network` replies `{"message":"token is required"}`). So a pull
 # with no credential cannot succeed, and before this Secret there was no `imagePullSecret`
 # anywhere in the tree. See decisions/2026-10-05-pm-agent-registry-pull-prereq.md.
 #
 # THE OTHER HALF, ALREADY FIXED: the nodes used to resolve this registry's hostname to the
 # non-hairpinned WAN address and fail TLS against the router's `CN=%D` placeholder
 # certificate. That is the `/etc/hosts` pin in
 # infra/ansible/inventory/group_vars/k3s_prod/vars.yml
 # (decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md). Both halves are needed; a
 # pull fails on whichever is missing.
 #
 # =============================================================================
 # HAND-OFF STEPS (operator)
 # =============================================================================
 #
 # (1) USE THE SAME `pm-agent` ACCOUNT the agent already has (created per
 #     pm-agent-credentials.sops.yaml.example — the dedicated, NON-ADMIN one). A second
 #     account would be a second thing to keep non-admin for no gain: this token reads one
 #     package and nothing else.
 #
 # (2) ADD THE PACKAGE SCOPE. Gitea token scopes are fixed at creation, so this means
 #     generating a token with all four scopes and using it in BOTH Secrets:
 #
 #       read:package        <- NEW, what the kubelet needs to pull
 #       read:user           <- the dispatcher's startup GET /user (Gitea 1.24.3 scopes it)
 #       write:repository    <- the agent's clone / branch push / pull-request create
 #       write:issue         <- the agent's issue comments
 #
 #     THE SCOPE IS NOT THE PERMISSION. `read:package` only lets the token use whatever
 #     package access the ACCOUNT has, and these packages belong to the ORGANISATION
 #     (`home-network/…`), not to a repository. Gitea 1.24.3 decides org-package access in
 #     services/context/package.go `determineAccessMode`: the highest Packages-unit mode of
 #     any org team the user is in, else READ if the org is visible to that user. Home-Network
 #     is `public` today (measured 2026-10-30), so a signed-in non-member already gets read —
 #     but that is a side effect of a visibility setting nobody chose for this. Make it
 #     explicit, and independent of visibility:
 #
 #       Organisation Home-Network -> Teams -> New Team
 #         name: pm-agent-pull     permission: none (unit-level)
 #         repositories: NONE (do not tick "all repositories")
 #         units: Packages = Read; every other unit = No Access
 #       then add `pm-agent` as its only member.
 #
 #     That grants package read on the org and nothing on any repository; the repo `Write`
 #     comes only from the collaborator entry in the credentials template.
 #
 #     Gitea -> (as pm-agent) Settings -> Applications -> Generate New Token.
 #     Put the SAME value in `pm-agent-credentials` (`gitea-token`) and in the
 #     `.dockerconfigjson` below. One token, two consumers, two formats — the same
 #     one-value-in-two-files pattern the Keycloak client secret already uses here, and for
 #     the same reason: a Secret cannot be shared across formats.
 #
 # (3) BUILD THE DOCKERCONFIGJSON. `auth` is base64 of `<username>:<token>` — NOT a hash,
 #     just encoding, so treat this file exactly as secret as the token itself:
 #
 #       USER=pm-agent
 #       TOKEN='<the token from step 2>'
 #       AUTH=$(printf '%s:%s' "$USER" "$TOKEN" | base64 -w0)
 #       cat > /tmp/dockerconfig.json <<JSON
 #       {"auths":{"gitea.home.keithjackson.dev":{"username":"$USER","password":"$TOKEN","auth":"$AUTH"}}}
 #       JSON
 #
 #     THE REGISTRY KEY MUST BE THE HOST AS THE IMAGE REFERENCE SPELLS IT —
 #     `gitea.home.keithjackson.dev`, no scheme, no port, no path. The kubelet matches the
 #     pull host against these keys literally; `https://…` or a trailing `/v2/` silently
 #     matches nothing and the pull is attempted anonymously, which 401s exactly as if this
 #     Secret were absent.
 #
 # (4) COPY, FILL, ENCRYPT, COMMIT:
 #
 #     `sops` IS IN THE PINNED TOOLBOX, NOT ON THE HOST (measured 2026-10-30: absent from the
 #     workstation's PATH; sops 3.9.4 and vi in the toolbox). The simplest path needs no
 #     plaintext file at all — edit the committed, already-encrypted file in place, from a
 #     worktree, the way runbooks/vpn-remote-access.md does for its key:
 #
 #       ./infra/devcontainer/dev.sh run sops platform/pm-agent/pm-agent-registry.sops.yaml
 #
 #     It opens vi on the decrypted document; replace the REPLACE-ME value(s), save, quit, and
 #     sops re-encrypts on write. If you take the cp route below instead, run the `sops -e -i`
 #     line through `./infra/devcontainer/dev.sh run` as well, and delete any plaintext temp file.
 #
 #       cp platform/pm-agent/pm-agent-registry.sops.yaml.example \
 #          platform/pm-agent/pm-agent-registry.sops.yaml
 #       # paste the one-line JSON from step 3 as the .dockerconfigjson value
 #       sops -e -i platform/pm-agent/pm-agent-registry.sops.yaml
 #
 #     The root `.sops.yaml` rule matches this path and encrypts `stringData`, so the diff
 #     stays reviewable. NEVER `sops -d` it into a terminal or a transcript (CLAUDE.md §6).
 #     It is already listed under `resources:` in platform/pm-agent/kustomization.yaml.
 #
 # (5) ROTATION. Rotating the Gitea token means updating BOTH Secrets, because it is one
 #     token. The dispatcher must then be restarted to re-read its env; worker Jobs and the
 #     kubelet pick the new value up on their own.
 # =============================================================================
 apiVersion: v1
 kind: Secret
 metadata:
  name: pm-agent-registry
  namespace: pm-agent
 type: kubernetes.io/dockerconfigjson
 stringData:
  # One line. Registry key = the host exactly as the image reference spells it.
  .dockerconfigjson: '{"auths":{"gitea.home.keithjackson.dev":{"username":"pm-agent","password":"REPLACE-ME-the-same-gitea-token","auth":"REPLACE-ME-base64-of-username:token"}}}'
@@ -67,9 +67,11 @@ Work through these in order; the first three are by far the most likely.
 | Symptom in the log | Cause | Fix |
 |---|---|---|
 | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
-| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been published; the committed digest is the deliberate placeholder | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
+| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
 | `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
-| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
+| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
 | `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets |
 | a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
 | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push |
 | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
 | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
Author	SHA1	Message	Date
pm-agent	f6150b8957	docs: Massive amounts of historical data gate / gate (pull_request) Successful in 2m58s Details gate / conformance (pull_request) Has been skipped Details From Gitea issue #97. Authored by the PM agent's worker; the merge gate is the approver. Refs #97. Agent: pm-agent (issue #97, mode implement)	2026-10-31 17:24:23 +00:00
jakman85	466f63ec3c	Merge pull request 'fix(pm-agent): log the CLI's stdout envelope when triage fails; add PM_DECISION_MAX_TURNS' (#99 ) from fix/pm-agent-triage-exit into main gate / gate (push) Successful in 2m56s Details images / images (push) Successful in 16s Details gate / conformance (push) Has been skipped Details Reviewed-on: #99	2026-10-31 17:12:40 +00:00
jakman85andClaude Opus 5.5	664ee82b9a	fix(pm-agent): log the CLI's stdout envelope when triage fails; add PM_DECISION_MAX_TURNS gate / gate (pull_request) Successful in 2m56s Details images / images (pull_request) Successful in 13s Details gate / conformance (pull_request) Has been skipped Details `claude --print --output-format json` reports its own failures (API error, usage limit, error_max_turns) as the result envelope on stdout and leaves stderr empty, so ask_pm()'s stderr-only message logged "claude exited 1: " with nothing after it, twice on 2026-10-31. describe_cli_failure() now logs the envelope's diagnostic fields plus stdout/stderr tails, redacted (sk-ant-, Bearer/token, URL userinfo). PM_DECISION_MAX_TURNS (default 1, unchanged) makes the max-turns remedy a ConfigMap change if the envelope turns out to say error_max_turns. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-31 12:04:09 -05:00
jakman85	95e08da2f0	Merge pull request 'chore(pm-agent): take the dispatcher out of dry run (operator-approved), config-generation 4' (#98 ) from chore/pm-agent-go-live into main gate / gate (push) Successful in 2m58s Details gate / conformance (push) Has been skipped Details Reviewed-on: #98 Reviewed-by: Keith Jackson <[email protected]>	2026-10-31 16:52:16 +00:00
jakman85andClaude Opus 5.5	0d3e51acc4	chore(pm-agent): turn PM_DRY_RUN off (operator-approved), bump config-generation to 4 gate / gate (pull_request) Successful in 2m58s Details gate / conformance (pull_request) Has been skipped Details The run-383 worker image (df36d9b, pinned in #96) bakes imagePullSecrets: pm-agent-registry, which admission requires, and the dispatcher's dry-run triage of #97 was read and approved. PM_REQUIRE_LABEL=agent and the worker caps are unchanged. The dry-run status text in the configmap comments, the README and the plan's follow-up 3 is updated to match. Pre-flight (read-only, k3s-prod): pm-agent-job-shape Ready+Enforce; SA/Role/RoleBinding present and can-i create jobs = yes; pm-agent-registry and pm-agent-credentials exist; both egress CNPs Valid. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-31 11:45:52 -05:00
jakman85	11267409d7	Merge pull request 'chore(pm-agent): pin agent-runner to the run-383 digest (`ddd04d4`), bump config-generation' (#96 ) from chore/pm-agent-runner-digest into main gate / gate (push) Successful in 2m58s Details gate / conformance (push) Has been skipped Details Reviewed-on: #96 Reviewed-by: Keith Jackson <[email protected]>	2026-10-31 16:34:49 +00:00
jakman85andClaude Opus 5.5	6a192ede2a	chore(pm-agent): pin agent-runner to the run-383 digest (`ddd04d4`), bump config-generation gate / gate (pull_request) Successful in 2m58s Details gate / conformance (pull_request) Has been skipped Details c4eba5b (run 340) bakes a worker template without imagePullSecrets, so the new admission rule refuses worker Jobs. df36d9b is agent-runner:main == :sha-ddd04d460ef6 (measured on the registry), and its /etc/pm-agent/worker-job-template.yaml carries imagePullSecrets: pm-agent-registry (read from the published layer). Renovate found df36d9b but holds it under "Awaiting Schedule" on dashboard #91: the operator's dispatch (run 385) was a real run at 12:14 New York, outside the 03:00-06:00 window. The renovate-digest markers are unchanged, so later bumps remain Renovate's. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-31 11:22:52 -05:00
jakman85	06d4a9a1fc	Merge pull request 'Added the credentials' (#95 ) from feature/pm-agent-credentials into main gate / gate (push) Successful in 2m56s Details gate / conformance (push) Has been skipped Details Reviewed-on: #95	2026-10-31 16:20:12 +00:00
jakman85	9db20e8ca0	Added the credentials gate / gate (pull_request) Successful in 2m57s Details gate / conformance (pull_request) Has been skipped Details	2026-10-31 11:11:22 -05:00
jakman85	ddd04d460e	Merge pull request 'feat(pm-agent): pull credential + admission rule 7, first published digest pinned, dry-run first start, token scopes fixed' (#61 ) from feat/pm-agent-registry-pull-secret into main gate / gate (push) Successful in 2m53s Details images / images (push) Successful in 15s Details gate / conformance (push) Has been skipped Details Reviewed-on: #61 Reviewed-by: Keith Jackson <[email protected]>	2026-10-31 15:46:54 +00:00
jakman85	f0760d1024	Merge pull request 'feat(images): pin pihole, jackett and plex to the digests already running; Renovate follows them' (#94 ) from feat/pin-pihole-jackett-plex into main gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details Reviewed-on: #94	2026-10-31 15:45:17 +00:00
jakman85	553d027f7c	Merge branch 'main' into feat/pm-agent-registry-pull-secret gate / gate (pull_request) Successful in 2m58s Details images / images (pull_request) Successful in 34s Details gate / conformance (pull_request) Has been skipped Details	2026-10-31 15:43:51 +00:00
jakman85andClaude Opus 5.5	7f0cbc7ba2	feat(pm-agent): pin the first published agent-runner digest, start dry, and fix the token scopes gate / gate (pull_request) Failing after 2m56s Details images / images (pull_request) Successful in 32s Details gate / conformance (pull_request) Has been skipped Details Rebased #61 onto the CI image pipeline (#81). This commit makes the remaining in-git blockers to going live go away: - deployment.yaml and PM_WORKER_IMAGE pin images run 340's digest (sha256:c4eba5b3..., tag sha-523cd9118f95 = main), read from the registry. Renovate markers unchanged. - PM_DRY_RUN=true and PM_REQUIRE_LABEL=agent for the first run. Dry run is required, not just cautious: the run-340 image's baked worker template predates imagePullSecrets, so admission would deny its Jobs until CI re-publishes after this merge and Renovate bumps. - Token scopes: Gitea 1.24.3 scopes GET /api/v1/user (the dispatcher's first call) under read:user. The documented list was missing it. Now read:user, write:repository, write:issue, read:package everywhere. - Org-package read documented as a team (Packages=Read, no repos), not just a scope. - sops is toolbox-only; the .examples say so and give the in-place edit. - runbook: key-name check no longer emits Secret values before the pipe. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-30 01:16:21 -05:00
jakman85andClaude Opus 5	a7eadd4054	feat(pm-agent): the kubelet's pull credential, and the seventh field admission had not read The registry had two blockers. #59 fixed the first (nodes resolved the hostname to the non-hairpinned WAN and failed TLS against the router's CN=%D cert). This is the second: the pull is unauthenticated, and this registry refuses unauthenticated clients outright — measured 2026-10-05, its anonymous token carries access: null and every path 401s. pm-agent-registry is a dockerconfigjson Secret (SOPS, placeholders) referenced by imagePullSecrets on the dispatcher Deployment and on the committed worker Job template. It holds the SAME Gitea token as pm-agent-credentials, because the two consumers need two formats: the application reads a raw token from env, the kubelet reads a dockerconfigjson before any container starts. Same one-value-in-two-files pattern as the Keycloak client secret, documented in both .example files. The operator's part is the value plus one new scope: read:package. ADMISSION GAINED A SEVENTH RULE. imagePullSecrets is a pod-spec field no container references, which is exactly why it is easy to leave unread — the same shape of omission as initContainers, ephemeralContainers, env and envFrom before it. The kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now REQUIRES the committed entry; rule 7 forbids any other and any second one. Both halves are needed: a foreach over an absent list yields no result. The leverage is small and the policy says so: the image is digest-pinned, so a different pull credential cannot change WHAT runs. Pinned anyway — 'only the committed shape' is the premise, and the rule costs four lines. kyverno test: 33 passed, 0 failed (was 28). Three new fixtures, each a real path: the committed entry plus a second one, a different Secret, and none at all. The gate also caught me shipping the Secret in plaintext — I never ran sops -e -i on the placeholder. Recorded in the handoff entry, because the file looked finished. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-30 01:09:39 -05:00