chore(pm-agent): pin agent-runner to the run-394 digest (466f63e, PR #99), bump config-generation

Dispatcher image and PM_WORKER_IMAGE both move to
sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2, the digest the
registry returns for agent-runner:sha-466f63ec3c3d and :main. config-generation 4 -> 5.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

apps/wave-0/jackett.yaml

@@ -78,7 +78,7 @@ spec:
     spec:
       containers:
       - name: jackett
-        image: lscr.io/linuxserver/jackett:latest
+        image: lscr.io/linuxserver/jackett:v0.24.2865-ls7
         ports:
         - containerPort: 9117
         volumeMounts:

apps/wave-0/pihole.yaml

@@ -62,7 +62,7 @@ spec:
     spec:
       containers:
       - name: pihole
-        image: pihole/pihole
+        image: pihole/pihole:2026.10.0
         ports:
         - containerPort: 80
         env:

apps/wave-0/plex.yaml

@@ -120,7 +120,7 @@ spec:
     spec:
       containers:
       - name: plex
-        image: plexinc/pms-docker:latest
+        image: plexinc/pms-docker:1.43.4.10903-e5521bd8c
         ports:
         - containerPort: 32400
         env:

catalog/jackett.yaml

@@ -16,7 +16,7 @@
 name: jackett
 kind: container
 source:
-  image: lscr.io/linuxserver/jackett:latest
+  image: lscr.io/linuxserver/jackett:v0.24.2865-ls7
 exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -70,7 +70,8 @@ auth:
     - ^/api/
 update:
   channel: stable               # R-UPD-01
-  range: ">=0.22.0"             # stay on the Jackett 0.22.x stable line
+  range: ">=0.22.0"             # floor only, no ceiling (0.24.x runs as of 2026-10-30; Jackett has no
+                                # major line to hold back from, so renovate.json5 mirrors no cap)
   cadence: "0 4 * * *"
   autoRollbackOnFailedHealth: true   # R-UPD-03
 observability:

catalog/pihole.yaml

@@ -5,7 +5,7 @@
 name: pihole
 kind: container
 source:
-  image: pihole/pihole
+  image: pihole/pihole:2026.10.0
 exposure: internal            # LAN-only admin UI (docs/13 row 6; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:

catalog/plex.yaml

@@ -12,7 +12,7 @@
 name: plex
 kind: container
 source:
-  image: plexinc/pms-docker:latest
+  image: plexinc/pms-docker:1.43.4.10903-e5521bd8c
 # plexinc/pms-docker env. Rendered VERBATIM onto the app container by render.py (schema env list).
 env:
   # PLEX_UID/PLEX_GID: the image boots as root (s6 PID1), chowns /config to this uid/gid, then drops

decisions/2026-10-30-pin-pihole-jackett-plex.md

@@ -0,0 +1,96 @@
+---
+date: 2026-10-30
+branch: feat/pin-pihole-jackett-plex
+title: Pin pihole, jackett and plex to the digests already running, and make Renovate able to follow them
+---
+
+## What changed
+
+- **`catalog/{pihole,jackett,plex}.yaml` `source.image`** pinned to the tag whose digest was
+  already running, so the *image* does not change:
+  - `pihole/pihole` (no tag) -> `pihole/pihole:2026.10.0`
+  - `lscr.io/linuxserver/jackett:latest` -> `lscr.io/linuxserver/jackett:v0.24.2865-ls7`
+  - `plexinc/pms-docker:latest` -> `plexinc/pms-docker:1.43.4.10903-e5521bd8c`
+- **`apps/wave-0/{pihole,jackett,plex}.yaml`** re-rendered (`render.py --all catalog -o apps
+  --wave-layout --no-image-automation`, in the toolbox). The diff is exactly the three app-container
+  `image:` lines; nothing else moved.
+- **`catalog/jackett.yaml`** `update.range` comment: "stay on the Jackett 0.22.x stable line" was
+  false (0.24 runs, and `>=0.22.0` has no ceiling). Range unchanged; comment corrected.
+- **`renovate.json5`**, four packageRules plus a comment block:
+  - jackett `versioning: regex:^v(major).(minor).(patch)-ls(build)$` (Renovate's linuxserver example).
+  - jackett **`overridePackageName: ghcr.io/linuxserver/jackett`**: not in the brief, and needed.
+    Renovate's docker datasource reads at most `dockerMaxPages` (20, a global-only option) tag
+    pages from any host except a short list that includes ghcr.io (1000 pages). lscr.io is not on
+    that list. Jackett has **32,093** tags, oldest first, ~1000 per page; the running tag is #31,980.
+    Before this rule the dry run extracted jackett at `v0.24.2865-ls7` and **proposed nothing**,
+    with no warning. The override changes only the lookup; the manifests keep `lscr.io/...`.
+    Every jackett rule matches on `depName`, because the override rewrites `packageName`.
+  - jackett `schedule: after 3am and before 6am on sunday`, i.e. weekly. Measured: 13 ls builds in
+    12 days (ls7 built 2026-10-17, ls20 built 2026-10-29). Each is a `patch` and auto-merges, so on
+    the daily window it would mean a PR, a gate run and a jackett restart (Recreate) every day.
+    Weekly picks the newest tag, so it skips nothing. `catalog/jackett.yaml update.cadence` still
+    says daily. That field fed the retired Flux image-automation, and the rule's description
+    says the rule is now the cadence that takes effect.
+  - plex `versioning: regex:^(major).(minor).(patch).(build)-[0-9a-f]+$`.
+  - pihole: **no rule**, as the operator decided. CalVer parses under default docker versioning.
+    Patches and minors auto-merge, and a year rollover is a major (dashboard approval). This is
+    written in a comment.
+  - Corrected the "WHAT RENOVATE STILL CANNOT FIX" comment. It listed these three as floating;
+    they publish immutable version tags.
+  - No `allowedVersions` mirror was added. None of the three catalog ranges has a ceiling
+    (`>=2024.0.0`, `>=0.22.0`, `>=1.40.0`), so #86's mirror has nothing to encode for them.
+- **Runbooks:** `image-policy-stalled.md` no longer says jackett and pms-docker are
+  `latest`-only. In `plex-quicksync.md`, "the floating-image caveat" became "the image-version
+  caveat", because the deployment no longer runs `:latest` with an implicit `Always`, and the symptom
+  is now "after a pms-docker bump". `plex-claim.md`'s two "running `:latest`" mentions are
+  now in the past tense.
+
+## What I verified
+
+- **Live digests** (read-only `kubectl --context k3s-prod`, pod `.status.containerStatuses[].imageID`):
+  pihole-0 and pihole-1 `sha256:d34134716fc3…`, jackett `sha256:4fcf3cc22dd3…`, plex
+  `sha256:e0ab27395614…`.
+- **Each chosen tag resolves to the same digest** (anonymous registry HEAD, OCI index + manifest-list
+  Accept): `pihole/pihole:2026.10.0` = `sha256:d34134716fc3…`; `ghcr.io/linuxserver/jackett:v0.24.2865-ls7`
+  = `sha256:4fcf3cc22dd3…`; `plexinc/pms-docker:1.43.4.10903-e5521bd8c` = `sha256:e0ab27395614…`.
+  (For contrast, `pms-docker:latest` is now `sha256:6d778d41…` and `jackett:latest` `sha256:ee6ae915…`.
+  The pods would have drifted on their next restart.)
+- `renovate-config-validator` (ghcr.io/renovatebot/renovate:44.167.0) reports "Config validated
+  successfully". The only warning is the existing `docker` block migration.
+- **Renovate dry run** (`RENOVATE_PLATFORM=local`, `--dry-run=full`, 44.167.0, anonymous, includePaths
+  limited to the six files). It extracted all three from both `catalog/` (custom regex) and
+  `apps/wave-0/` (kubernetes) under the same depName, so each lands in one branch:
+  - plex `1.43.4.10903-e5521bd8c` -> `1.43.5.11079-b3b13e506` (patch, `renovate/plexinc-pms-docker-1.x`)
+  - jackett `v0.24.2865-ls7` -> `v0.24.2961-ls20` (patch, `renovate/lscr.io-linuxserver-jackett-0.x`).
+    This needs the ghcr override; without it there was no proposal.
+  - pihole `2026.10.0` -> none. That is correct, because 2026.10.0 is the newest numbered tag on Docker Hub.
+- **Automerge, schedule and dashboard flags** come from Renovate's own `applyPackageRules`
+  (datasource-merge, then update-type-merge, the same order as `flatten.js`), run on the resolved
+  config. The local platform does not run the branch worker:
+  - plex patch: automerge **true**, daily window.
+  - jackett patch: automerge **true**, schedule **sunday** window.
+  - pihole (synthetic updates) `2026.10.1` patch and `2026.11.0` minor: automerge **true**. `2027.01.0`
+    major: automerge **false**, `dependencyDashboardApproval` **true**, +needs-review/breaking.
+- **Versioning filters** (Renovate's versioning module): the jackett regex rejects `latest`, bare
+  `0.24.2961`, `amd64-…` and `development-…`. The plex regex rejects `latest`, `public`, `beta`,
+  `plexpass` and `…-amd64`/`…-arm64`. Docker versioning ranks `2026.09.0 < 2026.10.0`.
+- Touched paths (`catalog/`, `apps/`, `renovate.json5`, `runbooks/`, `decisions/`) are not in
+  `infra/gitea/approval-policy.json` `sensitive_paths`.
+
+## What I did NOT verify
+
+- **That the merge is restart-free. It is not.** The pod-template `image` string changes, so Flux
+  rolls all three: plex and jackett are `Recreate`, so each has a brief outage, and pihole is a
+  2-replica StatefulSet that rolls one pod at a time. The digest is identical, so the image's
+  behaviour should not change, but the kubelet resolves the new tag name and may pull it. The
+  rollout itself was not observed.
+- The implicit `imagePullPolicy` moves from `Always` (`:latest` or no tag) to `IfNotPresent`. This is
+  intended, and was not observed live.
+- A real (non-dry) Renovate run against Gitea, and whether the Sunday window actually opens a jackett
+  PR. The schedule was only evaluated, not exercised.
+- Other `lscr.io/linuxserver/*` images (radarr, sonarr, qbittorrent) may hit the same 20-page
+  truncation once their tag counts grow. Not measured here, and not changed.
+- Plex not self-updating on a version tag was checked only against the running pod (read-only
+  `kubectl exec cat`). `/etc/cont-init.d/50-plex-update` does `exit 0` when `version` from
+  `/version.txt` equals the installed `plexmediaserver` package. Both are
+  `1.43.4.10903-e5521bd8c`. I did not boot a pod on a *new* tag to watch that short-circuit.

decisions/2026-10-31-pm-agent-digest-99.md

@@ -0,0 +1,41 @@
+---
+date: 2026-10-31
+branch: chore/pm-agent-digest-99
+title: Pin pm-agent dispatcher and worker to the agent-runner digest built from PR #99's merge
+---
+
+## What changed
+
+- `platform/pm-agent/deployment.yaml`: the dispatcher `image:` moves from
+  `agent-runner@sha256:df36d9b7…` (images run 383) to
+  `agent-runner@sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2`.
+  `config-generation` goes from "4" to "5" so the pod rolls.
+- `platform/pm-agent/configmap.yaml`: `PM_WORKER_IMAGE` gets the same digest. A comment line
+  records where it came from. The `# renovate-digest:` marker still sits directly above both
+  values.
+- No other values changed. `PM_DRY_RUN` stays "false". `PM_DECISION_MAX_TURNS` is not set, so
+  it keeps its in-code default of 1. No `*.sops.yaml` was touched.
+
+## What I verified
+
+- PR #99's merge commit is `466f63ec3c3df968a1dab25a8811dcd41961efde` (origin/main HEAD at
+  branch time).
+- The Gitea Actions tasks API shows the `images` workflow (images.yaml) on that commit as
+  **run 394** (task 643), with event `push` and status `success`.
+- I resolved the digest read-only from the registry by HEADing the manifests with OCI and
+  Docker Accept headers. Both `agent-runner:sha-466f63ec3c3d` and `agent-runner:main` returned
+  `200`, an `application/vnd.oci.image.manifest.v1+json` content type, and
+  `docker-content-digest: sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2`.
+  The two are identical.
+- Toolbox gate: the rc and skipped stages are in the PR description.
+
+## What I did NOT verify
+
+- That the dispatcher actually rolls to the new digest after merge, and that Flux applies it.
+  Flux's `lastAppliedRevision` and the pod's image were not checked. That is post-merge work.
+- That the new image surfaces the triage failure's real cause. It will show only on the next
+  failing triage, in the dispatcher log.
+- That admission (rule 7, `imagePullSecrets`) admits worker Jobs from this digest. The
+  template comes from `infra/agent/` at PR #99's merge, which still carries the
+  `imagePullSecrets` entry, but no Job was created to prove it.
+- I did not run the image's own selftest against this digest.

decisions/2026-10-31-pm-agent-go-live.md

@@ -0,0 +1,55 @@
+---
+date: 2026-10-31
+branch: chore/pm-agent-go-live
+title: Take pm-agent out of dry run (operator-approved); label scope and worker caps unchanged
+---
+
+## What changed
+
+- `platform/pm-agent/configmap.yaml`: `PM_DRY_RUN` `"true"` -> `"false"`. The operator approved
+  this after reading the dry-run triage of issue #97. `PM_REQUIRE_LABEL: agent`,
+  `PM_MAX_ACTIVE_WORKERS: 2`, `PM_MAX_NEW_WORKERS_PER_CYCLE: 1`, the worker image digest and every
+  other knob are unchanged. The two comments that said dry run was the only mode that worked
+  (because the run-340 digest lacked `imagePullSecrets`) now describe the run-383 digest that
+  PR #96 pinned.
+- `platform/pm-agent/deployment.yaml`: `config-generation` `"3"` -> `"4"`. Without the bump the
+  dispatcher keeps its startup config and stays in dry run.
+- `platform/pm-agent/README.md` and `docs/plans/pm-agent-issue-to-pr.md` (follow-up 3): the
+  "currently dry run" status text now says dry run is off. `runbooks/pm-agent.md` is unchanged.
+  Its "stop it: set `PM_DRY_RUN: "true"`" step and its admission-refusal troubleshooting row
+  are still accurate.
+
+## What I verified
+
+Read-only, `kubectl --context k3s-prod` through the toolbox, 2026-10-31 ~11:45 local, with Flux
+`pm-agent` and `platform-config` both at `main@1126740`:
+
+- ClusterPolicy `pm-agent-job-shape`: `Ready=True`, `validationFailureAction=Enforce`,
+  `background=false`. It has seven rules, including `worker-pull-secret-must-be-the-committed-one`.
+- RBAC: ServiceAccounts `pm-agent` and `pm-agent-worker`, Role `pm-agent` and RoleBinding
+  `pm-agent` (roleRef `pm-agent`, subject `ServiceAccount/pm-agent`) all exist. `auth can-i`
+  as `system:serviceaccount:pm-agent:pm-agent` returned `yes` for `create jobs.batch` and `no`
+  for `delete jobs.batch`.
+- Secrets (key names only): `pm-agent-registry` is `kubernetes.io/dockerconfigjson` with key
+  `.dockerconfigjson`. `pm-agent-credentials` is `Opaque` with keys `claude-code-oauth-token`
+  and `gitea-token`.
+- Egress: `default-deny` NetworkPolicy exists. CiliumNetworkPolicies `pm-agent-allow-egress`
+  and `pm-agent-worker-allow-egress` both report `Valid=True`.
+- Deployment `pm-agent` is 1/1 on `agent-runner@sha256:df36d9b7…` with
+  `config-generation: "3"` live. No Jobs exist in the namespace.
+- `platform/pm-agent/{configmap,deployment}.yaml`, README and docs do not match any
+  `sensitive_paths` glob in `infra/gitea/approval-policy.json`, so no review or override applies.
+
+## What I did NOT verify
+
+- **That a real worker Job is admitted and completes.** Nothing can show this until a labelled
+  issue gets a `start` decision after the merge. The L0 Kyverno suite shows the policy accepts
+  the committed template. Whether the API server admits the Job built from the df36d9b image's
+  baked template has not been tested live.
+- That the worker pod can pull its image with `pm-agent-registry`, reach Gitea and the model
+  through its egress policy, and push a branch and open a PR. None of this has run before.
+- The pod list also shows a stale `pm-agent-65db599649-v678g` (25d, `ContainerStatusUnknown`,
+  from an old ReplicaSet). It does not block anything, but I did not investigate it. Clearing
+  it is an operator action.
+- CI: see the PR for the gate result. Locally the best possible result is rc 3, with only
+  `gitea-settings` not run.

decisions/2026-10-31-pm-agent-runner-digest.md

@@ -0,0 +1,78 @@
+---
+date: 2026-10-31
+branch: chore/pm-agent-runner-digest
+title: Bump pm-agent to the run-383 agent-runner digest by hand; Renovate saw it but the schedule withheld it
+---
+
+## What changed
+
+`platform/pm-agent/deployment.yaml` (the dispatcher `image:`) and `platform/pm-agent/configmap.yaml`
+(`PM_WORKER_IMAGE`) move:
+
+- from `agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`, the
+  run-340 publish;
+- to `agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d`, the
+  images run 383 publish of ddd04d4.
+
+Both `# renovate-digest:` markers are unchanged, directly above their values.
+
+`pm-agent.homelab.keithjackson.dev/config-generation` goes `"2"` → `"3"`. That is the convention in
+`deployment.yaml`'s own comment: "bump it in the same commit as a change to configmap.yaml". The
+image change alone would roll the pod, but the convention is followed regardless.
+
+No `*.sops.yaml` file is touched. Branched from 06d4a9a, which already contains the operator's #95.
+
+### Why Renovate did not do this: the schedule, not a config defect
+
+- **Run 385** (dispatch, 2026-10-31 16:14 UTC, on ddd04d4) was a **real** run. It logged
+  `Deleting orphan branch (… renovate/pin-dependencies)`, not a `DRY-RUN:` line. It ran at **12:14
+  America/New_York**, outside `"schedule": ["after 3am and before 6am"]`, so it created and updated
+  no branch.
+- **The lookup DID see the new digest.** Dependency Dashboard issue #91 lists, under **"Awaiting
+  Schedule"**:
+  `chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to df36d9b`.
+- **Nothing else blocked it.** There are no open Renovate PRs and no `renovate/*` branch on Gitea
+  (111 branches listed across all pages). The previous agent-runner PR, #90 (to c4eba5b), was
+  auto-merged by renovate-bot at 08:22 on 10-31. No PR limit was hit, and digests are not
+  dashboard-approval gated.
+- **The registry, read-only, with Docker/OCI `Accept` headers:**
+  - `HEAD …/agent-runner/manifests/main` → `docker-content-digest: sha256:df36d9b7…771d`
+  - `HEAD …/agent-runner/manifests/sha-ddd04d460ef6` → the same digest.
+
+**Without this PR** the next scheduled run (08:15 UTC, 04:15 EDT) would open it, and it would
+auto-merge as a `digest` update. Ticking that "Awaiting Schedule" checkbox on #91 makes the next
+run of any kind create it out of schedule.
+
+### Found on the way: Renovate's labels have never been applied
+
+- The repo has exactly one label, `agent`. The org has none.
+- Every Renovate PR so far (#89, #90, #92, #93) was created with `"labels": []`.
+- On Gitea, Renovate drops a label that does not exist in the repo. So `needs-review`,
+  `blast-radius`, `range-widen`, `own-image`, `dependencies`, `automated-update`, `breaking` and
+  `ci-tooling` are never attached.
+- The **automerge flags still work.** Only the labels, the visible review signal, are missing.
+- **Operator fix:** create those labels on the repo. This was not changed here.
+
+## What I verified
+
+- **The new image carries the fix.** I fetched both images' layers read-only from the registry and
+  read `/etc/pm-agent/worker-job-template.yaml` from each (`infra/agent/Containerfile:116` copies
+  it there):
+  - df36d9b's has `imagePullSecrets:` / `- name: pm-agent-registry`;
+  - c4eba5b's has no `imagePullSecrets` at all.
+
+  This matches `infra/agent/worker-job-template.yaml:76-77` at ddd04d4.
+- **`main` = `sha-ddd04d460ef6` = df36d9b**, both measured.
+- **No other file references c4eba5b.** No admission policy pins it.
+- **No override is needed.** `infra/gitea/approval-policy.json` lists only `platform/kyverno/**`
+  under `platform/`, so `platform/pm-agent/**` is not sensitive.
+- I read the whole job logs of runs 362, 367 and 385.
+
+## What I did NOT verify
+
+- **That the rolled dispatcher starts on df36d9b and that a worker Job is admitted.** That needs
+  Flux to apply this and a real dispatch. Both are operator actions.
+- **That the `pm-agent-registry` Secret exists and decrypts in-cluster** (#95's sops file). I did not
+  read it.
+- **That Renovate will treat this hand bump as current.** It should: its next lookup finds
+  `main` == the pinned digest, so it proposes nothing and drops the dashboard entry. Not observed.

decisions/2026-10-31-pm-agent-triage-exit.md

@@ -0,0 +1,77 @@
+---
+date: 2026-10-31
+branch: fix/pm-agent-triage-exit
+title: pm-agent triage fails with "claude exited 1:" and no message; log the CLI's stdout envelope so the cause is visible
+---
+
+## What changed
+
+- `infra/agent/pm-dispatch.py`, `ask_pm()`: on a non-zero exit the error now comes from a new
+  `describe_cli_failure()`, not from `proc.stderr` alone. With `--output-format json` the CLI
+  reports its own failures as the result envelope on **stdout** and writes nothing to stderr,
+  which is why both failures today logged `claude exited 1: ` with an empty message. The new
+  line carries the envelope's `subtype`, `is_error`, `api_error_status`, `num_turns`,
+  `duration_ms`, `total_cost_usd`, `stop_reason`, its `result`/`error` text, and the tails of
+  stdout and stderr. It names silence explicitly when both streams are empty.
+- `redact()`: everything in that message, and in `extract_decision()`'s "no decision object"
+  error, passes through it. It covers `sk-ant-…`, `Bearer`/`token` values, and `user:pass@` in
+  both `http://` and `https://` URLs.
+- `PM_DECISION_MAX_TURNS`: a new knob. It defaults to 1, today's value, so behaviour is
+  unchanged, and it is bounded to 1–5. If the newly visible envelope says `error_max_turns`,
+  the remedy is a ConfigMap change rather than another image build. Documented in
+  `platform/pm-agent/README.md`.
+- Selftest: four new assertions. They cover an empty stderr with the cause on stdout, a
+  stderr-only failure, total silence, and redaction.
+
+## What I verified
+
+- **Timeline (Loki, `{namespace="pm-agent"}`, and `kubectl logs`):**
+  - **16:37:38 to 16:38:00**: pod `pm-agent-898fd8f5f-dp5hs`, image **c4eba5b3**,
+    config-generation 2, dry run. It triaged #97 in **22 s**, logged `PM says comment`, and
+    posted the 16:38:00Z comment. The thread then held only the issue body.
+  - **16:43:09 to 16:44:00**: pod `pm-agent-f7c9b7fcf-j8mzn`, image **df36d9b7**,
+    config-generation 3, **still dry run**. It logged `claude exited 1: ` with an empty
+    message after **51 s**.
+  - **16:55:42 to 16:56:36**: pod `pm-agent-7796db8bcb-p5dsm`, df36d9b7, config-generation
+    4, live. The same empty failure, after **54 s**.
+  - The only successful triage ever was the 16:38 one. Every earlier `898fd8f5f`/`5db55d558c`
+    log line is a startup crash (Gitea `GET /user` 401), not a triage.
+- **Going live is NOT the cause.** The first failure happened in dry run. `ask_pm()` has no
+  dry-run branch: the prompt, flags, tools (`--tools ""`) and cwd are identical in both
+  modes. `PM_DRY_RUN` gates only `create_job()`, which runs after the model has answered.
+- **Ruled out, with evidence:**
+  - **Timeout**: 51 to 54 s against a 900 s limit, and a timeout raises `TimeoutExpired`,
+    not rc 1.
+  - **Budget cap**: the PM call passes no `--max-budget-usd`.
+  - **Prompt size**: the thread is the issue body (293 chars) plus three comments
+    (1871 + 404 + 77 chars), and the persona is 8 KB.
+  - **Writable HOME**: HOME=`/workspace` (emptyDir), and `CLAUDE_CONFIG_DIR=/workspace/.claude`
+    was written at 16:56.
+  - **Egress**: a TLS 1.3 handshake to `api.anthropic.com:443` from the dispatcher pod
+    succeeds in 0.06 s.
+  - **Model name and CLI version**: the same `claude-opus-5` and CLI 2.1.285 succeeded at
+    16:38. Between c4eba5b and df36d9b, `infra/agent/` differs only by a template selftest
+    assertion and the toolbox base digest.
+- **A hint, not proof:** `/workspace/.claude/.claude.json` in the live pod has
+  `cachedExtraUsageDisabledReason: 'out_of_credits'`. No token was read or printed.
+- Offline selftest: `PM_SELFTEST=1` against the worktree's template and persona prints
+  `selftest OK`, including the 4 new assertions.
+- Toolbox gate: see the PR description for the rc and the stages that were skipped.
+
+## What I did NOT verify
+
+- **The root cause. UNVERIFIED.** The CLI's own message was discarded, `--no-session-persistence`
+  leaves no transcript, and I did not run `claude` in the pod. The candidates still open:
+  - **Usage or rate limit on the subscription seat.** The seat is shared with the operator's
+    interactive use. The `out_of_credits` extra-usage flag fits this. So does a ~50 s run,
+    which matches the CLI exhausting its 429/529 retries.
+  - **`error_max_turns`.** With `--max-turns 1` and `--json-schema`, an answer that fails the
+    schema cannot be retried. The failing thread now invites `implement` (a branch plus a
+    long brief), where the 16:38 one produced a short `comment`. ~50 s is about two
+    generations of the 22 s success.
+- **The 17:11 retry will NOT reveal the cause.** It runs the old image, which still discards
+  stdout. The cause becomes visible only after this PR merges, CI re-publishes agent-runner,
+  and a follow-up PR bumps the dispatcher digest (`deployment.yaml` and the `PM_WORKER_IMAGE`
+  in `configmap.yaml`, plus a `config-generation` bump).
+- That the CLI's error envelope carries exactly these key names. The parser takes whichever
+  keys are present and falls back to raw tails.

docs/plans/pm-agent-issue-to-pr.md

@@ -257,7 +257,9 @@ about runtime behaviour in this document is a design claim, not a measurement.
    bot's permission is `write`, not `admin`** — that measurement is the control that keeps
    the agent out of its own approval gate.
 3. **First run with `PM_DRY_RUN=true` and `PM_REQUIRE_LABEL=agent`**, on one issue written
-   for the purpose. Read what the PM says before letting it write anything.
+   for the purpose. Read what the PM says before letting it write anything. *Done:* dry-run
+   triage of issue #97 read; `PM_DRY_RUN=false` on 2026-10-31
+   (`decisions/2026-10-31-pm-agent-go-live.md`), `PM_REQUIRE_LABEL=agent` kept.
 4. **Tighten the admission image pin** from the `@sha256:*` wildcard to the literal digest.
    One line, reviewed.
 5. **Narrow `world:443` to `toFQDNs: api.anthropic.com`** — after verifying Cilium's DNS

infra/agent/pm-dispatch.py

@@ -214,6 +214,12 @@ class Config:
         # ConfigMap without rebuilding the image.
         self.push_on_red = env("PM_PUSH_ON_RED_GATE", "false")
         self.pm_timeout = env_int("PM_DECISION_TIMEOUT_SECONDS", 900)
+        # The PM holds no tools, so the only thing a second turn can be spent on is the CLI's
+        # structured-output step re-asking the model after an answer that failed the schema.
+        # 1 is the long-standing value; it is a knob so that, if the failure envelope now
+        # logged by describe_cli_failure() says `error_max_turns`, the remedy is a ConfigMap
+        # change rather than another image build.
+        self.pm_max_turns = env_int("PM_DECISION_MAX_TURNS", 1)
         # Opt-in gate. Empty (the default) means every open issue is in scope, which is the
         # behaviour the service was asked for. Setting it to a label name — `agent` is the
         # conventional one — narrows the agent to issues a human has explicitly handed it,
@@ -231,6 +237,8 @@ class Config:
 
         if self.max_active_workers < 1:
             die("PM_MAX_ACTIVE_WORKERS must be >= 1")
+        if not 1 <= self.pm_max_turns <= 5:
+            die("PM_DECISION_MAX_TURNS must be between 1 and 5")
         if "@sha256:" not in self.worker_image:
             # The admission policy requires a digest too, so a tag here would produce a Job
             # that is created and then DENIED — a failure visible only in the dispatcher's
@@ -437,7 +445,7 @@ def ask_pm(cfg: Config, thread: str) -> dict:
         "--system-prompt-file", cfg.persona_file,
         "--tools", "",
         "--permission-prompts", "none",
-        "--max-turns", "1",
+        "--max-turns", str(cfg.pm_max_turns),
         "--output-format", "json",
         "--json-schema", json.dumps(DECISION_SCHEMA),
         "--no-session-persistence",
@@ -445,10 +453,76 @@ def ask_pm(cfg: Config, thread: str) -> dict:
     proc = subprocess.run(cmd, input=thread, capture_output=True, text=True,
                           timeout=cfg.pm_timeout)
     if proc.returncode != 0:
-        raise RuntimeError(f"claude exited {proc.returncode}: {proc.stderr.strip()[:800]}")
+        raise RuntimeError(describe_cli_failure(proc.returncode, proc.stdout, proc.stderr))
     return extract_decision(proc.stdout)
 
 
+# Anything that could be a credential in the CLI's output. The CLI does not normally echo its
+# token, but an API error body or a proxy's HTML page is not something to bet a leak on. Both
+# schemes are matched (the http one is the lesson of a scheme-specific redaction that leaked
+# ~/.git-credentials), and `token ...` is Gitea's own header scheme.
+_SECRET_PATTERNS = [
+    (re.compile(r"sk-ant-[A-Za-z0-9_\-]+"), "sk-ant-[REDACTED]"),
+    (re.compile(r"(?i)\b(bearer|token)(\s+|=|:\s*)[A-Za-z0-9._~+/=\-]{8,}"), r"\1\2[REDACTED]"),
+    (re.compile(r"(?i)(https?://)[^/\s:@]+:[^/\s@]+@"), r"\1[REDACTED]@"),
+]
+
+
+def redact(text: str) -> str:
+    for pat, repl in _SECRET_PATTERNS:
+        text = pat.sub(repl, text)
+    return text
+
+
+def describe_cli_failure(rc: int, stdout: str, stderr: str, limit: int = 800) -> str:
+    """Say WHY `claude --print` failed, in the dispatcher log, redacted.
+
+    WHY THIS EXISTS: with `--output-format json` the CLI reports its own failures — an API
+    error, a usage limit, `error_max_turns`, a schema the model could not satisfy — as the
+    JSON result envelope on STDOUT (`is_error`, `subtype`, `result`), and writes nothing to
+    stderr. Logging only stderr therefore produced `claude exited 1: ` with an empty message
+    on 2026-10-31, twice, with the cause sitting unread in stdout. So: the envelope's
+    diagnostic fields first, then the raw tails of both streams.
+    """
+    parts = [f"claude exited {rc}"]
+    out = (stdout or "").strip()
+    env = None
+    if out:
+        try:
+            env = json.loads(out)
+        except ValueError:
+            # A stream of JSON lines, or a log line ahead of the envelope: the last line that
+            # parses as an object is the result.
+            for line in reversed(out.splitlines()):
+                try:
+                    cand = json.loads(line)
+                except ValueError:
+                    continue
+                if isinstance(cand, dict):
+                    env = cand
+                    break
+    if isinstance(env, dict):
+        fields = []
+        for key in ("subtype", "is_error", "api_error_status", "num_turns", "duration_ms",
+                    "duration_api_ms", "total_cost_usd", "stop_reason"):
+            if key in env:
+                fields.append(f"{key}={env[key]!r}")
+        if fields:
+            parts.append("envelope: " + " ".join(fields))
+        for key in ("result", "error", "errors", "message"):
+            if env.get(key):
+                val = env[key] if isinstance(env[key], str) else json.dumps(env[key])
+                parts.append(f"{key}: {val[:limit]}")
+    if out and not isinstance(env, dict):
+        parts.append(f"stdout tail: {out[-limit:]}")
+    err_text = (stderr or "").strip()
+    if err_text:
+        parts.append(f"stderr tail: {err_text[-limit:]}")
+    if len(parts) == 1:
+        parts.append("(no output on stdout or stderr)")
+    return redact(" | ".join(parts))
+
+
 def extract_decision(stdout: str) -> dict:
     """Pull the decision object out of whatever `claude --print` wrapped it in.
 
@@ -501,7 +575,7 @@ def extract_decision(stdout: str) -> dict:
         obj = as_obj(c)
         if obj and isinstance(obj.get("action"), str):
             return obj
-    raise RuntimeError(f"no decision object in claude output: {stdout[:400]!r}")
+    raise RuntimeError(redact(f"no decision object in claude output: {stdout[-800:]!r}"))
 
 
 # --------------------------------------------------------------------------- thread
@@ -988,8 +1062,25 @@ def selftest() -> int:
         except RuntimeError:
             check(True, f"bad output refused: {label}")
 
+    # -- a failed CLI run must say why, and must not leak ---------------------
+    # The 2026-10-31 failure: rc 1, empty stderr, the cause in the stdout envelope.
+    env_err = json.dumps({"type": "result", "subtype": "error_max_turns", "is_error": True,
+                          "num_turns": 2, "result": "API Error: 429 rate_limit_error"})
+    msg = describe_cli_failure(1, env_err, "")
+    check("error_max_turns" in msg and "429" in msg,
+          "CLI failure: stdout envelope surfaced when stderr is empty")
+    msg = describe_cli_failure(1, "", "boom on stderr")
+    check("boom on stderr" in msg, "CLI failure: stderr tail surfaced")
+    msg = describe_cli_failure(1, "", "")
+    check("no output" in msg, "CLI failure: silence is named, not an empty message")
+    msg = describe_cli_failure(1, "plain text sk-ant-oat01-abcdefABCDEF_123 here",
+                               "Authorization: Bearer abcdefghijklmnop "
+                               "http://u:[email protected]/x")
+    check("sk-ant-oat01" not in msg and "abcdefghijklmnop" not in msg and ":pw@" not in msg,
+          "CLI failure: token, bearer and URL credential redacted")
+
     # -- the committed Job template -----------------------------------------
-    tmpl = os.environ.get("PM_WORKER_JOB_TEMPLATE", "/etc/pm-agent/worker-job-template.yaml")
+    tmpl =os.environ.get("PM_WORKER_JOB_TEMPLATE", "/etc/pm-agent/worker-job-template.yaml")
     try:
         selftest_template(tmpl)
         check(True, f"worker Job template satisfies its contract ({tmpl})")

platform/pm-agent/README.md

@@ -46,11 +46,12 @@ waive the sensitive-path human review on its own pull request.
 Both are deliberate, and both fail **visibly** rather than silently:
 
 1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
-   `PM_WORKER_IMAGE` in `configmap.yaml` carry the digest of the **first** publish (images
-   run 340, `sha-523cd9118f95`), pinned once on 2026-10-30 so going live did not wait on
-   Renovate's first run; every later bump is Renovate's. That digest predates the worker
-   template's `imagePullSecrets` entry, so worker Jobs built from it are refused by
-   admission — which is why `PM_DRY_RUN` starts `true` (see `configmap.yaml`). `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
+   `PM_WORKER_IMAGE` in `configmap.yaml` first carried the run-340 publish
+   (`sha-523cd9118f95`), whose baked template predated the `imagePullSecrets` entry admission
+   requires, so the service ran with `PM_DRY_RUN=true`. PR #96 pinned the run-383 digest
+   (`df36d9b`, `sha-ddd04d460ef6`), whose template carries the entry, and dry run was turned
+   off on 2026-10-31 (`decisions/2026-10-31-pm-agent-go-live.md`); every later bump is
+   Renovate's. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
    (`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
    `infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
    `gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
@@ -249,6 +250,12 @@ five that matter:
 | `PM_MAX_ACTIVE_WORKERS` | `2` | concurrent workers. RAM is the binding constraint (2Gi limit each) and the subscription seat's rate limits are shared with every other use of that seat |
 | `PM_MAX_NEW_WORKERS_PER_CYCLE` | `1` | drains a backlog one issue at a time instead of in one stampede the first minute the credentials work |
 | `PM_PUSH_ON_RED_GATE` | `false` | a change the gate refused is not pushed; the issue gets the gate log instead |
+| `PM_DECISION_MAX_TURNS` | `1` (unset) | turns the triage call may take (1–5). The PM holds no tools, so a second turn is only ever the CLI re-asking after an answer that failed the decision schema. Raise it if a triage error logs `subtype='error_max_turns'` |
+
+**When a triage fails**, the `[error] issue #N: claude exited …` line carries the CLI's result
+envelope (`subtype`, `is_error`, `api_error_status`, `num_turns`, `result`) and the tails of
+stdout and stderr, redacted. Before 2026-10-31 it logged stderr only, which is empty under
+`--output-format json` — see `decisions/2026-10-31-pm-agent-triage-exit.md`.
 
 Changing a knob requires bumping the `config-generation` annotation in `deployment.yaml` in
 the same commit — the process reads its configuration once, at startup, so without the bump

platform/pm-agent/configmap.yaml

@@ -58,13 +58,16 @@ data:
   # request that also moves the Deployment's image (same marker, same branch). Nobody pushes
   # this image from a workstation.
   #
-  # READ BEFORE TURNING PM_DRY_RUN OFF: this digest PREDATES the worker template's
-  # `imagePullSecrets` entry (PR #61). Its baked template has none, and admission now REQUIRES
-  # it — so a worker Job built from this image is DENIED. The merge that adds the entry
-  # changes infra/agent/**, CI re-publishes agent-runner, and Renovate's bump to that digest
-  # is what makes worker Jobs admissible. Until then, dry run is the only mode that works.
+  # The run-340 digest predated the worker template's `imagePullSecrets` entry (PR #61), which
+  # admission REQUIRES, so its worker Jobs were DENIED. The digest below (df36d9b, run 383,
+  # agent-runner:sha-ddd04d460ef6, pinned in PR #96) bakes a template that carries
+  # `imagePullSecrets: pm-agent-registry`, which is what made turning PM_DRY_RUN off possible
+  # (decisions/2026-10-31-pm-agent-runner-digest.md). A digest older than that would be DENIED.
+  # The digest below (14a42ba, images run 394, agent-runner:sha-466f63ec3c3d = PR #99's merge)
+  # adds the CLI failure envelope to triage errors and the PM_DECISION_MAX_TURNS knob
+  # (decisions/2026-10-31-pm-agent-digest-99.md).
   # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
-  PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42"
+  PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2"
 
   # --- the model -------------------------------------------------------------
   # The PM only reads a thread and answers with one JSON object, so it holds no tools and
@@ -120,10 +123,11 @@ data:
   # no worker Job and so changes no code. The honest way to watch this service for a day
   # before letting it write anything.
   #
-  # "true" FOR THE FIRST RUN, and not only for caution: the pinned digest above cannot yet
-  # create an admissible worker Job (see the PM_WORKER_IMAGE comment). Flip to "false" in the
-  # same pull request as — or after — Renovate's first agent-runner digest bump.
-  PM_DRY_RUN: "true"
+  # "true" for the first run (2026-10-30 .. 2026-10-31); dry-run triage of issue #97 was read
+  # and the operator approved going live. "false" since decisions/2026-10-31-pm-agent-go-live.md.
+  # Setting it back to "true" (plus a config-generation bump) is the gentlest stop — see
+  # runbooks/pm-agent.md.
+  PM_DRY_RUN: "false"
 
   # If the gate refuses a worker's change, the branch is NOT pushed by default and the issue
   # gets the gate log instead. Set to "true" to push anyway and open a red pull request —

platform/pm-agent/deployment.yaml

@@ -48,7 +48,7 @@ spec:
         # It is a hand-maintained marker rather than a computed hash because Flux's
         # Kustomization here does not run a configmap generator; bump it in the same commit
         # as a change to configmap.yaml.
-        pm-agent.homelab.keithjackson.dev/config-generation: "2"
+        pm-agent.homelab.keithjackson.dev/config-generation: "5"
     spec:
       serviceAccountName: pm-agent
       # The kubelet's credential for pulling this image. Not the agent's Gitea credential —
@@ -78,7 +78,7 @@ spec:
           # the registry's manifest endpoint on 2026-10-30 and pinned by hand once so the
           # dispatcher does not wait on Renovate's first run; every later bump is Renovate's.
           # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
-          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42
+          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2
           imagePullPolicy: IfNotPresent
           command:
             - /usr/local/bin/pm-dispatch

platform/pm-agent/pm-agent-credentials.sops.yaml

@@ -11,8 +11,8 @@ metadata:
         app.kubernetes.io/part-of: pm-agent
 type: Opaque
 stringData:
-    gitea-token: ENC[AES256_GCM,data:Vs6TkiNocU+qzvlv8mndXhtiqp1RyY00KhIBDuL8w0WoX72B8DxfPK8eWISy9gtx9TBlYpu2FHP7,iv:2z8+gzWuZDFACb06rnYg071h4wO7YlcTFNZDQzDxRNs=,tag:cUhj+uQUwu44HpPtFTVjVA==,type:str]
-    claude-code-oauth-token: ENC[AES256_GCM,data:2PjA5MOx8Enm4wxqYdJ3QY8juqPkXh4adQv3IRtw1Nb2Dw6SkOFc,iv:ZSlefJZqCN7g17D+27VkkVYu8F1umM7JOG+f/crZtsE=,tag:QyNYIbHr+ANS0Rx4USa0Iw==,type:str]
+    gitea-token: ENC[AES256_GCM,data:nkQT4m0DYykNcEUPV+1A4APuraekHN+8tY74HnRIbwU8SwYbRRqfKQ==,iv:d6Lj4Q/ys381msrrRtIIYbV9Ey3Nhua4zmX/7aCMlSk=,tag:xKKmc1i9TRuUVc/C2P1zHg==,type:str]
+    claude-code-oauth-token: ENC[AES256_GCM,data:Z5elUXr2wiu1S3FvSsHPNfhyrh+dGAdBf4HEmvg+Ky3QuCDpLv/XcMG5wWNywYVR4jlx/z2rm2FjjRYo+8rG0+I3SzriKG2MD7Zri4bDUf+DFksa6pI2oGDmgTjp8Zjz8oGapaMurRnuYcMm,iv:tXADrr2PD1zTpmTuaCA224wzbsq2RKzx0/ylU2k6suE=,tag:DMh31sAj0E/8ndcP1M6zAw==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -28,8 +28,8 @@ sops:
             V3d1UGVsTlljS3lHbFRkUWE4aVdRSDAK3vlChFXHg+jOJyjzUBZ4F4NqIn8rr5xl
             WAH9bgY14yvX8qm3P4iLeFecNssqUvs2hvDW82vhMfUQlqYpTTXyjQ==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-10-06T03:09:21Z"
-    mac: ENC[AES256_GCM,data:lKj3CtZ3fCvOwuJSeNb5x8dj+5O0Uhfyoir+ZfIGD52xgeTWtmQ6CmOVbtrYjp1h1dgZnWopbibfMIKE9AmjSAHA7UJOen++zcy4qrA8Yh8xGzfhb1tOJplxZP+Lp9319VYQHqXqOScs7at8Osnw2RYuibW/j6eIY0B17JzIXvo=,iv:14aKtsMd9Q6QqnWVszoi+W2+z4QPfXUd7oZfK46+gRM=,tag:NwBg43/vNeFlgHmNLB7rYw==,type:str]
+    lastmodified: "2026-10-31T16:07:12Z"
+    mac: ENC[AES256_GCM,data:rK1GR6FoIsh2Y6VSQSkLhGwf1U2kBrV3VYZT/5X+TxbkURza7DOTUyMJsY0oLKApvkTvOxLd3/WfzBNW/O6Tho1S4otvi4UC7fY6DjC46LmrfSlAxoRp7ov/oIRfwLBUanpNzfT94fu7sjoNM0lRlFp9iSpBU44upcIWwclK5WY=,iv:EujucxcPU+zfN9KmOhbqfVT5anv4g/jRg41hu5BgGx8=,tag:fViHjTT6w20b2AsXZodq0g==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

platform/pm-agent/pm-agent-registry.sops.yaml

@@ -16,7 +16,7 @@ metadata:
         app.kubernetes.io/part-of: pm-agent
 type: kubernetes.io/dockerconfigjson
 stringData:
-    .dockerconfigjson: ENC[AES256_GCM,data:4OJS0P/oTcaXYuqlrc/936l+5/ZSqA9dlNMPx8HkDX9zmQ4q1qIumYmlXYMbcVm2c2byeYt0II0mGVLVVNOBjb5jgDB4vjyhSLN4HhoYBxgB+//hNcYe+uobmrdJTBJLEbd+lXamf1vfsEUTmLL7I4VQ2l/mjJbvqu8p2YNw25Ska9iO9mkxCzjOTfRdzMD0Pn1Z09fjo8GpYVo=,iv:AEysZLEqrybR+3l9kReHEeTbVFApzKPb8kAkibEYerE=,tag:TgE7T1zasye1bsTZ2BTFrA==,type:str]
+    .dockerconfigjson: ENC[AES256_GCM,data:NuKobTM0S0TK8U961uNTIIYiyAI2L3QCopZXvKxXHLC2tjpdyV+REKzYPmv+orCMiRsY8GNtfaHb8Hret4sQWHySAAKN0sEtL+OkzLyRH+u7irAGFYUBf8q+vIb+hnhdwejHfL1UEaHovJlZuIjqOdSItqlxqXURGtvUtBlkD7cnzDajzLyW+QTp5Gs3Fc8ps4XxRkv/589bKpn0DIEQiXUujydBZzKVnO5unxMea+4IBwcC6yaeQ+fXCoILrKD3PP2Igz0=,iv:vrQqH4dIUwDLSZGe1JKnsajtvUb7lJe3st/9K+4/w0M=,tag:P9VLwOqnTfM1uQnbousYqQ==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -32,8 +32,8 @@ sops:
             cG1Fb0V3VncrTm9SQ091a0J0Z3RURWMKpdzNVub0zH5S8nnCSCt2H6NcCqDbV24L
             Mf8K2eIfu/ixn2NfwoVVj2QO+n+3/P6PLphdoiKMrL+ms0+JqWFpkQ==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-10-06T06:01:43Z"
-    mac: ENC[AES256_GCM,data:zwmmMbM9Me7wE8EOvrKD6hEMLVHNCxI85FHvMjtRcHHm+mtdACjofDNfpynFVc1xFMMchcoOfipxEfAAJcwbH0FtfLeZq6M+XNHAf81qIk3NJxrS5Kgi/9FXZJAAwhqo/eced89Uozq8ntwt+mgxOuyjMds5z4h/VfN7/RmsWHg=,iv:VlvAEqozM9VV50vVx7r5aq3BdaaLLubhHJyHt8SSgS0=,tag:4ji7UqfRfm9P+qJrezFpmw==,type:str]
+    lastmodified: "2026-10-31T16:11:11Z"
+    mac: ENC[AES256_GCM,data:0GIYwtVcNngq2Zpz3haCx6L+j2SrgHFL5pRYeRM4Ac1T9kjKSwHik6Q4+T8FU+EJa9GP1yXrpdsjggsjzG9dI2zRyAF1C8x1znD5ls6OwT5jLak8QrHNcTavRlm6N6y1yOsQCTFMc+zeCywTnFJldTqPKH+6K/bNMNBuD14BNv8=,iv:oqGygkOmFgRw0TYbWO75/sJ/doXtorAWAg89LtaqIsg=,tag:vBO/DWV93ekZB86G+Hca2A==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

renovate.json5

@@ -325,6 +325,37 @@
     { "matchPackageNames": ["lscr.io/linuxserver/radarr"], "allowedVersions": "<6.0.0" },
     { "matchPackageNames": ["lscr.io/linuxserver/sonarr"], "allowedVersions": "<5.0.0" },
     { "matchPackageNames": ["semaphoreui/semaphore"], "allowedVersions": "<3.0.0" },
+    // ADDED 2026-10-30 — pihole, jackett and plex, PINNED to the digests that were already running
+    // (catalog/{pihole,jackett,plex}.yaml; each tag was resolved against the registry and matched the
+    // pods' imageID, so the pin changed nothing at runtime). None of the three catalog `update.range`
+    // values has a ceiling (pihole >=2024.0.0, jackett >=0.22.0, plex >=1.40.0), so the
+    // allowedVersions mirror above needs no entry for them.
+    //
+    // pihole/pihole needs NO rule: its CalVer tags (YYYY.MM.N, e.g. 2026.10.0) parse under the
+    // default `docker` versioning. Patches and minors (2026.10.0 -> 2026.10.1 / 2026.11.0) therefore
+    // AUTO-MERGE on a green gate under rule 1 — the operator's explicit choice, not an oversight; do
+    // not add a hold here. A YEAR ROLLOVER (2026.x -> 2027.x) is a MAJOR, so it waits for a tick on
+    // the Dependency Dashboard (rule below) and is never auto-merged.
+    {
+      "description": "ADDED 2026-10-30. LinuxServer tags: `vX.Y.Z-lsN` is the IMMUTABLE build tag (bare `X.Y.Z` and `latest` are re-pushed for every ls rebuild). Docker versioning would treat `-ls7` as a compatibility suffix and only ever compare tags with the SAME suffix, so it would never move. This is Renovate's own regex-versioning example for linuxserver images: the ls build number becomes the `build` component, so an ls rebuild of the same Jackett is an update too. The `^v` and `$` anchors exclude `latest`, `development`, the bare mutable `X.Y.Z` tags and the arch-prefixed (`amd64-v...`) tags.",
+      "matchDepNames": ["lscr.io/linuxserver/jackett"],
+      "versioning": "regex:^v(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)-ls(?<build>\\d+)$"
+    },
+    {
+      "description": "ADDED 2026-10-30. LOOK JACKETT UP ON ghcr.io, NOT lscr.io. lscr.io is LinuxServer's redirector in front of ghcr.io/linuxserver (same manifests: v0.24.2865-ls7 is sha256:4fcf3cc2... on both). Renovate's docker datasource fetches up to 1000 pages of tags from ghcr.io but only `dockerMaxPages` (default 20, a GLOBAL-only option) from any other host (lib/modules/datasource/docker/index.js getTags). Jackett has 32,093 tags at ~1000 per page (measured 2026-10-30), oldest first; the running tag is #31,980, so a lookup via lscr.io never sees it or anything newer and silently proposes NOTHING (measured in a dry run before this rule). overridePackageName changes only the lookup; the file keeps `lscr.io/...`, so the deployed image string does not change. Matched by depName because packageName is what this rule rewrites.",
+      "matchDepNames": ["lscr.io/linuxserver/jackett"],
+      "overridePackageName": "ghcr.io/linuxserver/jackett"
+    },
+    {
+      "description": "ADDED 2026-10-30. Jackett publishes ~daily (13 ls builds in 12 days, measured 2026-10-30) and every one is a `patch` under the regex above, i.e. auto-mergeable. Left on the global daily window that is a PR, a gate run and a jackett pod restart EVERY DAY for an indexer proxy whose individual builds nobody needs promptly. WEEKLY instead: Renovate proposes only the newest tag when the window opens, so nothing is skipped, it is just batched. The window is still inside the global 03:00-06:00 local one that .gitea/workflows/renovate.yaml's 08:15 UTC cron lands in. NOTE: catalog/jackett.yaml `update.cadence` still says daily (`0 4 * * *`); that field drove Flux image-automation, which is retired, and this rule is the effective cadence now.",
+      "matchDepNames": ["lscr.io/linuxserver/jackett"],
+      "schedule": ["after 3am and before 6am on sunday"]
+    },
+    {
+      "description": "ADDED 2026-10-30. Plex tags are `<major>.<minor>.<patch>.<build>-<commit hex>` (e.g. 1.43.4.10903-e5521bd8c). Docker versioning reads `-e5521bd8c` as a compatibility suffix that differs on EVERY release, so it would never find a successor. The trailing `$` excludes the per-arch variants and the floating `latest`/`public`/`beta`/`plexpass` tags. A version tag does NOT self-update inside the container: cont-init 50-plex-update exits early when /version.txt matches the installed package, so what is pinned is what runs.",
+      "matchPackageNames": ["plexinc/pms-docker"],
+      "versioning": "regex:^(?<major>\\d+)\\.(?<minor>\\d+)\\.(?<patch>\\d+)\\.(?<build>\\d+)-[0-9a-f]+$"
+    },
     {
       "description": "ADDED 2026-10-30. A CNPG Postgres MAJOR is a migration (new cluster + import), not an image bump; the dry run proposed 16.4 -> 18.6. Minors stay (and are blast-radius, above).",
       "matchPackageNames": ["ghcr.io/cloudnative-pg/postgresql"],
@@ -379,6 +410,10 @@
   // ARE maintained (the first dry run proposed 35-apache and 29-dind; the suffix is kept).
   // CORRECTED 2026-10-30: quay.io/keycloak/keycloak was never a running reference — the operator
   // picks the server image — and the Keycloak version is now tracked via the operator URLs.
+  // CORRECTED 2026-10-30 (feat/pin-pihole-jackett-plex): pms-docker, jackett and pihole are now
+  // PINNED (1.43.4.10903-e5521bd8c, v0.24.2865-ls7, 2026.10.0 — the digests that were running) and
+  // maintained via the versioning rules above. None of the three publishes "only latest": all three
+  // publish immutable version tags, they were simply never referenced.
   // Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on
   // kube-prometheus-stack): THIS FILE USED TO SAY they cap Renovate at patches. They do not —
   // Renovate WIDENS the range (runs 353/354). The `range-widen` rule above keeps that a reviewed

runbooks/image-policy-stalled.md

@@ -112,10 +112,12 @@ A private repository with no `secretRef`, or an expired token. The message names
 ### 4. The policy's range matches nothing
 `ImagePolicy` semver ranges come from each ServiceDefinition's `update.range` (for example
 `">=2024.0.0"` for pihole, `">=24.0.0"` for keycloak). A range that no published tag satisfies — or a
-repository that publishes only `latest` (this platform has `lscr.io/linuxserver/jackett:latest` and
-`plexinc/pms-docker:latest`) — leaves the policy with nothing to select. Remediation: fix the range in
-`catalog/<service>.yaml` and re-render; a `latest`-only image cannot participate in automation at all
-and should not carry a policy.
+repository that publishes only `latest` — leaves the policy with nothing to select. Remediation: fix
+the range in `catalog/<service>.yaml` and re-render; a `latest`-only image cannot participate in
+automation at all and should not carry a policy. (CORRECTED 2026-10-30: this page used to name
+`lscr.io/linuxserver/jackett` and `plexinc/pms-docker` as `latest`-only. Neither is — both publish
+immutable version tags (`v0.24.2865-ls7`, `1.43.4.10903-e5521bd8c`) — they were merely *deployed* as
+`:latest`. Both are now pinned to those tags and maintained by Renovate, not by an ImagePolicy.)
 
 ### 5. Egress is blocked for that namespace
 Default-deny everywhere (R-NET-01) means outbound registry access is an explicit allow. Cross-read

runbooks/plex-claim.md

@@ -21,7 +21,8 @@ which is why this page exists: the loop is scripted and the reconcile is **force
 waited for. Measured end to end on 2026-09-29: **~2m45s**.
 
 **An expired or wrong token does not break anything.** Verified against the image's own first-run
-script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
+script (and the copy inside the image then running as `plexinc/pms-docker:latest`; pinned to a version
+tag since 2026-10-30): a failed exchange gets a 403 with an empty
 body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
 no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
 serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
@@ -186,4 +187,5 @@ When the claim does not take, in order of how often it is the answer here:
 Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
 Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
 `root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
-the running `plexinc/pms-docker:latest` image.
+the image that was running at the time as `plexinc/pms-docker:latest` (the deployment has been pinned
+to a version tag, `catalog/plex.yaml` `source.image`, since 2026-10-30).

runbooks/plex-quicksync.md

@@ -371,8 +371,8 @@ verify".)
    2026-10-01, so this means a node changed — a rebuild, a BIOS reset, or a driver regression.
    **Fix the node.**
 
-4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
-   mechanism this feature leans on with no commit in this repo.
+4. **The image changed under you.** See "The image-version caveat". A pms-docker version bump (a
+   Renovate PR, possibly auto-merged) can remove the mechanism this feature leans on.
 
 5. **The session was never transcoding.** Direct Play / Direct Stream never reaches the transcoder,
    so no `hardware transcoding:` line is written for it at all. An *absent* line is this, not a
@@ -446,17 +446,21 @@ revert you):
 Re-adding the field later is the same edit in reverse. Nothing about the Plex-side toggle or the
 `/config` PVC is affected either way — do **not** delete the PVC as part of this.
 
-## The floating-image caveat
+## The image-version caveat
 
 Everything this page states about the image's behaviour — the `cont-init.d` hook that adds the app
 user to the device's group, the absence of `PLEX_PREFERENCE*`, the six `setPref` keys, the
 `HardwareAcceleratedCodecs` string — was read from digest **`sha256:e0ab27395614…`**
 (`plexmediaserver 1.43.4.10903`).
 
-The deployment runs **`plexinc/pms-docker:latest` with `imagePullPolicy: Always`**. A silent pull can
-therefore change the behaviour this feature depends on **with no commit in this repo**, while the
-manifest still looks identical. Symptom to watch for: hardware transcoding that worked and quietly
-stopped, with no git change to point at.
+**Since 2026-10-30 that digest is pinned:** `catalog/plex.yaml` names
+`plexinc/pms-docker:1.43.4.10903-e5521bd8c`, which resolved to exactly `sha256:e0ab27395614…` on
+Docker Hub. Before that the deployment ran the floating `:latest` (implicitly `imagePullPolicy:
+Always`), and a silent pull could change this behaviour with no commit in the repo. That is no longer
+the failure mode: the image now changes only through a **Renovate PR** bumping that tag (patch/minor
+ones auto-merge on a green gate, and the gate does not exercise QuickSync). So the symptom to watch
+for is hardware transcoding that quietly stopped **after a `chore(deps)` pms-docker bump** — check
+`git log -- catalog/plex.yaml` first.
 
 ```
 infra/devcontainer/dev.sh run bash -c \