Compare commits

...
Author SHA1 Message Date
renovate-bot fe1c50009f chore(deps): Update Helm release kube-prometheus-stack to v92
gate / gate (pull_request) Successful in 2m56s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:19:12 +00:00
jakman85 5b484f6b78 Merge pull request 'chore(deps): Update Helm release cilium to v1.20.4' (#104) from renovate/cilium-1.x into main
gate / gate (push) Successful in 2m48s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 4m59s
Reviewed-on: #104
2026-11-07 03:07:19 +00:00
renovate-bot 640edb758d Merge pull request 'chore(deps): Update plexinc/pms-docker Docker tag to v1.43.6.11085-d1e52e169' (#111) from renovate/plexinc-pms-docker-1.x into main
gate / gate (push) Successful in 2m56s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 1m58s
2026-11-05 08:27:21 +00:00
renovate-bot e6cd064f5c Merge pull request 'chore(deps): Update lscr.io/linuxserver/jackett Docker tag to v0.24.3009-ls26' (#110) from renovate/lscr.io-linuxserver-jackett-0.x into main
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-11-05 08:24:23 +00:00
renovate-bot dd9d8da86a chore(deps): Update plexinc/pms-docker Docker tag to v1.43.6.11085-d1e52e169
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Has been skipped
2026-11-05 08:17:16 +00:00
renovate-bot c3ce4db767 chore(deps): Update lscr.io/linuxserver/jackett Docker tag to v0.24.3009-ls26
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Has been skipped
2026-11-05 08:16:08 +00:00
renovate-bot 38076f1cff chore(deps): Update Helm release cilium to v1.20.4
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-05 08:15:53 +00:00
renovate-bot 552330398b Merge pull request 'chore(deps): Update pihole/pihole Docker tag to v2026.10.1' (#109) from renovate/pihole-pihole-2026.x into main
gate / gate (push) Successful in 2m56s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 3m19s
2026-11-04 08:22:16 +00:00
renovate-bot e49aebc803 Merge pull request 'chore(deps): Update docker.io/moby/buildkit Docker tag to v0.34.2' (#108) from renovate/docker.io-moby-buildkit-0.x into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
2026-11-04 08:19:20 +00:00
renovate-bot 9c0292001e chore(deps): Update pihole/pihole Docker tag to v2026.10.1
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-04 08:16:12 +00:00
renovate-bot db9416d3b5 chore(deps): Update docker.io/moby/buildkit Docker tag to v0.34.2
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Has been skipped
2026-11-04 08:15:52 +00:00
renovate-bot cb4a1bf2de Merge pull request 'chore(deps): Update lscr.io/linuxserver/sonarr Docker tag to v4.0.20' (#107) from renovate/lscr.io-linuxserver-sonarr-4.x into main
gate / gate (push) Successful in 3m3s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 1m12s
2026-11-03 08:22:33 +00:00
renovate-bot 028f5229f0 Merge pull request 'chore(deps): Update kubevirt-cdi' (#106) from renovate/kubevirt-cdi into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
2026-11-03 08:19:36 +00:00
renovate-bot 44e2cd3c64 chore(deps): Update lscr.io/linuxserver/sonarr Docker tag to v4.0.20
gate / gate (pull_request) Successful in 2m56s
gate / conformance (pull_request) Has been skipped
2026-11-03 08:16:12 +00:00
renovate-bot b30b7e6f67 chore(deps): Update kubevirt-cdi
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Has been skipped
2026-11-03 08:16:06 +00:00
renovate-bot 5a3a5f8df2 Merge pull request 'chore(deps): Update intel/intel-gpu-plugin Docker tag to v0.37.1' (#105) from renovate/intel-intel-gpu-plugin-0.x into main
gate / gate (push) Successful in 2m57s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 1m31s
2026-11-02 08:22:07 +00:00
renovate-bot f698bc82a0 chore(deps): Update intel/intel-gpu-plugin Docker tag to v0.37.1
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-02 08:16:06 +00:00
renovate-bot 6bc10fb60d Merge pull request 'chore(deps): Update dependency intel/intel-device-plugins-for-kubernetes to v0.37.1' (#102) from renovate/intel-intel-device-plugins-for-kubernetes-0.x into main
gate / gate (push) Successful in 2m58s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 1m4s
2026-11-01 08:18:56 +00:00
renovate-bot d8c330fb8c chore(deps): Update dependency intel/intel-device-plugins-for-kubernetes to v0.37.1
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-01 08:15:52 +00:00
jakman85 a1d89f4a5f Merge pull request 'chore(pm-agent): pin agent-runner to the run-394 digest (466f63e, PR #99), bump config-generation' (#100) from chore/pm-agent-digest-99 into main
gate / gate (push) Successful in 2m56s
gate / conformance (push) Has been skipped
renovate / renovate (push) Successful in 51s
Reviewed-on: #100
Reviewed-by: Keith Jackson <[email protected]>
2026-10-31 17:23:18 +00:00
jakman85andClaude Opus 5.5 19097e4d4f chore(pm-agent): pin agent-runner to the run-394 digest (466f63e, PR #99), bump config-generation
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
Dispatcher image and PM_WORKER_IMAGE both move to
sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2, the digest the
registry returns for agent-runner:sha-466f63ec3c3d and :main. config-generation 4 -> 5.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 12:17:00 -05:00
jakman85 466f63ec3c Merge pull request 'fix(pm-agent): log the CLI's stdout envelope when triage fails; add PM_DECISION_MAX_TURNS' (#99) from fix/pm-agent-triage-exit into main
gate / gate (push) Successful in 2m56s
images / images (push) Successful in 16s
gate / conformance (push) Has been skipped
Reviewed-on: #99
2026-10-31 17:12:40 +00:00
jakman85andClaude Opus 5.5 664ee82b9a fix(pm-agent): log the CLI's stdout envelope when triage fails; add PM_DECISION_MAX_TURNS
gate / gate (pull_request) Successful in 2m56s
images / images (pull_request) Successful in 13s
gate / conformance (pull_request) Has been skipped
`claude --print --output-format json` reports its own failures (API error, usage limit,
error_max_turns) as the result envelope on stdout and leaves stderr empty, so ask_pm()'s
stderr-only message logged "claude exited 1: " with nothing after it, twice on 2026-10-31.
describe_cli_failure() now logs the envelope's diagnostic fields plus stdout/stderr tails,
redacted (sk-ant-, Bearer/token, URL userinfo). PM_DECISION_MAX_TURNS (default 1, unchanged)
makes the max-turns remedy a ConfigMap change if the envelope turns out to say error_max_turns.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 12:04:09 -05:00
jakman85 95e08da2f0 Merge pull request 'chore(pm-agent): take the dispatcher out of dry run (operator-approved), config-generation 4' (#98) from chore/pm-agent-go-live into main
gate / gate (push) Successful in 2m58s
gate / conformance (push) Has been skipped
Reviewed-on: #98
Reviewed-by: Keith Jackson <[email protected]>
2026-10-31 16:52:16 +00:00
jakman85andClaude Opus 5.5 0d3e51acc4 chore(pm-agent): turn PM_DRY_RUN off (operator-approved), bump config-generation to 4
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
The run-383 worker image (df36d9b, pinned in #96) bakes imagePullSecrets: pm-agent-registry,
which admission requires, and the dispatcher's dry-run triage of #97 was read and approved.
PM_REQUIRE_LABEL=agent and the worker caps are unchanged. The dry-run status text in the
configmap comments, the README and the plan's follow-up 3 is updated to match.

Pre-flight (read-only, k3s-prod): pm-agent-job-shape Ready+Enforce; SA/Role/RoleBinding
present and can-i create jobs = yes; pm-agent-registry and pm-agent-credentials exist; both
egress CNPs Valid.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 11:45:52 -05:00
jakman85 11267409d7 Merge pull request 'chore(pm-agent): pin agent-runner to the run-383 digest (ddd04d4), bump config-generation' (#96) from chore/pm-agent-runner-digest into main
gate / gate (push) Successful in 2m58s
gate / conformance (push) Has been skipped
Reviewed-on: #96
Reviewed-by: Keith Jackson <[email protected]>
2026-10-31 16:34:49 +00:00
jakman85andClaude Opus 5.5 6a192ede2a chore(pm-agent): pin agent-runner to the run-383 digest (ddd04d4), bump config-generation
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
c4eba5b (run 340) bakes a worker template without imagePullSecrets, so the new admission rule
refuses worker Jobs. df36d9b is agent-runner:main == :sha-ddd04d460ef6 (measured on the registry),
and its /etc/pm-agent/worker-job-template.yaml carries imagePullSecrets: pm-agent-registry (read
from the published layer).

Renovate found df36d9b but holds it under "Awaiting Schedule" on dashboard #91: the operator's
dispatch (run 385) was a real run at 12:14 New York, outside the 03:00-06:00 window. The
renovate-digest markers are unchanged, so later bumps remain Renovate's.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 11:22:52 -05:00
jakman85 06d4a9a1fc Merge pull request 'Added the credentials' (#95) from feature/pm-agent-credentials into main
gate / gate (push) Successful in 2m56s
gate / conformance (push) Has been skipped
Reviewed-on: #95
2026-10-31 16:20:12 +00:00
23 changed files with 392 additions and 37 deletions

No files matched your search

+1 -1
View File
@@ -78,7 +78,7 @@ spec:
spec:
containers:
- name: jackett
image: lscr.io/linuxserver/jackett:v0.24.2865-ls7
image: lscr.io/linuxserver/jackett:v0.24.3009-ls26
ports:
- containerPort: 9117
volumeMounts:
+1 -1
View File
@@ -62,7 +62,7 @@ spec:
spec:
containers:
- name: pihole
image: pihole/pihole:2026.10.0
image: pihole/pihole:2026.10.1
ports:
- containerPort: 80
env:
+1 -1
View File
@@ -120,7 +120,7 @@ spec:
spec:
containers:
- name: plex
image: plexinc/pms-docker:1.43.4.10903-e5521bd8c
image: plexinc/pms-docker:1.43.6.11085-d1e52e169
ports:
- containerPort: 32400
env:
+1 -1
View File
@@ -122,7 +122,7 @@ spec:
spec:
containers:
- name: sonarr
image: lscr.io/linuxserver/sonarr:4.0.15
image: lscr.io/linuxserver/sonarr:4.0.20
ports:
- containerPort: 8989
env:
+1 -1
View File
@@ -16,7 +16,7 @@
name: jackett
kind: container
source:
image: lscr.io/linuxserver/jackett:v0.24.2865-ls7
image: lscr.io/linuxserver/jackett:v0.24.3009-ls26
exposure: internal # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
egress: vpn # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
storage:
+1 -1
View File
@@ -5,7 +5,7 @@
name: pihole
kind: container
source:
image: pihole/pihole:2026.10.0
image: pihole/pihole:2026.10.1
exposure: internal # LAN-only admin UI (docs/13 row 6; R-DNS/R-NET)
egress: internet # normal WAN egress (docs/03; R-NET-05)
storage:
+1 -1
View File
@@ -12,7 +12,7 @@
name: plex
kind: container
source:
image: plexinc/pms-docker:1.43.4.10903-e5521bd8c
image: plexinc/pms-docker:1.43.6.11085-d1e52e169
# plexinc/pms-docker env. Rendered VERBATIM onto the app container by render.py (schema env list).
env:
# PLEX_UID/PLEX_GID: the image boots as root (s6 PID1), chowns /config to this uid/gid, then drops
+1 -1
View File
@@ -8,7 +8,7 @@
name: sonarr
kind: container
source:
image: lscr.io/linuxserver/sonarr:4.0.15 # pinned stable v4 tag (R-UPD-01); update.range governs advancement
image: lscr.io/linuxserver/sonarr:4.0.20 # pinned stable v4 tag (R-UPD-01); update.range governs advancement
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
# which cannot write a 1000:3000-owned 0755 tree.
@@ -0,0 +1,41 @@
---
date: 2026-10-31
branch: chore/pm-agent-digest-99
title: Pin pm-agent dispatcher and worker to the agent-runner digest built from PR #99's merge
---
## What changed
- `platform/pm-agent/deployment.yaml`: the dispatcher `image:` moves from
`agent-runner@sha256:df36d9b7…` (images run 383) to
`agent-runner@sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2`.
`config-generation` goes from "4" to "5" so the pod rolls.
- `platform/pm-agent/configmap.yaml`: `PM_WORKER_IMAGE` gets the same digest. A comment line
records where it came from. The `# renovate-digest:` marker still sits directly above both
values.
- No other values changed. `PM_DRY_RUN` stays "false". `PM_DECISION_MAX_TURNS` is not set, so
it keeps its in-code default of 1. No `*.sops.yaml` was touched.
## What I verified
- PR #99's merge commit is `466f63ec3c3df968a1dab25a8811dcd41961efde` (origin/main HEAD at
branch time).
- The Gitea Actions tasks API shows the `images` workflow (images.yaml) on that commit as
**run 394** (task 643), with event `push` and status `success`.
- I resolved the digest read-only from the registry by HEADing the manifests with OCI and
Docker Accept headers. Both `agent-runner:sha-466f63ec3c3d` and `agent-runner:main` returned
`200`, an `application/vnd.oci.image.manifest.v1+json` content type, and
`docker-content-digest: sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2`.
The two are identical.
- Toolbox gate: the rc and skipped stages are in the PR description.
## What I did NOT verify
- That the dispatcher actually rolls to the new digest after merge, and that Flux applies it.
Flux's `lastAppliedRevision` and the pod's image were not checked. That is post-merge work.
- That the new image surfaces the triage failure's real cause. It will show only on the next
failing triage, in the dispatcher log.
- That admission (rule 7, `imagePullSecrets`) admits worker Jobs from this digest. The
template comes from `infra/agent/` at PR #99's merge, which still carries the
`imagePullSecrets` entry, but no Job was created to prove it.
- I did not run the image's own selftest against this digest.
+55
View File
@@ -0,0 +1,55 @@
---
date: 2026-10-31
branch: chore/pm-agent-go-live
title: Take pm-agent out of dry run (operator-approved); label scope and worker caps unchanged
---
## What changed
- `platform/pm-agent/configmap.yaml`: `PM_DRY_RUN` `"true"` -> `"false"`. The operator approved
this after reading the dry-run triage of issue #97. `PM_REQUIRE_LABEL: agent`,
`PM_MAX_ACTIVE_WORKERS: 2`, `PM_MAX_NEW_WORKERS_PER_CYCLE: 1`, the worker image digest and every
other knob are unchanged. The two comments that said dry run was the only mode that worked
(because the run-340 digest lacked `imagePullSecrets`) now describe the run-383 digest that
PR #96 pinned.
- `platform/pm-agent/deployment.yaml`: `config-generation` `"3"` -> `"4"`. Without the bump the
dispatcher keeps its startup config and stays in dry run.
- `platform/pm-agent/README.md` and `docs/plans/pm-agent-issue-to-pr.md` (follow-up 3): the
"currently dry run" status text now says dry run is off. `runbooks/pm-agent.md` is unchanged.
Its "stop it: set `PM_DRY_RUN: "true"`" step and its admission-refusal troubleshooting row
are still accurate.
## What I verified
Read-only, `kubectl --context k3s-prod` through the toolbox, 2026-10-31 ~11:45 local, with Flux
`pm-agent` and `platform-config` both at `main@1126740`:
- ClusterPolicy `pm-agent-job-shape`: `Ready=True`, `validationFailureAction=Enforce`,
`background=false`. It has seven rules, including `worker-pull-secret-must-be-the-committed-one`.
- RBAC: ServiceAccounts `pm-agent` and `pm-agent-worker`, Role `pm-agent` and RoleBinding
`pm-agent` (roleRef `pm-agent`, subject `ServiceAccount/pm-agent`) all exist. `auth can-i`
as `system:serviceaccount:pm-agent:pm-agent` returned `yes` for `create jobs.batch` and `no`
for `delete jobs.batch`.
- Secrets (key names only): `pm-agent-registry` is `kubernetes.io/dockerconfigjson` with key
`.dockerconfigjson`. `pm-agent-credentials` is `Opaque` with keys `claude-code-oauth-token`
and `gitea-token`.
- Egress: `default-deny` NetworkPolicy exists. CiliumNetworkPolicies `pm-agent-allow-egress`
and `pm-agent-worker-allow-egress` both report `Valid=True`.
- Deployment `pm-agent` is 1/1 on `agent-runner@sha256:df36d9b7…` with
`config-generation: "3"` live. No Jobs exist in the namespace.
- `platform/pm-agent/{configmap,deployment}.yaml`, README and docs do not match any
`sensitive_paths` glob in `infra/gitea/approval-policy.json`, so no review or override applies.
## What I did NOT verify
- **That a real worker Job is admitted and completes.** Nothing can show this until a labelled
issue gets a `start` decision after the merge. The L0 Kyverno suite shows the policy accepts
the committed template. Whether the API server admits the Job built from the df36d9b image's
baked template has not been tested live.
- That the worker pod can pull its image with `pm-agent-registry`, reach Gitea and the model
through its egress policy, and push a branch and open a PR. None of this has run before.
- The pod list also shows a stale `pm-agent-65db599649-v678g` (25d, `ContainerStatusUnknown`,
from an old ReplicaSet). It does not block anything, but I did not investigate it. Clearing
it is an operator action.
- CI: see the PR for the gate result. Locally the best possible result is rc 3, with only
`gitea-settings` not run.
@@ -0,0 +1,78 @@
---
date: 2026-10-31
branch: chore/pm-agent-runner-digest
title: Bump pm-agent to the run-383 agent-runner digest by hand; Renovate saw it but the schedule withheld it
---
## What changed
`platform/pm-agent/deployment.yaml` (the dispatcher `image:`) and `platform/pm-agent/configmap.yaml`
(`PM_WORKER_IMAGE`) move:
- from `agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`, the
run-340 publish;
- to `agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d`, the
images run 383 publish of ddd04d4.
Both `# renovate-digest:` markers are unchanged, directly above their values.
`pm-agent.homelab.keithjackson.dev/config-generation` goes `"2"` → `"3"`. That is the convention in
`deployment.yaml`'s own comment: "bump it in the same commit as a change to configmap.yaml". The
image change alone would roll the pod, but the convention is followed regardless.
No `*.sops.yaml` file is touched. Branched from 06d4a9a, which already contains the operator's #95.
### Why Renovate did not do this: the schedule, not a config defect
- **Run 385** (dispatch, 2026-10-31 16:14 UTC, on ddd04d4) was a **real** run. It logged
`Deleting orphan branch (… renovate/pin-dependencies)`, not a `DRY-RUN:` line. It ran at **12:14
America/New_York**, outside `"schedule": ["after 3am and before 6am"]`, so it created and updated
no branch.
- **The lookup DID see the new digest.** Dependency Dashboard issue #91 lists, under **"Awaiting
Schedule"**:
`chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to df36d9b`.
- **Nothing else blocked it.** There are no open Renovate PRs and no `renovate/*` branch on Gitea
(111 branches listed across all pages). The previous agent-runner PR, #90 (to c4eba5b), was
auto-merged by renovate-bot at 08:22 on 10-31. No PR limit was hit, and digests are not
dashboard-approval gated.
- **The registry, read-only, with Docker/OCI `Accept` headers:**
- `HEAD …/agent-runner/manifests/main` → `docker-content-digest: sha256:df36d9b7…771d`
- `HEAD …/agent-runner/manifests/sha-ddd04d460ef6` → the same digest.
**Without this PR** the next scheduled run (08:15 UTC, 04:15 EDT) would open it, and it would
auto-merge as a `digest` update. Ticking that "Awaiting Schedule" checkbox on #91 makes the next
run of any kind create it out of schedule.
### Found on the way: Renovate's labels have never been applied
- The repo has exactly one label, `agent`. The org has none.
- Every Renovate PR so far (#89, #90, #92, #93) was created with `"labels": []`.
- On Gitea, Renovate drops a label that does not exist in the repo. So `needs-review`,
`blast-radius`, `range-widen`, `own-image`, `dependencies`, `automated-update`, `breaking` and
`ci-tooling` are never attached.
- The **automerge flags still work.** Only the labels, the visible review signal, are missing.
- **Operator fix:** create those labels on the repo. This was not changed here.
## What I verified
- **The new image carries the fix.** I fetched both images' layers read-only from the registry and
read `/etc/pm-agent/worker-job-template.yaml` from each (`infra/agent/Containerfile:116` copies
it there):
- df36d9b's has `imagePullSecrets:` / `- name: pm-agent-registry`;
- c4eba5b's has no `imagePullSecrets` at all.
This matches `infra/agent/worker-job-template.yaml:76-77` at ddd04d4.
- **`main` = `sha-ddd04d460ef6` = df36d9b**, both measured.
- **No other file references c4eba5b.** No admission policy pins it.
- **No override is needed.** `infra/gitea/approval-policy.json` lists only `platform/kyverno/**`
under `platform/`, so `platform/pm-agent/**` is not sensitive.
- I read the whole job logs of runs 362, 367 and 385.
## What I did NOT verify
- **That the rolled dispatcher starts on df36d9b and that a worker Job is admitted.** That needs
Flux to apply this and a real dispatch. Both are operator actions.
- **That the `pm-agent-registry` Secret exists and decrypts in-cluster** (#95's sops file). I did not
read it.
- **That Renovate will treat this hand bump as current.** It should: its next lookup finds
`main` == the pinned digest, so it proposes nothing and drops the dashboard entry. Not observed.
@@ -0,0 +1,77 @@
---
date: 2026-10-31
branch: fix/pm-agent-triage-exit
title: pm-agent triage fails with "claude exited 1:" and no message; log the CLI's stdout envelope so the cause is visible
---
## What changed
- `infra/agent/pm-dispatch.py`, `ask_pm()`: on a non-zero exit the error now comes from a new
`describe_cli_failure()`, not from `proc.stderr` alone. With `--output-format json` the CLI
reports its own failures as the result envelope on **stdout** and writes nothing to stderr,
which is why both failures today logged `claude exited 1: ` with an empty message. The new
line carries the envelope's `subtype`, `is_error`, `api_error_status`, `num_turns`,
`duration_ms`, `total_cost_usd`, `stop_reason`, its `result`/`error` text, and the tails of
stdout and stderr. It names silence explicitly when both streams are empty.
- `redact()`: everything in that message, and in `extract_decision()`'s "no decision object"
error, passes through it. It covers `sk-ant-…`, `Bearer`/`token` values, and `user:pass@` in
both `http://` and `https://` URLs.
- `PM_DECISION_MAX_TURNS`: a new knob. It defaults to 1, today's value, so behaviour is
unchanged, and it is bounded to 1–5. If the newly visible envelope says `error_max_turns`,
the remedy is a ConfigMap change rather than another image build. Documented in
`platform/pm-agent/README.md`.
- Selftest: four new assertions. They cover an empty stderr with the cause on stdout, a
stderr-only failure, total silence, and redaction.
## What I verified
- **Timeline (Loki, `{namespace="pm-agent"}`, and `kubectl logs`):**
- **16:37:38 to 16:38:00**: pod `pm-agent-898fd8f5f-dp5hs`, image **c4eba5b3**,
config-generation 2, dry run. It triaged #97 in **22 s**, logged `PM says comment`, and
posted the 16:38:00Z comment. The thread then held only the issue body.
- **16:43:09 to 16:44:00**: pod `pm-agent-f7c9b7fcf-j8mzn`, image **df36d9b7**,
config-generation 3, **still dry run**. It logged `claude exited 1: ` with an empty
message after **51 s**.
- **16:55:42 to 16:56:36**: pod `pm-agent-7796db8bcb-p5dsm`, df36d9b7, config-generation
4, live. The same empty failure, after **54 s**.
- The only successful triage ever was the 16:38 one. Every earlier `898fd8f5f`/`5db55d558c`
log line is a startup crash (Gitea `GET /user` 401), not a triage.
- **Going live is NOT the cause.** The first failure happened in dry run. `ask_pm()` has no
dry-run branch: the prompt, flags, tools (`--tools ""`) and cwd are identical in both
modes. `PM_DRY_RUN` gates only `create_job()`, which runs after the model has answered.
- **Ruled out, with evidence:**
- **Timeout**: 51 to 54 s against a 900 s limit, and a timeout raises `TimeoutExpired`,
not rc 1.
- **Budget cap**: the PM call passes no `--max-budget-usd`.
- **Prompt size**: the thread is the issue body (293 chars) plus three comments
(1871 + 404 + 77 chars), and the persona is 8 KB.
- **Writable HOME**: HOME=`/workspace` (emptyDir), and `CLAUDE_CONFIG_DIR=/workspace/.claude`
was written at 16:56.
- **Egress**: a TLS 1.3 handshake to `api.anthropic.com:443` from the dispatcher pod
succeeds in 0.06 s.
- **Model name and CLI version**: the same `claude-opus-5` and CLI 2.1.285 succeeded at
16:38. Between c4eba5b and df36d9b, `infra/agent/` differs only by a template selftest
assertion and the toolbox base digest.
- **A hint, not proof:** `/workspace/.claude/.claude.json` in the live pod has
`cachedExtraUsageDisabledReason: 'out_of_credits'`. No token was read or printed.
- Offline selftest: `PM_SELFTEST=1` against the worktree's template and persona prints
`selftest OK`, including the 4 new assertions.
- Toolbox gate: see the PR description for the rc and the stages that were skipped.
## What I did NOT verify
- **The root cause. UNVERIFIED.** The CLI's own message was discarded, `--no-session-persistence`
leaves no transcript, and I did not run `claude` in the pod. The candidates still open:
- **Usage or rate limit on the subscription seat.** The seat is shared with the operator's
interactive use. The `out_of_credits` extra-usage flag fits this. So does a ~50 s run,
which matches the CLI exhausting its 429/529 retries.
- **`error_max_turns`.** With `--max-turns 1` and `--json-schema`, an answer that fails the
schema cannot be retried. The failing thread now invites `implement` (a branch plus a
long brief), where the 16:38 one produced a short `comment`. ~50 s is about two
generations of the 22 s success.
- **The 17:11 retry will NOT reveal the cause.** It runs the old image, which still discards
stdout. The cause becomes visible only after this PR merges, CI re-publishes agent-runner,
and a follow-up PR bumps the dispatcher digest (`deployment.yaml` and the `PM_WORKER_IMAGE`
in `configmap.yaml`, plus a `config-generation` bump).
- That the CLI's error envelope carries exactly these key names. The parser takes whichever
keys are present and falls back to raw tails.
+3 -1
View File
@@ -257,7 +257,9 @@ about runtime behaviour in this document is a design claim, not a measurement.
bot's permission is `write`, not `admin`** — that measurement is the control that keeps
the agent out of its own approval gate.
3. **First run with `PM_DRY_RUN=true` and `PM_REQUIRE_LABEL=agent`**, on one issue written
for the purpose. Read what the PM says before letting it write anything.
for the purpose. Read what the PM says before letting it write anything. *Done:* dry-run
triage of issue #97 read; `PM_DRY_RUN=false` on 2026-10-31
(`decisions/2026-10-31-pm-agent-go-live.md`), `PM_REQUIRE_LABEL=agent` kept.
4. **Tighten the admission image pin** from the `@sha256:*` wildcard to the literal digest.
One line, reviewed.
5. **Narrow `world:443` to `toFQDNs: api.anthropic.com`** — after verifying Cilium's DNS
+95 -4
View File
@@ -214,6 +214,12 @@ class Config:
# ConfigMap without rebuilding the image.
self.push_on_red = env("PM_PUSH_ON_RED_GATE", "false")
self.pm_timeout = env_int("PM_DECISION_TIMEOUT_SECONDS", 900)
# The PM holds no tools, so the only thing a second turn can be spent on is the CLI's
# structured-output step re-asking the model after an answer that failed the schema.
# 1 is the long-standing value; it is a knob so that, if the failure envelope now
# logged by describe_cli_failure() says `error_max_turns`, the remedy is a ConfigMap
# change rather than another image build.
self.pm_max_turns = env_int("PM_DECISION_MAX_TURNS", 1)
# Opt-in gate. Empty (the default) means every open issue is in scope, which is the
# behaviour the service was asked for. Setting it to a label name — `agent` is the
# conventional one — narrows the agent to issues a human has explicitly handed it,
@@ -231,6 +237,8 @@ class Config:
if self.max_active_workers < 1:
die("PM_MAX_ACTIVE_WORKERS must be >= 1")
if not 1 <= self.pm_max_turns <= 5:
die("PM_DECISION_MAX_TURNS must be between 1 and 5")
if "@sha256:" not in self.worker_image:
# The admission policy requires a digest too, so a tag here would produce a Job
# that is created and then DENIED — a failure visible only in the dispatcher's
@@ -437,7 +445,7 @@ def ask_pm(cfg: Config, thread: str) -> dict:
"--system-prompt-file", cfg.persona_file,
"--tools", "",
"--permission-prompts", "none",
"--max-turns", "1",
"--max-turns", str(cfg.pm_max_turns),
"--output-format", "json",
"--json-schema", json.dumps(DECISION_SCHEMA),
"--no-session-persistence",
@@ -445,10 +453,76 @@ def ask_pm(cfg: Config, thread: str) -> dict:
proc = subprocess.run(cmd, input=thread, capture_output=True, text=True,
timeout=cfg.pm_timeout)
if proc.returncode != 0:
raise RuntimeError(f"claude exited {proc.returncode}: {proc.stderr.strip()[:800]}")
raise RuntimeError(describe_cli_failure(proc.returncode, proc.stdout, proc.stderr))
return extract_decision(proc.stdout)
# Anything that could be a credential in the CLI's output. The CLI does not normally echo its
# token, but an API error body or a proxy's HTML page is not something to bet a leak on. Both
# schemes are matched (the http one is the lesson of a scheme-specific redaction that leaked
# ~/.git-credentials), and `token ...` is Gitea's own header scheme.
_SECRET_PATTERNS = [
(re.compile(r"sk-ant-[A-Za-z0-9_\-]+"), "sk-ant-[REDACTED]"),
(re.compile(r"(?i)\b(bearer|token)(\s+|=|:\s*)[A-Za-z0-9._~+/=\-]{8,}"), r"\1\2[REDACTED]"),
(re.compile(r"(?i)(https?://)[^/\s:@]+:[^/\s@]+@"), r"\1[REDACTED]@"),
]
def redact(text: str) -> str:
for pat, repl in _SECRET_PATTERNS:
text = pat.sub(repl, text)
return text
def describe_cli_failure(rc: int, stdout: str, stderr: str, limit: int = 800) -> str:
"""Say WHY `claude --print` failed, in the dispatcher log, redacted.
WHY THIS EXISTS: with `--output-format json` the CLI reports its own failures — an API
error, a usage limit, `error_max_turns`, a schema the model could not satisfy — as the
JSON result envelope on STDOUT (`is_error`, `subtype`, `result`), and writes nothing to
stderr. Logging only stderr therefore produced `claude exited 1: ` with an empty message
on 2026-10-31, twice, with the cause sitting unread in stdout. So: the envelope's
diagnostic fields first, then the raw tails of both streams.
"""
parts = [f"claude exited {rc}"]
out = (stdout or "").strip()
env = None
if out:
try:
env = json.loads(out)
except ValueError:
# A stream of JSON lines, or a log line ahead of the envelope: the last line that
# parses as an object is the result.
for line in reversed(out.splitlines()):
try:
cand = json.loads(line)
except ValueError:
continue
if isinstance(cand, dict):
env = cand
break
if isinstance(env, dict):
fields = []
for key in ("subtype", "is_error", "api_error_status", "num_turns", "duration_ms",
"duration_api_ms", "total_cost_usd", "stop_reason"):
if key in env:
fields.append(f"{key}={env[key]!r}")
if fields:
parts.append("envelope: " + " ".join(fields))
for key in ("result", "error", "errors", "message"):
if env.get(key):
val = env[key] if isinstance(env[key], str) else json.dumps(env[key])
parts.append(f"{key}: {val[:limit]}")
if out and not isinstance(env, dict):
parts.append(f"stdout tail: {out[-limit:]}")
err_text = (stderr or "").strip()
if err_text:
parts.append(f"stderr tail: {err_text[-limit:]}")
if len(parts) == 1:
parts.append("(no output on stdout or stderr)")
return redact(" | ".join(parts))
def extract_decision(stdout: str) -> dict:
"""Pull the decision object out of whatever `claude --print` wrapped it in.
@@ -501,7 +575,7 @@ def extract_decision(stdout: str) -> dict:
obj = as_obj(c)
if obj and isinstance(obj.get("action"), str):
return obj
raise RuntimeError(f"no decision object in claude output: {stdout[:400]!r}")
raise RuntimeError(redact(f"no decision object in claude output: {stdout[-800:]!r}"))
# --------------------------------------------------------------------------- thread
@@ -988,8 +1062,25 @@ def selftest() -> int:
except RuntimeError:
check(True, f"bad output refused: {label}")
# -- a failed CLI run must say why, and must not leak ---------------------
# The 2026-10-31 failure: rc 1, empty stderr, the cause in the stdout envelope.
env_err = json.dumps({"type": "result", "subtype": "error_max_turns", "is_error": True,
"num_turns": 2, "result": "API Error: 429 rate_limit_error"})
msg = describe_cli_failure(1, env_err, "")
check("error_max_turns" in msg and "429" in msg,
"CLI failure: stdout envelope surfaced when stderr is empty")
msg = describe_cli_failure(1, "", "boom on stderr")
check("boom on stderr" in msg, "CLI failure: stderr tail surfaced")
msg = describe_cli_failure(1, "", "")
check("no output" in msg, "CLI failure: silence is named, not an empty message")
msg = describe_cli_failure(1, "plain text sk-ant-oat01-abcdefABCDEF_123 here",
"Authorization: Bearer abcdefghijklmnop "
"http://u:[email protected]/x")
check("sk-ant-oat01" not in msg and "abcdefghijklmnop" not in msg and ":pw@" not in msg,
"CLI failure: token, bearer and URL credential redacted")
# -- the committed Job template -----------------------------------------
tmpl = os.environ.get("PM_WORKER_JOB_TEMPLATE", "/etc/pm-agent/worker-job-template.yaml")
tmpl =os.environ.get("PM_WORKER_JOB_TEMPLATE", "/etc/pm-agent/worker-job-template.yaml")
try:
selftest_template(tmpl)
check(True, f"worker Job template satisfies its contract ({tmpl})")
+1 -1
View File
@@ -60,7 +60,7 @@ spec:
# Pinned by digest (the multi-arch index). The kubernetes manager in renovate.json5
# tracks platform/**, so Renovate proposes bumps; the CI client (buildx, pinned in
# versions.env) talks to any recent daemon.
image: docker.io/moby/buildkit:v0.34.1-rootless@sha256:f68752a5a94acfcdad2d07867b7075601a421638e17fc5f28efc1c814d0c4f89
image: docker.io/moby/buildkit:v0.34.2-rootless@sha256:c8252268760e1baba80780d63d738ef9f260fb7c7c0973c78494cb9ddd99eda1
args:
- --addr
- unix:///run/user/1000/buildkit/buildkitd.sock # local only: the probes below
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
chart: cilium
# [parity] Pinned to the exact version the Ansible bootstrap installs
# (cilium_version in roles/cilium/defaults/main.yml). Renovate bumps both together.
version: "1.20.1"
version: "1.20.4"
sourceRef:
kind: HelmRepository
name: cilium
@@ -63,7 +63,7 @@ spec:
valueFrom:
fieldRef:
fieldPath: status.hostIP
image: intel/intel-gpu-plugin:0.37.0
image: intel/intel-gpu-plugin:0.37.1
imagePullPolicy: IfNotPresent
name: intel-gpu-plugin
resources:
+1 -1
View File
@@ -96,4 +96,4 @@ images:
- name: intel/intel-gpu-plugin
newName: docker.io/intel/intel-gpu-plugin
# renovate: datasource=github-releases depName=intel/intel-device-plugins-for-kubernetes
newTag: "0.37.0"
newTag: "0.37.1"
+2 -2
View File
@@ -33,6 +33,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
# renovate: datasource=github-releases depName=kubevirt/kubevirt
- https://github.com/kubevirt/kubevirt/releases/download/v1.9.0/kubevirt-operator.yaml
- https://github.com/kubevirt/kubevirt/releases/download/v1.9.1/kubevirt-operator.yaml
# renovate: datasource=github-releases depName=kubevirt/containerized-data-importer
- https://github.com/kubevirt/containerized-data-importer/releases/download/v1.66.0/cdi-operator.yaml
- https://github.com/kubevirt/containerized-data-importer/releases/download/v1.66.1/cdi-operator.yaml
@@ -24,7 +24,7 @@ spec:
chart:
spec:
chart: kube-prometheus-stack
version: ">=88.0.0 <89.0.0" # k8s-1.36 catch-up (po v0.93; CRDs pre-applied server-side, Helm won't update them). Renovate maintains
version: "<93.0.0" # k8s-1.36 catch-up (po v0.93; CRDs pre-applied server-side, Helm won't update them). Renovate maintains
sourceRef:
kind: HelmRepository
name: prometheus-community
+12 -5
View File
@@ -46,11 +46,12 @@ waive the sensitive-path human review on its own pull request.
Both are deliberate, and both fail **visibly** rather than silently:
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
`PM_WORKER_IMAGE` in `configmap.yaml` carry the digest of the **first** publish (images
run 340, `sha-523cd9118f95`), pinned once on 2026-10-30 so going live did not wait on
Renovate's first run; every later bump is Renovate's. That digest predates the worker
template's `imagePullSecrets` entry, so worker Jobs built from it are refused by
admission — which is why `PM_DRY_RUN` starts `true` (see `configmap.yaml`). `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
`PM_WORKER_IMAGE` in `configmap.yaml` first carried the run-340 publish
(`sha-523cd9118f95`), whose baked template predated the `imagePullSecrets` entry admission
requires, so the service ran with `PM_DRY_RUN=true`. PR #96 pinned the run-383 digest
(`df36d9b`, `sha-ddd04d460ef6`), whose template carries the entry, and dry run was turned
off on 2026-10-31 (`decisions/2026-10-31-pm-agent-go-live.md`); every later bump is
Renovate's. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
(`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
`infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
@@ -249,6 +250,12 @@ five that matter:
| `PM_MAX_ACTIVE_WORKERS` | `2` | concurrent workers. RAM is the binding constraint (2Gi limit each) and the subscription seat's rate limits are shared with every other use of that seat |
| `PM_MAX_NEW_WORKERS_PER_CYCLE` | `1` | drains a backlog one issue at a time instead of in one stampede the first minute the credentials work |
| `PM_PUSH_ON_RED_GATE` | `false` | a change the gate refused is not pushed; the issue gets the gate log instead |
| `PM_DECISION_MAX_TURNS` | `1` (unset) | turns the triage call may take (1–5). The PM holds no tools, so a second turn is only ever the CLI re-asking after an answer that failed the decision schema. Raise it if a triage error logs `subtype='error_max_turns'` |
**When a triage fails**, the `[error] issue #N: claude exited …` line carries the CLI's result
envelope (`subtype`, `is_error`, `api_error_status`, `num_turns`, `result`) and the tails of
stdout and stderr, redacted. Before 2026-10-31 it logged stderr only, which is empty under
`--output-format json` — see `decisions/2026-10-31-pm-agent-triage-exit.md`.
Changing a knob requires bumping the `config-generation` annotation in `deployment.yaml` in
the same commit — the process reads its configuration once, at startup, so without the bump
+14 -10
View File
@@ -58,13 +58,16 @@ data:
# request that also moves the Deployment's image (same marker, same branch). Nobody pushes
# this image from a workstation.
#
# READ BEFORE TURNING PM_DRY_RUN OFF: this digest PREDATES the worker template's
# `imagePullSecrets` entry (PR #61). Its baked template has none, and admission now REQUIRES
# it — so a worker Job built from this image is DENIED. The merge that adds the entry
# changes infra/agent/**, CI re-publishes agent-runner, and Renovate's bump to that digest
# is what makes worker Jobs admissible. Until then, dry run is the only mode that works.
# The run-340 digest predated the worker template's `imagePullSecrets` entry (PR #61), which
# admission REQUIRES, so its worker Jobs were DENIED. The digest below (df36d9b, run 383,
# agent-runner:sha-ddd04d460ef6, pinned in PR #96) bakes a template that carries
# `imagePullSecrets: pm-agent-registry`, which is what made turning PM_DRY_RUN off possible
# (decisions/2026-10-31-pm-agent-runner-digest.md). A digest older than that would be DENIED.
# The digest below (14a42ba, images run 394, agent-runner:sha-466f63ec3c3d = PR #99's merge)
# adds the CLI failure envelope to triage errors and the PM_DECISION_MAX_TURNS knob
# (decisions/2026-10-31-pm-agent-digest-99.md).
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42"
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2"
# --- the model -------------------------------------------------------------
# The PM only reads a thread and answers with one JSON object, so it holds no tools and
@@ -120,10 +123,11 @@ data:
# no worker Job and so changes no code. The honest way to watch this service for a day
# before letting it write anything.
#
# "true" FOR THE FIRST RUN, and not only for caution: the pinned digest above cannot yet
# create an admissible worker Job (see the PM_WORKER_IMAGE comment). Flip to "false" in the
# same pull request as — or after — Renovate's first agent-runner digest bump.
PM_DRY_RUN: "true"
# "true" for the first run (2026-10-30 .. 2026-10-31); dry-run triage of issue #97 was read
# and the operator approved going live. "false" since decisions/2026-10-31-pm-agent-go-live.md.
# Setting it back to "true" (plus a config-generation bump) is the gentlest stop — see
# runbooks/pm-agent.md.
PM_DRY_RUN: "false"
# If the gate refuses a worker's change, the branch is NOT pushed by default and the issue
# gets the gate log instead. Set to "true" to push anyway and open a red pull request —
+2 -2
View File
@@ -48,7 +48,7 @@ spec:
# It is a hand-maintained marker rather than a computed hash because Flux's
# Kustomization here does not run a configmap generator; bump it in the same commit
# as a change to configmap.yaml.
pm-agent.homelab.keithjackson.dev/config-generation: "2"
pm-agent.homelab.keithjackson.dev/config-generation: "5"
spec:
serviceAccountName: pm-agent
# The kubelet's credential for pulling this image. Not the agent's Gitea credential —
@@ -78,7 +78,7 @@ spec:
# the registry's manifest endpoint on 2026-10-30 and pinned by hand once so the
# dispatcher does not wait on Renovate's first run; every later bump is Renovate's.
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2
imagePullPolicy: IfNotPresent
command:
- /usr/local/bin/pm-dispatch