Compare commits

...
Author SHA1 Message Date
pm-agent f6150b8957 docs: Massive amounts of historical data
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
From Gitea issue #97.

Authored by the PM agent's worker; the merge gate is the approver. Refs #97.

Agent: pm-agent (issue #97, mode implement)
2026-10-31 17:24:23 +00:00
jakman85 466f63ec3c Merge pull request 'fix(pm-agent): log the CLI's stdout envelope when triage fails; add PM_DECISION_MAX_TURNS' (#99) from fix/pm-agent-triage-exit into main
gate / gate (push) Successful in 2m56s
images / images (push) Successful in 16s
gate / conformance (push) Has been skipped
Reviewed-on: #99
2026-10-31 17:12:40 +00:00
jakman85andClaude Opus 5.5 664ee82b9a fix(pm-agent): log the CLI's stdout envelope when triage fails; add PM_DECISION_MAX_TURNS
gate / gate (pull_request) Successful in 2m56s
images / images (pull_request) Successful in 13s
gate / conformance (pull_request) Has been skipped
`claude --print --output-format json` reports its own failures (API error, usage limit,
error_max_turns) as the result envelope on stdout and leaves stderr empty, so ask_pm()'s
stderr-only message logged "claude exited 1: " with nothing after it, twice on 2026-10-31.
describe_cli_failure() now logs the envelope's diagnostic fields plus stdout/stderr tails,
redacted (sk-ant-, Bearer/token, URL userinfo). PM_DECISION_MAX_TURNS (default 1, unchanged)
makes the max-turns remedy a ConfigMap change if the envelope turns out to say error_max_turns.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 12:04:09 -05:00
jakman85 95e08da2f0 Merge pull request 'chore(pm-agent): take the dispatcher out of dry run (operator-approved), config-generation 4' (#98) from chore/pm-agent-go-live into main
gate / gate (push) Successful in 2m58s
gate / conformance (push) Has been skipped
Reviewed-on: #98
Reviewed-by: Keith Jackson <[email protected]>
2026-10-31 16:52:16 +00:00
4 changed files with 618 additions and 3 deletions

No files matched your search

@@ -0,0 +1,77 @@
---
date: 2026-10-31
branch: fix/pm-agent-triage-exit
title: pm-agent triage fails with "claude exited 1:" and no message; log the CLI's stdout envelope so the cause is visible
---
## What changed
- `infra/agent/pm-dispatch.py`, `ask_pm()`: on a non-zero exit the error now comes from a new
`describe_cli_failure()`, not from `proc.stderr` alone. With `--output-format json` the CLI
reports its own failures as the result envelope on **stdout** and writes nothing to stderr,
which is why both failures today logged `claude exited 1: ` with an empty message. The new
line carries the envelope's `subtype`, `is_error`, `api_error_status`, `num_turns`,
`duration_ms`, `total_cost_usd`, `stop_reason`, its `result`/`error` text, and the tails of
stdout and stderr. It names silence explicitly when both streams are empty.
- `redact()`: everything in that message, and in `extract_decision()`'s "no decision object"
error, passes through it. It covers `sk-ant-…`, `Bearer`/`token` values, and `user:pass@` in
both `http://` and `https://` URLs.
- `PM_DECISION_MAX_TURNS`: a new knob. It defaults to 1, today's value, so behaviour is
unchanged, and it is bounded to 1–5. If the newly visible envelope says `error_max_turns`,
the remedy is a ConfigMap change rather than another image build. Documented in
`platform/pm-agent/README.md`.
- Selftest: four new assertions. They cover an empty stderr with the cause on stdout, a
stderr-only failure, total silence, and redaction.
## What I verified
- **Timeline (Loki, `{namespace="pm-agent"}`, and `kubectl logs`):**
- **16:37:38 to 16:38:00**: pod `pm-agent-898fd8f5f-dp5hs`, image **c4eba5b3**,
config-generation 2, dry run. It triaged #97 in **22 s**, logged `PM says comment`, and
posted the 16:38:00Z comment. The thread then held only the issue body.
- **16:43:09 to 16:44:00**: pod `pm-agent-f7c9b7fcf-j8mzn`, image **df36d9b7**,
config-generation 3, **still dry run**. It logged `claude exited 1: ` with an empty
message after **51 s**.
- **16:55:42 to 16:56:36**: pod `pm-agent-7796db8bcb-p5dsm`, df36d9b7, config-generation
4, live. The same empty failure, after **54 s**.
- The only successful triage ever was the 16:38 one. Every earlier `898fd8f5f`/`5db55d558c`
log line is a startup crash (Gitea `GET /user` 401), not a triage.
- **Going live is NOT the cause.** The first failure happened in dry run. `ask_pm()` has no
dry-run branch: the prompt, flags, tools (`--tools ""`) and cwd are identical in both
modes. `PM_DRY_RUN` gates only `create_job()`, which runs after the model has answered.
- **Ruled out, with evidence:**
- **Timeout**: 51 to 54 s against a 900 s limit, and a timeout raises `TimeoutExpired`,
not rc 1.
- **Budget cap**: the PM call passes no `--max-budget-usd`.
- **Prompt size**: the thread is the issue body (293 chars) plus three comments
(1871 + 404 + 77 chars), and the persona is 8 KB.
- **Writable HOME**: HOME=`/workspace` (emptyDir), and `CLAUDE_CONFIG_DIR=/workspace/.claude`
was written at 16:56.
- **Egress**: a TLS 1.3 handshake to `api.anthropic.com:443` from the dispatcher pod
succeeds in 0.06 s.
- **Model name and CLI version**: the same `claude-opus-5` and CLI 2.1.285 succeeded at
16:38. Between c4eba5b and df36d9b, `infra/agent/` differs only by a template selftest
assertion and the toolbox base digest.
- **A hint, not proof:** `/workspace/.claude/.claude.json` in the live pod has
`cachedExtraUsageDisabledReason: 'out_of_credits'`. No token was read or printed.
- Offline selftest: `PM_SELFTEST=1` against the worktree's template and persona prints
`selftest OK`, including the 4 new assertions.
- Toolbox gate: see the PR description for the rc and the stages that were skipped.
## What I did NOT verify
- **The root cause. UNVERIFIED.** The CLI's own message was discarded, `--no-session-persistence`
leaves no transcript, and I did not run `claude` in the pod. The candidates still open:
- **Usage or rate limit on the subscription seat.** The seat is shared with the operator's
interactive use. The `out_of_credits` extra-usage flag fits this. So does a ~50 s run,
which matches the CLI exhausting its 429/529 retries.
- **`error_max_turns`.** With `--max-turns 1` and `--json-schema`, an answer that fails the
schema cannot be retried. The failing thread now invites `implement` (a branch plus a
long brief), where the 16:38 one produced a short `comment`. ~50 s is about two
generations of the 22 s success.
- **The 17:11 retry will NOT reveal the cause.** It runs the old image, which still discards
stdout. The cause becomes visible only after this PR merges, CI re-publishes agent-runner,
and a follow-up PR bumps the dispatcher digest (`deployment.yaml` and the `PM_WORKER_IMAGE`
in `configmap.yaml`, plus a `config-generation` bump).
- That the CLI's error envelope carries exactly these key names. The parser takes whichever
keys are present and falls back to raw tails.
@@ -0,0 +1,441 @@
---
date: 2026-10-31
branch: docs/source-comment-audit
title: Audit — historical narrative in source comments (read-only inventory, sample not exhaustive)
---
## What changed
**Nothing but this file.** This is the read-only audit asked for in issue #97. No source
file was edited; the pruning is to be briefed afterwards, per directory, from this list.
### What was asked, and the calibration applied
From the reporter (issue #97, 2026-10-31T16:42:30Z): *"'historical' talks about previous
issues or intent that have changed and why/how it is used today within comments of source
files rather than keeping them in the decisions and documentation. Source file comments
should only talk about the current files and sections of code/yaml purposes, uses,
expectations."*
Each row below is classified:
- **`historical`** — narrative about a superseded state, a past incident, a corrected
earlier belief, or a closed issue. Says nothing about what the line does today.
- **`load-bearing`** — phrased as history, but the sentence is the only record of a
**current** non-obvious constraint. Per the brief these get **rewrite**, never delete.
- **`ambiguous`** — I could not rule without the operator. Left for a human, deliberately
not resolved here.
### THIS IS A SAMPLE, NOT AN EXHAUSTIVE INVENTORY
The brief caps the audit at roughly 40–50 findings and asks for honesty over padding.
Measured candidate volume, tree-wide, with the keyword sweep below:
```
git ls-files -z | xargs -0 grep -nEi '#.*(used to|previously|formerly|no longer|we had|
before we|historically|in the past|originally|superseded|turned out|was wrong|predates|
legacy|issue #[0-9]+|PR #[0-9]+|measured 20|as of 20[0-9][0-9]-|see decisions/|DECISIONS-LOG)'
```
| Directory | candidate hits | files | coverage here | findings listed |
|---|---|---|---|---|
| `platform/` | 107 | 56 | **sampled** | 9 |
| `infra/` | 75 | 44 | **sampled** | 8 |
| `conformance/` (excl. `runs/`) | 72 | ~31 | **sampled** | 5 |
| `clusters/` | 43 | 19 | **sampled** | 7 |
| `ci/` | 42 | 15 | **sampled** | 6 |
| `catalog/` | 39 | 9 | **sampled** | 7 |
| `.gitea/` | 10 | 5 | **full** | 5 |
| `bootstrap/` | 4 | 3 | **full** | 3 |
| `platform-config/` | 3 | 1 | **full** | 3 |
| `apps/vms/` (hand-authored) | 3 | 2 | **full** | 2 |
| `apps/wave-*` (generated) | 15 | — | **excluded — see below** | 0 |
| `conformance/runs/` (archived logs) | 8 | 18 files | **excluded — see below** | 0 |
| `runbooks/` | 11 | — | **not audited** (prose docs, not source comments) | 0 |
| **total** | **~420** | **~190** | — | **55** |
Covered **fully**: `.gitea/`, `bootstrap/`, `platform-config/`, `apps/vms/`.
Covered **by sample only**: `platform/`, `infra/`, `conformance/`, `clusters/`, `ci/`,
`catalog/`. Roughly **365 candidate hits remain unexamined**, and candidate ≠ finding —
the keyword sweep's precision on the parts I did read was well under half, so the true
residual is smaller than 365 but is certainly in the low hundreds.
**The issue's premise holds**, with one qualification the operator should price in: of the
55 rows below, **22 are `load-bearing`** and get a rewrite rather than a deletion. The
narrative is genuinely excessive, but a mechanical "delete the history" sweep would destroy
real constraints. That ratio is the single most important number in this report.
### Two structural warnings that change how the follow-up PRs must be written
**1. `apps/wave-0/`, `apps/wave-1/`, `apps/wave-2/` are GENERATED. Do not edit comments there.**
They are goldens produced by one canonical invocation of `catalog/render.py`
(`ci/render-golden.sh:21`), and `ci/render-golden.sh:136-150` byte-compares every rendered
file against its committed golden — **any diff is a hard `exit 1` that blocks the merge**.
So the 15 candidate hits under `apps/wave-*` are *copies* of comments whose source is
`catalog/*.yaml` or `catalog/render.py`; e.g. `apps/wave-0/gitea.yaml:130` is the rendered
form of `catalog/gitea.yaml:100`. Edit the catalog source, then re-render with the canonical
command and commit both. A pruning PR that hand-edits `apps/wave-*` will sit red.
**2. `conformance/runs/*/run.log` are archived evidence artifacts, not source.** 8 candidate
hits sit in 18 committed run logs. They are the recorded output of past conformance runs —
the comments inside them are copies of the test bodies as they stood at run time. Rewriting
them would falsify evidence. Excluded from scope entirely; recommend the operator confirm.
### A dangling pointer, replicated across 18 files — fix this before any "replace with a pointer" work
**18 files cite `DECISIONS-LOG: "Headlamp gating stall"` as their rationale pointer, and no
such entry exists.** `grep -ci "gating stall" DECISIONS-LOG.md` → **0**; the phrase appears
nowhere in `decisions/` either. Citers:
```
clusters/base/crafty.yaml clusters/base/headlamp.yaml
clusters/base/homepage.yaml clusters/base/kustomization.yaml
clusters/base/runbooks.yaml clusters/base/vpn-ingress.yaml
platform-config/kustomization.yaml platform/crafty/deployment.yaml
platform/crafty/kustomization.yaml platform/headlamp/kustomization.yaml
platform/homepage/kustomization.yaml platform/homepage/README.md
runbooks/kustomization.yaml runbooks/flux-reconcile.md
conformance/tests/l0-static/alert-runbooks-resolvable/check.sh (+3 more)
```
This is the audit's clearest warning about its own remedy: "replace the narrative with a
pointer" is only an improvement if the target exists and keeps existing. The archive is
frozen and has no index, so a title-based citation into it cannot be checked by anything.
**Recommendation:** when the operator takes the pruning PRs, either write a real
`decisions/` entry for the leaf-gating lesson and repoint all 18 citations at its filename,
or drop the citation. Pointer targets of the form `decisions/<file>.md` are checkable;
`DECISIONS-LOG: "<title>"` is not. Nothing below recommends a pointer target I did not
confirm exists.
---
### `platform/` — 107 candidates, 9 listed (sampled). NOT sensitive, except `platform/kyverno/**`
| # | path:lines | comment (gist) | class | action | sensitive |
|---|---|---|---|---|---|
| 1 | `platform/crafty/service-l4.yaml:33-91` | "CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE … This file used to say `Local`" — 58 lines of outage narrative | load-bearing | **rewrite.** Keep the current constraints (Cilium's L2 announcer is not backend-aware; SNAT mode loses client IP; `/ban-ip` degrades so use `/ban`; `remote-node` is subsumed by `cluster` so do NOT add it; no `loadBalancerSourceRanges` deliberately). Drop the outage story, the lease-age detail and the per-protocol tell | no |
| 2 | `platform/longhorn/helmrelease.yaml:191-200` | "`orphanAutoDeletion: true` USED TO BE HERE AND WAS INERT. Removed 2026-09-13" | load-bearing | **rewrite** to the forward-looking rule: Longhorn ≥1.9 replaced it with `orphanResourceAutoDeletion` and the chart silently drops unknown `defaultSettings` keys — so do not set it | no |
| 3 | `platform/longhorn/helmrelease.yaml:249-254` | "CORRECTION, re-measured 2026-09-13 later the same day: the line originally recorded here read …" — corrects a superseded measurement | historical | **delete.** The correct reading is already stated at `:246-248` | no |
| 4 | `platform/monitoring/alerts.yaml:625-636` | "Both rules below used to read `== 2` / `== 3`, with the comment 0=unknown 1=healthy…" | load-bearing | **rewrite.** Keep "`longhorn_volume_robustness` is a one-hot `state`-labelled gauge, value 0 or 1" — that is why the expressions look as they do. Drop the old-enum story | no |
| 5 | `platform/monitoring/alerts.yaml:869-875` | "This alert originally used `severity: info`. That was WRONG, and wrong in the silent direction" | load-bearing | **rewrite** to the current rule: carries no `severity` deliberately, because `general.rules`' `InfoInhibitor` would swallow an info-severity deploy notification | no |
| 6 | `platform/monitoring/pvc-capacity-alerts.yaml:12-27` | "`LonghornVolumeCapacityHigh` … never did, for two different reasons in sequence, and it is now GONE" — dated 2026-09-06/2026-09-13 timeline | load-bearing | **rewrite.** Keep only "these rules key on `kubelet_volume_stats_*` because `longhorn_volume_actual_size_bytes` measures allocation, not filesystem use". Delete the sequence | no |
| 7 | `platform/headlamp/helmrelease.yaml:77-85` | "These two used to be keys INSIDE headlamp-oidc-credentials.sops.yaml. That was the bug…" + the dev-Keycloak `i/o timeout` story | load-bearing | **rewrite** to the current rule: issuer and callback URLs are public and MUST be cleartext, because Flux's `${BASE_DOMAIN}` postBuild substitution cannot reach inside an encrypted Secret. Keep the `[parity]` note at `:87-92` as-is | no |
| 8 | `platform/vpn-egress/egress-steering.yaml:79` | "CORRECTION 2026-09-12 — this comment used to say the host nftables kill-switch 'is the…'" | load-bearing | **rewrite** to state the actual mechanism (Cilium's eBPF egress-gateway drop) without the correction framing | no |
| 9 | `platform/keycloak/passkey-reconciler.yaml:18` | "It used to be duplicated here almost verbatim" | historical | **delete**, or replace with pointer `decisions/2026-10-18-keycloak-passkey-reconciler.md` (confirmed present) | no |
Also seen in `platform/` and left unlisted for the cap, same shapes: `platform/cilium/helmrelease.yaml:57-65`
(load-bearing — the `k3s_api_host`/`k3s_apiserver_endpoint` split is the current parity contract),
`platform/kubevirt/kubevirt-cr.yaml:7` (`useEmulation was previously true`), `platform/external-snapshotter/kustomization.yaml:35`
(`Previously these were REMOTE`), `platform/kustomization.yaml:7` (`the same pattern by which kyverno was previously deferred`),
`platform/pm-agent/configmap.yaml:22`, `platform/monitoring/tests/run.sh:61` and `:129`,
`platform/cnpg/alerts.promtool-test.yaml:20`, `platform/oauth2-proxy/auth-visibility-alerts.yaml:36-42`
and `:98-101`, `platform/vpn-ingress/daemonset.yaml:165`.
### `infra/` — 75 candidates, 8 listed (sampled). NOT sensitive, except `infra/gitea/**`
| # | path:lines | comment (gist) | class | action | sensitive |
|---|---|---|---|---|---|
| 10 | `infra/plex/claim-ship.sh:29-32` | "Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered feature/** … Those per-branch push triggers were deleted" | historical | **delete.** The live trigger set is already documented at `.gitea/workflows/gate.yml:48-56` and `CLAUDE.md` §4 | no |
| 11 | `infra/ansible/inventory/hosts.ini:105` | "GROUP-NAME GAP: CLOSED 2026-08-31. The roles and playbooks used to hardcode the…" | historical | **delete** — the gap is closed and the current variable is named at `playbooks/node-health.yml:12` | no |
| 12 | `infra/ansible/inventory/hosts.ini:147-153` | "NOTE (CORRECTED 2026-09-12): this used to read 'node 3 also has wlp1s0 up on 192.168.2.195 (wifi)'. It does not" | load-bearing | **rewrite** as a positive present-tense assertion: node 3 has no `wl*` netdev and a single IPv4 uplink on `eno1`. That fact is load-bearing for the kill-switch datapath; only the correction framing should go | no |
| 13 | `infra/ansible/roles/node_health/defaults/main.yml:110-121` | "The first real run of this role FAILED k3s-prod-1's Crucial/Micron C400 on `reallocated=61440` … That verdict was WRONG" | load-bearing | **rewrite.** Keep the current rule (SMART raw values are vendor-defined; Micron TN-FD-20 scales ID 5 by `Block_Sector_Count`, so ID 170/196 are the trustworthy corroborators). Drop "the first real run failed" | no |
| 14 | `infra/ansible/roles/unattended_upgrades/tasks/main.yml:304-309` | "This was originally a `find` over /etc/kernel/postinst.d … That theory was wrong" | load-bearing | **rewrite** to the current justification: the sentinel writer is ours at two known paths, and two exact `stat`s are strictly stronger than a glob that would match Debian's own conffile | no |
| 15 | `infra/ansible/roles/node_health/tasks/assess.yml:398` | "measurement was right; the inference from the log line was wrong — the memattr block below…" | historical | **delete** | no |
| 16 | `infra/devcontainer/dev.sh:103-108` | "SUPERSEDED but still honoured: `DEV_KUBECONFIG` used to be the only way to reach a cluster other…" | load-bearing | **rewrite.** "Still honoured" is current behaviour and `CLAUDE.md` §3 tells every agent to set `DEV_KUBECONFIG=/dev/null`, so this must keep documenting the variable — just not as history | no |
| 17 | `infra/vpn-ingress/new-peer.sh:136` | "COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the…" | load-bearing | **rewrite**, replacing the history with pointer `decisions/2026-10-06-new-peer-ignores-template-comments.md` (confirmed present) | no |
Unlisted, same shapes: `infra/ansible/inventory/group_vars/all.yml:31`,
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml:86` ("CORRECTION 2026-09-06: an earlier
version of this comment claimed…"), `infra/ansible/roles/node_baseline/tasks/disks.yml:559`,
`infra/ansible/roles/node_baseline/tasks/main.yml:801`, `infra/ansible/roles/kube_vip/tasks/main.yml:68`,
`infra/ansible/roles/cilium/defaults/main.yml:30`, `infra/ansible/playbooks/node-health.yml:12`,
`infra/ansible/roles/vpn_egress_gateway/templates/vpn-egress-up.sh.j2:39` and
`vpn-egress-down.sh.j2:15`, `infra/devcontainer/kubeconfig.sh:26`, `infra/agent/Containerfile:216`,
`infra/agent/build-image.sh:30`, `infra/agent/pm-worker:97`, `infra/agent/probe-validation.sh:98`.
### `catalog/` — 39 candidates, 7 listed (sampled). NOT sensitive — but every edit here re-renders `apps/wave-*`
| # | path:lines | comment (gist) | class | action | sensitive |
|---|---|---|---|---|---|
| 18 | `catalog/qbittorrent.yaml:7-12` | "BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record). This entry used to ship config-only, with a comment promising … 'once OQ-2 is resolved'" | historical | **delete.** Self-describes as "kept for the record", which is precisely the issue's complaint | no |
| 19 | `catalog/qbittorrent.yaml:141-151` | "It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var … That MISDIAGNOSIS is preserved in this service's own marker log at 2026-09-13T05:35Z" | load-bearing | **rewrite.** Keep "PORT is a literal matching `ingressRoutes[0].port`; a literal cannot be mangled by envsubst". Drop the misdiagnosis story (it is already preserved in the marker log the comment itself cites) | no |
| 20 | `catalog/plex.yaml:43-46` | "(This comment previously asserted '~4 minutes' as established fact. It was … Measured 2026-09-29: the full loop is ~2m45s and fits.)" | historical | **rewrite** to the measured figure alone; delete the self-correction parenthesis | no |
| 21 | `catalog/plex.yaml:65` | "So the earlier 'a placeholder claim Secret would crashloop the pod' note was WRONG about the…" | historical | **delete** | no |
| 22 | `catalog/plex.yaml:318` | "actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep" | historical | **rewrite** with pointer `decisions/2026-10-01-quicksync-via-device-plugin.md` and/or `decisions/2026-10-03-cilium-cgroup-hostroot.md` (both confirmed present) | no |
| 23 | `catalog/gitea.yaml:14` | "Headless-install bootstrap (formerly `apps/gitea-bootstrap-patch.yaml`; now first-class via the…)" | historical | **delete** the "formerly" clause; keep the description of what the block does | no |
| 24 | `catalog/render.py:159-166` | "THE SCRAPE WORKS AS OF 2026-10-05 — but it took THREE independent pieces … (PR #28) … (PR #49)" | load-bearing | **rewrite.** The three-part requirement and "do not read this flag alone as 'metrics are working'" are current and genuinely non-obvious; drop the PR numbers and the "63.7h" incident, pointing at `decisions/2026-10-05-oauth2-sidecar-scrape-verified.md` (confirmed present) | no |
`catalog/pihole.yaml:210` and `:218` are the **exemplar to copy**: narrative already reduced to
`see decisions/2026-10-04-pihole-multi-replica.md` / `-pihole-declare-state.md`, both of which
exist. Unlisted: `catalog/gitea.yaml:100-130`, `catalog/nextcloud.yaml:211` and `:226-240`,
`catalog/semaphore.yaml:84`, `catalog/pihole.yaml:179`, `catalog/qbittorrent.yaml:389-417`,
`catalog/sonarr.yaml:120`, `catalog/render.py:672`, `:1371`, `:2161`.
`catalog/jackett.yaml:73` ("floor only, no ceiling; 0.24.x runs as of 2026-10-30") is
**load-bearing, leave alone** — a dated version constraint, exactly the carve-out the brief names.
### `.gitea/` — 10 candidates, 5 listed (FULL coverage). **SENSITIVE** (`.gitea/**`)
| # | path:lines | comment (gist) | class | action | sensitive |
|---|---|---|---|---|---|
| 25 | `.gitea/workflows/gate.yml:14-16` | "(it used to print '— not green' and then 'MERGE GATE: PASS', exit 0 — fail-open in the gate itself)" | historical | **delete** the parenthesis; the surrounding `GATE_STRICT_SKIPS` rationale is current and stays | **yes** |
| 26 | `.gitea/workflows/gate.yml:48-56` | "There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/** …) and it was a net negative here" + the 2026-09-29 plex-claim cost | load-bearing | **rewrite.** "Do not re-add a push trigger; `act_runner` is capacity 1 and the duplicate run serialises ahead of the PR run" is a live constraint a future editor needs. Drop the plex-claim anecdote | **yes** |
| 27 | `.gitea/workflows/gate.yml:118-123` | "WIDENED TOLERANCE, DECLARED: `\|\| [ "$?" -eq 3 ]` used to cover exactly one skip (spec-immutability)" | load-bearing | **rewrite.** This documents a deliberate *current* weakening plus its exit condition (`REQUIRE_GITEA_SETTINGS=true` once the secret is seen green). Keep the instruction, drop "used to" | **yes** |
| 28 | `.gitea/workflows/renovate.yaml:72-76` | "CORRECTED 2026-10-30. This list used to name only write:repository + read:user … Run 350 then failed at `GET /api/v1/orgs/… 403`" | load-bearing | **rewrite** to the required scope list as a plain present-tense requirement; point at `decisions/2026-10-30-renovate-auth.md` (confirmed present) for the run-350 story | **yes** |
| 29 | `.gitea/workflows/merge-gate.yaml:15` | "The text that used to sit here…" | historical | **delete** | **yes** |
Remaining `.gitea/` candidates examined and judged **not findings** (current-state
measurements, correctly phrased): `renovate.yaml:6`, `:34`, `:259`, `images.yaml:44`,
`scripts/images.sh:11`.
### `ci/` — 42 candidates, 6 listed (sampled). **SENSITIVE** (`ci/**`)
| # | path:lines | comment (gist) | class | action | sensitive |
|---|---|---|---|---|---|
| 30 | `ci/decisions-log.sh:5-23` | 19-line "WHY THIS STAGE EXISTS" block reproducing the four-PR conflict table verbatim | historical | **replace with a pointer.** It is a near-verbatim duplicate of `decisions/README.md:9-30`, which the comment itself already cites at `:23`. Cut to the assertion list + that pointer (and `decisions/2026-09-29-decisions-log-conflict-free.md`, confirmed present) | **yes** |
| 31 | `ci/gitea-settings.sh:7-23` | "On 2026-09-29 that blind spot was measured … PR #1 … sat `pending` for 30.6 minutes … PRs #5-#9 merged with zero approvals" | load-bearing | **rewrite.** "No other check can see Gitea's own database; assert-scripts do not detect drift from their own target" is the current reason the stage exists. Drop the PR-by-PR history | **yes** |
| 32 | `ci/l0.sh:15-23` | "That union replaces what used to be a hand-written 'UNCONDITIONAL' stage per whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e bulk-tier)" | historical | **rewrite** to describe the present selection rule only (touched-requirement subset ∪ every test declaring `scope: whole-repo`) | **yes** |
| 33 | `ci/lib.sh:138-142` | "WHY THIS EXISTS: `ci/l0.sh` used to carry a hand-written stage per whole-repo check, each with its own `summary …` line" | historical | **rewrite** to "names every offline check that ran and its verdict" | **yes** |
| 34 | `ci/approval-policy.sh:107-112` | "Blocking on that turned 'CI cannot read your override' into a message that said 'you did not post one', and cost a real debugging session on PR #38" | load-bearing | **rewrite.** The token-scope fact and the fail-direction guarantee ("an unreadable source can only cause a false BLOCK, never a false pass") are current; the debugging-session cost is not | **yes** |
| 35 | `ci/touched-requirements.sh:288-293` | "R-TEST-06 added 2026-09-12. The comment in `changed_diff()`'s rule 3 above records why it was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE" | load-bearing | **rewrite.** Keep why a `ci/` change must attribute R-TEST-06 today; drop the absence history | **yes** |
Unlisted: `ci/approval-policy.sh:30-38` and `:182`, `ci/conformance.sh:73`, `ci/gate.sh:133`,
`:233-234`, `ci/l0.sh:86`, `ci/l1.sh:33`, `ci/lib.sh:65`, `ci/render-golden.sh:5-6` and `:24`,
`ci/touched-requirements.sh:39`, `:316`, plus `ci/tests/*.test.sh` (5 files — note these are
test *rationale* headers, where a "this used to fail on main" line arguably earns its keep;
I would send the whole `ci/tests/` group to `ambiguous`).
### `clusters/` — 43 candidates, 7 listed (sampled). **SENSITIVE** (`clusters/**`)
| # | path:lines | comment (gist) | class | action | sensitive |
|---|---|---|---|---|---|
| 36 | `clusters/base/cluster-vars.yaml:244-251` | "IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read 'the two clusters are on OPPOSITE SIDES of pfSense'…" | load-bearing | **rewrite.** Keep "exactly one storage host, 192.168.2.30; still a knob only because the renderer reads it per-environment" and the immutability warning at `:253+`. Replace the pfSense/DNAT history with pointer `decisions/2026-10-07-bulk-nfs-truenas.md` (confirmed present) | **yes** |
| 37 | `clusters/prod/kustomization.yaml:335-351` | "APEX AS OF 2026-09-13 … Prerequisites completed and VERIFIED before this flip, in this order: 1. dev's 18 A records released … 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted by hand … 3. dev's Certificate deleted" | historical | **rewrite.** The three completed prerequisites are done and unrepeatable. Keep only the still-true `NOTE` at `:348-351` (the zone's two wildcards point at the legacy stack's public IP and prod's specific records shadow them) | **yes** |
| 38 | `clusters/prod/kustomization.yaml:472-479` | "*** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously read 'Not yet decided' while shipping .10 …)" | load-bearing | **rewrite.** Keep "this is the control-plane VIP, NOT .20 the ingress VIP and NOT .10 the single node address" and why a node-pinned CNI endpoint fails silently. Drop the previous-wording parenthesis | **yes** |
| 39 | `clusters/base/apps-vms.yaml:51-55` | "postBuild.substituteFrom ENABLED 2026-08-31 (R-ENV-02). It was previously omitted because Flux's envsubst runs over the ENTIRE manifest…" | load-bearing | **rewrite.** The envsubst hazard over VM cloud-init and the `$${...}` escape obligation are live constraints for anyone adding a VM manifest; the enablement date and prior omission are not | **yes** |
| 40 | `clusters/dev/kustomization.yaml:282-287` | "THIS VALUE IS INERT — dev is decommissioned, its nodes no longer exist, and the Proxmox host this used to name (192.168.1.9) went with the pools it held" | load-bearing | **keep, trim.** "Inert, restated per this overlay's supply-every-knob convention, unproven because there is no dev node" is honest current state and the kind of comment the issue wants | **yes** |
| 41 | `clusters/base/apps-wave-0.yaml:4`, `apps-wave-1.yaml:4`, `apps-wave-2.yaml:4` | "wait:true health-gates ONLY this wave's resources, so a later wave can no longer stall an earlier one" (×3, and the same sentence is emitted by `catalog/render.py:2385`) | load-bearing | **rewrite** the "no longer" to plain present tense. Note the renderer emits this string, so fix `render.py:2385` and re-render rather than editing the three files by hand | **yes** |
| 42 | `clusters/base/headlamp.yaml:9-12`, `homepage.yaml:10`, `crafty.yaml:11`, `kustomization.yaml:53`, `runbooks.yaml:15`, `vpn-ingress.yaml:9` | six files citing `DECISIONS-LOG: "Headlamp gating stall"` — **target does not exist** (see above) | historical | **rewrite** to the present-tense rule (a leaf UI belongs in its own post-`platform-config` Kustomization so its crashloop cannot stall the chain) and drop or repoint the dead citation | **yes** |
Unlisted: `clusters/base/cilium.yaml:52`, `cluster-vars.yaml:29`, `:51`, `:264`,
`platform.yaml:17`, `:56`, `platform-config.yaml:6`, `platform-config-cdi.yaml:13`,
`oauth2-proxy.yaml:10`, `clusters/dev/kustomization.yaml:67`, `:81`, `:222`,
`clusters/dev/sync.yaml:8`, `clusters/prod/kustomization.yaml:277-283`, `:317`, `:363`, `:546`, `:554`.
### `conformance/` — 72 candidates (excl. `runs/`), 5 listed (sampled). **SENSITIVE** (`conformance/**`)
| # | path:lines | comment (gist) | class | action | sensitive |
|---|---|---|---|---|---|
| 43 | `conformance/tests/l3-fault-injection/vpn-killswitch/chainsaw-test.yaml:16-24` | "MECHANISM CORRECTED 2026-09-12: this comment used to credit 'the host nftables kill-switch' … The ASSERTION below is unchanged and still the right one — only the explanation was wrong" | load-bearing | **rewrite** to state the real mechanism (Cilium's eBPF egress-gateway drop, netfilter never invoked) in the present tense. Pairs with finding #8 — same correction duplicated in `platform/vpn-egress/` | **yes** |
| 44 | `conformance/tests/l3-fault-injection/vm-data-survival/chainsaw-test.yaml:18-27` | "(Feature-gate note, revalidated 2026-08: an earlier revision of this comment also claimed the containerDisk BOOT path was broken … That premise is now STALE: the cluster is k3s v1.36.4, where ImageVolume is ENABLED)" | historical | **rewrite** to the one current sentence that matters — DataVolume is used because only it survives a rebuild — and drop the whole stale-premise parenthesis | **yes** |
| 45 | `conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml:51-60` | "TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible)" | historical | **rewrite.** Both gaps are closed and now asserted; state the assertions, drop the gap history. Same in `check.sh:98` and `:500` ("This section used to be an ESCALATION in the header's gap list") | **yes** |
| 46 | `conformance/lib/status.py:303-310` | "It used to be, and that mis-reported a SAFETY-CRITICAL requirement … A previous engineer worked around it by rewording prose" | load-bearing | **rewrite.** "Scaffold detection must name the idioms it hunts for, because a placeholder-detection test legitimately contains the words" is the current design constraint; the R-STATE-03 misreport and the previous engineer are not | **yes** |
| 47 | `conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh:447-455` | "It used to assert `… in L0`: ci/l0.sh carried a hand-written 'UNCONDITIONAL' stage per whole-repo check" — **the same paragraph is duplicated verbatim in 3 files** (also `flux-postbuild-substitution/check.sh:604`, `keycloak-client-secret-reconciler/check.sh:678`) | historical | **rewrite** all three to the present mechanism (the test declares `scope: whole-repo`; `ci/l0.sh` unions declaring tests via `--include-scope`; both halves checked). Good candidate for one small PR — identical edit ×3 | **yes** |
Unlisted: `conformance/incluster/chainsaw-job.yaml:44`, `conformance/incluster/rbac.yaml:105`
(**also sensitive as `**/rbac.yaml`** — "superseded design" for the gluetun Deployment;
load-bearing, since it explains why the fault Job needs no RBAC), `conformance/lib/common.sh:33`,
`conformance/lib/status.py:114`, and ~10 further `tests/l0-static/*/check.sh` headers of the
same shape (`alert-metrics-have-scrape-targets`, `alert-rules-unit-tested`,
`alert-runbooks-resolvable`, `auditability-proofs-producible`, `keycloak-passkey-reconciler`).
### `bootstrap/` — 4 candidates, 3 listed (FULL coverage). **SENSITIVE** (`bootstrap/**`)
| # | path:lines | comment (gist) | class | action | sensitive |
|---|---|---|---|---|---|
| 48 | `bootstrap/flux/components/kustomization.yaml:36-38` | "On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry)" | load-bearing | **rewrite.** The fail-open default and what the feature gate changes are the current justification; the incident is one clause. Pointer is vague ("INCIDENT entry") — `grep` finds 2026-09-01 material in the archive, so if a citation is kept it should name a locatable heading, not "INCIDENT entry" | **yes** |
| 49 | `bootstrap/flux/bootstrap.sh:84-86` | "see ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md" | historical | **rewrite** to keep only the cross-reference to `./components/kustomization.yaml`, which is checkable; drop the archive citation (duplicate of #48) | **yes** |
| 50 | `bootstrap/sops/secret-templates/pia-custom-ovpn.yaml.example:6` | `See DECISIONS-LOG "Phase 3 VPN — custom .ovpn"` | ambiguous | **operator call.** The target *does* exist (10 hits for "Phase 3 VPN" in the archive), so this is a working pointer of the discouraged form. Keep, or repoint if a `decisions/` entry is written. Cited by path+line only; no content quoted (secret-adjacent `.example` template) | no (`*.sops.yaml.example` is not matched by `**/*.sops.yaml`) |
Remaining candidate `bootstrap/flux/bootstrap.sh:122` ("a patch target that no longer exists
after a Flux…") examined — **not a finding**, it describes current failure behaviour.
### `platform-config/` — 3 candidates, 3 listed (FULL coverage). **SENSITIVE** (`platform-config/kustomization.yaml`)
| # | path:lines | comment (gist) | class | action | sensitive |
|---|---|---|---|---|---|
| 51 | `platform-config/kustomization.yaml:8` | "See `clusters/base/platform-config.yaml` + DECISIONS-LOG." | historical | **rewrite** to keep the checkable half (`clusters/base/platform-config.yaml`) and drop the bare, untargeted `DECISIONS-LOG` reference — it names no entry and cannot be resolved | **yes** |
| 52 | `platform-config/kustomization.yaml:93` | "Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05." | load-bearing | **rewrite.** "THE RULE" at `:95-96` is the keeper; the dev-invisibility observation is stale (dev is decommissioned — see finding #40) and should go | **yes** |
| 53 | `platform-config/kustomization.yaml:167` | "headlamp: httproute MOVED OUT … Keeping it here … meant a headlamp crashloop stalled platform-config. See DECISIONS-LOG 'Headlamp gating stall'." | historical | **rewrite** to the present-tense statement of where the HTTPRoute lives now; **dead pointer**, see the dangling-pointer section | **yes** |
### `apps/vms/` — 3 candidates, 2 listed (FULL coverage). Hand-authored, NOT generated
| # | path:lines | comment (gist) | class | action | sensitive |
|---|---|---|---|---|---|
| 54 | `apps/vms/unifi/50-service-l4.yaml:27-78` | "CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE … This file used to say `Local` … BOTH HALVES OF THAT WERE FALSE" — ~52 lines | load-bearing | **rewrite**, same treatment as #1 (its near-twin). Keep: L2 announcer is not backend-aware; KubeVirt `masquerade` already destroyed the client IP so `Cluster` costs nothing here; `cluster` subsumes `remote-node` so do NOT "fix" the CNP; no `loadBalancerSourceRanges` deliberately. Drop the outage story | no |
| 55 | `apps/vms/unifi/30-virtualmachine.yaml:119`, `40-networkpolicy.yaml:36` | `# legacy Network UI (still served)`, `# legacy UI` on port 8443 | load-bearing | **keep.** "legacy" here is the upstream product's own name for a port that is still served — it describes current behaviour, not history. Listed so a keyword-driven sweep does not delete it | no |
---
### Per-directory counts (findings listed above)
| Directory | historical | load-bearing | ambiguous | total | sensitive-path? |
|---|---|---|---|---|---|
| `platform/` | 2 | 7 | 0 | 9 | no (except `kyverno/**`) |
| `infra/` | 3 | 5 | 0 | 8 | no (except `gitea/**`) |
| `catalog/` | 4 | 3 | 0 | 7 | no |
| `clusters/` | 2 | 5 | 0 | 7 | **yes** |
| `ci/` | 3 | 3 | 0 | 6 | **yes** |
| `conformance/` | 3 | 2 | 0 | 5 | **yes** |
| `.gitea/` | 2 | 3 | 0 | 5 | **yes** |
| `bootstrap/` | 1 | 1 | 1 | 3 | **yes** (×2) |
| `platform-config/` | 2 | 1 | 0 | 3 | **yes** |
| `apps/vms/` | 0 | 2 | 0 | 2 | no |
| **total** | **22** | **32** | **1** | **55** | 29 of 55 sensitive |
The `ambiguous` bucket is thin because I could rule on most rows against the reporter's
calibration. Two groups I am **deliberately not ruling on** and want the operator's call:
1. **`ci/tests/*.test.sh` headers** (5 files). These say things like "this suite exists
because `ci/gate.sh` used to set BLOCK=1 for a FAILED stage but…" and "a stage was added
to gate.sh by PR #17, never added here, so 15 of 50 assertions failed on `main`". For a
*regression test*, the regression it guards is arguably the test's subject rather than
fluff. Ruling these `historical` would strip the only record of what each assertion is
for. **Operator to decide.**
2. **In-repo `README.md` prose** (`platform/vpn-egress/README.md:91`,
`platform/crafty/README.md:52`, `infra/devcontainer/README.md:126`,
`infra/gitea/README.md:253`). These are documentation, not source comments, so arguably
already in the place the issue wants narrative to live. I did not audit them as findings.
### Suggested order for the follow-up PRs — cheapest and least sensitive first
1. **`catalog/` + re-render `apps/wave-*`** (7 findings, not sensitive). Highest value per
unit of review: it also fixes the generated copies under `apps/wave-*`. Must re-run
`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`,
drop the not-onboarded outputs, and commit the goldens, or `render-golden` fails.
2. **`platform/` excluding `kyverno/**`** (9 findings, not sensitive). Biggest residual too
(107 candidates) — worth splitting by component (`monitoring/`, `longhorn/`, `crafty/`,
`headlamp/`, `vpn-egress/`) into 2–3 PRs rather than one sweep.
3. **`infra/` excluding `gitea/**`** (8 findings, not sensitive). `ansible/roles/*` is most
of it; `infra/plex/claim-ship.sh:29-32` (#10) is a clean one-paragraph deletion and a
good first commit.
4. **`apps/vms/`** (2 findings, not sensitive). Self-contained; #54 is the twin of #1, so do
it right after `platform/crafty/service-l4.yaml` while the reasoning is fresh.
5. **The dangling-pointer fix** (18 files, crosses sensitive and non-sensitive). Either
write one `decisions/` entry for the leaf-gating lesson and repoint all 18, or drop the
citations. Doing this *before* steps 6–9 means those PRs do not have to re-litigate it.
6. **`platform-config/kustomization.yaml`** (3 findings, **sensitive**). One file, three
edits — the smallest possible sensitive-path PR, so it is the cheapest review.
7. **`bootstrap/`** (3 findings, **sensitive**). Two files.
8. **`.gitea/`** (5 findings, **sensitive**). Keep #27 (`gate.yml:118-123`) in its own commit
— it documents a live, deliberate tolerance-widening and is the one most likely to be
misread as a behaviour change.
9. **`ci/`** then **`clusters/`** then **`conformance/`** (6 + 7 + 5, all **sensitive**,
largest residuals). Last, because each needs a non-author approving review or an operator
`/approve-override`, and because `ci/` edits touch the thing doing the approving.
A note on review cost: **29 of the 55 findings are on sensitive paths**, so roughly half
this work cannot merge on a green gate alone. Steps 1–4 (26 findings) are the portion that
can. That is the argument for the ordering above.
## What I verified
Everything below was run in this container, offline, against the tree at
`466f63e` on branch `docs/source-comment-audit`.
- **The tree layout, against the brief's believed list.** `git ls-files` → **855** tracked
files. The brief's list was broadly right but differs in three ways worth recording:
(a) there is **no `scripts/` and no `tools/` directory**; (b) `docs/` **does exist in this
repo** (9 files: `docs/BULK-NFS-STORAGE.md` + `docs/plans/*.md`) and is *not* the sibling
spec repo's `docs/` — I audited neither, since both are prose, not source comments;
(c) `platform-config-cdi/` exists as a separate top-level directory (1 file, 0 candidate
hits).
- **Candidate volume, per directory**, with the keyword sweep quoted verbatim above:
`platform/` 107 hits / 56 files, `infra/` 75 / 44, `conformance/` 80 / 40 (**72 excluding
the 8 in `conformance/runs/`**), `clusters/` 43 / 19, `ci/` 42 / 15, `catalog/` 39 / 9,
`apps/` 18 (**15 under the generated `apps/wave-*`, 3 under hand-authored `apps/vms/`**),
`runbooks/` 11, `.gitea/` 10 / 5, `bootstrap/` 4 / 3, `platform-config/` 3 / 1,
`platform-config-cdi/` 0. Total ≈ **420 candidate hits in ≈190 files**.
- **`apps/wave-*` is generated and byte-enforced.** Read `ci/render-golden.sh`: the canonical
command is at `:21`/`:120`, and the comparison loop at `:136-150` iterates the *rendered*
output, requiring each golden to exist and to be byte-identical, `exit 1` on any diff.
Confirmed the duplication concretely: the "HARDENED (was a silent first-boot RACE)"
paragraph appears at both `catalog/gitea.yaml:100` and `apps/wave-0/gitea.yaml:130`.
Confirmed `apps/vms/` is *not* produced by that loop (the renderer emits `wave-N/` paths
only), so it is hand-authored and directly editable.
- **The dangling pointer.** `grep -ci "gating stall" DECISIONS-LOG.md` → **0**;
`grep -ci "Headlamp gating stall" DECISIONS-LOG.md` → **0**;
`grep -ril "gating stall" decisions/` → **no output**; `git grep -il "gating stall"` →
**18 files**. So 18 files cite a rationale entry that exists in neither the frozen archive
nor `decisions/`.
- **Every `decisions/...md` pointer I recommend exists.** Checked against `ls decisions/`
(71 entries + `README.md` + `ARCHIVE.sha256`): `2026-10-18-keycloak-passkey-reconciler.md`,
`2026-10-06-new-peer-ignores-template-comments.md`, `2026-10-01-quicksync-via-device-plugin.md`,
`2026-10-03-cilium-cgroup-hostroot.md`, `2026-10-05-oauth2-sidecar-scrape-verified.md`,
`2026-10-07-bulk-nfs-truenas.md`, `2026-10-30-renovate-auth.md`,
`2026-09-29-decisions-log-conflict-free.md`, `2026-10-04-pihole-multi-replica.md`,
`2026-10-04-pihole-declare-state.md` — all present. I recommend **no pointer target I did
not list in `ls` output**.
- **Archive-citation targets**, for the rows where I say a `DECISIONS-LOG` pointer does or
does not resolve: `"Remote Path"` 5 hits, `"Phase 3 VPN"` 10, `"Phase 4 image-automation"` 1,
`F-DNS1` 3, `F-OPS1` 1, `2026-09-01` 6 — all present in the 5,733-line archive.
`"Headlamp gating stall"` 0, as above.
- **The sensitive-path column is from the policy data, not the brief.** Read
`infra/gitea/approval-policy.json`: `sensitive_paths` = `ci/**`, `.gitea/**`,
`conformance/**`, `clusters/**`, `bootstrap/**`, `infra/gitea/**`, `platform/kyverno/**`,
`platform-config/kustomization.yaml`, `.sops.yaml`, `**/*.sops.yaml`, `**/rbac.yaml`,
`CLAUDE.md`. This matches the brief, and adds the root `.sops.yaml` and the explicit
`**/` glob forms. Note `**/*.sops.yaml` does **not** match `*.sops.yaml.example`, which is
why findings #50 and the `platform/headlamp/...sops.yaml.example` row are marked not
sensitive.
- **Every cited `path:line-range` was opened and read**, not matched and trusted. All 55
rows quote text I read with `sed -n` at the line numbers given, at `466f63e`. Line numbers
are from that commit and this PR changes no source file, so they remain valid on merge.
- **Secret-adjacent files**: no `*.sops.yaml` content is quoted anywhere in this report, and
no `.data`/`.stringData` value or key name appears. The two `.sops.yaml.example` rows
(#50, and `platform/headlamp/headlamp-oidc-credentials.sops.yaml.example:15` in the
unlisted set) are cited by path and line only. I ran no `sops -d`.
- **The gate.** Result recorded in the PR; see "What I did NOT verify" for the stages that
cannot run in this container.
- **`git diff --stat` against `main`**: one added file, `decisions/2026-10-31-source-comment-audit.md`,
and nothing else — which is the brief's "done means" condition.
## What I did NOT verify
- **Anything about the live cluster. I hold no cluster credentials** — no kubeconfig and no
ServiceAccount token is mounted in this container, so `kubectl` cannot work here. This
audit is a pure text analysis of the tree and makes **no claim about the running
platform**. That is appropriate (no behaviour changes), but it also means every *factual*
assertion quoted inside the comments I audited — the measured Cilium lease holders, the
`longhorn_volume_robustness` series counts, the 192.168.2.30 NFS reachability, the k3s
version, the oauth2-proxy scrape targets — is **unverified by me**. I classified comments
by what they *talk about*, not by whether their measurements are still true. A rewrite PR
that restates one of those measurements as current fact should re-measure it first.
- **The two gate stages that cannot run in this container**, per the environment notes and
`CLAUDE.md` §3: `spec-immutability` (the sibling spec repo has no git remote and is not
reachable from the cluster) and `gitea-settings` (needs `SETTINGS_API_TOKEN`, which I
deliberately do not hold). So R-TEST-06 is **not** actually exercised on this run.
- **~365 of the ~420 candidate hits were not examined.** The per-directory table marks which
directories are full and which are sampled. For `platform/`, `infra/`, `conformance/`,
`clusters/`, `ci/` and `catalog/` the listed findings are **representative, not complete**,
and the `historical : load-bearing` ratio I report (22:32) is measured over the 55 rows I
judged, **not** over the whole corpus — do not project it onto the residual.
- **The keyword sweep is not a complete detector.** It finds English phrasings I thought of.
Historical narrative that avoids all of them — a dated war story with no "used to", a
stale rationale phrased entirely in the present tense, a comment that is simply wrong
about today — is **invisible to it**. The most dangerous category (a confidently
present-tense comment describing a superseded state) is precisely the one this method
cannot find. A complete audit needs a reader per directory, not a grep.
- **`runbooks/` (11 candidate hits), the repo-root markdown (`README.md`, `DESIGN-waves.md`),
and the in-repo `docs/` tree were not audited.** They are prose documentation rather than
source comments, so they fall outside the reporter's wording — but `CLAUDE.md` §1 already
records that `runbooks/` contains stale `git push origin main` instructions, so there is
known stale narrative there that this audit does not cover.
- **The sibling spec repo (`R-*` requirements, ADRs) was not read or assessed at all** — it
is not present in this container and is human-owned and immutable to agents (R-TEST-06).
Nothing here proposes a change to it. Where a comment cites a spec document
(e.g. `docs/06/R-STATE-01` in finding #44), I did **not** verify that the citation
resolves.
- **I did not verify that any recommended rewrite is correct, only that the current comment
is historical.** Each "rewrite" row names what I believe is the load-bearing residue; the
engineer doing the pruning must confirm that residue is still true before writing it as
present-tense fact. I wrote no replacement text.
- **Whether the operator agrees with my `historical` / `load-bearing` line.** I applied the
calibration from issue #97 literally plus the brief's one carve-out. The 22:32 split is a
judgement call, and the two groups in the `ambiguous` discussion above are explicitly
unresolved and need a human ruling before the `ci/tests/` and README rows are touched.
- **`conformance/runs/*/run.log` and `apps/wave-*` were excluded by my own judgement**, on
the structural grounds given (archived evidence; gate-enforced goldens). The operator has
not confirmed either exclusion.
- **Nothing here traces to an `R-*` requirement**, deliberately — this is housekeeping, per
the brief. I did not attempt to attribute one, and I did not check whether
`ci/touched-requirements.sh` expects an attribution for a `decisions/`-only change.
+94 -3
View File
@@ -214,6 +214,12 @@ class Config:
# ConfigMap without rebuilding the image. # ConfigMap without rebuilding the image.
self.push_on_red = env("PM_PUSH_ON_RED_GATE", "false") self.push_on_red = env("PM_PUSH_ON_RED_GATE", "false")
self.pm_timeout = env_int("PM_DECISION_TIMEOUT_SECONDS", 900) self.pm_timeout = env_int("PM_DECISION_TIMEOUT_SECONDS", 900)
# The PM holds no tools, so the only thing a second turn can be spent on is the CLI's
# structured-output step re-asking the model after an answer that failed the schema.
# 1 is the long-standing value; it is a knob so that, if the failure envelope now
# logged by describe_cli_failure() says `error_max_turns`, the remedy is a ConfigMap
# change rather than another image build.
self.pm_max_turns = env_int("PM_DECISION_MAX_TURNS", 1)
# Opt-in gate. Empty (the default) means every open issue is in scope, which is the # Opt-in gate. Empty (the default) means every open issue is in scope, which is the
# behaviour the service was asked for. Setting it to a label name — `agent` is the # behaviour the service was asked for. Setting it to a label name — `agent` is the
# conventional one — narrows the agent to issues a human has explicitly handed it, # conventional one — narrows the agent to issues a human has explicitly handed it,
@@ -231,6 +237,8 @@ class Config:
if self.max_active_workers < 1: if self.max_active_workers < 1:
die("PM_MAX_ACTIVE_WORKERS must be >= 1") die("PM_MAX_ACTIVE_WORKERS must be >= 1")
if not 1 <= self.pm_max_turns <= 5:
die("PM_DECISION_MAX_TURNS must be between 1 and 5")
if "@sha256:" not in self.worker_image: if "@sha256:" not in self.worker_image:
# The admission policy requires a digest too, so a tag here would produce a Job # The admission policy requires a digest too, so a tag here would produce a Job
# that is created and then DENIED — a failure visible only in the dispatcher's # that is created and then DENIED — a failure visible only in the dispatcher's
@@ -437,7 +445,7 @@ def ask_pm(cfg: Config, thread: str) -> dict:
"--system-prompt-file", cfg.persona_file, "--system-prompt-file", cfg.persona_file,
"--tools", "", "--tools", "",
"--permission-prompts", "none", "--permission-prompts", "none",
"--max-turns", "1", "--max-turns", str(cfg.pm_max_turns),
"--output-format", "json", "--output-format", "json",
"--json-schema", json.dumps(DECISION_SCHEMA), "--json-schema", json.dumps(DECISION_SCHEMA),
"--no-session-persistence", "--no-session-persistence",
@@ -445,10 +453,76 @@ def ask_pm(cfg: Config, thread: str) -> dict:
proc = subprocess.run(cmd, input=thread, capture_output=True, text=True, proc = subprocess.run(cmd, input=thread, capture_output=True, text=True,
timeout=cfg.pm_timeout) timeout=cfg.pm_timeout)
if proc.returncode != 0: if proc.returncode != 0:
raise RuntimeError(f"claude exited {proc.returncode}: {proc.stderr.strip()[:800]}") raise RuntimeError(describe_cli_failure(proc.returncode, proc.stdout, proc.stderr))
return extract_decision(proc.stdout) return extract_decision(proc.stdout)
# Anything that could be a credential in the CLI's output. The CLI does not normally echo its
# token, but an API error body or a proxy's HTML page is not something to bet a leak on. Both
# schemes are matched (the http one is the lesson of a scheme-specific redaction that leaked
# ~/.git-credentials), and `token ...` is Gitea's own header scheme.
_SECRET_PATTERNS = [
(re.compile(r"sk-ant-[A-Za-z0-9_\-]+"), "sk-ant-[REDACTED]"),
(re.compile(r"(?i)\b(bearer|token)(\s+|=|:\s*)[A-Za-z0-9._~+/=\-]{8,}"), r"\1\2[REDACTED]"),
(re.compile(r"(?i)(https?://)[^/\s:@]+:[^/\s@]+@"), r"\1[REDACTED]@"),
]
def redact(text: str) -> str:
for pat, repl in _SECRET_PATTERNS:
text = pat.sub(repl, text)
return text
def describe_cli_failure(rc: int, stdout: str, stderr: str, limit: int = 800) -> str:
"""Say WHY `claude --print` failed, in the dispatcher log, redacted.
WHY THIS EXISTS: with `--output-format json` the CLI reports its own failures — an API
error, a usage limit, `error_max_turns`, a schema the model could not satisfy — as the
JSON result envelope on STDOUT (`is_error`, `subtype`, `result`), and writes nothing to
stderr. Logging only stderr therefore produced `claude exited 1: ` with an empty message
on 2026-10-31, twice, with the cause sitting unread in stdout. So: the envelope's
diagnostic fields first, then the raw tails of both streams.
"""
parts = [f"claude exited {rc}"]
out = (stdout or "").strip()
env = None
if out:
try:
env = json.loads(out)
except ValueError:
# A stream of JSON lines, or a log line ahead of the envelope: the last line that
# parses as an object is the result.
for line in reversed(out.splitlines()):
try:
cand = json.loads(line)
except ValueError:
continue
if isinstance(cand, dict):
env = cand
break
if isinstance(env, dict):
fields = []
for key in ("subtype", "is_error", "api_error_status", "num_turns", "duration_ms",
"duration_api_ms", "total_cost_usd", "stop_reason"):
if key in env:
fields.append(f"{key}={env[key]!r}")
if fields:
parts.append("envelope: " + " ".join(fields))
for key in ("result", "error", "errors", "message"):
if env.get(key):
val = env[key] if isinstance(env[key], str) else json.dumps(env[key])
parts.append(f"{key}: {val[:limit]}")
if out and not isinstance(env, dict):
parts.append(f"stdout tail: {out[-limit:]}")
err_text = (stderr or "").strip()
if err_text:
parts.append(f"stderr tail: {err_text[-limit:]}")
if len(parts) == 1:
parts.append("(no output on stdout or stderr)")
return redact(" | ".join(parts))
def extract_decision(stdout: str) -> dict: def extract_decision(stdout: str) -> dict:
"""Pull the decision object out of whatever `claude --print` wrapped it in. """Pull the decision object out of whatever `claude --print` wrapped it in.
@@ -501,7 +575,7 @@ def extract_decision(stdout: str) -> dict:
obj = as_obj(c) obj = as_obj(c)
if obj and isinstance(obj.get("action"), str): if obj and isinstance(obj.get("action"), str):
return obj return obj
raise RuntimeError(f"no decision object in claude output: {stdout[:400]!r}") raise RuntimeError(redact(f"no decision object in claude output: {stdout[-800:]!r}"))
# --------------------------------------------------------------------------- thread # --------------------------------------------------------------------------- thread
@@ -988,6 +1062,23 @@ def selftest() -> int:
except RuntimeError: except RuntimeError:
check(True, f"bad output refused: {label}") check(True, f"bad output refused: {label}")
# -- a failed CLI run must say why, and must not leak ---------------------
# The 2026-10-31 failure: rc 1, empty stderr, the cause in the stdout envelope.
env_err = json.dumps({"type": "result", "subtype": "error_max_turns", "is_error": True,
"num_turns": 2, "result": "API Error: 429 rate_limit_error"})
msg = describe_cli_failure(1, env_err, "")
check("error_max_turns" in msg and "429" in msg,
"CLI failure: stdout envelope surfaced when stderr is empty")
msg = describe_cli_failure(1, "", "boom on stderr")
check("boom on stderr" in msg, "CLI failure: stderr tail surfaced")
msg = describe_cli_failure(1, "", "")
check("no output" in msg, "CLI failure: silence is named, not an empty message")
msg = describe_cli_failure(1, "plain text sk-ant-oat01-abcdefABCDEF_123 here",
"Authorization: Bearer abcdefghijklmnop "
"http://u:[email protected]/x")
check("sk-ant-oat01" not in msg and "abcdefghijklmnop" not in msg and ":pw@" not in msg,
"CLI failure: token, bearer and URL credential redacted")
# -- the committed Job template ----------------------------------------- # -- the committed Job template -----------------------------------------
tmpl =os.environ.get("PM_WORKER_JOB_TEMPLATE", "/etc/pm-agent/worker-job-template.yaml") tmpl =os.environ.get("PM_WORKER_JOB_TEMPLATE", "/etc/pm-agent/worker-job-template.yaml")
try: try:
+6
View File
@@ -250,6 +250,12 @@ five that matter:
| `PM_MAX_ACTIVE_WORKERS` | `2` | concurrent workers. RAM is the binding constraint (2Gi limit each) and the subscription seat's rate limits are shared with every other use of that seat | | `PM_MAX_ACTIVE_WORKERS` | `2` | concurrent workers. RAM is the binding constraint (2Gi limit each) and the subscription seat's rate limits are shared with every other use of that seat |
| `PM_MAX_NEW_WORKERS_PER_CYCLE` | `1` | drains a backlog one issue at a time instead of in one stampede the first minute the credentials work | | `PM_MAX_NEW_WORKERS_PER_CYCLE` | `1` | drains a backlog one issue at a time instead of in one stampede the first minute the credentials work |
| `PM_PUSH_ON_RED_GATE` | `false` | a change the gate refused is not pushed; the issue gets the gate log instead | | `PM_PUSH_ON_RED_GATE` | `false` | a change the gate refused is not pushed; the issue gets the gate log instead |
| `PM_DECISION_MAX_TURNS` | `1` (unset) | turns the triage call may take (1–5). The PM holds no tools, so a second turn is only ever the CLI re-asking after an answer that failed the decision schema. Raise it if a triage error logs `subtype='error_max_turns'` |
**When a triage fails**, the `[error] issue #N: claude exited …` line carries the CLI's result
envelope (`subtype`, `is_error`, `api_error_status`, `num_turns`, `result`) and the tails of
stdout and stderr, redacted. Before 2026-10-31 it logged stderr only, which is empty under
`--output-format json` — see `decisions/2026-10-31-pm-agent-triage-exit.md`.
Changing a knob requires bumping the `config-generation` annotation in `deployment.yaml` in Changing a knob requires bumping the `config-generation` annotation in `deployment.yaml` in
the same commit — the process reads its configuration once, at startup, so without the bump the same commit — the process reads its configuration once, at startup, so without the bump