home-network-k3s-impl/decisions/2026-10-31-pm-agent-triage-exit.md

date, branch, title

date	branch	title
2026-10-31	fix/pm-agent-triage-exit	pm-agent triage fails with "claude exited 1:" and no message; log the CLI's stdout envelope so the cause is visible

What changed

• infra/agent/pm-dispatch.py, ask_pm(): on a non-zero exit the error now comes from a new describe_cli_failure(), not from proc.stderr alone. With --output-format json the CLI reports its own failures as the result envelope on stdout and writes nothing to stderr, which is why both failures today logged claude exited 1: with an empty message. The new line carries the envelope's subtype, is_error, api_error_status, num_turns, duration_ms, total_cost_usd, stop_reason, its result/error text, and the tails of stdout and stderr. It names silence explicitly when both streams are empty.
• redact(): everything in that message, and in extract_decision()'s "no decision object" error, passes through it. It covers sk-ant-…, Bearer/token values, and user:pass@ in both http:// and https:// URLs.
• PM_DECISION_MAX_TURNS: a new knob. It defaults to 1, today's value, so behaviour is unchanged, and it is bounded to 1–5. If the newly visible envelope says error_max_turns, the remedy is a ConfigMap change rather than another image build. Documented in platform/pm-agent/README.md.
• Selftest: four new assertions. They cover an empty stderr with the cause on stdout, a stderr-only failure, total silence, and redaction.

What I verified

• Timeline (Loki, {namespace="pm-agent"}, and kubectl logs):
  • 16:37:38 to 16:38:00: pod pm-agent-898fd8f5f-dp5hs, image c4eba5b3, config-generation 2, dry run. It triaged #97 in 22 s, logged PM says comment, and posted the 16:38:00Z comment. The thread then held only the issue body.
  • 16:43:09 to 16:44:00: pod pm-agent-f7c9b7fcf-j8mzn, image df36d9b7, config-generation 3, still dry run. It logged claude exited 1: with an empty message after 51 s.
  • 16:55:42 to 16:56:36: pod pm-agent-7796db8bcb-p5dsm, df36d9b7, config-generation 4, live. The same empty failure, after 54 s.
  • The only successful triage ever was the 16:38 one. Every earlier 898fd8f5f/5db55d558c log line is a startup crash (Gitea GET /user 401), not a triage.
• Going live is NOT the cause. The first failure happened in dry run. ask_pm() has no dry-run branch: the prompt, flags, tools (--tools "") and cwd are identical in both modes. PM_DRY_RUN gates only create_job(), which runs after the model has answered.
• Ruled out, with evidence:
  • Timeout: 51 to 54 s against a 900 s limit, and a timeout raises TimeoutExpired, not rc 1.
  • Budget cap: the PM call passes no --max-budget-usd.
  • Prompt size: the thread is the issue body (293 chars) plus three comments (1871 + 404 + 77 chars), and the persona is 8 KB.
  • Writable HOME: HOME=/workspace (emptyDir), and CLAUDE_CONFIG_DIR=/workspace/.claude was written at 16:56.
  • Egress: a TLS 1.3 handshake to api.anthropic.com:443 from the dispatcher pod succeeds in 0.06 s.
  • Model name and CLI version: the same claude-opus-5 and CLI 2.1.285 succeeded at 16:38. Between c4eba5b and df36d9b, infra/agent/ differs only by a template selftest assertion and the toolbox base digest.
• A hint, not proof: /workspace/.claude/.claude.json in the live pod has cachedExtraUsageDisabledReason: 'out_of_credits'. No token was read or printed.
• Offline selftest: PM_SELFTEST=1 against the worktree's template and persona prints selftest OK, including the 4 new assertions.
• Toolbox gate: see the PR description for the rc and the stages that were skipped.

What I did NOT verify

• The root cause. UNVERIFIED. The CLI's own message was discarded, --no-session-persistence leaves no transcript, and I did not run claude in the pod. The candidates still open:
  • Usage or rate limit on the subscription seat. The seat is shared with the operator's interactive use. The out_of_credits extra-usage flag fits this. So does a ~50 s run, which matches the CLI exhausting its 429/529 retries.
  • error_max_turns. With --max-turns 1 and --json-schema, an answer that fails the schema cannot be retried. The failing thread now invites implement (a branch plus a long brief), where the 16:38 one produced a short comment. ~50 s is about two generations of the 22 s success.
• The 17:11 retry will NOT reveal the cause. It runs the old image, which still discards stdout. The cause becomes visible only after this PR merges, CI re-publishes agent-runner, and a follow-up PR bumps the dispatcher digest (deployment.yaml and the PM_WORKER_IMAGE in configmap.yaml, plus a config-generation bump).
• That the CLI's error envelope carries exactly these key names. The parser takes whichever keys are present and falls back to raw tails.