Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
664ee82b9a | ||
|
|
95e08da2f0 | ||
|
|
0d3e51acc4 | ||
|
|
11267409d7 | ||
|
|
6a192ede2a | ||
|
|
06d4a9a1fc |
No files matched your search
@@ -0,0 +1,55 @@
|
|||||||
|
---
|
||||||
|
date: 2026-10-31
|
||||||
|
branch: chore/pm-agent-go-live
|
||||||
|
title: Take pm-agent out of dry run (operator-approved); label scope and worker caps unchanged
|
||||||
|
---
|
||||||
|
|
||||||
|
## What changed
|
||||||
|
|
||||||
|
- `platform/pm-agent/configmap.yaml`: `PM_DRY_RUN` `"true"` -> `"false"`. The operator approved
|
||||||
|
this after reading the dry-run triage of issue #97. `PM_REQUIRE_LABEL: agent`,
|
||||||
|
`PM_MAX_ACTIVE_WORKERS: 2`, `PM_MAX_NEW_WORKERS_PER_CYCLE: 1`, the worker image digest and every
|
||||||
|
other knob are unchanged. The two comments that said dry run was the only mode that worked
|
||||||
|
(because the run-340 digest lacked `imagePullSecrets`) now describe the run-383 digest that
|
||||||
|
PR #96 pinned.
|
||||||
|
- `platform/pm-agent/deployment.yaml`: `config-generation` `"3"` -> `"4"`. Without the bump the
|
||||||
|
dispatcher keeps its startup config and stays in dry run.
|
||||||
|
- `platform/pm-agent/README.md` and `docs/plans/pm-agent-issue-to-pr.md` (follow-up 3): the
|
||||||
|
"currently dry run" status text now says dry run is off. `runbooks/pm-agent.md` is unchanged.
|
||||||
|
Its "stop it: set `PM_DRY_RUN: "true"`" step and its admission-refusal troubleshooting row
|
||||||
|
are still accurate.
|
||||||
|
|
||||||
|
## What I verified
|
||||||
|
|
||||||
|
Read-only, `kubectl --context k3s-prod` through the toolbox, 2026-10-31 ~11:45 local, with Flux
|
||||||
|
`pm-agent` and `platform-config` both at `main@1126740`:
|
||||||
|
|
||||||
|
- ClusterPolicy `pm-agent-job-shape`: `Ready=True`, `validationFailureAction=Enforce`,
|
||||||
|
`background=false`. It has seven rules, including `worker-pull-secret-must-be-the-committed-one`.
|
||||||
|
- RBAC: ServiceAccounts `pm-agent` and `pm-agent-worker`, Role `pm-agent` and RoleBinding
|
||||||
|
`pm-agent` (roleRef `pm-agent`, subject `ServiceAccount/pm-agent`) all exist. `auth can-i`
|
||||||
|
as `system:serviceaccount:pm-agent:pm-agent` returned `yes` for `create jobs.batch` and `no`
|
||||||
|
for `delete jobs.batch`.
|
||||||
|
- Secrets (key names only): `pm-agent-registry` is `kubernetes.io/dockerconfigjson` with key
|
||||||
|
`.dockerconfigjson`. `pm-agent-credentials` is `Opaque` with keys `claude-code-oauth-token`
|
||||||
|
and `gitea-token`.
|
||||||
|
- Egress: `default-deny` NetworkPolicy exists. CiliumNetworkPolicies `pm-agent-allow-egress`
|
||||||
|
and `pm-agent-worker-allow-egress` both report `Valid=True`.
|
||||||
|
- Deployment `pm-agent` is 1/1 on `agent-runner@sha256:df36d9b7…` with
|
||||||
|
`config-generation: "3"` live. No Jobs exist in the namespace.
|
||||||
|
- `platform/pm-agent/{configmap,deployment}.yaml`, README and docs do not match any
|
||||||
|
`sensitive_paths` glob in `infra/gitea/approval-policy.json`, so no review or override applies.
|
||||||
|
|
||||||
|
## What I did NOT verify
|
||||||
|
|
||||||
|
- **That a real worker Job is admitted and completes.** Nothing can show this until a labelled
|
||||||
|
issue gets a `start` decision after the merge. The L0 Kyverno suite shows the policy accepts
|
||||||
|
the committed template. Whether the API server admits the Job built from the df36d9b image's
|
||||||
|
baked template has not been tested live.
|
||||||
|
- That the worker pod can pull its image with `pm-agent-registry`, reach Gitea and the model
|
||||||
|
through its egress policy, and push a branch and open a PR. None of this has run before.
|
||||||
|
- The pod list also shows a stale `pm-agent-65db599649-v678g` (25d, `ContainerStatusUnknown`,
|
||||||
|
from an old ReplicaSet). It does not block anything, but I did not investigate it. Clearing
|
||||||
|
it is an operator action.
|
||||||
|
- CI: see the PR for the gate result. Locally the best possible result is rc 3, with only
|
||||||
|
`gitea-settings` not run.
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
---
|
||||||
|
date: 2026-10-31
|
||||||
|
branch: chore/pm-agent-runner-digest
|
||||||
|
title: Bump pm-agent to the run-383 agent-runner digest by hand; Renovate saw it but the schedule withheld it
|
||||||
|
---
|
||||||
|
|
||||||
|
## What changed
|
||||||
|
|
||||||
|
`platform/pm-agent/deployment.yaml` (the dispatcher `image:`) and `platform/pm-agent/configmap.yaml`
|
||||||
|
(`PM_WORKER_IMAGE`) move:
|
||||||
|
|
||||||
|
- from `agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`, the
|
||||||
|
run-340 publish;
|
||||||
|
- to `agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d`, the
|
||||||
|
images run 383 publish of ddd04d4.
|
||||||
|
|
||||||
|
Both `# renovate-digest:` markers are unchanged, directly above their values.
|
||||||
|
|
||||||
|
`pm-agent.homelab.keithjackson.dev/config-generation` goes `"2"` → `"3"`. That is the convention in
|
||||||
|
`deployment.yaml`'s own comment: "bump it in the same commit as a change to configmap.yaml". The
|
||||||
|
image change alone would roll the pod, but the convention is followed regardless.
|
||||||
|
|
||||||
|
No `*.sops.yaml` file is touched. Branched from 06d4a9a, which already contains the operator's #95.
|
||||||
|
|
||||||
|
### Why Renovate did not do this: the schedule, not a config defect
|
||||||
|
|
||||||
|
- **Run 385** (dispatch, 2026-10-31 16:14 UTC, on ddd04d4) was a **real** run. It logged
|
||||||
|
`Deleting orphan branch (… renovate/pin-dependencies)`, not a `DRY-RUN:` line. It ran at **12:14
|
||||||
|
America/New_York**, outside `"schedule": ["after 3am and before 6am"]`, so it created and updated
|
||||||
|
no branch.
|
||||||
|
- **The lookup DID see the new digest.** Dependency Dashboard issue #91 lists, under **"Awaiting
|
||||||
|
Schedule"**:
|
||||||
|
`chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to df36d9b`.
|
||||||
|
- **Nothing else blocked it.** There are no open Renovate PRs and no `renovate/*` branch on Gitea
|
||||||
|
(111 branches listed across all pages). The previous agent-runner PR, #90 (to c4eba5b), was
|
||||||
|
auto-merged by renovate-bot at 08:22 on 10-31. No PR limit was hit, and digests are not
|
||||||
|
dashboard-approval gated.
|
||||||
|
- **The registry, read-only, with Docker/OCI `Accept` headers:**
|
||||||
|
- `HEAD …/agent-runner/manifests/main` → `docker-content-digest: sha256:df36d9b7…771d`
|
||||||
|
- `HEAD …/agent-runner/manifests/sha-ddd04d460ef6` → the same digest.
|
||||||
|
|
||||||
|
**Without this PR** the next scheduled run (08:15 UTC, 04:15 EDT) would open it, and it would
|
||||||
|
auto-merge as a `digest` update. Ticking that "Awaiting Schedule" checkbox on #91 makes the next
|
||||||
|
run of any kind create it out of schedule.
|
||||||
|
|
||||||
|
### Found on the way: Renovate's labels have never been applied
|
||||||
|
|
||||||
|
- The repo has exactly one label, `agent`. The org has none.
|
||||||
|
- Every Renovate PR so far (#89, #90, #92, #93) was created with `"labels": []`.
|
||||||
|
- On Gitea, Renovate drops a label that does not exist in the repo. So `needs-review`,
|
||||||
|
`blast-radius`, `range-widen`, `own-image`, `dependencies`, `automated-update`, `breaking` and
|
||||||
|
`ci-tooling` are never attached.
|
||||||
|
- The **automerge flags still work.** Only the labels, the visible review signal, are missing.
|
||||||
|
- **Operator fix:** create those labels on the repo. This was not changed here.
|
||||||
|
|
||||||
|
## What I verified
|
||||||
|
|
||||||
|
- **The new image carries the fix.** I fetched both images' layers read-only from the registry and
|
||||||
|
read `/etc/pm-agent/worker-job-template.yaml` from each (`infra/agent/Containerfile:116` copies
|
||||||
|
it there):
|
||||||
|
- df36d9b's has `imagePullSecrets:` / `- name: pm-agent-registry`;
|
||||||
|
- c4eba5b's has no `imagePullSecrets` at all.
|
||||||
|
|
||||||
|
This matches `infra/agent/worker-job-template.yaml:76-77` at ddd04d4.
|
||||||
|
- **`main` = `sha-ddd04d460ef6` = df36d9b**, both measured.
|
||||||
|
- **No other file references c4eba5b.** No admission policy pins it.
|
||||||
|
- **No override is needed.** `infra/gitea/approval-policy.json` lists only `platform/kyverno/**`
|
||||||
|
under `platform/`, so `platform/pm-agent/**` is not sensitive.
|
||||||
|
- I read the whole job logs of runs 362, 367 and 385.
|
||||||
|
|
||||||
|
## What I did NOT verify
|
||||||
|
|
||||||
|
- **That the rolled dispatcher starts on df36d9b and that a worker Job is admitted.** That needs
|
||||||
|
Flux to apply this and a real dispatch. Both are operator actions.
|
||||||
|
- **That the `pm-agent-registry` Secret exists and decrypts in-cluster** (#95's sops file). I did not
|
||||||
|
read it.
|
||||||
|
- **That Renovate will treat this hand bump as current.** It should: its next lookup finds
|
||||||
|
`main` == the pinned digest, so it proposes nothing and drops the dashboard entry. Not observed.
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
---
|
||||||
|
date: 2026-10-31
|
||||||
|
branch: fix/pm-agent-triage-exit
|
||||||
|
title: pm-agent triage fails with "claude exited 1:" and no message; log the CLI's stdout envelope so the cause is visible
|
||||||
|
---
|
||||||
|
|
||||||
|
## What changed
|
||||||
|
|
||||||
|
- `infra/agent/pm-dispatch.py`, `ask_pm()`: on a non-zero exit the error now comes from a new
|
||||||
|
`describe_cli_failure()`, not from `proc.stderr` alone. With `--output-format json` the CLI
|
||||||
|
reports its own failures as the result envelope on **stdout** and writes nothing to stderr,
|
||||||
|
which is why both failures today logged `claude exited 1: ` with an empty message. The new
|
||||||
|
line carries the envelope's `subtype`, `is_error`, `api_error_status`, `num_turns`,
|
||||||
|
`duration_ms`, `total_cost_usd`, `stop_reason`, its `result`/`error` text, and the tails of
|
||||||
|
stdout and stderr. It names silence explicitly when both streams are empty.
|
||||||
|
- `redact()`: everything in that message, and in `extract_decision()`'s "no decision object"
|
||||||
|
error, passes through it. It covers `sk-ant-…`, `Bearer`/`token` values, and `user:pass@` in
|
||||||
|
both `http://` and `https://` URLs.
|
||||||
|
- `PM_DECISION_MAX_TURNS`: a new knob. It defaults to 1, today's value, so behaviour is
|
||||||
|
unchanged, and it is bounded to 1–5. If the newly visible envelope says `error_max_turns`,
|
||||||
|
the remedy is a ConfigMap change rather than another image build. Documented in
|
||||||
|
`platform/pm-agent/README.md`.
|
||||||
|
- Selftest: four new assertions. They cover an empty stderr with the cause on stdout, a
|
||||||
|
stderr-only failure, total silence, and redaction.
|
||||||
|
|
||||||
|
## What I verified
|
||||||
|
|
||||||
|
- **Timeline (Loki, `{namespace="pm-agent"}`, and `kubectl logs`):**
|
||||||
|
- **16:37:38 to 16:38:00**: pod `pm-agent-898fd8f5f-dp5hs`, image **c4eba5b3**,
|
||||||
|
config-generation 2, dry run. It triaged #97 in **22 s**, logged `PM says comment`, and
|
||||||
|
posted the 16:38:00Z comment. The thread then held only the issue body.
|
||||||
|
- **16:43:09 to 16:44:00**: pod `pm-agent-f7c9b7fcf-j8mzn`, image **df36d9b7**,
|
||||||
|
config-generation 3, **still dry run**. It logged `claude exited 1: ` with an empty
|
||||||
|
message after **51 s**.
|
||||||
|
- **16:55:42 to 16:56:36**: pod `pm-agent-7796db8bcb-p5dsm`, df36d9b7, config-generation
|
||||||
|
4, live. The same empty failure, after **54 s**.
|
||||||
|
- The only successful triage ever was the 16:38 one. Every earlier `898fd8f5f`/`5db55d558c`
|
||||||
|
log line is a startup crash (Gitea `GET /user` 401), not a triage.
|
||||||
|
- **Going live is NOT the cause.** The first failure happened in dry run. `ask_pm()` has no
|
||||||
|
dry-run branch: the prompt, flags, tools (`--tools ""`) and cwd are identical in both
|
||||||
|
modes. `PM_DRY_RUN` gates only `create_job()`, which runs after the model has answered.
|
||||||
|
- **Ruled out, with evidence:**
|
||||||
|
- **Timeout**: 51 to 54 s against a 900 s limit, and a timeout raises `TimeoutExpired`,
|
||||||
|
not rc 1.
|
||||||
|
- **Budget cap**: the PM call passes no `--max-budget-usd`.
|
||||||
|
- **Prompt size**: the thread is the issue body (293 chars) plus three comments
|
||||||
|
(1871 + 404 + 77 chars), and the persona is 8 KB.
|
||||||
|
- **Writable HOME**: HOME=`/workspace` (emptyDir), and `CLAUDE_CONFIG_DIR=/workspace/.claude`
|
||||||
|
was written at 16:56.
|
||||||
|
- **Egress**: a TLS 1.3 handshake to `api.anthropic.com:443` from the dispatcher pod
|
||||||
|
succeeds in 0.06 s.
|
||||||
|
- **Model name and CLI version**: the same `claude-opus-5` and CLI 2.1.285 succeeded at
|
||||||
|
16:38. Between c4eba5b and df36d9b, `infra/agent/` differs only by a template selftest
|
||||||
|
assertion and the toolbox base digest.
|
||||||
|
- **A hint, not proof:** `/workspace/.claude/.claude.json` in the live pod has
|
||||||
|
`cachedExtraUsageDisabledReason: 'out_of_credits'`. No token was read or printed.
|
||||||
|
- Offline selftest: `PM_SELFTEST=1` against the worktree's template and persona prints
|
||||||
|
`selftest OK`, including the 4 new assertions.
|
||||||
|
- Toolbox gate: see the PR description for the rc and the stages that were skipped.
|
||||||
|
|
||||||
|
## What I did NOT verify
|
||||||
|
|
||||||
|
- **The root cause. UNVERIFIED.** The CLI's own message was discarded, `--no-session-persistence`
|
||||||
|
leaves no transcript, and I did not run `claude` in the pod. The candidates still open:
|
||||||
|
- **Usage or rate limit on the subscription seat.** The seat is shared with the operator's
|
||||||
|
interactive use. The `out_of_credits` extra-usage flag fits this. So does a ~50 s run,
|
||||||
|
which matches the CLI exhausting its 429/529 retries.
|
||||||
|
- **`error_max_turns`.** With `--max-turns 1` and `--json-schema`, an answer that fails the
|
||||||
|
schema cannot be retried. The failing thread now invites `implement` (a branch plus a
|
||||||
|
long brief), where the 16:38 one produced a short `comment`. ~50 s is about two
|
||||||
|
generations of the 22 s success.
|
||||||
|
- **The 17:11 retry will NOT reveal the cause.** It runs the old image, which still discards
|
||||||
|
stdout. The cause becomes visible only after this PR merges, CI re-publishes agent-runner,
|
||||||
|
and a follow-up PR bumps the dispatcher digest (`deployment.yaml` and the `PM_WORKER_IMAGE`
|
||||||
|
in `configmap.yaml`, plus a `config-generation` bump).
|
||||||
|
- That the CLI's error envelope carries exactly these key names. The parser takes whichever
|
||||||
|
keys are present and falls back to raw tails.
|
||||||
@@ -257,7 +257,9 @@ about runtime behaviour in this document is a design claim, not a measurement.
|
|||||||
bot's permission is `write`, not `admin`** — that measurement is the control that keeps
|
bot's permission is `write`, not `admin`** — that measurement is the control that keeps
|
||||||
the agent out of its own approval gate.
|
the agent out of its own approval gate.
|
||||||
3. **First run with `PM_DRY_RUN=true` and `PM_REQUIRE_LABEL=agent`**, on one issue written
|
3. **First run with `PM_DRY_RUN=true` and `PM_REQUIRE_LABEL=agent`**, on one issue written
|
||||||
for the purpose. Read what the PM says before letting it write anything.
|
for the purpose. Read what the PM says before letting it write anything. *Done:* dry-run
|
||||||
|
triage of issue #97 read; `PM_DRY_RUN=false` on 2026-10-31
|
||||||
|
(`decisions/2026-10-31-pm-agent-go-live.md`), `PM_REQUIRE_LABEL=agent` kept.
|
||||||
4. **Tighten the admission image pin** from the `@sha256:*` wildcard to the literal digest.
|
4. **Tighten the admission image pin** from the `@sha256:*` wildcard to the literal digest.
|
||||||
One line, reviewed.
|
One line, reviewed.
|
||||||
5. **Narrow `world:443` to `toFQDNs: api.anthropic.com`** — after verifying Cilium's DNS
|
5. **Narrow `world:443` to `toFQDNs: api.anthropic.com`** — after verifying Cilium's DNS
|
||||||
|
|||||||
@@ -214,6 +214,12 @@ class Config:
|
|||||||
# ConfigMap without rebuilding the image.
|
# ConfigMap without rebuilding the image.
|
||||||
self.push_on_red = env("PM_PUSH_ON_RED_GATE", "false")
|
self.push_on_red = env("PM_PUSH_ON_RED_GATE", "false")
|
||||||
self.pm_timeout = env_int("PM_DECISION_TIMEOUT_SECONDS", 900)
|
self.pm_timeout = env_int("PM_DECISION_TIMEOUT_SECONDS", 900)
|
||||||
|
# The PM holds no tools, so the only thing a second turn can be spent on is the CLI's
|
||||||
|
# structured-output step re-asking the model after an answer that failed the schema.
|
||||||
|
# 1 is the long-standing value; it is a knob so that, if the failure envelope now
|
||||||
|
# logged by describe_cli_failure() says `error_max_turns`, the remedy is a ConfigMap
|
||||||
|
# change rather than another image build.
|
||||||
|
self.pm_max_turns = env_int("PM_DECISION_MAX_TURNS", 1)
|
||||||
# Opt-in gate. Empty (the default) means every open issue is in scope, which is the
|
# Opt-in gate. Empty (the default) means every open issue is in scope, which is the
|
||||||
# behaviour the service was asked for. Setting it to a label name — `agent` is the
|
# behaviour the service was asked for. Setting it to a label name — `agent` is the
|
||||||
# conventional one — narrows the agent to issues a human has explicitly handed it,
|
# conventional one — narrows the agent to issues a human has explicitly handed it,
|
||||||
@@ -231,6 +237,8 @@ class Config:
|
|||||||
|
|
||||||
if self.max_active_workers < 1:
|
if self.max_active_workers < 1:
|
||||||
die("PM_MAX_ACTIVE_WORKERS must be >= 1")
|
die("PM_MAX_ACTIVE_WORKERS must be >= 1")
|
||||||
|
if not 1 <= self.pm_max_turns <= 5:
|
||||||
|
die("PM_DECISION_MAX_TURNS must be between 1 and 5")
|
||||||
if "@sha256:" not in self.worker_image:
|
if "@sha256:" not in self.worker_image:
|
||||||
# The admission policy requires a digest too, so a tag here would produce a Job
|
# The admission policy requires a digest too, so a tag here would produce a Job
|
||||||
# that is created and then DENIED — a failure visible only in the dispatcher's
|
# that is created and then DENIED — a failure visible only in the dispatcher's
|
||||||
@@ -437,7 +445,7 @@ def ask_pm(cfg: Config, thread: str) -> dict:
|
|||||||
"--system-prompt-file", cfg.persona_file,
|
"--system-prompt-file", cfg.persona_file,
|
||||||
"--tools", "",
|
"--tools", "",
|
||||||
"--permission-prompts", "none",
|
"--permission-prompts", "none",
|
||||||
"--max-turns", "1",
|
"--max-turns", str(cfg.pm_max_turns),
|
||||||
"--output-format", "json",
|
"--output-format", "json",
|
||||||
"--json-schema", json.dumps(DECISION_SCHEMA),
|
"--json-schema", json.dumps(DECISION_SCHEMA),
|
||||||
"--no-session-persistence",
|
"--no-session-persistence",
|
||||||
@@ -445,10 +453,76 @@ def ask_pm(cfg: Config, thread: str) -> dict:
|
|||||||
proc = subprocess.run(cmd, input=thread, capture_output=True, text=True,
|
proc = subprocess.run(cmd, input=thread, capture_output=True, text=True,
|
||||||
timeout=cfg.pm_timeout)
|
timeout=cfg.pm_timeout)
|
||||||
if proc.returncode != 0:
|
if proc.returncode != 0:
|
||||||
raise RuntimeError(f"claude exited {proc.returncode}: {proc.stderr.strip()[:800]}")
|
raise RuntimeError(describe_cli_failure(proc.returncode, proc.stdout, proc.stderr))
|
||||||
return extract_decision(proc.stdout)
|
return extract_decision(proc.stdout)
|
||||||
|
|
||||||
|
|
||||||
|
# Anything that could be a credential in the CLI's output. The CLI does not normally echo its
|
||||||
|
# token, but an API error body or a proxy's HTML page is not something to bet a leak on. Both
|
||||||
|
# schemes are matched (the http one is the lesson of a scheme-specific redaction that leaked
|
||||||
|
# ~/.git-credentials), and `token ...` is Gitea's own header scheme.
|
||||||
|
_SECRET_PATTERNS = [
|
||||||
|
(re.compile(r"sk-ant-[A-Za-z0-9_\-]+"), "sk-ant-[REDACTED]"),
|
||||||
|
(re.compile(r"(?i)\b(bearer|token)(\s+|=|:\s*)[A-Za-z0-9._~+/=\-]{8,}"), r"\1\2[REDACTED]"),
|
||||||
|
(re.compile(r"(?i)(https?://)[^/\s:@]+:[^/\s@]+@"), r"\1[REDACTED]@"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def redact(text: str) -> str:
|
||||||
|
for pat, repl in _SECRET_PATTERNS:
|
||||||
|
text = pat.sub(repl, text)
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def describe_cli_failure(rc: int, stdout: str, stderr: str, limit: int = 800) -> str:
|
||||||
|
"""Say WHY `claude --print` failed, in the dispatcher log, redacted.
|
||||||
|
|
||||||
|
WHY THIS EXISTS: with `--output-format json` the CLI reports its own failures — an API
|
||||||
|
error, a usage limit, `error_max_turns`, a schema the model could not satisfy — as the
|
||||||
|
JSON result envelope on STDOUT (`is_error`, `subtype`, `result`), and writes nothing to
|
||||||
|
stderr. Logging only stderr therefore produced `claude exited 1: ` with an empty message
|
||||||
|
on 2026-10-31, twice, with the cause sitting unread in stdout. So: the envelope's
|
||||||
|
diagnostic fields first, then the raw tails of both streams.
|
||||||
|
"""
|
||||||
|
parts = [f"claude exited {rc}"]
|
||||||
|
out = (stdout or "").strip()
|
||||||
|
env = None
|
||||||
|
if out:
|
||||||
|
try:
|
||||||
|
env = json.loads(out)
|
||||||
|
except ValueError:
|
||||||
|
# A stream of JSON lines, or a log line ahead of the envelope: the last line that
|
||||||
|
# parses as an object is the result.
|
||||||
|
for line in reversed(out.splitlines()):
|
||||||
|
try:
|
||||||
|
cand = json.loads(line)
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
if isinstance(cand, dict):
|
||||||
|
env = cand
|
||||||
|
break
|
||||||
|
if isinstance(env, dict):
|
||||||
|
fields = []
|
||||||
|
for key in ("subtype", "is_error", "api_error_status", "num_turns", "duration_ms",
|
||||||
|
"duration_api_ms", "total_cost_usd", "stop_reason"):
|
||||||
|
if key in env:
|
||||||
|
fields.append(f"{key}={env[key]!r}")
|
||||||
|
if fields:
|
||||||
|
parts.append("envelope: " + " ".join(fields))
|
||||||
|
for key in ("result", "error", "errors", "message"):
|
||||||
|
if env.get(key):
|
||||||
|
val = env[key] if isinstance(env[key], str) else json.dumps(env[key])
|
||||||
|
parts.append(f"{key}: {val[:limit]}")
|
||||||
|
if out and not isinstance(env, dict):
|
||||||
|
parts.append(f"stdout tail: {out[-limit:]}")
|
||||||
|
err_text = (stderr or "").strip()
|
||||||
|
if err_text:
|
||||||
|
parts.append(f"stderr tail: {err_text[-limit:]}")
|
||||||
|
if len(parts) == 1:
|
||||||
|
parts.append("(no output on stdout or stderr)")
|
||||||
|
return redact(" | ".join(parts))
|
||||||
|
|
||||||
|
|
||||||
def extract_decision(stdout: str) -> dict:
|
def extract_decision(stdout: str) -> dict:
|
||||||
"""Pull the decision object out of whatever `claude --print` wrapped it in.
|
"""Pull the decision object out of whatever `claude --print` wrapped it in.
|
||||||
|
|
||||||
@@ -501,7 +575,7 @@ def extract_decision(stdout: str) -> dict:
|
|||||||
obj = as_obj(c)
|
obj = as_obj(c)
|
||||||
if obj and isinstance(obj.get("action"), str):
|
if obj and isinstance(obj.get("action"), str):
|
||||||
return obj
|
return obj
|
||||||
raise RuntimeError(f"no decision object in claude output: {stdout[:400]!r}")
|
raise RuntimeError(redact(f"no decision object in claude output: {stdout[-800:]!r}"))
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- thread
|
# --------------------------------------------------------------------------- thread
|
||||||
@@ -988,8 +1062,25 @@ def selftest() -> int:
|
|||||||
except RuntimeError:
|
except RuntimeError:
|
||||||
check(True, f"bad output refused: {label}")
|
check(True, f"bad output refused: {label}")
|
||||||
|
|
||||||
|
# -- a failed CLI run must say why, and must not leak ---------------------
|
||||||
|
# The 2026-10-31 failure: rc 1, empty stderr, the cause in the stdout envelope.
|
||||||
|
env_err = json.dumps({"type": "result", "subtype": "error_max_turns", "is_error": True,
|
||||||
|
"num_turns": 2, "result": "API Error: 429 rate_limit_error"})
|
||||||
|
msg = describe_cli_failure(1, env_err, "")
|
||||||
|
check("error_max_turns" in msg and "429" in msg,
|
||||||
|
"CLI failure: stdout envelope surfaced when stderr is empty")
|
||||||
|
msg = describe_cli_failure(1, "", "boom on stderr")
|
||||||
|
check("boom on stderr" in msg, "CLI failure: stderr tail surfaced")
|
||||||
|
msg = describe_cli_failure(1, "", "")
|
||||||
|
check("no output" in msg, "CLI failure: silence is named, not an empty message")
|
||||||
|
msg = describe_cli_failure(1, "plain text sk-ant-oat01-abcdefABCDEF_123 here",
|
||||||
|
"Authorization: Bearer abcdefghijklmnop "
|
||||||
|
"http://u:[email protected]/x")
|
||||||
|
check("sk-ant-oat01" not in msg and "abcdefghijklmnop" not in msg and ":pw@" not in msg,
|
||||||
|
"CLI failure: token, bearer and URL credential redacted")
|
||||||
|
|
||||||
# -- the committed Job template -----------------------------------------
|
# -- the committed Job template -----------------------------------------
|
||||||
tmpl = os.environ.get("PM_WORKER_JOB_TEMPLATE", "/etc/pm-agent/worker-job-template.yaml")
|
tmpl =os.environ.get("PM_WORKER_JOB_TEMPLATE", "/etc/pm-agent/worker-job-template.yaml")
|
||||||
try:
|
try:
|
||||||
selftest_template(tmpl)
|
selftest_template(tmpl)
|
||||||
check(True, f"worker Job template satisfies its contract ({tmpl})")
|
check(True, f"worker Job template satisfies its contract ({tmpl})")
|
||||||
|
|||||||
@@ -46,11 +46,12 @@ waive the sensitive-path human review on its own pull request.
|
|||||||
Both are deliberate, and both fail **visibly** rather than silently:
|
Both are deliberate, and both fail **visibly** rather than silently:
|
||||||
|
|
||||||
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
|
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
|
||||||
`PM_WORKER_IMAGE` in `configmap.yaml` carry the digest of the **first** publish (images
|
`PM_WORKER_IMAGE` in `configmap.yaml` first carried the run-340 publish
|
||||||
run 340, `sha-523cd9118f95`), pinned once on 2026-10-30 so going live did not wait on
|
(`sha-523cd9118f95`), whose baked template predated the `imagePullSecrets` entry admission
|
||||||
Renovate's first run; every later bump is Renovate's. That digest predates the worker
|
requires, so the service ran with `PM_DRY_RUN=true`. PR #96 pinned the run-383 digest
|
||||||
template's `imagePullSecrets` entry, so worker Jobs built from it are refused by
|
(`df36d9b`, `sha-ddd04d460ef6`), whose template carries the entry, and dry run was turned
|
||||||
admission — which is why `PM_DRY_RUN` starts `true` (see `configmap.yaml`). `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
|
off on 2026-10-31 (`decisions/2026-10-31-pm-agent-go-live.md`); every later bump is
|
||||||
|
Renovate's. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
|
||||||
(`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
|
(`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
|
||||||
`infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
|
`infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
|
||||||
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
|
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
|
||||||
@@ -249,6 +250,12 @@ five that matter:
|
|||||||
| `PM_MAX_ACTIVE_WORKERS` | `2` | concurrent workers. RAM is the binding constraint (2Gi limit each) and the subscription seat's rate limits are shared with every other use of that seat |
|
| `PM_MAX_ACTIVE_WORKERS` | `2` | concurrent workers. RAM is the binding constraint (2Gi limit each) and the subscription seat's rate limits are shared with every other use of that seat |
|
||||||
| `PM_MAX_NEW_WORKERS_PER_CYCLE` | `1` | drains a backlog one issue at a time instead of in one stampede the first minute the credentials work |
|
| `PM_MAX_NEW_WORKERS_PER_CYCLE` | `1` | drains a backlog one issue at a time instead of in one stampede the first minute the credentials work |
|
||||||
| `PM_PUSH_ON_RED_GATE` | `false` | a change the gate refused is not pushed; the issue gets the gate log instead |
|
| `PM_PUSH_ON_RED_GATE` | `false` | a change the gate refused is not pushed; the issue gets the gate log instead |
|
||||||
|
| `PM_DECISION_MAX_TURNS` | `1` (unset) | turns the triage call may take (1–5). The PM holds no tools, so a second turn is only ever the CLI re-asking after an answer that failed the decision schema. Raise it if a triage error logs `subtype='error_max_turns'` |
|
||||||
|
|
||||||
|
**When a triage fails**, the `[error] issue #N: claude exited …` line carries the CLI's result
|
||||||
|
envelope (`subtype`, `is_error`, `api_error_status`, `num_turns`, `result`) and the tails of
|
||||||
|
stdout and stderr, redacted. Before 2026-10-31 it logged stderr only, which is empty under
|
||||||
|
`--output-format json` — see `decisions/2026-10-31-pm-agent-triage-exit.md`.
|
||||||
|
|
||||||
Changing a knob requires bumping the `config-generation` annotation in `deployment.yaml` in
|
Changing a knob requires bumping the `config-generation` annotation in `deployment.yaml` in
|
||||||
the same commit — the process reads its configuration once, at startup, so without the bump
|
the same commit — the process reads its configuration once, at startup, so without the bump
|
||||||
|
|||||||
@@ -58,13 +58,13 @@ data:
|
|||||||
# request that also moves the Deployment's image (same marker, same branch). Nobody pushes
|
# request that also moves the Deployment's image (same marker, same branch). Nobody pushes
|
||||||
# this image from a workstation.
|
# this image from a workstation.
|
||||||
#
|
#
|
||||||
# READ BEFORE TURNING PM_DRY_RUN OFF: this digest PREDATES the worker template's
|
# The run-340 digest predated the worker template's `imagePullSecrets` entry (PR #61), which
|
||||||
# `imagePullSecrets` entry (PR #61). Its baked template has none, and admission now REQUIRES
|
# admission REQUIRES, so its worker Jobs were DENIED. The digest below (df36d9b, run 383,
|
||||||
# it — so a worker Job built from this image is DENIED. The merge that adds the entry
|
# agent-runner:sha-ddd04d460ef6, pinned in PR #96) bakes a template that carries
|
||||||
# changes infra/agent/**, CI re-publishes agent-runner, and Renovate's bump to that digest
|
# `imagePullSecrets: pm-agent-registry`, which is what made turning PM_DRY_RUN off possible
|
||||||
# is what makes worker Jobs admissible. Until then, dry run is the only mode that works.
|
# (decisions/2026-10-31-pm-agent-runner-digest.md). A digest older than that would be DENIED.
|
||||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
|
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
|
||||||
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42"
|
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d"
|
||||||
|
|
||||||
# --- the model -------------------------------------------------------------
|
# --- the model -------------------------------------------------------------
|
||||||
# The PM only reads a thread and answers with one JSON object, so it holds no tools and
|
# The PM only reads a thread and answers with one JSON object, so it holds no tools and
|
||||||
@@ -120,10 +120,11 @@ data:
|
|||||||
# no worker Job and so changes no code. The honest way to watch this service for a day
|
# no worker Job and so changes no code. The honest way to watch this service for a day
|
||||||
# before letting it write anything.
|
# before letting it write anything.
|
||||||
#
|
#
|
||||||
# "true" FOR THE FIRST RUN, and not only for caution: the pinned digest above cannot yet
|
# "true" for the first run (2026-10-30 .. 2026-10-31); dry-run triage of issue #97 was read
|
||||||
# create an admissible worker Job (see the PM_WORKER_IMAGE comment). Flip to "false" in the
|
# and the operator approved going live. "false" since decisions/2026-10-31-pm-agent-go-live.md.
|
||||||
# same pull request as — or after — Renovate's first agent-runner digest bump.
|
# Setting it back to "true" (plus a config-generation bump) is the gentlest stop — see
|
||||||
PM_DRY_RUN: "true"
|
# runbooks/pm-agent.md.
|
||||||
|
PM_DRY_RUN: "false"
|
||||||
|
|
||||||
# If the gate refuses a worker's change, the branch is NOT pushed by default and the issue
|
# If the gate refuses a worker's change, the branch is NOT pushed by default and the issue
|
||||||
# gets the gate log instead. Set to "true" to push anyway and open a red pull request —
|
# gets the gate log instead. Set to "true" to push anyway and open a red pull request —
|
||||||
|
|||||||
@@ -48,7 +48,7 @@ spec:
|
|||||||
# It is a hand-maintained marker rather than a computed hash because Flux's
|
# It is a hand-maintained marker rather than a computed hash because Flux's
|
||||||
# Kustomization here does not run a configmap generator; bump it in the same commit
|
# Kustomization here does not run a configmap generator; bump it in the same commit
|
||||||
# as a change to configmap.yaml.
|
# as a change to configmap.yaml.
|
||||||
pm-agent.homelab.keithjackson.dev/config-generation: "2"
|
pm-agent.homelab.keithjackson.dev/config-generation: "4"
|
||||||
spec:
|
spec:
|
||||||
serviceAccountName: pm-agent
|
serviceAccountName: pm-agent
|
||||||
# The kubelet's credential for pulling this image. Not the agent's Gitea credential —
|
# The kubelet's credential for pulling this image. Not the agent's Gitea credential —
|
||||||
@@ -78,7 +78,7 @@ spec:
|
|||||||
# the registry's manifest endpoint on 2026-10-30 and pinned by hand once so the
|
# the registry's manifest endpoint on 2026-10-30 and pinned by hand once so the
|
||||||
# dispatcher does not wait on Renovate's first run; every later bump is Renovate's.
|
# dispatcher does not wait on Renovate's first run; every later bump is Renovate's.
|
||||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
|
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
|
||||||
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42
|
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d
|
||||||
imagePullPolicy: IfNotPresent
|
imagePullPolicy: IfNotPresent
|
||||||
command:
|
command:
|
||||||
- /usr/local/bin/pm-dispatch
|
- /usr/local/bin/pm-dispatch
|
||||||
|
|||||||
Reference in new issue
Block a user