home-network-k3s-impl/runbooks/pm-agent.md

# pm-agent — the Program Manager agent service

Operator procedure, not alert-linked. The service reads Gitea issues, decides what to do
about each thread as the Program Manager, and runs one worker agent per assignment that
authors a change, runs the merge gate and opens a pull request.

- What it is and why it is shaped this way: [`platform/pm-agent/README.md`](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/src/branch/main/platform/pm-agent/README.md)
- The design and the rejected alternatives: `docs/plans/pm-agent-issue-to-pr.md`

**Everything below is read-only except where it says otherwise.** Mutating `kubectl` and
`flux reconcile` are operator actions, not agent actions.

---

## Stop it. Right now.

In order of increasing severity. Each is an operator action.

**1. Stop it taking new work, keep it running.** Set `PM_DRY_RUN: "true"` in
`platform/pm-agent/configmap.yaml`, bump the `config-generation` annotation in
`deployment.yaml`, merge. It still reads threads and still posts what it *would* have done,
and creates no worker Jobs. This is also the right setting for watching it for a day.

**2. Stop it immediately, without a merge.**

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent scale deploy/pm-agent --replicas=0'
```

Flux will put it back at the next reconcile (10m). To hold it down, suspend the tier:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'flux --context k3s-prod suspend kustomization pm-agent'
```

Running worker Jobs are **not** stopped by either. They finish, or:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent delete job -l app.kubernetes.io/name=pm-agent-worker'
```

**3. Revoke its reach.** Delete the Gitea token in Gitea (as the `pm-agent` user: Settings →
Applications → the token → Delete). Every worker and the dispatcher then fail
authentication, nothing can be pushed, and no comment can be posted. This is the one that
does not depend on the cluster behaving.

A branch the agent already pushed is not undone by any of the above. A pull request it
opened cannot merge without a green gate, and — if it touched a sensitive path — not without
a human review either.

---

## It is not doing anything

Work through these in order; the first three are by far the most likely.

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get deploy,pod,job'
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent logs deploy/pm-agent --tail=60'
```

| Symptom in the log | Cause | Fix |
|---|---|---|
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) |
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
| `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
| `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle |
| nothing at all, not even `cycle:` lines | the process is not running | check the pod, then the events |

**It will not answer an issue whose newest comment is its own.** That is the trigger, not a
bug: it acts only when the newest thing in a thread was written by somebody else. If you want
it to look again, comment on the issue.

**It will not re-answer an issue within `PM_ISSUE_COOLDOWN_SECONDS`** (15 min) of its last
attempt on that issue. Restarting the pod clears that timer.

---

## A worker failed

Every worker posts its own outcome on the issue, including failures — so **read the issue
first**; it usually names the step. Then:

```bash
# which workers ran, and how they ended (Jobs are kept 48h)
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get jobs -l app.kubernetes.io/name=pm-agent-worker \
     -L pm-agent.homelab.keithjackson.dev/issue'
# the full log of one
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent logs job/pm-agent-task-<issue>-<ts> --tail=200'
```

| What you see | Meaning |
|---|---|
| issue comment: *"the merge gate refused it"* with a gate log | working as intended. Nothing was pushed. Comment on the issue with guidance and the PM will send a worker back |
| `unexpected skipped stage '<x>' — rc 3 is NOT acceptable` | the gate could not run a stage that is not one of the two expected skips (`spec-immutability`, `gitea-settings`). Treated as a refusal, correctly: a stage that did not run is not green |
| `branch '<x>' already exists on the remote, but this assignment was 'implement'` | the PM thought this was new work. Nothing was forced. Ask for a revision of the existing branch instead |
| `is not an allowed git host` | `PM_GITEA_URL` was changed to something outside the allowlist baked into `pm-worker`. That allowlist is deliberate — changing it means rebuilding the image |
| `Error from server (Forbidden)` creating a Job, in the **dispatcher** log | admission denied the Job. Read the message: it names the rule. Either the image digest is not under the agent-runner path, or the template drifted out of the shape the policy admits |
| Job `DeadlineExceeded` after 90 min | a wedged session. Nothing was pushed (the push is the last step) |
| nothing on the issue at all and the Job failed | the worker died before it could comment. The pod log is the only record; the Job is kept 48h |

---

## The Anthropic token expires after a year

`claude setup-token` mints a **one-year** OAuth token, and **nothing warns you**: Claude
Code's expiry notice only appears for an interactive login, never for the
`CLAUDE_CODE_OAUTH_TOKEN` variable. When it lapses, every session fails at once and the
symptom is workers failing with an authentication error.

Rotate it exactly as it was created — `claude setup-token` on a machine with a browser, then
`sops -e -i platform/pm-agent/pm-agent-credentials.sops.yaml`, then a pull request. Restart
the dispatcher pod afterwards so it re-reads the Secret (a Secret change does not roll a
Deployment by itself); worker Jobs pick the new value up on their next run with no action.

The date it was minted belongs in the handoff entry. Check it now:

```bash
grep -rn 'claude setup-token' decisions/ | head
```

---

## Checks worth making, and the one you must not

**Confirm the bot account is NOT an administrator.** This is the control that stops the
service clearing its own sensitive-path review — `infra/gitea/approval-policy.json`
authorises `/approve-override` by live repository permission alone, so an admin-permission
bot could waive the human review on its own pull request. It must print `write`:

```bash
curl -s -H "Authorization: token <the pm-agent token>" \
  https://gitea.home.keithjackson.dev/api/v1/repos/Home-Network/home-network-k3s-impl/collaborators/pm-agent/permission \
  | python3 -c 'import json,sys; print(json.load(sys.stdin).get("permission"))'
```

**Confirm the admission policy is actually applied** — a policy that is in git and not in
the applied set is the exact failure this repository has already had once:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusterpolicy pm-agent-job-shape \
     -o jsonpath="{.spec.validationFailureAction}{\"\n\"}"'
```

`Enforce`, or the boundary is not there.

**Never** `sops -d` the credentials file into a terminal, a log or a transcript, and never
print the Secret's `.data` to check that it exists. Key names only:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get secret pm-agent-credentials \
     -o jsonpath="{.data}" | tr "," "\n" | cut -d\" -f2'
```

(That prints the **keys**. A password has been leaked into a transcript in this repository by
printing the values.)