home-network-k3s-impl/runbooks/pm-agent.md

# pm-agent — the Program Manager agent service

Operator procedure, not alert-linked. The service reads Gitea issues, decides what to do
about each thread as the Program Manager, and runs one worker agent per assignment that
authors a change, runs the merge gate and opens a pull request.

- What it is and why it is shaped this way: [`platform/pm-agent/README.md`](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/src/branch/main/platform/pm-agent/README.md)
- The design and the rejected alternatives: `docs/plans/pm-agent-issue-to-pr.md`

**Everything below is read-only except where it says otherwise.** Mutating `kubectl` and
`flux reconcile` are operator actions, not agent actions.

---

## Stop it. Right now.

In order of increasing severity. Each is an operator action.

**1. Stop it taking new work, keep it running.** Set `PM_DRY_RUN: "true"` in
`platform/pm-agent/configmap.yaml`, bump the `config-generation` annotation in
`deployment.yaml`, merge. It still reads threads and still posts what it *would* have done,
and creates no worker Jobs. This is also the right setting for watching it for a day.

`PM_DRY_RUN` also silences the **pull-request mention** path completely — that path only ever
posts a comment, so leaving it posting would make a dry run not dry, and its reply carries the
marker that stops the same mention being answered again. To silence just that path and leave
issue triage alone, set `PM_PR_MENTIONS: "false"` instead (that is the default; see
`platform/pm-agent/README.md`, "Mentions on a pull request").

**2. Stop it immediately, without a merge.**

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent scale deploy/pm-agent --replicas=0'
```

Flux will put it back at the next reconcile (10m). To hold it down, suspend the tier:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'flux --context k3s-prod suspend kustomization pm-agent'
```

Running worker Jobs are **not** stopped by either. They finish, or:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent delete job -l app.kubernetes.io/name=pm-agent-worker'
```

**3. Revoke its reach.** Delete the Gitea token in Gitea (as the `pm-agent` user: Settings →
Applications → the token → Delete). Every worker and the dispatcher then fail
authentication, nothing can be pushed, and no comment can be posted. This is the one that
does not depend on the cluster behaving.

A branch the agent already pushed is not undone by any of the above. A pull request it
opened cannot merge without a green gate, and — if it touched a sensitive path — not without
a human review either.

---

## It is not doing anything

Work through these in order; the first three are by far the most likely.

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get deploy,pod,job'
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent logs deploy/pm-agent --tail=60'
```

| Symptom in the log | Cause | Fix |
|---|---|---|
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and (before the 2026-11-08 Tunnel cutover; the public answer is now Cloudflare) got the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets. Add `read:notification` if you want the cheap pull-request mention query (`platform/pm-agent/README.md`) |
| a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push |
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
| `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
| `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle |
| nothing at all, not even `cycle:` lines | the process is not running | check the pod, then the events |
| `pull-request mentions: off` | `PM_PR_MENTIONS` is `false`, which is the default | read `PM_PLAN` output once, then set it to `"true"` (ConfigMap + `config-generation` bump) |
| `pr-mentions: /notifications was refused …` | expected on today's token: Gitea puts that endpoint behind its `notification` scope and the bot's token is minted `read:user, write:repository, write:issue, read:package`. The dispatcher has fallen back to listing open pull requests, so it still works — just not cheaply | re-mint the token with `read:notification` added and update **both** Secrets (below), or set `PM_PR_MENTION_DISCOVERY: "pulls"` to stop trying |
| `pr-mentions: 0 candidate pull request(s) from notifications-unavailable` | `PM_PR_MENTION_DISCOVERY` is `notifications` and that query is refused, so the path deliberately does nothing rather than widening into a full scan | fix the scope, or set the knob to `auto` |
| `[PM_PLAN: nothing will be posted]` | `PM_PLAN` or `PM_DRY_RUN` is set; the payload is printed and no reply is posted | expected; clear both to let it reply |
| `pull #N: inline reply refused …; falling back to the conversation` | Gitea would not accept a comment at that diff position (usually the position no longer exists on the current head) | nothing to fix — the reply was posted in the conversation, naming the file and position |

**It will not answer an issue whose newest comment is its own.** That is the trigger, not a
bug: it acts only when the newest thing in a thread was written by somebody else. If you want
it to look again, comment on the issue.

**It will not re-answer an issue within `PM_ISSUE_COOLDOWN_SECONDS`** (15 min) of its last
attempt on that issue. Restarting the pod clears that timer.

**On a pull request it answers explicit mentions and nothing else.** A comment that does not
write `@`-the-bot-account is never answered and costs no model call, however recently the bot
was mentioned or replied on that pull request. If you asked something and got no reply, the
likely cause is a missing `@` — tag it again. (Near misses do not count either:
`@pm-agent-worker` and `[email protected]` are deliberately not mentions.)

**It will not answer the same pull-request comment twice.** Every reply it posts on a pull
request carries a hidden marker naming the comment ids it answered
(`<!-- pm-agent:replied-to=… -->`), and it skips anything already marked. To make it answer
again, write a *new* comment that mentions it again — editing the old one does not change its
id. `PM_PR_COOLDOWN_SECONDS` (15 min) bounds it to one decision per pull request per window.

---

## A worker failed

Every worker posts its own outcome on the issue, including failures — so **read the issue
first**; it usually names the step. Then:

```bash
# which workers ran, and how they ended (Jobs are kept 48h)
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get jobs -l app.kubernetes.io/name=pm-agent-worker \
     -L pm-agent.homelab.keithjackson.dev/issue'
# the full log of one
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent logs job/pm-agent-task-<issue>-<ts> --tail=200'
```

| What you see | Meaning |
|---|---|
| issue comment: *"the merge gate refused it"* with a gate log | working as intended. Nothing was pushed. Comment on the issue with guidance and the PM will send a worker back |
| `unexpected skipped stage '<x>' — rc 3 is NOT acceptable` | the gate could not run a stage that is not one of the two expected skips (`spec-immutability`, `gitea-settings`). Treated as a refusal, correctly: a stage that did not run is not green |
| `branch '<x>' already exists on the remote, but this assignment was 'implement'` | the PM thought this was new work. Nothing was forced. Ask for a revision of the existing branch instead |
| `is not an allowed git host` | `PM_GITEA_URL` was changed to something outside the allowlist baked into `pm-worker`. That allowlist is deliberate — changing it means rebuilding the image |
| `Error from server (Forbidden)` creating a Job, in the **dispatcher** log | admission denied the Job. Read the message: it names the rule. Either the image digest is not under the agent-runner path, or the template drifted out of the shape the policy admits |
| Job `DeadlineExceeded` after 90 min | a wedged session. Nothing was pushed (the push is the last step) |
| nothing on the issue at all and the Job failed | the worker died before it could comment. The pod log is the only record; the Job is kept 48h |

---

## The Anthropic token expires after a year

`claude setup-token` mints a **one-year** OAuth token, and **nothing warns you**: Claude
Code's expiry notice only appears for an interactive login, never for the
`CLAUDE_CODE_OAUTH_TOKEN` variable. When it lapses, every session fails at once and the
symptom is workers failing with an authentication error.

Rotate it exactly as it was created — `claude setup-token` on a machine with a browser, then
`sops -e -i platform/pm-agent/pm-agent-credentials.sops.yaml`, then a pull request. Restart
the dispatcher pod afterwards so it re-reads the Secret (a Secret change does not roll a
Deployment by itself); worker Jobs pick the new value up on their next run with no action.

The date it was minted belongs in the handoff entry. Check it now:

```bash
grep -rn 'claude setup-token' decisions/ | head
```

---

## Checks worth making, and the one you must not

**Confirm the bot account is NOT an administrator.** This is the control that stops the
service clearing its own sensitive-path review — `infra/gitea/approval-policy.json`
authorises `/approve-override` by live repository permission alone, so an admin-permission
bot could waive the human review on its own pull request. It must print `write`:

```bash
curl -s -H "Authorization: token <the pm-agent token>" \
  https://gitea.home.keithjackson.dev/api/v1/repos/Home-Network/home-network-k3s-impl/collaborators/pm-agent/permission \
  | python3 -c 'import json,sys; print(json.load(sys.stdin).get("permission"))'
```

**Confirm the admission policy is actually applied** — a policy that is in git and not in
the applied set is the exact failure this repository has already had once:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusterpolicy pm-agent-job-shape \
     -o jsonpath="{.spec.validationFailureAction}{\"\n\"}"'
```

`Enforce`, or the boundary is not there.

**Never** `sops -d` the credentials file into a terminal, a log or a transcript, and never
print the Secret's `.data` to check that it exists. Key names only:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get secret pm-agent-credentials \
     -o jsonpath="{.data}" | tr "," "\n" | cut -d\" -f2'
```

(That prints the **keys**. A password has been leaked into a transcript in this repository by
printing the values.)