# pm-agent — the Program Manager agent service

Operator procedure, not alert-linked. The service reads Gitea issues, decides what to do
about each thread as the Program Manager, and runs one worker agent per assignment that
authors a change, runs the merge gate and opens a pull request.

- What it is and why it is shaped this way: [`platform/pm-agent/README.md`](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/src/branch/main/platform/pm-agent/README.md)
- The design and the rejected alternatives: `docs/plans/pm-agent-issue-to-pr.md`

**Everything below is read-only except where it says otherwise.** Mutating `kubectl` and
`flux reconcile` are operator actions, not agent actions.

---

## Stop it. Right now.

In order of increasing severity. Each is an operator action.

**1. Stop it taking new work, keep it running.** Set `PM_DRY_RUN: "true"` in
`platform/pm-agent/configmap.yaml`, bump the `config-generation` annotation in
`deployment.yaml`, merge. It still reads threads and still posts what it *would* have done,
and creates no worker Jobs. This is also the right setting for watching it for a day.

**2. Stop it immediately, without a merge.**

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent scale deploy/pm-agent --replicas=0'
```

Flux will put it back at the next reconcile (10m). To hold it down, suspend the tier:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'flux --context k3s-prod suspend kustomization pm-agent'
```

Running worker Jobs are **not** stopped by either. They finish, or:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent delete job -l app.kubernetes.io/name=pm-agent-worker'
```

**3. Revoke its reach.** Delete the Gitea token in Gitea (as the `pm-agent` user: Settings →
Applications → the token → Delete). Every worker and the dispatcher then fail
authentication, nothing can be pushed, and no comment can be posted. This is the one that
does not depend on the cluster behaving.

A branch the agent already pushed is not undone by any of the above. A pull request it
opened cannot merge without a green gate, and — if it touched a sensitive path — not without
a human review either.

---

## It is not doing anything

Work through these in order; the first three are by far the most likely.

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get deploy,pod,job'
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent logs deploy/pm-agent --tail=60'
```

| Symptom in the log | Cause | Fix |
|---|---|---|
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets |
| a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push |
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
| `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
| `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle |
| nothing at all, not even `cycle:` lines | the process is not running | check the pod, then the events |

**It will not answer an issue whose newest comment is its own.** That is the trigger, not a
bug: it acts only when the newest thing in a thread was written by somebody else. If you want
it to look again, comment on the issue.

**It will not re-answer an issue within `PM_ISSUE_COOLDOWN_SECONDS`** (15 min) of its last
attempt on that issue. Restarting the pod clears that timer.

---

## A worker failed

Every worker posts its own outcome on the issue, including failures — so **read the issue
first**; it usually names the step. Then:

```bash
# which workers ran, and how they ended (Jobs are kept 48h)
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get jobs -l app.kubernetes.io/name=pm-agent-worker \
     -L pm-agent.homelab.keithjackson.dev/issue'
# the full log of one
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent logs job/pm-agent-task-<issue>-<ts> --tail=200'
```

| What you see | Meaning |
|---|---|
| issue comment: *"the merge gate refused it"* with a gate log | working as intended. Nothing was pushed. Comment on the issue with guidance and the PM will send a worker back |
| `unexpected skipped stage '<x>' — rc 3 is NOT acceptable` | the gate could not run a stage that is not one of the two expected skips (`spec-immutability`, `gitea-settings`). Treated as a refusal, correctly: a stage that did not run is not green |
| `branch '<x>' already exists on the remote, but this assignment was 'implement'` | the PM thought this was new work. Nothing was forced. Ask for a revision of the existing branch instead |
| `is not an allowed git host` | `PM_GITEA_URL` was changed to something outside the allowlist baked into `pm-worker`. That allowlist is deliberate — changing it means rebuilding the image |
| `Error from server (Forbidden)` creating a Job, in the **dispatcher** log | admission denied the Job. Read the message: it names the rule. Either the image digest is not under the agent-runner path, or the template drifted out of the shape the policy admits |
| Job `DeadlineExceeded` after 90 min | a wedged session. Nothing was pushed (the push is the last step) |
| nothing on the issue at all and the Job failed | the worker died before it could comment. The pod log is the only record; the Job is kept 48h |

---

## The Anthropic token expires after a year

`claude setup-token` mints a **one-year** OAuth token, and **nothing warns you**: Claude
Code's expiry notice only appears for an interactive login, never for the
`CLAUDE_CODE_OAUTH_TOKEN` variable. When it lapses, every session fails at once and the
symptom is workers failing with an authentication error.

Rotate it exactly as it was created — `claude setup-token` on a machine with a browser, then
`sops -e -i platform/pm-agent/pm-agent-credentials.sops.yaml`, then a pull request. Restart
the dispatcher pod afterwards so it re-reads the Secret (a Secret change does not roll a
Deployment by itself); worker Jobs pick the new value up on their next run with no action.

The date it was minted belongs in the handoff entry. Check it now:

```bash
grep -rn 'claude setup-token' decisions/ | head
```

---

## Checks worth making, and the one you must not

**Confirm the bot account is NOT an administrator.** This is the control that stops the
service clearing its own sensitive-path review — `infra/gitea/approval-policy.json`
authorises `/approve-override` by live repository permission alone, so an admin-permission
bot could waive the human review on its own pull request. It must print `write`:

```bash
curl -s -H "Authorization: token <the pm-agent token>" \
  https://gitea.home.keithjackson.dev/api/v1/repos/Home-Network/home-network-k3s-impl/collaborators/pm-agent/permission \
  | python3 -c 'import json,sys; print(json.load(sys.stdin).get("permission"))'
```

**Confirm the admission policy is actually applied** — a policy that is in git and not in
the applied set is the exact failure this repository has already had once:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusterpolicy pm-agent-job-shape \
     -o jsonpath="{.spec.validationFailureAction}{\"\n\"}"'
```

`Enforce`, or the boundary is not there.

**Never** `sops -d` the credentials file into a terminal, a log or a transcript, and never
print the Secret's `.data` to check that it exists. Key names only:

```bash
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get secret pm-agent-credentials \
     -o jsonpath="{.data}" | tr "," "\n" | cut -d\" -f2'
```

(That prints the **keys**. A password has been leaked into a transcript in this repository by
printing the values.)