home-network-k3s-impl/runbooks/pm-agent.md

pm-agent — the Program Manager agent service

Operator procedure, not alert-linked. The service reads Gitea issues, decides what to do about each thread as the Program Manager, and runs one worker agent per assignment that authors a change, runs the merge gate and opens a pull request.

• What it is and why it is shaped this way: platform/pm-agent/README.md
• The design and the rejected alternatives: docs/plans/pm-agent-issue-to-pr.md

Everything below is read-only except where it says otherwise. Mutating kubectl and flux reconcile are operator actions, not agent actions.

---

Stop it. Right now.

In order of increasing severity. Each is an operator action.

1. Stop it taking new work, keep it running. Set PM_DRY_RUN: "true" in platform/pm-agent/configmap.yaml, bump the config-generation annotation in deployment.yaml, merge. It still reads threads and still posts what it would have done, and creates no worker Jobs. This is also the right setting for watching it for a day.

PM_DRY_RUN also silences the pull-request mention path completely — that path only ever posts a comment, so leaving it posting would make a dry run not dry, and its reply carries the marker that stops the same mention being answered again. To silence just that path and leave issue triage alone, set PM_PR_MENTIONS: "false" instead (that is the default; see platform/pm-agent/README.md, "Mentions on a pull request").

2. Stop it immediately, without a merge.

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent scale deploy/pm-agent --replicas=0'

Flux will put it back at the next reconcile (10m). To hold it down, suspend the tier:

./infra/devcontainer/dev.sh run bash -c \
  'flux --context k3s-prod suspend kustomization pm-agent'

Running worker Jobs are not stopped by either. They finish, or:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent delete job -l app.kubernetes.io/name=pm-agent-worker'

3. Revoke its reach. Delete the Gitea token in Gitea (as the pm-agent user: Settings → Applications → the token → Delete). Every worker and the dispatcher then fail authentication, nothing can be pushed, and no comment can be posted. This is the one that does not depend on the cluster behaving.

A branch the agent already pushed is not undone by any of the above. A pull request it opened cannot merge without a green gate, and — if it touched a sensitive path — not without a human review either.

---

It is not doing anything

Work through these in order; the first three are by far the most likely.

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get deploy,pod,job'
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent logs deploy/pm-agent --tail=60'

Symptom in the log	Cause	Fix
GET /user returned no login — is GITEA_TOKEN valid? and a CrashLoop	the credentials are still the committed placeholders, or the token was revoked	fill them in per platform/pm-agent/pm-agent-credentials.sops.yaml.example
ImagePullBackOff, digest sha256:0000…	(historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named	CI publishes it, not a person: check the latest images workflow run on main (it fails loudly if the REGISTRY_PUSH_TOKEN secret is missing; re-run it with workflow_dispatch), then merge Renovate's digest-bump PR, which updates both deployment.yaml and configmap.yaml (PM_WORKER_IMAGE) together — but read the row below first
ImagePullBackOff with x509: certificate is not valid for any names	diagnosed. containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own CN=%D placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert	the /etc/hosts pin for gitea.{{ k3s_base_domain }} → 192.168.2.20 in infra/ansible/inventory/group_vars/k3s_prod/vars.yml fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts. Full evidence: decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md
ImagePullBackOff with 401 Unauthorized / unauthorized	the pull credential is wrong, absent or scoped without read:package. The pm-agent-registry Secret exists and is referenced; its value is what starts as a placeholder	check the key names only, never values: kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}' should list .dockerconfigjson. Then confirm the token carries read:package, that its account can read the org's packages (a team with the Packages unit — see the .example), and that the registry key inside the JSON is the bare host gitea.home.keithjackson.dev — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: platform/pm-agent/pm-agent-registry.sops.yaml.example
GET /user answers 403 / token does not have at least one of required scope(s): [read:user] at startup, then a CrashLoop	the token lacks read:user — Gitea 1.24.3 scopes the whole /api/v1/user group, and the dispatcher's first call is GET /user	mint a new token with all four scopes (read:user, write:repository, write:issue, read:package) and put it in both Secrets. Add read:notification if you want the cheap pull-request mention query (platform/pm-agent/README.md)
a worker Job is refused by admission (pm-agent-job-shape, imagePullSecrets)	PM_WORKER_IMAGE is a digest built before the template carried imagePullSecrets (PR #61), so its baked template lacks the entry admission requires	pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep PM_DRY_RUN=true
PM_WORKER_IMAGE must be digest-pinned	someone put a tag in the ConfigMap	restore the digest form; the digest comes from CI's publish (.gitea/workflows/images.yaml job summary) and Renovate's bump PR, never from a hand push
scope: ONLY issues labelled 'agent'	PM_REQUIRE_LABEL is set; unlabelled issues are out of scope by configuration	label the issue, or clear the knob
cycle: 2 worker(s) active and nothing new	at PM_MAX_ACTIVE_WORKERS	wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi)
at the worker cap / worker limit reached	the per-cycle cap, draining a backlog one issue at a time	expected; it will continue next cycle
nothing at all, not even cycle: lines	the process is not running	check the pod, then the events
pull-request mentions: off	PM_PR_MENTIONS is false, which is the default	read PM_PLAN output once, then set it to "true" (ConfigMap + config-generation bump)
pr-mentions: /notifications was refused …	expected on today's token: Gitea puts that endpoint behind its notification scope and the bot's token is minted read:user, write:repository, write:issue, read:package. The dispatcher has fallen back to listing open pull requests, so it still works — just not cheaply	re-mint the token with read:notification added and update both Secrets (below), or set PM_PR_MENTION_DISCOVERY: "pulls" to stop trying
pr-mentions: 0 candidate pull request(s) from notifications-unavailable	PM_PR_MENTION_DISCOVERY is notifications and that query is refused, so the path deliberately does nothing rather than widening into a full scan	fix the scope, or set the knob to auto
[PM_PLAN: nothing will be posted]	PM_PLAN or PM_DRY_RUN is set; the payload is printed and no reply is posted	expected; clear both to let it reply
pull #N: inline reply refused …; falling back to the conversation	Gitea would not accept a comment at that diff position (usually the position no longer exists on the current head)	nothing to fix — the reply was posted in the conversation, naming the file and position

It will not answer an issue whose newest comment is its own. That is the trigger, not a bug: it acts only when the newest thing in a thread was written by somebody else. If you want it to look again, comment on the issue.

It will not re-answer an issue within PM_ISSUE_COOLDOWN_SECONDS (15 min) of its last attempt on that issue. Restarting the pod clears that timer.

On a pull request it answers explicit mentions and nothing else. A comment that does not write @-the-bot-account is never answered and costs no model call, however recently the bot was mentioned or replied on that pull request. If you asked something and got no reply, the likely cause is a missing @ — tag it again. (Near misses do not count either: @pm-agent-worker and [email protected] are deliberately not mentions.)

It will not answer the same pull-request comment twice. Every reply it posts on a pull request carries a hidden marker naming the comment ids it answered (<!-- pm-agent:replied-to=… -->), and it skips anything already marked. To make it answer again, write a new comment that mentions it again — editing the old one does not change its id. PM_PR_COOLDOWN_SECONDS (15 min) bounds it to one decision per pull request per window.

---

A worker failed

Every worker posts its own outcome on the issue, including failures — so read the issue first; it usually names the step. Then:

# which workers ran, and how they ended (Jobs are kept 48h)
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get jobs -l app.kubernetes.io/name=pm-agent-worker \
     -L pm-agent.homelab.keithjackson.dev/issue'
# the full log of one
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent logs job/pm-agent-task-<issue>-<ts> --tail=200'

What you see	Meaning
issue comment: "the merge gate refused it" with a gate log	working as intended. Nothing was pushed. Comment on the issue with guidance and the PM will send a worker back
unexpected skipped stage '<x>' — rc 3 is NOT acceptable	the gate could not run a stage that is not one of the two expected skips (spec-immutability, gitea-settings). Treated as a refusal, correctly: a stage that did not run is not green
branch '<x>' already exists on the remote, but this assignment was 'implement'	the PM thought this was new work. Nothing was forced. Ask for a revision of the existing branch instead
is not an allowed git host	PM_GITEA_URL was changed to something outside the allowlist baked into pm-worker. That allowlist is deliberate — changing it means rebuilding the image
Error from server (Forbidden) creating a Job, in the dispatcher log	admission denied the Job. Read the message: it names the rule. Either the image digest is not under the agent-runner path, or the template drifted out of the shape the policy admits
Job DeadlineExceeded after 90 min	a wedged session. Nothing was pushed (the push is the last step)
nothing on the issue at all and the Job failed	the worker died before it could comment. The pod log is the only record; the Job is kept 48h

---

The Anthropic token expires after a year

claude setup-token mints a one-year OAuth token, and nothing warns you: Claude Code's expiry notice only appears for an interactive login, never for the CLAUDE_CODE_OAUTH_TOKEN variable. When it lapses, every session fails at once and the symptom is workers failing with an authentication error.

Rotate it exactly as it was created — claude setup-token on a machine with a browser, then sops -e -i platform/pm-agent/pm-agent-credentials.sops.yaml, then a pull request. Restart the dispatcher pod afterwards so it re-reads the Secret (a Secret change does not roll a Deployment by itself); worker Jobs pick the new value up on their next run with no action.

The date it was minted belongs in the handoff entry. Check it now:

grep -rn 'claude setup-token' decisions/ | head

---

Checks worth making, and the one you must not

Confirm the bot account is NOT an administrator. This is the control that stops the service clearing its own sensitive-path review — infra/gitea/approval-policy.json authorises /approve-override by live repository permission alone, so an admin-permission bot could waive the human review on its own pull request. It must print write:

curl -s -H "Authorization: token <the pm-agent token>" \
  https://gitea.home.keithjackson.dev/api/v1/repos/Home-Network/home-network-k3s-impl/collaborators/pm-agent/permission \
  | python3 -c 'import json,sys; print(json.load(sys.stdin).get("permission"))'

Confirm the admission policy is actually applied — a policy that is in git and not in the applied set is the exact failure this repository has already had once:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusterpolicy pm-agent-job-shape \
     -o jsonpath="{.spec.validationFailureAction}{\"\n\"}"'

Enforce, or the boundary is not there.

Never sops -d the credentials file into a terminal, a log or a transcript, and never print the Secret's .data to check that it exists. Key names only:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get secret pm-agent-credentials \
     -o jsonpath="{.data}" | tr "," "\n" | cut -d\" -f2'

(That prints the keys. A password has been leaked into a transcript in this repository by printing the values.)