home-network-k3s-impl/runbooks/pm-agent.md

pm-agent — the Program Manager agent service

Operator procedure, not alert-linked. The service reads Gitea issues, decides what to do about each thread as the Program Manager, and runs one worker agent per assignment that authors a change, runs the merge gate and opens a pull request.

• What it is and why it is shaped this way: platform/pm-agent/README.md
• The design and the rejected alternatives: docs/plans/pm-agent-issue-to-pr.md

Everything below is read-only except where it says otherwise. Mutating kubectl and flux reconcile are operator actions, not agent actions.

---

Stop it. Right now.

In order of increasing severity. Each is an operator action.

1. Stop it taking new work, keep it running. Set PM_DRY_RUN: "true" in platform/pm-agent/configmap.yaml, bump the config-generation annotation in deployment.yaml, merge. It still reads threads and still posts what it would have done, and creates no worker Jobs. This is also the right setting for watching it for a day.

2. Stop it immediately, without a merge.

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent scale deploy/pm-agent --replicas=0'

Flux will put it back at the next reconcile (10m). To hold it down, suspend the tier:

./infra/devcontainer/dev.sh run bash -c \
  'flux --context k3s-prod suspend kustomization pm-agent'

Running worker Jobs are not stopped by either. They finish, or:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent delete job -l app.kubernetes.io/name=pm-agent-worker'

3. Revoke its reach. Delete the Gitea token in Gitea (as the pm-agent user: Settings → Applications → the token → Delete). Every worker and the dispatcher then fail authentication, nothing can be pushed, and no comment can be posted. This is the one that does not depend on the cluster behaving.

A branch the agent already pushed is not undone by any of the above. A pull request it opened cannot merge without a green gate, and — if it touched a sensitive path — not without a human review either.

---

It is not doing anything

Work through these in order; the first three are by far the most likely.

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get deploy,pod,job'
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent logs deploy/pm-agent --tail=60'

Symptom in the log	Cause	Fix
GET /user returned no login — is GITEA_TOKEN valid? and a CrashLoop	the credentials are still the committed placeholders, or the token was revoked	fill them in per platform/pm-agent/pm-agent-credentials.sops.yaml.example
ImagePullBackOff, digest sha256:0000…	the image has never been pushed; the committed digest is the deliberate placeholder	infra/agent/build-image.sh, operator pushes, then one PR updates the digest in both deployment.yaml and configmap.yaml (PM_WORKER_IMAGE) — but read the row below first
ImagePullBackOff with x509: certificate is not valid for any names	diagnosed. containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own CN=%D placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert	the /etc/hosts pin for gitea.{{ k3s_base_domain }} → 192.168.2.20 in infra/ansible/inventory/group_vars/k3s_prod/vars.yml fixes it — the same pin the Keycloak issuer already has. Run the node_baseline play for it to reach the nodes. Full evidence: decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md
ImagePullBackOff with 401 Unauthorized once TLS works	the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no imagePullSecret exists anywhere in the tree	add one (a dockerconfigjson, SOPS-encrypted like pm-agent-credentials, referenced from the Deployment and the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours
PM_WORKER_IMAGE must be digest-pinned	someone put a tag in the ConfigMap	use the digest from podman push --digestfile
scope: ONLY issues labelled 'agent'	PM_REQUIRE_LABEL is set; unlabelled issues are out of scope by configuration	label the issue, or clear the knob
cycle: 2 worker(s) active and nothing new	at PM_MAX_ACTIVE_WORKERS	wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi)
at the worker cap / worker limit reached	the per-cycle cap, draining a backlog one issue at a time	expected; it will continue next cycle
nothing at all, not even cycle: lines	the process is not running	check the pod, then the events

It will not answer an issue whose newest comment is its own. That is the trigger, not a bug: it acts only when the newest thing in a thread was written by somebody else. If you want it to look again, comment on the issue.

It will not re-answer an issue within PM_ISSUE_COOLDOWN_SECONDS (15 min) of its last attempt on that issue. Restarting the pod clears that timer.

---

A worker failed

Every worker posts its own outcome on the issue, including failures — so read the issue first; it usually names the step. Then:

# which workers ran, and how they ended (Jobs are kept 48h)
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get jobs -l app.kubernetes.io/name=pm-agent-worker \
     -L pm-agent.homelab.keithjackson.dev/issue'
# the full log of one
./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent logs job/pm-agent-task-<issue>-<ts> --tail=200'

What you see	Meaning
issue comment: "the merge gate refused it" with a gate log	working as intended. Nothing was pushed. Comment on the issue with guidance and the PM will send a worker back
unexpected skipped stage '<x>' — rc 3 is NOT acceptable	the gate could not run a stage that is not one of the two expected skips (spec-immutability, gitea-settings). Treated as a refusal, correctly: a stage that did not run is not green
branch '<x>' already exists on the remote, but this assignment was 'implement'	the PM thought this was new work. Nothing was forced. Ask for a revision of the existing branch instead
is not an allowed git host	PM_GITEA_URL was changed to something outside the allowlist baked into pm-worker. That allowlist is deliberate — changing it means rebuilding the image
Error from server (Forbidden) creating a Job, in the dispatcher log	admission denied the Job. Read the message: it names the rule. Either the image digest is not under the agent-runner path, or the template drifted out of the shape the policy admits
Job DeadlineExceeded after 90 min	a wedged session. Nothing was pushed (the push is the last step)
nothing on the issue at all and the Job failed	the worker died before it could comment. The pod log is the only record; the Job is kept 48h

---

The Anthropic token expires after a year

claude setup-token mints a one-year OAuth token, and nothing warns you: Claude Code's expiry notice only appears for an interactive login, never for the CLAUDE_CODE_OAUTH_TOKEN variable. When it lapses, every session fails at once and the symptom is workers failing with an authentication error.

Rotate it exactly as it was created — claude setup-token on a machine with a browser, then sops -e -i platform/pm-agent/pm-agent-credentials.sops.yaml, then a pull request. Restart the dispatcher pod afterwards so it re-reads the Secret (a Secret change does not roll a Deployment by itself); worker Jobs pick the new value up on their next run with no action.

The date it was minted belongs in the handoff entry. Check it now:

grep -rn 'claude setup-token' decisions/ | head

---

Checks worth making, and the one you must not

Confirm the bot account is NOT an administrator. This is the control that stops the service clearing its own sensitive-path review — infra/gitea/approval-policy.json authorises /approve-override by live repository permission alone, so an admin-permission bot could waive the human review on its own pull request. It must print write:

curl -s -H "Authorization: token <the pm-agent token>" \
  https://gitea.home.keithjackson.dev/api/v1/repos/Home-Network/home-network-k3s-impl/collaborators/pm-agent/permission \
  | python3 -c 'import json,sys; print(json.load(sys.stdin).get("permission"))'

Confirm the admission policy is actually applied — a policy that is in git and not in the applied set is the exact failure this repository has already had once:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get clusterpolicy pm-agent-job-shape \
     -o jsonpath="{.spec.validationFailureAction}{\"\n\"}"'

Enforce, or the boundary is not there.

Never sops -d the credentials file into a terminal, a log or a transcript, and never print the Secret's .data to check that it exists. Key names only:

./infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n pm-agent get secret pm-agent-credentials \
     -o jsonpath="{.data}" | tr "," "\n" | cut -d\" -f2'

(That prints the keys. A password has been leaked into a transcript in this repository by printing the values.)