Merge pull request 'feat(pm-agent): an in-cluster AI Program Manager that turns Gitea issues into pull requests' (#57) from feat/pm-agent-issue-to-pr into main

Reviewed-on: #57
Reviewed-by: Keith Jackson <[email protected]>

runbooks/kustomization.yaml

@@ -108,6 +108,12 @@ configMapGenerator:
       # the Plex UI — the one step GitOps cannot reach — and a page nobody can find is how a shipped
       # feature stays switched off for months.
       - plex-quicksync=plex-quicksync.md
+      # Operator procedure, not alert-linked: how to STOP the PM agent service (three ways,
+      # increasing severity), how to read why it is idle, how to tell a worker that was
+      # correctly refused by the gate from one that broke, and the one-year expiry of its
+      # Anthropic subscription token — which nothing warns about, because Claude Code's expiry
+      # notice only fires for an interactive login, never for CLAUDE_CODE_OAUTH_TOKEN.
+      - pm-agent=pm-agent.md
       - pvc-capacity=pvc-capacity.md
       - targets-down=targets-down.md
       - update-rollback=update-rollback.md

runbooks/pm-agent.md

@@ -0,0 +1,167 @@
+# pm-agent — the Program Manager agent service
+
+Operator procedure, not alert-linked. The service reads Gitea issues, decides what to do
+about each thread as the Program Manager, and runs one worker agent per assignment that
+authors a change, runs the merge gate and opens a pull request.
+
+- What it is and why it is shaped this way: [`platform/pm-agent/README.md`](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/src/branch/main/platform/pm-agent/README.md)
+- The design and the rejected alternatives: `docs/plans/pm-agent-issue-to-pr.md`
+
+**Everything below is read-only except where it says otherwise.** Mutating `kubectl` and
+`flux reconcile` are operator actions, not agent actions.
+
+---
+
+## Stop it. Right now.
+
+In order of increasing severity. Each is an operator action.
+
+**1. Stop it taking new work, keep it running.** Set `PM_DRY_RUN: "true"` in
+`platform/pm-agent/configmap.yaml`, bump the `config-generation` annotation in
+`deployment.yaml`, merge. It still reads threads and still posts what it *would* have done,
+and creates no worker Jobs. This is also the right setting for watching it for a day.
+
+**2. Stop it immediately, without a merge.**
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n pm-agent scale deploy/pm-agent --replicas=0'
+```
+
+Flux will put it back at the next reconcile (10m). To hold it down, suspend the tier:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'flux --context k3s-prod suspend kustomization pm-agent'
+```
+
+Running worker Jobs are **not** stopped by either. They finish, or:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n pm-agent delete job -l app.kubernetes.io/name=pm-agent-worker'
+```
+
+**3. Revoke its reach.** Delete the Gitea token in Gitea (as the `pm-agent` user: Settings →
+Applications → the token → Delete). Every worker and the dispatcher then fail
+authentication, nothing can be pushed, and no comment can be posted. This is the one that
+does not depend on the cluster behaving.
+
+A branch the agent already pushed is not undone by any of the above. A pull request it
+opened cannot merge without a green gate, and — if it touched a sensitive path — not without
+a human review either.
+
+---
+
+## It is not doing anything
+
+Work through these in order; the first three are by far the most likely.
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n pm-agent get deploy,pod,job'
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n pm-agent logs deploy/pm-agent --tail=60'
+```
+
+| Symptom in the log | Cause | Fix |
+|---|---|---|
+| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
+| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) |
+| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
+| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
+| `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
+| `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle |
+| nothing at all, not even `cycle:` lines | the process is not running | check the pod, then the events |
+
+**It will not answer an issue whose newest comment is its own.** That is the trigger, not a
+bug: it acts only when the newest thing in a thread was written by somebody else. If you want
+it to look again, comment on the issue.
+
+**It will not re-answer an issue within `PM_ISSUE_COOLDOWN_SECONDS`** (15 min) of its last
+attempt on that issue. Restarting the pod clears that timer.
+
+---
+
+## A worker failed
+
+Every worker posts its own outcome on the issue, including failures — so **read the issue
+first**; it usually names the step. Then:
+
+```bash
+# which workers ran, and how they ended (Jobs are kept 48h)
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n pm-agent get jobs -l app.kubernetes.io/name=pm-agent-worker \
+     -L pm-agent.homelab.keithjackson.dev/issue'
+# the full log of one
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n pm-agent logs job/pm-agent-task-<issue>-<ts> --tail=200'
+```
+
+| What you see | Meaning |
+|---|---|
+| issue comment: *"the merge gate refused it"* with a gate log | working as intended. Nothing was pushed. Comment on the issue with guidance and the PM will send a worker back |
+| `unexpected skipped stage '<x>' — rc 3 is NOT acceptable` | the gate could not run a stage that is not one of the two expected skips (`spec-immutability`, `gitea-settings`). Treated as a refusal, correctly: a stage that did not run is not green |
+| `branch '<x>' already exists on the remote, but this assignment was 'implement'` | the PM thought this was new work. Nothing was forced. Ask for a revision of the existing branch instead |
+| `is not an allowed git host` | `PM_GITEA_URL` was changed to something outside the allowlist baked into `pm-worker`. That allowlist is deliberate — changing it means rebuilding the image |
+| `Error from server (Forbidden)` creating a Job, in the **dispatcher** log | admission denied the Job. Read the message: it names the rule. Either the image digest is not under the agent-runner path, or the template drifted out of the shape the policy admits |
+| Job `DeadlineExceeded` after 90 min | a wedged session. Nothing was pushed (the push is the last step) |
+| nothing on the issue at all and the Job failed | the worker died before it could comment. The pod log is the only record; the Job is kept 48h |
+
+---
+
+## The Anthropic token expires after a year
+
+`claude setup-token` mints a **one-year** OAuth token, and **nothing warns you**: Claude
+Code's expiry notice only appears for an interactive login, never for the
+`CLAUDE_CODE_OAUTH_TOKEN` variable. When it lapses, every session fails at once and the
+symptom is workers failing with an authentication error.
+
+Rotate it exactly as it was created — `claude setup-token` on a machine with a browser, then
+`sops -e -i platform/pm-agent/pm-agent-credentials.sops.yaml`, then a pull request. Restart
+the dispatcher pod afterwards so it re-reads the Secret (a Secret change does not roll a
+Deployment by itself); worker Jobs pick the new value up on their next run with no action.
+
+The date it was minted belongs in the handoff entry. Check it now:
+
+```bash
+grep -rn 'claude setup-token' decisions/ | head
+```
+
+---
+
+## Checks worth making, and the one you must not
+
+**Confirm the bot account is NOT an administrator.** This is the control that stops the
+service clearing its own sensitive-path review — `infra/gitea/approval-policy.json`
+authorises `/approve-override` by live repository permission alone, so an admin-permission
+bot could waive the human review on its own pull request. It must print `write`:
+
+```bash
+curl -s -H "Authorization: token <the pm-agent token>" \
+  https://gitea.home.keithjackson.dev/api/v1/repos/Home-Network/home-network-k3s-impl/collaborators/pm-agent/permission \
+  | python3 -c 'import json,sys; print(json.load(sys.stdin).get("permission"))'
+```
+
+**Confirm the admission policy is actually applied** — a policy that is in git and not in
+the applied set is the exact failure this repository has already had once:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get clusterpolicy pm-agent-job-shape \
+     -o jsonpath="{.spec.validationFailureAction}{\"\n\"}"'
+```
+
+`Enforce`, or the boundary is not there.
+
+**Never** `sops -d` the credentials file into a terminal, a log or a transcript, and never
+print the Secret's `.data` to check that it exists. Key names only:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n pm-agent get secret pm-agent-credentials \
+     -o jsonpath="{.data}" | tr "," "\n" | cut -d\" -f2'
+```
+
+(That prints the **keys**. A password has been leaked into a transcript in this repository by
+printing the values.)