jakman85andClaude Opus 5
90a79d29d3
Node configuration in this project is Ansible's — declared in the inventory, reviewed,
version-controlled — and for prod it is the ONLY surface: infra/PROD-PROVISIONING.md
records prod as 3x bare metal with the dev OpenTofu -> Proxmox -> cloud-init path
explicitly not applying, so there is no tofu/cloud-init half to keep in parity for an
/etc/hosts pin.
The pin was already in the right place; what the docs lacked was the command. All four
places now carry the scoped, idempotent one:
cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
-e k3s_hosts=k3s_prod --tags hosts
--tags hosts runs ONLY the /etc/hosts task, so applying this pin cannot drift any other
baseline setting.
Also verified the mechanism is not inert, which is the failure mode this repo keeps
meeting: roles/node_baseline/tasks/main.yml renders node_baseline_hosts_pins into a marked
blockinfile with backup: true, and state: absent when the list empties — so a removed
entry really does leave the node.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
170 lines
9.7 KiB
Markdown
170 lines
9.7 KiB
Markdown
# pm-agent — the Program Manager agent service
|
|
|
|
Operator procedure, not alert-linked. The service reads Gitea issues, decides what to do
|
|
about each thread as the Program Manager, and runs one worker agent per assignment that
|
|
authors a change, runs the merge gate and opens a pull request.
|
|
|
|
- What it is and why it is shaped this way: [`platform/pm-agent/README.md`](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/src/branch/main/platform/pm-agent/README.md)
|
|
- The design and the rejected alternatives: `docs/plans/pm-agent-issue-to-pr.md`
|
|
|
|
**Everything below is read-only except where it says otherwise.** Mutating `kubectl` and
|
|
`flux reconcile` are operator actions, not agent actions.
|
|
|
|
---
|
|
|
|
## Stop it. Right now.
|
|
|
|
In order of increasing severity. Each is an operator action.
|
|
|
|
**1. Stop it taking new work, keep it running.** Set `PM_DRY_RUN: "true"` in
|
|
`platform/pm-agent/configmap.yaml`, bump the `config-generation` annotation in
|
|
`deployment.yaml`, merge. It still reads threads and still posts what it *would* have done,
|
|
and creates no worker Jobs. This is also the right setting for watching it for a day.
|
|
|
|
**2. Stop it immediately, without a merge.**
|
|
|
|
```bash
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent scale deploy/pm-agent --replicas=0'
|
|
```
|
|
|
|
Flux will put it back at the next reconcile (10m). To hold it down, suspend the tier:
|
|
|
|
```bash
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'flux --context k3s-prod suspend kustomization pm-agent'
|
|
```
|
|
|
|
Running worker Jobs are **not** stopped by either. They finish, or:
|
|
|
|
```bash
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent delete job -l app.kubernetes.io/name=pm-agent-worker'
|
|
```
|
|
|
|
**3. Revoke its reach.** Delete the Gitea token in Gitea (as the `pm-agent` user: Settings →
|
|
Applications → the token → Delete). Every worker and the dispatcher then fail
|
|
authentication, nothing can be pushed, and no comment can be posted. This is the one that
|
|
does not depend on the cluster behaving.
|
|
|
|
A branch the agent already pushed is not undone by any of the above. A pull request it
|
|
opened cannot merge without a green gate, and — if it touched a sensitive path — not without
|
|
a human review either.
|
|
|
|
---
|
|
|
|
## It is not doing anything
|
|
|
|
Work through these in order; the first three are by far the most likely.
|
|
|
|
```bash
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent get deploy,pod,job'
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent logs deploy/pm-agent --tail=60'
|
|
```
|
|
|
|
| Symptom in the log | Cause | Fix |
|
|
|---|---|---|
|
|
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
|
|
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
|
|
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
|
| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
|
|
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
|
|
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
|
|
| `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
|
|
| `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle |
|
|
| nothing at all, not even `cycle:` lines | the process is not running | check the pod, then the events |
|
|
|
|
**It will not answer an issue whose newest comment is its own.** That is the trigger, not a
|
|
bug: it acts only when the newest thing in a thread was written by somebody else. If you want
|
|
it to look again, comment on the issue.
|
|
|
|
**It will not re-answer an issue within `PM_ISSUE_COOLDOWN_SECONDS`** (15 min) of its last
|
|
attempt on that issue. Restarting the pod clears that timer.
|
|
|
|
---
|
|
|
|
## A worker failed
|
|
|
|
Every worker posts its own outcome on the issue, including failures — so **read the issue
|
|
first**; it usually names the step. Then:
|
|
|
|
```bash
|
|
# which workers ran, and how they ended (Jobs are kept 48h)
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent get jobs -l app.kubernetes.io/name=pm-agent-worker \
|
|
-L pm-agent.homelab.keithjackson.dev/issue'
|
|
# the full log of one
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent logs job/pm-agent-task-<issue>-<ts> --tail=200'
|
|
```
|
|
|
|
| What you see | Meaning |
|
|
|---|---|
|
|
| issue comment: *"the merge gate refused it"* with a gate log | working as intended. Nothing was pushed. Comment on the issue with guidance and the PM will send a worker back |
|
|
| `unexpected skipped stage '<x>' — rc 3 is NOT acceptable` | the gate could not run a stage that is not one of the two expected skips (`spec-immutability`, `gitea-settings`). Treated as a refusal, correctly: a stage that did not run is not green |
|
|
| `branch '<x>' already exists on the remote, but this assignment was 'implement'` | the PM thought this was new work. Nothing was forced. Ask for a revision of the existing branch instead |
|
|
| `is not an allowed git host` | `PM_GITEA_URL` was changed to something outside the allowlist baked into `pm-worker`. That allowlist is deliberate — changing it means rebuilding the image |
|
|
| `Error from server (Forbidden)` creating a Job, in the **dispatcher** log | admission denied the Job. Read the message: it names the rule. Either the image digest is not under the agent-runner path, or the template drifted out of the shape the policy admits |
|
|
| Job `DeadlineExceeded` after 90 min | a wedged session. Nothing was pushed (the push is the last step) |
|
|
| nothing on the issue at all and the Job failed | the worker died before it could comment. The pod log is the only record; the Job is kept 48h |
|
|
|
|
---
|
|
|
|
## The Anthropic token expires after a year
|
|
|
|
`claude setup-token` mints a **one-year** OAuth token, and **nothing warns you**: Claude
|
|
Code's expiry notice only appears for an interactive login, never for the
|
|
`CLAUDE_CODE_OAUTH_TOKEN` variable. When it lapses, every session fails at once and the
|
|
symptom is workers failing with an authentication error.
|
|
|
|
Rotate it exactly as it was created — `claude setup-token` on a machine with a browser, then
|
|
`sops -e -i platform/pm-agent/pm-agent-credentials.sops.yaml`, then a pull request. Restart
|
|
the dispatcher pod afterwards so it re-reads the Secret (a Secret change does not roll a
|
|
Deployment by itself); worker Jobs pick the new value up on their next run with no action.
|
|
|
|
The date it was minted belongs in the handoff entry. Check it now:
|
|
|
|
```bash
|
|
grep -rn 'claude setup-token' decisions/ | head
|
|
```
|
|
|
|
---
|
|
|
|
## Checks worth making, and the one you must not
|
|
|
|
**Confirm the bot account is NOT an administrator.** This is the control that stops the
|
|
service clearing its own sensitive-path review — `infra/gitea/approval-policy.json`
|
|
authorises `/approve-override` by live repository permission alone, so an admin-permission
|
|
bot could waive the human review on its own pull request. It must print `write`:
|
|
|
|
```bash
|
|
curl -s -H "Authorization: token <the pm-agent token>" \
|
|
https://gitea.home.keithjackson.dev/api/v1/repos/Home-Network/home-network-k3s-impl/collaborators/pm-agent/permission \
|
|
| python3 -c 'import json,sys; print(json.load(sys.stdin).get("permission"))'
|
|
```
|
|
|
|
**Confirm the admission policy is actually applied** — a policy that is in git and not in
|
|
the applied set is the exact failure this repository has already had once:
|
|
|
|
```bash
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get clusterpolicy pm-agent-job-shape \
|
|
-o jsonpath="{.spec.validationFailureAction}{\"\n\"}"'
|
|
```
|
|
|
|
`Enforce`, or the boundary is not there.
|
|
|
|
**Never** `sops -d` the credentials file into a terminal, a log or a transcript, and never
|
|
print the Secret's `.data` to check that it exists. Key names only:
|
|
|
|
```bash
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent get secret pm-agent-credentials \
|
|
-o jsonpath="{.data}" | tr "," "\n" | cut -d\" -f2'
|
|
```
|
|
|
|
(That prints the **keys**. A password has been leaked into a transcript in this repository by
|
|
printing the values.)
|