jakman85andClaude Opus 5
7fa5e84d25
The registry had two blockers. #59 fixed the first (nodes resolved the hostname to the non-hairpinned WAN and failed TLS against the router's CN=%D cert). This is the second: the pull is unauthenticated, and this registry refuses unauthenticated clients outright — measured 2026-10-05, its anonymous token carries access: null and every path 401s. pm-agent-registry is a dockerconfigjson Secret (SOPS, placeholders) referenced by imagePullSecrets on the dispatcher Deployment and on the committed worker Job template. It holds the SAME Gitea token as pm-agent-credentials, because the two consumers need two formats: the application reads a raw token from env, the kubelet reads a dockerconfigjson before any container starts. Same one-value-in-two-files pattern as the Keycloak client secret, documented in both .example files. The operator's part is the value plus one new scope: read:package. ADMISSION GAINED A SEVENTH RULE. imagePullSecrets is a pod-spec field no container references, which is exactly why it is easy to leave unread — the same shape of omission as initContainers, ephemeralContainers, env and envFrom before it. The kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now REQUIRES the committed entry; rule 7 forbids any other and any second one. Both halves are needed: a foreach over an absent list yields no result. The leverage is small and the policy says so: the image is digest-pinned, so a different pull credential cannot change WHAT runs. Pinned anyway — 'only the committed shape' is the premise, and the rule costs four lines. kyverno test: 33 passed, 0 failed (was 28). Three new fixtures, each a real path: the committed entry plus a second one, a different Secret, and none at all. The gate also caught me shipping the Secret in plaintext — I never ran sops -e -i on the placeholder. Recorded in the handoff entry, because the file looked finished. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
170 lines
10 KiB
Markdown
170 lines
10 KiB
Markdown
# pm-agent — the Program Manager agent service
|
|
|
|
Operator procedure, not alert-linked. The service reads Gitea issues, decides what to do
|
|
about each thread as the Program Manager, and runs one worker agent per assignment that
|
|
authors a change, runs the merge gate and opens a pull request.
|
|
|
|
- What it is and why it is shaped this way: [`platform/pm-agent/README.md`](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl/src/branch/main/platform/pm-agent/README.md)
|
|
- The design and the rejected alternatives: `docs/plans/pm-agent-issue-to-pr.md`
|
|
|
|
**Everything below is read-only except where it says otherwise.** Mutating `kubectl` and
|
|
`flux reconcile` are operator actions, not agent actions.
|
|
|
|
---
|
|
|
|
## Stop it. Right now.
|
|
|
|
In order of increasing severity. Each is an operator action.
|
|
|
|
**1. Stop it taking new work, keep it running.** Set `PM_DRY_RUN: "true"` in
|
|
`platform/pm-agent/configmap.yaml`, bump the `config-generation` annotation in
|
|
`deployment.yaml`, merge. It still reads threads and still posts what it *would* have done,
|
|
and creates no worker Jobs. This is also the right setting for watching it for a day.
|
|
|
|
**2. Stop it immediately, without a merge.**
|
|
|
|
```bash
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent scale deploy/pm-agent --replicas=0'
|
|
```
|
|
|
|
Flux will put it back at the next reconcile (10m). To hold it down, suspend the tier:
|
|
|
|
```bash
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'flux --context k3s-prod suspend kustomization pm-agent'
|
|
```
|
|
|
|
Running worker Jobs are **not** stopped by either. They finish, or:
|
|
|
|
```bash
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent delete job -l app.kubernetes.io/name=pm-agent-worker'
|
|
```
|
|
|
|
**3. Revoke its reach.** Delete the Gitea token in Gitea (as the `pm-agent` user: Settings →
|
|
Applications → the token → Delete). Every worker and the dispatcher then fail
|
|
authentication, nothing can be pushed, and no comment can be posted. This is the one that
|
|
does not depend on the cluster behaving.
|
|
|
|
A branch the agent already pushed is not undone by any of the above. A pull request it
|
|
opened cannot merge without a green gate, and — if it touched a sensitive path — not without
|
|
a human review either.
|
|
|
|
---
|
|
|
|
## It is not doing anything
|
|
|
|
Work through these in order; the first three are by far the most likely.
|
|
|
|
```bash
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent get deploy,pod,job'
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent logs deploy/pm-agent --tail=60'
|
|
```
|
|
|
|
| Symptom in the log | Cause | Fix |
|
|
|---|---|---|
|
|
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
|
|
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
|
|
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
|
| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o jsonpath='{.data}' \| tr ',' '\n' \| cut -d'"' -f2` should list `.dockerconfigjson`. Then confirm the token carries `read:package` and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
|
|
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
|
|
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
|
|
| `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
|
|
| `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle |
|
|
| nothing at all, not even `cycle:` lines | the process is not running | check the pod, then the events |
|
|
|
|
**It will not answer an issue whose newest comment is its own.** That is the trigger, not a
|
|
bug: it acts only when the newest thing in a thread was written by somebody else. If you want
|
|
it to look again, comment on the issue.
|
|
|
|
**It will not re-answer an issue within `PM_ISSUE_COOLDOWN_SECONDS`** (15 min) of its last
|
|
attempt on that issue. Restarting the pod clears that timer.
|
|
|
|
---
|
|
|
|
## A worker failed
|
|
|
|
Every worker posts its own outcome on the issue, including failures — so **read the issue
|
|
first**; it usually names the step. Then:
|
|
|
|
```bash
|
|
# which workers ran, and how they ended (Jobs are kept 48h)
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent get jobs -l app.kubernetes.io/name=pm-agent-worker \
|
|
-L pm-agent.homelab.keithjackson.dev/issue'
|
|
# the full log of one
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent logs job/pm-agent-task-<issue>-<ts> --tail=200'
|
|
```
|
|
|
|
| What you see | Meaning |
|
|
|---|---|
|
|
| issue comment: *"the merge gate refused it"* with a gate log | working as intended. Nothing was pushed. Comment on the issue with guidance and the PM will send a worker back |
|
|
| `unexpected skipped stage '<x>' — rc 3 is NOT acceptable` | the gate could not run a stage that is not one of the two expected skips (`spec-immutability`, `gitea-settings`). Treated as a refusal, correctly: a stage that did not run is not green |
|
|
| `branch '<x>' already exists on the remote, but this assignment was 'implement'` | the PM thought this was new work. Nothing was forced. Ask for a revision of the existing branch instead |
|
|
| `is not an allowed git host` | `PM_GITEA_URL` was changed to something outside the allowlist baked into `pm-worker`. That allowlist is deliberate — changing it means rebuilding the image |
|
|
| `Error from server (Forbidden)` creating a Job, in the **dispatcher** log | admission denied the Job. Read the message: it names the rule. Either the image digest is not under the agent-runner path, or the template drifted out of the shape the policy admits |
|
|
| Job `DeadlineExceeded` after 90 min | a wedged session. Nothing was pushed (the push is the last step) |
|
|
| nothing on the issue at all and the Job failed | the worker died before it could comment. The pod log is the only record; the Job is kept 48h |
|
|
|
|
---
|
|
|
|
## The Anthropic token expires after a year
|
|
|
|
`claude setup-token` mints a **one-year** OAuth token, and **nothing warns you**: Claude
|
|
Code's expiry notice only appears for an interactive login, never for the
|
|
`CLAUDE_CODE_OAUTH_TOKEN` variable. When it lapses, every session fails at once and the
|
|
symptom is workers failing with an authentication error.
|
|
|
|
Rotate it exactly as it was created — `claude setup-token` on a machine with a browser, then
|
|
`sops -e -i platform/pm-agent/pm-agent-credentials.sops.yaml`, then a pull request. Restart
|
|
the dispatcher pod afterwards so it re-reads the Secret (a Secret change does not roll a
|
|
Deployment by itself); worker Jobs pick the new value up on their next run with no action.
|
|
|
|
The date it was minted belongs in the handoff entry. Check it now:
|
|
|
|
```bash
|
|
grep -rn 'claude setup-token' decisions/ | head
|
|
```
|
|
|
|
---
|
|
|
|
## Checks worth making, and the one you must not
|
|
|
|
**Confirm the bot account is NOT an administrator.** This is the control that stops the
|
|
service clearing its own sensitive-path review — `infra/gitea/approval-policy.json`
|
|
authorises `/approve-override` by live repository permission alone, so an admin-permission
|
|
bot could waive the human review on its own pull request. It must print `write`:
|
|
|
|
```bash
|
|
curl -s -H "Authorization: token <the pm-agent token>" \
|
|
https://gitea.home.keithjackson.dev/api/v1/repos/Home-Network/home-network-k3s-impl/collaborators/pm-agent/permission \
|
|
| python3 -c 'import json,sys; print(json.load(sys.stdin).get("permission"))'
|
|
```
|
|
|
|
**Confirm the admission policy is actually applied** — a policy that is in git and not in
|
|
the applied set is the exact failure this repository has already had once:
|
|
|
|
```bash
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get clusterpolicy pm-agent-job-shape \
|
|
-o jsonpath="{.spec.validationFailureAction}{\"\n\"}"'
|
|
```
|
|
|
|
`Enforce`, or the boundary is not there.
|
|
|
|
**Never** `sops -d` the credentials file into a terminal, a log or a transcript, and never
|
|
print the Secret's `.data` to check that it exists. Key names only:
|
|
|
|
```bash
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n pm-agent get secret pm-agent-credentials \
|
|
-o jsonpath="{.data}" | tr "," "\n" | cut -d\" -f2'
|
|
```
|
|
|
|
(That prints the **keys**. A password has been leaked into a transcript in this repository by
|
|
printing the values.)
|