Files
home-network-k3s-impl/platform/pm-agent/README.md
T
pm-agent e8e9de8611
gate / gate (pull_request) Successful in 2m58s
images / images (pull_request) Successful in 1m6s
gate / conformance (pull_request) Has been skipped
feat: PM-Agent also need to look at PR comments
From Gitea issue #116.

Authored by the PM agent's worker; the merge gate is the approver. Refs #116.

Agent: pm-agent (issue #116, mode revise)
2026-11-07 06:02:58 +00:00

406 lines
27 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# pm-agent — an AI Program Manager that turns Gitea issues into pull requests
A service that runs in the cluster, reads the issues of
`Home-Network/home-network-k3s-impl`, decides what to do about each thread **as the Program
Manager persona this repository already works under**, and runs one worker agent per
assignment that authors the change, runs the merge gate and opens a pull request.
It reacts to issue creation and to comments on issues. It can also — **off by default** —
reply when it is `@`-mentioned on a pull request; see
[Mentions on a pull request](#mentions-on-a-pull-request). Every assignment and every piece
of correspondence travels through the issue thread and the pull request, so the whole record
of what the agent did, and why, is in the same place a human is already looking.
It **never pushes to `main`** and **never merges a pull request**. The merge gate remains the
approver, and a sensitive-path change still needs the human review `ci/approval-policy.sh`
requires.
| | |
|---|---|
| Design, alternatives considered, threat model | [`docs/plans/pm-agent-issue-to-pr.md`](../../docs/plans/pm-agent-issue-to-pr.md) |
| Operator runbook — stopping it, reading its logs, rotating its tokens | [`runbooks/pm-agent.md`](../../runbooks/pm-agent.md) |
| The image, the two entrypoints, the PM persona | [`infra/agent/`](../../infra/agent/) |
| The security boundary | [`platform/kyverno/admission/pm-agent-job-shape.yaml`](../kyverno/admission/pm-agent-job-shape.yaml) |
---
## Why the agent authenticates to Gitea with a local account and a PAT
Not Keycloak, and that is a decision rather than an omission (operator, 2026-10-06): the
agents only ever talk to Gitea, so a local Gitea account with a scoped personal access token
is the whole requirement, and it is drastically simpler than federating a machine identity.
It is also what Gitea can actually do. Keycloak service accounts were considered and are not
available for this: Gitea's API accepts basic auth, PATs, SSH signatures and **its own**
OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization
Code Grant** — there is no client-credentials grant, and nothing in Gitea validates a token
issued by a third-party OIDC provider. So a Keycloak service-account token cannot
authenticate to Gitea's API or to git-over-HTTP at all. Keycloak remains what it is for human
access: the browser login source registered on Gitea (`catalog/gitea.yaml`).
What still matters, and is enforced by the hand-off rather than by Keycloak: **the account
must not be an administrator** — `infra/gitea/approval-policy.json` authorises
`/approve-override` on live repository permission alone, so an admin-permission bot could
waive the sensitive-path human review on its own pull request.
## It does nothing at all until an operator does two things
Both are deliberate, and both fail **visibly** rather than silently:
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
`PM_WORKER_IMAGE` in `configmap.yaml` first carried the run-340 publish
(`sha-523cd9118f95`), whose baked template predated the `imagePullSecrets` entry admission
requires, so the service ran with `PM_DRY_RUN=true`. PR #96 pinned the run-383 digest
(`df36d9b`, `sha-ddd04d460ef6`), whose template carries the entry, and dry run was turned
off on 2026-10-31 (`decisions/2026-10-31-pm-agent-go-live.md`); every later bump is
Renovate's. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
(`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
`infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
lines and moves them together in one pull request. Nobody builds-and-pushes this image from
a workstation. The one operator prerequisite on the publishing side is the
`REGISTRY_PUSH_TOKEN` Actions secret (see the header of `.gitea/workflows/images.yaml`);
without it the publish job fails loudly rather than skipping.
**Two things have to be true before any digest will pull, and neither is declared in this
repository.** Both were measured after PR #57 merged — this service is the first workload
in the cluster ever to reference the in-cluster registry, so nothing had exercised the
path. Details and the exact commands:
[`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md).
- **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
*node* process, so it resolves through the node's public upstreams
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
IP — which this router does not hairpin; it answers :443 with its own placeholder
certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
and the digest were never implicated. The fix is the pin this repository already uses
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
inventory, never typed on a node. Apply it scoped to that one task:
`cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
--tags hosts`.
- **The pull must be authenticated** once the node reaches the right endpoint: an
unauthenticated client gets nothing from this registry. **Now wired** —
`pm-agent-registry` is a `dockerconfigjson` Secret referenced by both the Deployment and
the committed worker Job template, and the admission policy pins it to exactly that
name. What is left to you is the value: the Gitea token must carry **`read:package`** in
addition to `read:user`, `write:repository` and `write:issue`, and the dockerconfigjson is built from
it. Steps:
[`pm-agent-registry.sops.yaml.example`](./pm-agent-registry.sops.yaml.example).
The same two apply to `conformance-runner`: same registry path, same publish-then-Renovate
model, never pulled either.
2. **The credentials are placeholders — two Secrets, one token.** Both are encrypted to the
repository's age recipient and both hold `REPLACE-ME-…` strings:
| Secret | type | read by | hand-off |
|---|---|---|---|
| `pm-agent-credentials` | Opaque | the application, as env | [`.example`](./pm-agent-credentials.sops.yaml.example) |
| `pm-agent-registry` | dockerconfigjson | the **kubelet**, before any container starts | [`.example`](./pm-agent-registry.sops.yaml.example) |
They hold the **same** Gitea token in two formats, because a Secret cannot be both — the
same one-value-in-two-files pattern the Keycloak client secret already uses here. So mint
it once, with all four scopes: `read:package` (the pull), `read:user` (the dispatcher's
startup `GET /user` — Gitea 1.24.3 scopes the whole `/user` group), `write:repository`
(clone, branch push, pull request) and `write:issue` (issue comments). The account also
needs package *permission*, not just the scope: a `pm-agent-pull` org team with Packages
= Read and no repositories (steps in the registry `.example`). The first `.example`
explains which account to create and why it must *not* be an administrator.
Until they are filled in: the pull fails `401`, and if you fix only the pull, the
dispatcher CrashLoops with `GET /user returned no login — is GITEA_TOKEN valid?`.
Its Flux Kustomization is `wait: false` precisely because of those two states: a workload
that is *expected* to be unhealthy for a while must never be able to gate anything.
---
## How it works
```
┌──────────────────────── pm-agent namespace ─────────────────────────┐
│ │
Gitea issue ───▶│ pm-dispatch (1 replica, Recreate) │
(poll, 60s) │ 1. list open issues │
│ 2. act iff the NEWEST thing in the thread is not mine │
│ 3. ask the model, under the PM persona, with NO tools │
│ → {action, branch, brief, comment} │
│ 4. validate it, then: create the Job, then post the comment │
│ │ │
│ ▼ │
│ pm-worker (one Job per assignment, no SA token, read-only rootfs) │
│ clone → branch → read the live thread → Claude Code session → │
│ ./ci/gate.sh → commit → push → open the PR → comment on the issue│
└─────────────────────────────────────────────────────────────────────┘
egress: gitea:3000 · kube-apiserver (dispatcher only) · tcp/443 · DNS. ingress: none.
```
**There is no database and no PVC.** The trigger is a question asked of the issue thread
itself — *act iff the newest thing in this thread was not written by me* — and since the PM
always answers in the thread, acting makes the thread quiet again until a human (or a worker)
says something else. The Gitea thread **is** the state. It survives restarts with no recovery
path to get wrong, it is readable by the operator in the place they already look, and it
creates no R-STATE-03 obligation for irreplaceable state.
**It polls; it does not receive webhooks.** Gitea's webhook SSRF guard
(`[webhook] ALLOWED_HOST_LIST`) does not by default permit calling a private in-cluster
address, a missed delivery is simply lost, and a receiver would need inbound reach into a
namespace whose entire purpose is running model-authored code. The cost is up to 60s of
latency. The full argument is in the header of
[`infra/agent/pm-dispatch.py`](../../infra/agent/pm-dispatch.py).
**The PM holds no tools.** It runs `claude --print --tools ""` against a JSON schema: text
in, one JSON object out, no Bash, no file access, no network of its own. The dispatcher then
validates every field — the action against a fixed set, the branch against `CLAUDE.md` §4's
pattern, `revise` against the branch that actually carries the issue's work — and builds the
Job from a committed template. **The model's output is data, never a command.**
---
## Mentions on a pull request
**Off by default** (`PM_PR_MENTIONS: "false"`), and nothing below has been exercised against
the live Gitea — see `decisions/2026-11-07-pm-agent-pr-mentions.md`.
With it on, the dispatcher runs a second, independent pass each poll. A mention of the bot
account on a **pull request** makes the PM read that pull request — its description, its
changed files, its diff, the thread it was mentioned in and the rest of the conversation —
and **reply where it was mentioned**. Three places count: the pull request's conversation, a
review's own body (Gitea renders it in the conversation, so the reply goes there), and an
inline review comment against a file and a line.
```
@pm-agent in a PR comment ─▶ discover (notifications, else open PRs)
or in a review body │
or on a line inline ├─▶ read /issues/{n}/comments, /pulls/{n}/reviews,
│ .../reviews/{id}/comments
├─▶ decide, per comment: self-authored? already
│ answered? does it name the bot?
├─▶ assemble the PR + diff + thread, every field fenced
│ and labelled UNTRUSTED
├─▶ ask the model: `comment` or `decline`, and nothing else
└─▶ post the reply in that thread, carrying its own marker
```
**It can only reply.** `implement` and `revise` are not in the vocabulary this path offers
the model, and the decision schema has no `branch` and no `brief` field, so the PM cannot
ask for work in a shape this program would accept. Nothing on the path calls `build_job()`.
The change-authoring half of issue #116 — revise a branch an agent owns, versus open a fresh
branch and PR against `main` for somebody else's — is a separate issue and a separate PR.
**An explicit mention is the only trigger.** A comment is answered if and only if it is not
the bot's own, it spells out `@`-the-bot-account, and no earlier reply of the bot's says it
was already answered. Nothing else is actionable: a comment with no mention produces **no
reply and no model call**, however recently the bot was mentioned or replied nearby. So a
mention is a discrete, self-limiting request — one `@` gets one reply in the place it was
written, and the rest of the pull request is read as *context* rather than as a queue of
questions.
That is the operator's decision on issue #116, taken once it was established that pull-request
conversation comments are a flat list rather than a hierarchy: *"since there is no threads and
the PR comments are flat, then the pm-agent should be looking for explicit mentions rather than
a single mention and continued scrapping of comments within the PR in perpetuity"*. It replaced
an earlier rule where the first mention in a thread armed replies to every later non-bot
comment in it. It is applied uniformly, including to inline review comments — those *are*
genuinely threaded in Gitea, but keeping continuation for them alone would be a second rule
nobody asked for. The fixtures that used to assert continuation are **inverted, not deleted**,
so the old behaviour cannot creep back unnoticed.
Two practical consequences. A mention is **repeatable**: ask again by tagging again, and each
tag gets its own reply. And a mention is **not a conversation** — the PM is told to answer as
if it is the last word it will see, and to ask you to tag it again if it needs something back.
What remains, unavoidably, is that *anyone* who writes the handle gets a reply — the operator's
answer (3) is "anyone can talk and have the agent work" — and Gitea cannot tell a quoted
mention (`> @pm-agent what do you think?`) from a fresh one, so re-quoting an old mention in a
new comment costs one more reply.
**Thread identity is now only about where the reply goes.** Inline review comments are grouped
by (path, position), which is Gitea's own grouping and is what makes the reply land in the
thread that was spoken to rather than beside it; conversation comments and review bodies are
answered in the conversation. None of that is part of the trigger any more.
**Loop prevention.** There is no author gate — anyone who can comment can start a run, which
is the operator's decision (issue #116, answer 3: *"anyone can talk and have the agent work.
Authorization of changes is through the CI pipeline"*). That makes the self-authored filter
load-bearing, so it is the first check made on every comment and it is structural: a
bot-authored comment can only ever *clear* work, never create it, even if it contains the
bot's own handle. The reply the PM posts also never contains `@`-the-bot. Both are asserted
by the selftest.
**Idempotency.** The record that a mention has been answered is a marker inside the reply:
```
<!-- pm-agent:replied-to=comment-9001,review-comment-77 -->
```
One atomic write with the reply, so the two cannot get out of step. The failure mode chosen,
deliberately, is **at-least-once model call, at-most-once reply**: a crash before posting
costs one repeated decision, a crash after posting is a no-op because the marker is already
in the thread, and a mention is never silently dropped. The marker is only believed in the
bot's *own* comments — pasting it into yours does not silence a mention.
### The cheap mention query, and the token scope it needs
The operator asked for "a way to check for '@'s for the bot account instead of scouring the
entire pull request list and associated threads". That is Gitea's notification list: a
mention subscribes the mentioned account to the thread, so one bounded call names the pull
requests worth reading.
It is a **hint, never the decision** — the trigger is re-derived by reading that pull
request's comments. That split is why the endpoint over-delivering is harmless: the
subscription keeps raising notifications for later comments on a pull request the bot was
once mentioned on, and since those comments carry no mention, they resolve to nothing.
**It will be refused on today's token.** `/api/v1/notifications` sits behind Gitea's
`notification` scope category, and the bot's token is minted
`read:user, write:repository, write:issue, read:package` (above). The dispatcher handles that
explicitly: it logs the refusal, names `read:notification` as the remedy, and falls back to
listing open pull requests, which needs only `read:repository`. Which query answered is in
the log every cycle. To get the cheap one, re-mint the token with `read:notification` added —
same account, same five other scopes, both Secrets updated (`runbooks/pm-agent.md`).
### Look before you turn it on
`PM_PLAN: "true"` makes this path discover, assemble and **print** the exact payload it would
send and the exact thread it would post into, without asking the model and without posting
anything:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n pm-agent logs deploy/pm-agent | sed -n "/^=\{20,\}/,\$p"'
```
Setting `PM_PLAN` also enables the path, so it is one knob. `PM_DRY_RUN` implies it.
To settle the API shapes by hand before enabling anything (read-only, and the bot's token is
enough for all but the first):
```bash
T=<the pm-agent token>; G=https://gitea.home.keithjackson.dev/api/v1
R=repos/Home-Network/home-network-k3s-impl
curl -s -H "Authorization: token $T" "$G/notifications?all=true&subject-type=pull" | head -c 2000
curl -s -H "Authorization: token $T" "$G/$R/pulls/116/reviews"
curl -s -H "Authorization: token $T" "$G/$R/pulls/116/reviews/1/comments"
curl -s -H "Authorization: token $T" "$G/$R/pulls/116/files"
```
The committed fixtures are `infra/agent/testdata/pr-mention-fixtures.json`; they are written
to Gitea's published 1.24 shapes rather than captured from the server, and the file says so
at the top. Replacing them with a real capture is the cheapest way to close that gap.
---
## What the worker can and cannot do
| | |
|---|---|
| **can** | change any file in its own clone; run any command in the container; run the gate; read the issue thread and the pull requests |
| **can** | push **one** branch (the one the PM named, validated twice against `^(feat\|fix\|docs\|chore)/…`) and open/update its pull request |
| **can** | post comments on its own issue |
| **cannot** | push to `main` — refused by `pm-worker` before anything runs, and by branch protection regardless |
| **cannot** | merge anything — the gate is the approver, and the service makes no merge call |
| **cannot** | reach the Kubernetes API: no ServiceAccount token is mounted, the `pm-agent-worker` identity is bound to nothing, and the egress policy does not include the API server. `kubectl` is on its `PATH` (the gate's toolchain needs it) and is inert |
| **cannot** | reach anything except this Gitea and tcp/443 — no package registry, no GitHub |
| **cannot** | clone from or push to anywhere but `gitea.gitea.svc.cluster.local:3000`: `pm-worker` validates `PM_GITEA_URL` against a one-entry allowlist baked into the image. In-cluster consumers reach a service by its cluster-local name — a different path by design from how a client outside the cluster reaches it — so the external hostname is not an accepted value |
| **cannot** | authenticate a `git push` **during the model session**: `GIT_ASKPASS` is unset for its duration, so the restriction is structural rather than a request in a prompt |
| **cannot** | be created in any shape but the committed one — see the admission policy |
| **cannot** | have the kubelet read any Secret but `pm-agent-registry` on its behalf: `imagePullSecrets` is pinned to exactly that one name |
### What is *not* defended, stated plainly
- **The worker's own credentials are inside the model's reach.** The session runs as the same
uid in the same container as the wrapper, so it can read the environment, including its
`GITEA_TOKEN` and `CLAUDE_CODE_OAUTH_TOKEN`. There is no isolation between them and none is
claimed. What bounds it: the Gitea token is scoped `read:user, write:repository, write:issue, read:package` on **one
repository** with no admin permission, and egress reaches only this Gitea and tcp/443.
- **`world:443` is wider than `api.anthropic.com`.** Cilium can express the tighter rule
(`toFQDNs`) but nothing in this repository uses FQDN policy and its prerequisite (Cilium's
DNS proxy) has not been measured here. Narrowing it is a tracked follow-up with that
measurement as its first step, not a one-line edit.
- **A `write` collaborator in Gitea can merge a pull request.** No token scope separates
"push a branch and open a PR" from "merge it". The service issues no merge call — that is
all of `pm-worker`'s API calls, readable in one file — but it is a property of the code, not
of the credential. What the non-admin permission *does* buy structurally is that the bot
**cannot** post `/approve-override`, so a sensitive-path pull request from the agent
correctly sits unmergeable until a human reviews it.
- **The PM cannot see the repository.** It triages from the thread alone, so its brief is a
belief about where the work lives, not a verified fact — which is why the persona requires
it to say so, and why the worker is told to investigate before changing anything.
- **Nothing here has run.** Everything in this directory is authored and gate-checked
offline. No part of this service has been built, pushed, deployed or observed; see the
handoff entry's "What I did NOT verify".
---
## The escalation this service is built around
Kubernetes places **no `use` check** on a Pod's `serviceAccountName`. So "may create Jobs in
namespace X" is equivalent to "may run arbitrary code as the most privileged identity in
namespace X" — no matter how narrow the creator's own Role looks. The dispatcher creates
Jobs. That is the whole problem, and this repository has met it before
(`conformance/incluster/rbac.yaml`).
Two things answer it, and neither is sufficient alone:
1. **There is nothing to escalate to.** `pm-agent-worker` is bound to no Role at all, and the
Job template mounts no token. The prize is empty.
2. **The shape is constrained at the API server.**
`platform/kyverno/admission/pm-agent-job-shape.yaml` is `Enforce` + `failurePolicy: Fail`
and admits only: the digest-pinned `agent-runner` image, exactly one container, the pinned
`command` with no args, that powerless SA, no mounted token, `restartPolicy: Never`,
non-root with no privilege escalation and a read-only root filesystem, only the
`workspace`/`tmp` **emptyDir** volumes, no init or ephemeral containers, only the
committed env **names**, no `envFrom`, and exactly one `imagePullSecret` — the committed
one.
Seven rules, because pinning `command` only moves the injection to whatever adjacent field
nobody enumerated — on the sibling policy that was `initContainers` (measured: a Job with a
compliant `containers[0]` and a privileged init container was **admitted**, pass 3 fail 0),
then `ephemeralContainers`, then `env`, then `envFrom`. All of them are closed here from the
start, and `platform/kyverno/tests/pm-agent/` proves each refusal offline: **33 assertions,
33 passing** under the pinned `kyverno` CLI. The worker's own input guards have their own
offline probe — `infra/agent/probe-validation.sh`, **17 refusals** — because a guard nothing
exercises is a claim rather than a control.
> The policy is applied by `platform-config/kustomization.yaml`, and it has to be: adding a
> ClusterPolicy only to `platform/kyverno/admission/kustomization.yaml` is a **silent
> no-op**. That is exactly how the conformance boundary shipped green and never reached the
> cluster on 2026-09-29.
---
## Operating it
Every knob is in [`configmap.yaml`](./configmap.yaml), with the reasoning next to it. The
five that matter:
| knob | default | what it is for |
|---|---|---|
| `PM_REQUIRE_LABEL` | `""` (every open issue) | set it to `agent` and only issues a human has labelled are in scope. **Recommended for the first weeks** — it bounds the agent to work somebody deliberately handed it, with no code change |
| `PM_DRY_RUN` | `false` | still triages and still posts what it would have done; creates no worker. The honest way to watch it for a day |
| `PM_MAX_ACTIVE_WORKERS` | `2` | concurrent workers. RAM is the binding constraint (2Gi limit each) and the subscription seat's rate limits are shared with every other use of that seat |
| `PM_MAX_NEW_WORKERS_PER_CYCLE` | `1` | drains a backlog one issue at a time instead of in one stampede the first minute the credentials work |
| `PM_PUSH_ON_RED_GATE` | `false` | a change the gate refused is not pushed; the issue gets the gate log instead |
| `PM_DECISION_MAX_TURNS` | `1` (unset) | turns the triage call may take (1–5). The PM holds no tools, so a second turn is only ever the CLI re-asking after an answer that failed the decision schema. Raise it if a triage error logs `subtype='error_max_turns'` |
| `PM_PR_MENTIONS` | `false` | whether a mention of the bot on a **pull request** triages it. See "Mentions on a pull request" above; read `PM_PLAN` output once before turning it on |
| `PM_PLAN` | `false` | the pull-request path's dry run: print the payload and the intended thread, ask no model, post nothing. Also enables the path. `PM_DRY_RUN` implies it |
**When a triage fails**, the `[error] issue #N: claude exited …` line carries the CLI's result
envelope (`subtype`, `is_error`, `api_error_status`, `num_turns`, `result`) and the tails of
stdout and stderr, redacted. Before 2026-10-31 it logged stderr only, which is empty under
`--output-format json` — see `decisions/2026-10-31-pm-agent-triage-exit.md`.
Changing a knob requires bumping the `config-generation` annotation in `deployment.yaml` in
the same commit — the process reads its configuration once, at startup, so without the bump
the ConfigMap changes and nothing does. (Same class of defect as the inert merged
`cilium-config`: `decisions/2026-10-05-cilium-roll-out-on-config-change.md`.)
**To stop it**, in increasing severity: `PM_DRY_RUN`, scale to zero, suspend the
Kustomization, or revoke the Gitea token. Step-by-step in
[`runbooks/pm-agent.md`](../../runbooks/pm-agent.md).