Merge pull request 'feat: PM-Agent also need to look at PR comments' (#117) from feat/pm-agent-pr-mentions into main
gate / gate (push) Successful in 2m55s
gate / conformance (push) Skipped
images / images (push) Successful in 1m25s

Reviewed-on: #117
This commit was merged in pull request #117.
This commit is contained in:
jakman85 committed 2026-11-07 12:07:34 +00:00
commit d7317606e7
9 files changed
+2337 -32

No files matched your search

+141 -3
View File
@@ -5,9 +5,11 @@ A service that runs in the cluster, reads the issues of
Manager persona this repository already works under**, and runs one worker agent per
assignment that authors the change, runs the merge gate and opens a pull request.
It reacts to issue creation and to comments on issues. Every assignment and every piece of
correspondence travels through the issue thread and the pull request, so the whole record of
what the agent did, and why, is in the same place a human is already looking.
It reacts to issue creation and to comments on issues. It can also — **off by default** —
reply when it is `@`-mentioned on a pull request; see
[Mentions on a pull request](#mentions-on-a-pull-request). Every assignment and every piece
of correspondence travels through the issue thread and the pull request, so the whole record
of what the agent did, and why, is in the same place a human is already looking.
It **never pushes to `main`** and **never merges a pull request**. The merge gate remains the
approver, and a sensitive-path change still needs the human review `ci/approval-policy.sh`
@@ -159,6 +161,140 @@ Job from a committed template. **The model's output is data, never a command.**
---
## Mentions on a pull request
**Off by default** (`PM_PR_MENTIONS: "false"`), and nothing below has been exercised against
the live Gitea — see `decisions/2026-11-07-pm-agent-pr-mentions.md`.
With it on, the dispatcher runs a second, independent pass each poll. A mention of the bot
account on a **pull request** makes the PM read that pull request — its description, its
changed files, its diff, the thread it was mentioned in and the rest of the conversation —
and **reply where it was mentioned**. Three places count: the pull request's conversation, a
review's own body (Gitea renders it in the conversation, so the reply goes there), and an
inline review comment against a file and a line.
```
@pm-agent in a PR comment ─▶ discover (notifications, else open PRs)
or in a review body │
or on a line inline ├─▶ read /issues/{n}/comments, /pulls/{n}/reviews,
│ .../reviews/{id}/comments
├─▶ decide, per comment: self-authored? already
│ answered? does it name the bot?
├─▶ assemble the PR + diff + thread, every field fenced
│ and labelled UNTRUSTED
├─▶ ask the model: `comment` or `decline`, and nothing else
└─▶ post the reply in that thread, carrying its own marker
```
**It can only reply.** `implement` and `revise` are not in the vocabulary this path offers
the model, and the decision schema has no `branch` and no `brief` field, so the PM cannot
ask for work in a shape this program would accept. Nothing on the path calls `build_job()`.
The change-authoring half of issue #116 — revise a branch an agent owns, versus open a fresh
branch and PR against `main` for somebody else's — is a separate issue and a separate PR.
**An explicit mention is the only trigger.** A comment is answered if and only if it is not
the bot's own, it spells out `@`-the-bot-account, and no earlier reply of the bot's says it
was already answered. Nothing else is actionable: a comment with no mention produces **no
reply and no model call**, however recently the bot was mentioned or replied nearby. So a
mention is a discrete, self-limiting request — one `@` gets one reply in the place it was
written, and the rest of the pull request is read as *context* rather than as a queue of
questions.
That is the operator's decision on issue #116, taken once it was established that pull-request
conversation comments are a flat list rather than a hierarchy: *"since there is no threads and
the PR comments are flat, then the pm-agent should be looking for explicit mentions rather than
a single mention and continued scrapping of comments within the PR in perpetuity"*. It replaced
an earlier rule where the first mention in a thread armed replies to every later non-bot
comment in it. It is applied uniformly, including to inline review comments — those *are*
genuinely threaded in Gitea, but keeping continuation for them alone would be a second rule
nobody asked for. The fixtures that used to assert continuation are **inverted, not deleted**,
so the old behaviour cannot creep back unnoticed.
Two practical consequences. A mention is **repeatable**: ask again by tagging again, and each
tag gets its own reply. And a mention is **not a conversation** — the PM is told to answer as
if it is the last word it will see, and to ask you to tag it again if it needs something back.
What remains, unavoidably, is that *anyone* who writes the handle gets a reply — the operator's
answer (3) is "anyone can talk and have the agent work" — and Gitea cannot tell a quoted
mention (`> @pm-agent what do you think?`) from a fresh one, so re-quoting an old mention in a
new comment costs one more reply.
**Thread identity is now only about where the reply goes.** Inline review comments are grouped
by (path, position), which is Gitea's own grouping and is what makes the reply land in the
thread that was spoken to rather than beside it; conversation comments and review bodies are
answered in the conversation. None of that is part of the trigger any more.
**Loop prevention.** There is no author gate — anyone who can comment can start a run, which
is the operator's decision (issue #116, answer 3: *"anyone can talk and have the agent work.
Authorization of changes is through the CI pipeline"*). That makes the self-authored filter
load-bearing, so it is the first check made on every comment and it is structural: a
bot-authored comment can only ever *clear* work, never create it, even if it contains the
bot's own handle. The reply the PM posts also never contains `@`-the-bot. Both are asserted
by the selftest.
**Idempotency.** The record that a mention has been answered is a marker inside the reply:
```
<!-- pm-agent:replied-to=comment-9001,review-comment-77 -->
```
One atomic write with the reply, so the two cannot get out of step. The failure mode chosen,
deliberately, is **at-least-once model call, at-most-once reply**: a crash before posting
costs one repeated decision, a crash after posting is a no-op because the marker is already
in the thread, and a mention is never silently dropped. The marker is only believed in the
bot's *own* comments — pasting it into yours does not silence a mention.
### The cheap mention query, and the token scope it needs
The operator asked for "a way to check for '@'s for the bot account instead of scouring the
entire pull request list and associated threads". That is Gitea's notification list: a
mention subscribes the mentioned account to the thread, so one bounded call names the pull
requests worth reading.
It is a **hint, never the decision** — the trigger is re-derived by reading that pull
request's comments. That split is why the endpoint over-delivering is harmless: the
subscription keeps raising notifications for later comments on a pull request the bot was
once mentioned on, and since those comments carry no mention, they resolve to nothing.
**It will be refused on today's token.** `/api/v1/notifications` sits behind Gitea's
`notification` scope category, and the bot's token is minted
`read:user, write:repository, write:issue, read:package` (above). The dispatcher handles that
explicitly: it logs the refusal, names `read:notification` as the remedy, and falls back to
listing open pull requests, which needs only `read:repository`. Which query answered is in
the log every cycle. To get the cheap one, re-mint the token with `read:notification` added —
same account, same five other scopes, both Secrets updated (`runbooks/pm-agent.md`).
### Look before you turn it on
`PM_PLAN: "true"` makes this path discover, assemble and **print** the exact payload it would
send and the exact thread it would post into, without asking the model and without posting
anything:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n pm-agent logs deploy/pm-agent | sed -n "/^=\{20,\}/,\$p"'
```
Setting `PM_PLAN` also enables the path, so it is one knob. `PM_DRY_RUN` implies it.
To settle the API shapes by hand before enabling anything (read-only, and the bot's token is
enough for all but the first):
```bash
T=<the pm-agent token>; G=https://gitea.home.keithjackson.dev/api/v1
R=repos/Home-Network/home-network-k3s-impl
curl -s -H "Authorization: token $T" "$G/notifications?all=true&subject-type=pull" | head -c 2000
curl -s -H "Authorization: token $T" "$G/$R/pulls/116/reviews"
curl -s -H "Authorization: token $T" "$G/$R/pulls/116/reviews/1/comments"
curl -s -H "Authorization: token $T" "$G/$R/pulls/116/files"
```
The committed fixtures are `infra/agent/testdata/pr-mention-fixtures.json`; they are written
to Gitea's published 1.24 shapes rather than captured from the server, and the file says so
at the top. Replacing them with a real capture is the cheapest way to close that gap.
---
## What the worker can and cannot do
| | |
@@ -251,6 +387,8 @@ five that matter:
| `PM_MAX_NEW_WORKERS_PER_CYCLE` | `1` | drains a backlog one issue at a time instead of in one stampede the first minute the credentials work |
| `PM_PUSH_ON_RED_GATE` | `false` | a change the gate refused is not pushed; the issue gets the gate log instead |
| `PM_DECISION_MAX_TURNS` | `1` (unset) | turns the triage call may take (1–5). The PM holds no tools, so a second turn is only ever the CLI re-asking after an answer that failed the decision schema. Raise it if a triage error logs `subtype='error_max_turns'` |
| `PM_PR_MENTIONS` | `false` | whether a mention of the bot on a **pull request** triages it. See "Mentions on a pull request" above; read `PM_PLAN` output once before turning it on |
| `PM_PLAN` | `false` | the pull-request path's dry run: print the payload and the intended thread, ask no model, post nothing. Also enables the path. `PM_DRY_RUN` implies it |
**When a triage fails**, the `[error] issue #N: claude exited …` line carries the CLI's result
envelope (`subtype`, `is_error`, `api_error_status`, `num_turns`, `result`) and the tails of
+58
View File
@@ -129,6 +129,64 @@ data:
# runbooks/pm-agent.md.
PM_DRY_RUN: "false"
# --- mentions on a pull request (issue #116) --------------------------------
# OFF. Turning it on makes the dispatcher also look at PULL REQUESTS: a mention of the bot
# account in a pull request's conversation, in a review's body, or on a line inside an
# inline review thread, makes the PM read that pull request and reply where it was
# mentioned. It can ONLY reply — a mention cannot start, revise or push a branch; that is
# the second half of issue #116 and has its own issue.
#
# AN EXPLICIT MENTION IS THE ONLY TRIGGER, and a mention is self-limiting: one `@` gets one
# reply, and no other comment on the pull request is picked up. A comment that does not
# name the bot costs nothing — no reply and no model call — however recently the bot was
# mentioned or replied nearby. So the spend on this path is bounded by how often somebody
# types the handle, not by how busy the pull request is.
#
# WHY IT SHIPS OFF. Not one call on this path has been made against the live Gitea — the
# branch that wrote it held no Gitea credential and no cluster — and the cheap mention
# query it prefers needs a token scope the bot does not have today (see
# PM_PR_MENTION_DISCOVERY). The intended order is: set PM_PLAN once and read what it would
# have replied to, then set this to "true". Both are ConfigMap edits plus a
# config-generation bump.
PM_PR_MENTIONS: "false"
# How candidate pull requests are found. `auto` tries Gitea's notification list for the bot
# account first — one bounded call, which is the "way to check for '@'s for the bot account
# instead of scouring the entire pull request list" the operator asked for — and falls back
# to listing open pull requests if that is refused. `notifications` uses only the cheap
# query and does nothing if it is unavailable (it will not silently widen into the scan the
# operator rejected). `pulls` never calls the notification endpoint.
#
# EXPECT THE FALLBACK AT FIRST. Gitea puts /api/v1/notifications behind its own
# `notification` token scope, and the bot's token is minted
# `read:user, write:repository, write:issue, read:package` — so until the operator re-mints
# it with `read:notification`, the cheap query is refused and the dispatcher logs exactly
# that, naming the scope. Which query answered is logged every cycle either way.
PM_PR_MENTION_DISCOVERY: "auto"
# How far back the notification query looks. Longer costs nothing but a bigger response;
# shorter risks missing a mention the dispatcher was down for. A day is well inside the
# idempotency marker's reach, which does not expire at all.
PM_PR_NOTIFICATION_LOOKBACK_SECONDS: "86400"
# The same throttle shape as the issue path: one decision per pull request per cooldown
# window, and at most this many replies per poll.
PM_PR_COOLDOWN_SECONDS: "900"
PM_PR_MAX_REPLIES_PER_CYCLE: "1"
# How much of the change the PM is shown. The changed-file summary is always included; this
# caps the diff quoted beneath it. Set to "0" to send the file list alone — the right
# setting if the diffs here turn out to dominate the decision's token cost.
PM_PR_DIFF_MAX_CHARS: "20000"
PM_PR_MAX_CHANGED_FILES: "200"
# Plan mode for the pull-request path: discover, assemble the payload, PRINT it to the
# dispatcher's stdout, ask no model and post nothing. Setting it also enables the path, so
# it is one knob to look before leaping. PM_DRY_RUN above implies it, because on this path
# the posted reply IS the action under test, and it also carries the idempotency marker —
# posting it would make a dry run not dry.
PM_PLAN: "false"
# If the gate refuses a worker's change, the branch is NOT pushed by default and the issue
# gets the gate log instead. Set to "true" to push anyway and open a red pull request —
# useful when a human wants to see the work, noisy as a standing setting.
+1 -1
View File
@@ -48,7 +48,7 @@ spec:
# It is a hand-maintained marker rather than a computed hash because Flux's
# Kustomization here does not run a configmap generator; bump it in the same commit
# as a change to configmap.yaml.
pm-agent.homelab.keithjackson.dev/config-generation: "5"
pm-agent.homelab.keithjackson.dev/config-generation: "6"
spec:
serviceAccountName: pm-agent
# The kubelet's credential for pulling this image. Not the agent's Gitea credential —