Merge pull request 'feat: PM-Agent also need to look at PR comments' (#117) from feat/pm-agent-pr-mentions into main
gate / gate (push) Successful in 2m55s
gate / conformance (push) Skipped
images / images (push) Successful in 1m25s

Reviewed-on: #117
This commit was merged in pull request #117.
This commit is contained in:
jakman85 committed 2026-11-07 12:07:34 +00:00
commit d7317606e7
9 files changed
+2337 -32

No files matched your search

+24 -1
View File
@@ -21,6 +21,12 @@ In order of increasing severity. Each is an operator action.
`deployment.yaml`, merge. It still reads threads and still posts what it *would* have done,
and creates no worker Jobs. This is also the right setting for watching it for a day.
`PM_DRY_RUN` also silences the **pull-request mention** path completely — that path only ever
posts a comment, so leaving it posting would make a dry run not dry, and its reply carries the
marker that stops the same mention being answered again. To silence just that path and leave
issue triage alone, set `PM_PR_MENTIONS: "false"` instead (that is the default; see
`platform/pm-agent/README.md`, "Mentions on a pull request").
**2. Stop it immediately, without a merge.**
```bash
@@ -70,13 +76,18 @@ Work through these in order; the first three are by far the most likely.
| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets |
| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets. Add `read:notification` if you want the cheap pull-request mention query (`platform/pm-agent/README.md`) |
| a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push |
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
| `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
| `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle |
| nothing at all, not even `cycle:` lines | the process is not running | check the pod, then the events |
| `pull-request mentions: off` | `PM_PR_MENTIONS` is `false`, which is the default | read `PM_PLAN` output once, then set it to `"true"` (ConfigMap + `config-generation` bump) |
| `pr-mentions: /notifications was refused …` | expected on today's token: Gitea puts that endpoint behind its `notification` scope and the bot's token is minted `read:user, write:repository, write:issue, read:package`. The dispatcher has fallen back to listing open pull requests, so it still works — just not cheaply | re-mint the token with `read:notification` added and update **both** Secrets (below), or set `PM_PR_MENTION_DISCOVERY: "pulls"` to stop trying |
| `pr-mentions: 0 candidate pull request(s) from notifications-unavailable` | `PM_PR_MENTION_DISCOVERY` is `notifications` and that query is refused, so the path deliberately does nothing rather than widening into a full scan | fix the scope, or set the knob to `auto` |
| `[PM_PLAN: nothing will be posted]` | `PM_PLAN` or `PM_DRY_RUN` is set; the payload is printed and no reply is posted | expected; clear both to let it reply |
| `pull #N: inline reply refused …; falling back to the conversation` | Gitea would not accept a comment at that diff position (usually the position no longer exists on the current head) | nothing to fix — the reply was posted in the conversation, naming the file and position |
**It will not answer an issue whose newest comment is its own.** That is the trigger, not a
bug: it acts only when the newest thing in a thread was written by somebody else. If you want
@@ -85,6 +96,18 @@ it to look again, comment on the issue.
**It will not re-answer an issue within `PM_ISSUE_COOLDOWN_SECONDS`** (15 min) of its last
attempt on that issue. Restarting the pod clears that timer.
**On a pull request it answers explicit mentions and nothing else.** A comment that does not
write `@`-the-bot-account is never answered and costs no model call, however recently the bot
was mentioned or replied on that pull request. If you asked something and got no reply, the
likely cause is a missing `@` — tag it again. (Near misses do not count either:
`@pm-agent-worker` and `[email protected]` are deliberately not mentions.)
**It will not answer the same pull-request comment twice.** Every reply it posts on a pull
request carries a hidden marker naming the comment ids it answered
(`<!-- pm-agent:replied-to=… -->`), and it skips anything already marked. To make it answer
again, write a *new* comment that mentions it again — editing the old one does not change its
id. `PM_PR_COOLDOWN_SECONDS` (15 min) bounds it to one decision per pull request per window.
---
## A worker failed