diff --git a/decisions/2026-10-31-pm-agent-go-live.md b/decisions/2026-10-31-pm-agent-go-live.md new file mode 100644 index 0000000..1f985f9 --- /dev/null +++ b/decisions/2026-10-31-pm-agent-go-live.md @@ -0,0 +1,55 @@ +--- +date: 2026-10-31 +branch: chore/pm-agent-go-live +title: Take pm-agent out of dry run (operator-approved); label scope and worker caps unchanged +--- + +## What changed + +- `platform/pm-agent/configmap.yaml`: `PM_DRY_RUN` `"true"` -> `"false"`. The operator approved + this after reading the dry-run triage of issue #97. `PM_REQUIRE_LABEL: agent`, + `PM_MAX_ACTIVE_WORKERS: 2`, `PM_MAX_NEW_WORKERS_PER_CYCLE: 1`, the worker image digest and every + other knob are unchanged. The two comments that said dry run was the only mode that worked + (because the run-340 digest lacked `imagePullSecrets`) now describe the run-383 digest that + PR #96 pinned. +- `platform/pm-agent/deployment.yaml`: `config-generation` `"3"` -> `"4"`. Without the bump the + dispatcher keeps its startup config and stays in dry run. +- `platform/pm-agent/README.md` and `docs/plans/pm-agent-issue-to-pr.md` (follow-up 3): the + "currently dry run" status text now says dry run is off. `runbooks/pm-agent.md` is unchanged. + Its "stop it: set `PM_DRY_RUN: "true"`" step and its admission-refusal troubleshooting row + are still accurate. + +## What I verified + +Read-only, `kubectl --context k3s-prod` through the toolbox, 2026-10-31 ~11:45 local, with Flux +`pm-agent` and `platform-config` both at `main@1126740`: + +- ClusterPolicy `pm-agent-job-shape`: `Ready=True`, `validationFailureAction=Enforce`, + `background=false`. It has seven rules, including `worker-pull-secret-must-be-the-committed-one`. +- RBAC: ServiceAccounts `pm-agent` and `pm-agent-worker`, Role `pm-agent` and RoleBinding + `pm-agent` (roleRef `pm-agent`, subject `ServiceAccount/pm-agent`) all exist. `auth can-i` + as `system:serviceaccount:pm-agent:pm-agent` returned `yes` for `create jobs.batch` and `no` + for `delete jobs.batch`. +- Secrets (key names only): `pm-agent-registry` is `kubernetes.io/dockerconfigjson` with key + `.dockerconfigjson`. `pm-agent-credentials` is `Opaque` with keys `claude-code-oauth-token` + and `gitea-token`. +- Egress: `default-deny` NetworkPolicy exists. CiliumNetworkPolicies `pm-agent-allow-egress` + and `pm-agent-worker-allow-egress` both report `Valid=True`. +- Deployment `pm-agent` is 1/1 on `agent-runner@sha256:df36d9b7…` with + `config-generation: "3"` live. No Jobs exist in the namespace. +- `platform/pm-agent/{configmap,deployment}.yaml`, README and docs do not match any + `sensitive_paths` glob in `infra/gitea/approval-policy.json`, so no review or override applies. + +## What I did NOT verify + +- **That a real worker Job is admitted and completes.** Nothing can show this until a labelled + issue gets a `start` decision after the merge. The L0 Kyverno suite shows the policy accepts + the committed template. Whether the API server admits the Job built from the df36d9b image's + baked template has not been tested live. +- That the worker pod can pull its image with `pm-agent-registry`, reach Gitea and the model + through its egress policy, and push a branch and open a PR. None of this has run before. +- The pod list also shows a stale `pm-agent-65db599649-v678g` (25d, `ContainerStatusUnknown`, + from an old ReplicaSet). It does not block anything, but I did not investigate it. Clearing + it is an operator action. +- CI: see the PR for the gate result. Locally the best possible result is rc 3, with only + `gitea-settings` not run. diff --git a/docs/plans/pm-agent-issue-to-pr.md b/docs/plans/pm-agent-issue-to-pr.md index db04b6a..bb844af 100644 --- a/docs/plans/pm-agent-issue-to-pr.md +++ b/docs/plans/pm-agent-issue-to-pr.md @@ -257,7 +257,9 @@ about runtime behaviour in this document is a design claim, not a measurement. bot's permission is `write`, not `admin`** — that measurement is the control that keeps the agent out of its own approval gate. 3. **First run with `PM_DRY_RUN=true` and `PM_REQUIRE_LABEL=agent`**, on one issue written - for the purpose. Read what the PM says before letting it write anything. + for the purpose. Read what the PM says before letting it write anything. *Done:* dry-run + triage of issue #97 read; `PM_DRY_RUN=false` on 2026-10-31 + (`decisions/2026-10-31-pm-agent-go-live.md`), `PM_REQUIRE_LABEL=agent` kept. 4. **Tighten the admission image pin** from the `@sha256:*` wildcard to the literal digest. One line, reviewed. 5. **Narrow `world:443` to `toFQDNs: api.anthropic.com`** — after verifying Cilium's DNS diff --git a/platform/pm-agent/README.md b/platform/pm-agent/README.md index baa768d..07f5981 100644 --- a/platform/pm-agent/README.md +++ b/platform/pm-agent/README.md @@ -46,11 +46,12 @@ waive the sensitive-path human review on its own pull request. Both are deliberate, and both fail **visibly** rather than silently: 1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and - `PM_WORKER_IMAGE` in `configmap.yaml` carry the digest of the **first** publish (images - run 340, `sha-523cd9118f95`), pinned once on 2026-10-30 so going live did not wait on - Renovate's first run; every later bump is Renovate's. That digest predates the worker - template's `imagePullSecrets` entry, so worker Jobs built from it are refused by - admission — which is why `PM_DRY_RUN` starts `true` (see `configmap.yaml`). `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit + `PM_WORKER_IMAGE` in `configmap.yaml` first carried the run-340 publish + (`sha-523cd9118f95`), whose baked template predated the `imagePullSecrets` entry admission + requires, so the service ran with `PM_DRY_RUN=true`. PR #96 pinned the run-383 digest + (`df36d9b`, `sha-ddd04d460ef6`), whose template carries the entry, and dry run was turned + off on 2026-10-31 (`decisions/2026-10-31-pm-agent-go-live.md`); every later bump is + Renovate's. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit (`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that `infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to `gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-` and `main`. diff --git a/platform/pm-agent/configmap.yaml b/platform/pm-agent/configmap.yaml index 9e668cc..0ccf073 100644 --- a/platform/pm-agent/configmap.yaml +++ b/platform/pm-agent/configmap.yaml @@ -58,11 +58,11 @@ data: # request that also moves the Deployment's image (same marker, same branch). Nobody pushes # this image from a workstation. # - # READ BEFORE TURNING PM_DRY_RUN OFF: this digest PREDATES the worker template's - # `imagePullSecrets` entry (PR #61). Its baked template has none, and admission now REQUIRES - # it — so a worker Job built from this image is DENIED. The merge that adds the entry - # changes infra/agent/**, CI re-publishes agent-runner, and Renovate's bump to that digest - # is what makes worker Jobs admissible. Until then, dry run is the only mode that works. + # The run-340 digest predated the worker template's `imagePullSecrets` entry (PR #61), which + # admission REQUIRES, so its worker Jobs were DENIED. The digest below (df36d9b, run 383, + # agent-runner:sha-ddd04d460ef6, pinned in PR #96) bakes a template that carries + # `imagePullSecrets: pm-agent-registry`, which is what made turning PM_DRY_RUN off possible + # (decisions/2026-10-31-pm-agent-runner-digest.md). A digest older than that would be DENIED. # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d" @@ -120,10 +120,11 @@ data: # no worker Job and so changes no code. The honest way to watch this service for a day # before letting it write anything. # - # "true" FOR THE FIRST RUN, and not only for caution: the pinned digest above cannot yet - # create an admissible worker Job (see the PM_WORKER_IMAGE comment). Flip to "false" in the - # same pull request as — or after — Renovate's first agent-runner digest bump. - PM_DRY_RUN: "true" + # "true" for the first run (2026-10-30 .. 2026-10-31); dry-run triage of issue #97 was read + # and the operator approved going live. "false" since decisions/2026-10-31-pm-agent-go-live.md. + # Setting it back to "true" (plus a config-generation bump) is the gentlest stop — see + # runbooks/pm-agent.md. + PM_DRY_RUN: "false" # If the gate refuses a worker's change, the branch is NOT pushed by default and the issue # gets the gate log instead. Set to "true" to push anyway and open a red pull request — diff --git a/platform/pm-agent/deployment.yaml b/platform/pm-agent/deployment.yaml index 9aaa563..3b958ed 100644 --- a/platform/pm-agent/deployment.yaml +++ b/platform/pm-agent/deployment.yaml @@ -48,7 +48,7 @@ spec: # It is a hand-maintained marker rather than a computed hash because Flux's # Kustomization here does not run a configmap generator; bump it in the same commit # as a change to configmap.yaml. - pm-agent.homelab.keithjackson.dev/config-generation: "3" + pm-agent.homelab.keithjackson.dev/config-generation: "4" spec: serviceAccountName: pm-agent # The kubelet's credential for pulling this image. Not the agent's Gitea credential —