Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5.5 0d3e51acc4 chore(pm-agent): turn PM_DRY_RUN off (operator-approved), bump config-generation to 4
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
The run-383 worker image (df36d9b, pinned in #96) bakes imagePullSecrets: pm-agent-registry,
which admission requires, and the dispatcher's dry-run triage of #97 was read and approved.
PM_REQUIRE_LABEL=agent and the worker caps are unchanged. The dry-run status text in the
configmap comments, the README and the plan's follow-up 3 is updated to match.

Pre-flight (read-only, k3s-prod): pm-agent-job-shape Ready+Enforce; SA/Role/RoleBinding
present and can-i create jobs = yes; pm-agent-registry and pm-agent-credentials exist; both
egress CNPs Valid.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 11:45:52 -05:00
jakman85 11267409d7 Merge pull request 'chore(pm-agent): pin agent-runner to the run-383 digest (ddd04d4), bump config-generation' (#96) from chore/pm-agent-runner-digest into main
gate / gate (push) Successful in 2m58s
gate / conformance (push) Has been skipped
Reviewed-on: #96
Reviewed-by: Keith Jackson <[email protected]>
2026-10-31 16:34:49 +00:00
jakman85andClaude Opus 5.5 6a192ede2a chore(pm-agent): pin agent-runner to the run-383 digest (ddd04d4), bump config-generation
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Has been skipped
c4eba5b (run 340) bakes a worker template without imagePullSecrets, so the new admission rule
refuses worker Jobs. df36d9b is agent-runner:main == :sha-ddd04d460ef6 (measured on the registry),
and its /etc/pm-agent/worker-job-template.yaml carries imagePullSecrets: pm-agent-registry (read
from the published layer).

Renovate found df36d9b but holds it under "Awaiting Schedule" on dashboard #91: the operator's
dispatch (run 385) was a real run at 12:14 New York, outside the 03:00-06:00 window. The
renovate-digest markers are unchanged, so later bumps remain Renovate's.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 11:22:52 -05:00
jakman85 06d4a9a1fc Merge pull request 'Added the credentials' (#95) from feature/pm-agent-credentials into main
gate / gate (push) Successful in 2m56s
gate / conformance (push) Has been skipped
Reviewed-on: #95
2026-10-31 16:20:12 +00:00
6 changed files with 155 additions and 18 deletions

No files matched your search

+55
View File
@@ -0,0 +1,55 @@
---
date: 2026-10-31
branch: chore/pm-agent-go-live
title: Take pm-agent out of dry run (operator-approved); label scope and worker caps unchanged
---
## What changed
- `platform/pm-agent/configmap.yaml`: `PM_DRY_RUN` `"true"` -> `"false"`. The operator approved
this after reading the dry-run triage of issue #97. `PM_REQUIRE_LABEL: agent`,
`PM_MAX_ACTIVE_WORKERS: 2`, `PM_MAX_NEW_WORKERS_PER_CYCLE: 1`, the worker image digest and every
other knob are unchanged. The two comments that said dry run was the only mode that worked
(because the run-340 digest lacked `imagePullSecrets`) now describe the run-383 digest that
PR #96 pinned.
- `platform/pm-agent/deployment.yaml`: `config-generation` `"3"` -> `"4"`. Without the bump the
dispatcher keeps its startup config and stays in dry run.
- `platform/pm-agent/README.md` and `docs/plans/pm-agent-issue-to-pr.md` (follow-up 3): the
"currently dry run" status text now says dry run is off. `runbooks/pm-agent.md` is unchanged.
Its "stop it: set `PM_DRY_RUN: "true"`" step and its admission-refusal troubleshooting row
are still accurate.
## What I verified
Read-only, `kubectl --context k3s-prod` through the toolbox, 2026-10-31 ~11:45 local, with Flux
`pm-agent` and `platform-config` both at `main@1126740`:
- ClusterPolicy `pm-agent-job-shape`: `Ready=True`, `validationFailureAction=Enforce`,
`background=false`. It has seven rules, including `worker-pull-secret-must-be-the-committed-one`.
- RBAC: ServiceAccounts `pm-agent` and `pm-agent-worker`, Role `pm-agent` and RoleBinding
`pm-agent` (roleRef `pm-agent`, subject `ServiceAccount/pm-agent`) all exist. `auth can-i`
as `system:serviceaccount:pm-agent:pm-agent` returned `yes` for `create jobs.batch` and `no`
for `delete jobs.batch`.
- Secrets (key names only): `pm-agent-registry` is `kubernetes.io/dockerconfigjson` with key
`.dockerconfigjson`. `pm-agent-credentials` is `Opaque` with keys `claude-code-oauth-token`
and `gitea-token`.
- Egress: `default-deny` NetworkPolicy exists. CiliumNetworkPolicies `pm-agent-allow-egress`
and `pm-agent-worker-allow-egress` both report `Valid=True`.
- Deployment `pm-agent` is 1/1 on `agent-runner@sha256:df36d9b7…` with
`config-generation: "3"` live. No Jobs exist in the namespace.
- `platform/pm-agent/{configmap,deployment}.yaml`, README and docs do not match any
`sensitive_paths` glob in `infra/gitea/approval-policy.json`, so no review or override applies.
## What I did NOT verify
- **That a real worker Job is admitted and completes.** Nothing can show this until a labelled
issue gets a `start` decision after the merge. The L0 Kyverno suite shows the policy accepts
the committed template. Whether the API server admits the Job built from the df36d9b image's
baked template has not been tested live.
- That the worker pod can pull its image with `pm-agent-registry`, reach Gitea and the model
through its egress policy, and push a branch and open a PR. None of this has run before.
- The pod list also shows a stale `pm-agent-65db599649-v678g` (25d, `ContainerStatusUnknown`,
from an old ReplicaSet). It does not block anything, but I did not investigate it. Clearing
it is an operator action.
- CI: see the PR for the gate result. Locally the best possible result is rc 3, with only
`gitea-settings` not run.
@@ -0,0 +1,78 @@
---
date: 2026-10-31
branch: chore/pm-agent-runner-digest
title: Bump pm-agent to the run-383 agent-runner digest by hand; Renovate saw it but the schedule withheld it
---
## What changed
`platform/pm-agent/deployment.yaml` (the dispatcher `image:`) and `platform/pm-agent/configmap.yaml`
(`PM_WORKER_IMAGE`) move:
- from `agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`, the
run-340 publish;
- to `agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d`, the
images run 383 publish of ddd04d4.
Both `# renovate-digest:` markers are unchanged, directly above their values.
`pm-agent.homelab.keithjackson.dev/config-generation` goes `"2"` → `"3"`. That is the convention in
`deployment.yaml`'s own comment: "bump it in the same commit as a change to configmap.yaml". The
image change alone would roll the pod, but the convention is followed regardless.
No `*.sops.yaml` file is touched. Branched from 06d4a9a, which already contains the operator's #95.
### Why Renovate did not do this: the schedule, not a config defect
- **Run 385** (dispatch, 2026-10-31 16:14 UTC, on ddd04d4) was a **real** run. It logged
`Deleting orphan branch (… renovate/pin-dependencies)`, not a `DRY-RUN:` line. It ran at **12:14
America/New_York**, outside `"schedule": ["after 3am and before 6am"]`, so it created and updated
no branch.
- **The lookup DID see the new digest.** Dependency Dashboard issue #91 lists, under **"Awaiting
Schedule"**:
`chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to df36d9b`.
- **Nothing else blocked it.** There are no open Renovate PRs and no `renovate/*` branch on Gitea
(111 branches listed across all pages). The previous agent-runner PR, #90 (to c4eba5b), was
auto-merged by renovate-bot at 08:22 on 10-31. No PR limit was hit, and digests are not
dashboard-approval gated.
- **The registry, read-only, with Docker/OCI `Accept` headers:**
- `HEAD …/agent-runner/manifests/main` → `docker-content-digest: sha256:df36d9b7…771d`
- `HEAD …/agent-runner/manifests/sha-ddd04d460ef6` → the same digest.
**Without this PR** the next scheduled run (08:15 UTC, 04:15 EDT) would open it, and it would
auto-merge as a `digest` update. Ticking that "Awaiting Schedule" checkbox on #91 makes the next
run of any kind create it out of schedule.
### Found on the way: Renovate's labels have never been applied
- The repo has exactly one label, `agent`. The org has none.
- Every Renovate PR so far (#89, #90, #92, #93) was created with `"labels": []`.
- On Gitea, Renovate drops a label that does not exist in the repo. So `needs-review`,
`blast-radius`, `range-widen`, `own-image`, `dependencies`, `automated-update`, `breaking` and
`ci-tooling` are never attached.
- The **automerge flags still work.** Only the labels, the visible review signal, are missing.
- **Operator fix:** create those labels on the repo. This was not changed here.
## What I verified
- **The new image carries the fix.** I fetched both images' layers read-only from the registry and
read `/etc/pm-agent/worker-job-template.yaml` from each (`infra/agent/Containerfile:116` copies
it there):
- df36d9b's has `imagePullSecrets:` / `- name: pm-agent-registry`;
- c4eba5b's has no `imagePullSecrets` at all.
This matches `infra/agent/worker-job-template.yaml:76-77` at ddd04d4.
- **`main` = `sha-ddd04d460ef6` = df36d9b**, both measured.
- **No other file references c4eba5b.** No admission policy pins it.
- **No override is needed.** `infra/gitea/approval-policy.json` lists only `platform/kyverno/**`
under `platform/`, so `platform/pm-agent/**` is not sensitive.
- I read the whole job logs of runs 362, 367 and 385.
## What I did NOT verify
- **That the rolled dispatcher starts on df36d9b and that a worker Job is admitted.** That needs
Flux to apply this and a real dispatch. Both are operator actions.
- **That the `pm-agent-registry` Secret exists and decrypts in-cluster** (#95's sops file). I did not
read it.
- **That Renovate will treat this hand bump as current.** It should: its next lookup finds
`main` == the pinned digest, so it proposes nothing and drops the dashboard entry. Not observed.
+3 -1
View File
@@ -257,7 +257,9 @@ about runtime behaviour in this document is a design claim, not a measurement.
bot's permission is `write`, not `admin`** — that measurement is the control that keeps
the agent out of its own approval gate.
3. **First run with `PM_DRY_RUN=true` and `PM_REQUIRE_LABEL=agent`**, on one issue written
for the purpose. Read what the PM says before letting it write anything.
for the purpose. Read what the PM says before letting it write anything. *Done:* dry-run
triage of issue #97 read; `PM_DRY_RUN=false` on 2026-10-31
(`decisions/2026-10-31-pm-agent-go-live.md`), `PM_REQUIRE_LABEL=agent` kept.
4. **Tighten the admission image pin** from the `@sha256:*` wildcard to the literal digest.
One line, reviewed.
5. **Narrow `world:443` to `toFQDNs: api.anthropic.com`** — after verifying Cilium's DNS
+6 -5
View File
@@ -46,11 +46,12 @@ waive the sensitive-path human review on its own pull request.
Both are deliberate, and both fail **visibly** rather than silently:
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
`PM_WORKER_IMAGE` in `configmap.yaml` carry the digest of the **first** publish (images
run 340, `sha-523cd9118f95`), pinned once on 2026-10-30 so going live did not wait on
Renovate's first run; every later bump is Renovate's. That digest predates the worker
template's `imagePullSecrets` entry, so worker Jobs built from it are refused by
admission — which is why `PM_DRY_RUN` starts `true` (see `configmap.yaml`). `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
`PM_WORKER_IMAGE` in `configmap.yaml` first carried the run-340 publish
(`sha-523cd9118f95`), whose baked template predated the `imagePullSecrets` entry admission
requires, so the service ran with `PM_DRY_RUN=true`. PR #96 pinned the run-383 digest
(`df36d9b`, `sha-ddd04d460ef6`), whose template carries the entry, and dry run was turned
off on 2026-10-31 (`decisions/2026-10-31-pm-agent-go-live.md`); every later bump is
Renovate's. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
(`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
`infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
+11 -10
View File
@@ -58,13 +58,13 @@ data:
# request that also moves the Deployment's image (same marker, same branch). Nobody pushes
# this image from a workstation.
#
# READ BEFORE TURNING PM_DRY_RUN OFF: this digest PREDATES the worker template's
# `imagePullSecrets` entry (PR #61). Its baked template has none, and admission now REQUIRES
# it — so a worker Job built from this image is DENIED. The merge that adds the entry
# changes infra/agent/**, CI re-publishes agent-runner, and Renovate's bump to that digest
# is what makes worker Jobs admissible. Until then, dry run is the only mode that works.
# The run-340 digest predated the worker template's `imagePullSecrets` entry (PR #61), which
# admission REQUIRES, so its worker Jobs were DENIED. The digest below (df36d9b, run 383,
# agent-runner:sha-ddd04d460ef6, pinned in PR #96) bakes a template that carries
# `imagePullSecrets: pm-agent-registry`, which is what made turning PM_DRY_RUN off possible
# (decisions/2026-10-31-pm-agent-runner-digest.md). A digest older than that would be DENIED.
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42"
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d"
# --- the model -------------------------------------------------------------
# The PM only reads a thread and answers with one JSON object, so it holds no tools and
@@ -120,10 +120,11 @@ data:
# no worker Job and so changes no code. The honest way to watch this service for a day
# before letting it write anything.
#
# "true" FOR THE FIRST RUN, and not only for caution: the pinned digest above cannot yet
# create an admissible worker Job (see the PM_WORKER_IMAGE comment). Flip to "false" in the
# same pull request as — or after — Renovate's first agent-runner digest bump.
PM_DRY_RUN: "true"
# "true" for the first run (2026-10-30 .. 2026-10-31); dry-run triage of issue #97 was read
# and the operator approved going live. "false" since decisions/2026-10-31-pm-agent-go-live.md.
# Setting it back to "true" (plus a config-generation bump) is the gentlest stop — see
# runbooks/pm-agent.md.
PM_DRY_RUN: "false"
# If the gate refuses a worker's change, the branch is NOT pushed by default and the issue
# gets the gate log instead. Set to "true" to push anyway and open a red pull request —
+2 -2
View File
@@ -48,7 +48,7 @@ spec:
# It is a hand-maintained marker rather than a computed hash because Flux's
# Kustomization here does not run a configmap generator; bump it in the same commit
# as a change to configmap.yaml.
pm-agent.homelab.keithjackson.dev/config-generation: "2"
pm-agent.homelab.keithjackson.dev/config-generation: "4"
spec:
serviceAccountName: pm-agent
# The kubelet's credential for pulling this image. Not the agent's Gitea credential —
@@ -78,7 +78,7 @@ spec:
# the registry's manifest endpoint on 2026-10-30 and pinned by hand once so the
# dispatcher does not wait on Renovate's first run; every later bump is Renovate's.
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:df36d9b7fef56d141220a387f64fb864cede8e5af30513ebfeb3ab566d3f771d
imagePullPolicy: IfNotPresent
command:
- /usr/local/bin/pm-dispatch