Merge pull request 'feat(pm-agent): pull credential + admission rule 7, first published digest pinned, dry-run first start, token scopes fixed' (#61) from feat/pm-agent-registry-pull-secret into main

Reviewed-on: #61 Reviewed-by: Keith Jackson <[email protected]>
2026-10-31 15:46:54 +00:00
commit ddd04d460e
@@ -0,0 +1,104 @@
+---
+date: 2026-10-06
+branch: feat/pm-agent-registry-pull-secret
+title: The kubelet's pull credential — the second half of making agent-runner pullable, and the seventh field the admission policy had not read
+---
+
+## What changed
+
+The registry had two blockers. The first — the nodes resolving the registry hostname to the
+non-hairpinned WAN and failing TLS against the router's `CN=%D` certificate — was diagnosed
+and fixed by an `/etc/hosts` pin, merged in #59 and applied by the operator. This is the
+second: **the pull is unauthenticated, and this registry refuses unauthenticated clients
+outright.**
+
+`pm-agent-registry` is a new `kubernetes.io/dockerconfigjson` Secret, SOPS-encrypted with
+placeholder values, referenced by `imagePullSecrets` on both the dispatcher Deployment and
+the committed worker Job template. It is a **second Secret holding the same Gitea token** as
+`pm-agent-credentials`, because the two consumers need two formats: the application reads a
+raw token from env, the kubelet reads a dockerconfigjson before any container starts. That
+duplication is the established pattern here (the Keycloak client secret is mirrored the same
+way, for the same reason) and it is documented in both `.example` files.
+
+The operator's part is the value, and one scope: the Gitea token must now carry
+**`read:package`** alongside `write:repository` and `write:issue`. Steps are in
+`platform/pm-agent/pm-agent-registry.sops.yaml.example`, including the trap that
+`.dockerconfigjson`'s registry key must be the bare host as the image reference spells it —
+with a scheme or a port it matches nothing, the kubelet pulls anonymously, and the failure is
+indistinguishable from the Secret being absent.
+
+**The admission policy gained a seventh rule.** `imagePullSecrets` is a pod-spec field that
+no container references, which is exactly why it is easy to leave unread — the same shape of
+omission as `initContainers`, `ephemeralContainers`, `env` and `envFrom` before it. The
+kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now
+REQUIRES the committed entry and rule 7 forbids any other or any second one; both halves are
+needed because a `foreach` over an absent list yields no result.
+
+The leverage is small and worth saying so: the image is digest-pinned, so a different pull
+credential cannot change *what* runs, and the only Secrets in this namespace are the agent's
+own. It is pinned anyway, because "only the committed shape" is this policy's whole premise
+and the rule costs four lines.
+
+## What I verified
+
+- **`kyverno test platform/kyverno/tests/pm-agent` → 33 passed, 0 failed** under the pinned
+  CLI (was 28 before this change). Three new fixtures, each a real path rather than a style
+  violation: `extra-pull-secret` (the committed entry PLUS a second one — rule 1's array
+  pattern is satisfied by the first element, which is why the count is asserted separately),
+  `wrong-pull-secret` (a different Secret in this namespace), and `no-pull-secret` (absent
+  entirely, refused by rule 1's pattern because rule 7's foreach cannot see it).
+- **`PM_SELFTEST=1 pm-dispatch` → exit 0**, now also asserting that the committed Job
+  template names exactly `pm-agent-registry` — so the template and the policy cannot drift
+  apart without the image build failing.
+- **`./ci/gate.sh` in the pinned toolbox, from a §0 worktree → exit 3 (UNDETERMINED)**, with
+  `gitea-settings` the only stage that did not run (no `SETTINGS_API_TOKEN` on this
+  workstation). Every other stage green, `spec-immutability` included. rc 3 is not a pass.
+- **The gate caught me shipping the Secret in plaintext.** I wrote the placeholder
+  `pm-agent-registry.sops.yaml` and never ran `sops -e -i` on it. `no-plaintext-secrets`
+  refused it with both halves named — "has no top-level `sops:` block with a mac" and
+  "Secret value '.dockerconfigjson' is not SOPS ciphertext" — which is exactly the check
+  earning its place: the file looked finished, and the value was a placeholder, so nothing
+  about reading the diff would have made it obvious. Encrypted, and the stage is green.
+- **The registry really does refuse anonymous clients**, measured before building this
+  (2026-10-05): the anonymous registry token's `access` claim is `null`, every `/v2/…`
+  path answers 401 whether it exists or not, and `/api/v1/packages/Home-Network` replies
+  `{"message":"token is required"}`. That is why this is a credential and not a posture
+  tweak.
+
+## What I did NOT verify
+
+- **That the pull now works.** Nothing has been pushed to that registry and the committed
+  digest is still the all-zero placeholder, so neither half of the registry fix has been
+  exercised end to end. The first real pull is what proves both at once — and if it fails, the
+  error names which half: `x509 … not valid for any names` is the resolution pin, `401` /
+  `unauthorized` is this credential.
+- **That the `/etc/hosts` pin from #59 took effect.** The operator applied it; I could not
+  confirm the effect, because the dispatcher pod's retained events are all repeating
+  `BackOff` with no fresh `Pulling`, so the `x509` message still visible is the one from
+  before the play ran. Not evidence either way.
+- **Whether `read:package` is sufficient** for a pull from a Gitea package of this type. It is
+  the documented read scope for packages; nothing here has exercised it, because no package
+  exists yet.
+- **The dockerconfigjson placeholder is not a valid credential and is not meant to be.** Its
+  `auth` field is a `REPLACE-ME` string rather than base64, so a pull with it will fail at
+  authentication rather than be mistaken for working.
+- **Anything about runtime.** Unchanged from the previous entries: the image has never been
+  built or pushed, no dispatcher has authenticated to Gitea, no issue has been triaged, no
+  worker Job has ever been created, and the admission policy — live, `Enforce`, now seven
+  rules — has still never admitted or refused a real Job.
+
+**Addendum 2026-10-30 (rebased onto #81/#82/#83; see `decisions/2026-10-30-pm-agent-live.md`).**
+Three statements above are now out of date, and one was wrong when written:
+- *"Nothing has been pushed to that registry"* — no longer true. CI published agent-runner in
+  images run 340 (`sha-523cd9118f95` = `main`, digest `sha256:c4eba5b3…6eb42`), and this
+  branch now pins it.
+- *"That the `/etc/hosts` pin from #59 took effect"* — now measured: all three prod nodes
+  carry the line, resolve the name to `192.168.2.20`, and get `401` with `ssl_verify_result=0`
+  from `https://gitea.home.keithjackson.dev/v2/` (see the new entry).
+- *"Whether `read:package` is sufficient"* — the scope is necessary but not the whole answer:
+  org-package access is decided by account permission (team Packages unit, or org
+  visibility). Handled in the registry `.example`.
+- **Wrong when written:** the scope list. Gitea 1.24.3 puts `GET /api/v1/user` — the
+  dispatcher's first call — behind the `user` scope category, so a token with only
+  `write:repository`, `write:issue` and `read:package` CrashLoops the dispatcher on startup.
+  The token needs `read:user` too.
@@ -0,0 +1,101 @@
+---
+date: 2026-10-30
+branch: feat/pm-agent-registry-pull-secret
+title: PM agent — #61 rebased onto the CI image pipeline, first published digest pinned, and the remaining path to live audited
+---
+
+## What changed
+
+PR #61 (`feat/pm-agent-registry-pull-secret`, the kubelet pull credential and admission rule
+7) was rebased onto `origin/main` after #81 (CI image pipeline), #82 and #83. Conflicts in
+`docs/plans/pm-agent-issue-to-pr.md`, `platform/pm-agent/README.md` and
+`runbooks/pm-agent.md` were resolved keeping both sides: #61's "pull credential now wired"
+and #81's "CI publishes, Renovate bumps". No manual build-and-push text came back.
+
+On top of the rebase, as a second commit:
+
+- **The first published digest is pinned**: `deployment.yaml` and `PM_WORKER_IMAGE` in
+  `configmap.yaml` now name
+  `gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`
+  (images run 340, tag `sha-523cd9118f95`, which is also `main`). The `# renovate-digest:`
+  markers are unchanged and still sit on the line directly above each value, so later bumps
+  stay Renovate's. Pinning by hand once removes the dependency on Renovate's first-ever run.
+- **The first run is dry, and scoped to a label**: `PM_DRY_RUN: "true"`,
+  `PM_REQUIRE_LABEL: "agent"` (the plan's follow-up 3); `config-generation` bumped to `2`.
+  Dry run is also a *correctness* requirement, not only caution: the pinned digest was
+  built before this PR added `imagePullSecrets` to the worker Job template, which is baked
+  into the image. Admission now requires that entry, so a worker Job built from the run-340
+  image would be denied. Merging this PR touches `infra/agent/**`, so `images.yaml`
+  re-publishes agent-runner; Renovate's bump to that digest is what makes workers
+  admissible. Only then flip `PM_DRY_RUN` to `false`.
+- **The token scope list was wrong; it is fixed.** Gitea 1.24.3's `routers/api/v1/api.go`
+  closes the `/user` route group with `tokenRequiresScopes(AccessTokenScopeCategoryUser)`,
+  and the dispatcher's first call is `GET /user`. The documented scopes are now
+  `read:user`, `write:repository`, `write:issue`, `read:package` everywhere (both
+  `.example`s, README, runbook row).
+- **Org-package permission is documented separately from the scope.** Per
+  `services/context/package.go` `determineAccessMode` (v1.24.3), a non-admin's access to an
+  org-owned package is the highest Packages-unit mode across its org teams, else read if the
+  org is visible to it. Home-Network is `public`, so it works today by visibility alone; the
+  registry `.example` now has the operator create a `pm-agent-pull` team (no repositories,
+  Packages = Read) so it does not depend on that.
+- **`sops` is toolbox-only**: both `.example`s now say so and give the in-place
+  `./infra/devcontainer/dev.sh run sops <file>` edit (the same procedure
+  `runbooks/vpn-remote-access.md` documents), which writes no plaintext file.
+- The runbook's Secret-key check used `-o jsonpath='{.data}'`, which prints values before the
+  pipe trims them; it is now a `go-template` that emits key names only. Two new runbook
+  rows: the `read:user` 403, and the admission denial from a pre-#61 digest.
+
+## What I verified
+
+- **Registry digest**, read-only from the registry API (bearer token from `/v2/token`,
+  `HEAD /v2/home-network/agent-runner/manifests/<tag>` with OCI and Docker accept headers):
+  `main` and `sha-523cd9118f95` both answer `200`,
+  `Docker-Content-Digest: sha256:c4eba5b3…6eb42`, `application/vnd.oci.image.manifest.v1+json`.
+  Tags list: `[main, sha-523cd9118f95]`. The admission pattern
+  `gitea.home.keithjackson.dev/home-network/agent-runner@sha256:*` matches the pinned value.
+- **#81's claim that the template's digest needs no tracking holds**: `pm-dispatch.build_job`
+  sets `containers[0].image = cfg.worker_image` (from `PM_WORKER_IMAGE`) on every Job.
+  Renovate's `renovate-digest` custom manager only matches `platform/pm-agent/*.yaml` and
+  `versions.env`, so the template is not touched by it.
+- **Node path to the registry, read-only over SSH, all three prod nodes**: `/etc/hosts` line 9
+  is `192.168.2.20 keycloak.home.keithjackson.dev gitea.home.keithjackson.dev`, nsswitch is
+  `hosts: files dns`, and `curl https://gitea.home.keithjackson.dev/v2/` from the node answers
+  `401` with `ssl_verify_result=0` and `remote_ip=192.168.2.20`. That is the host resolver,
+  which is what containerd (a node process) uses; a pod's view would not have shown it.
+- **Network path from the namespace**: dispatcher and worker both use
+  `PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000` (ConfigMap, Job template, and
+  `pm-worker`'s hard allowlist, which refuses anything else); the git remote and the API base
+  are both derived from it. So nothing goes through the Gateway and #83's
+  `toEndpoints`-through-Gateway issue does not apply. Egress: `toEndpoints` gitea pods `:3000`
+  (both CNPs), `kube-apiserver` (dispatcher only), `world:443` (the model). Ingress on the
+  Gitea side: `gitea-allow-pm-agent-ingress` exists live and is `VALID True`.
+- **Live state (read-only, `--context k3s-prod`)**: namespace `pm-agent` exists; Deployment
+  `0/1`, pod `ImagePullBackOff` for 24d on the all-zero digest; Secret `pm-agent-credentials`
+  (2 keys) exists, `pm-agent-registry` does not yet (it ships in this PR); Flux Kustomization
+  `pm-agent` Ready at `main@dfc432c`; ClusterPolicy `pm-agent-job-shape` Ready.
+- **Gitea state (read-only API)**: 1.24.3; org Home-Network `public`; no user `pm-agent`
+  exists; teams are `Owners` and `registry-push`; collaborators are `jakman85` (owner) and
+  `renovate-bot` (write); the repository has **no labels** and **no open issues**.
+- **Placeholders, without decrypting**: the registry ciphertext decodes to 155 bytes, exactly
+  the length of the `.example`'s placeholder JSON. The credentials values decode to 57 and 39
+  bytes — neither a 40-char Gitea token nor a ~100-char Claude OAuth token. Both files'
+  `lastmodified` is the commit that created them as placeholders.
+- `kyverno test`, `PM_SELFTEST=1 pm-dispatch` and the gate: numbers in the PR description and
+  the final handoff report for this run.
+
+## What I did NOT verify
+
+- **Any authenticated pull, poll or worker.** No `pm-agent` account or token exists, so the
+  scope set has not been exercised end to end. The `read:user` requirement is read from
+  Gitea's v1.24.3 source, not from a live 403.
+- **That the credentials ciphertext is a placeholder.** Lengths are strong evidence, not
+  proof; I did not decrypt.
+- **That Renovate will open the post-merge bump.** Its token was being re-minted at the time
+  of writing and it has never completed a run here. Renovate's `kubernetes` manager also
+  scans `infra/agent/*.yaml` and `platform/**`; what it does with a digest-only reference
+  with no tag (the template) is unmeasured.
+- **The dispatcher's `claude --print` call to `api.anthropic.com`** through `world:443`, and
+  whether the run-340 image's dispatcher behaves as this tree's code does (it predates #61's
+  selftest line, nothing else in `pm-dispatch.py`).
+- **The admission policy against a real Job.** Still never admitted or refused one live.