jakman85andClaude Opus 5.5
7f0cbc7ba2
Rebased #61 onto the CI image pipeline (#81). This commit makes the remaining in-git blockers to going live go away: - deployment.yaml and PM_WORKER_IMAGE pin images run 340's digest (sha256:c4eba5b3..., tag sha-523cd9118f95 = main), read from the registry. Renovate markers unchanged. - PM_DRY_RUN=true and PM_REQUIRE_LABEL=agent for the first run. Dry run is required, not just cautious: the run-340 image's baked worker template predates imagePullSecrets, so admission would deny its Jobs until CI re-publishes after this merge and Renovate bumps. - Token scopes: Gitea 1.24.3 scopes GET /api/v1/user (the dispatcher's first call) under read:user. The documented list was missing it. Now read:user, write:repository, write:issue, read:package everywhere. - Org-package read documented as a team (Packages=Read, no repos), not just a scope. - sops is toolbox-only; the .examples say so and give the in-place edit. - runbook: key-name check no longer emits Secret values before the pipe. Co-Authored-By: Claude Opus 5.5 <[email protected]>
105 lines
7.0 KiB
Markdown
105 lines
7.0 KiB
Markdown
---
|
|
date: 2026-10-06
|
|
branch: feat/pm-agent-registry-pull-secret
|
|
title: The kubelet's pull credential — the second half of making agent-runner pullable, and the seventh field the admission policy had not read
|
|
---
|
|
|
|
## What changed
|
|
|
|
The registry had two blockers. The first — the nodes resolving the registry hostname to the
|
|
non-hairpinned WAN and failing TLS against the router's `CN=%D` certificate — was diagnosed
|
|
and fixed by an `/etc/hosts` pin, merged in #59 and applied by the operator. This is the
|
|
second: **the pull is unauthenticated, and this registry refuses unauthenticated clients
|
|
outright.**
|
|
|
|
`pm-agent-registry` is a new `kubernetes.io/dockerconfigjson` Secret, SOPS-encrypted with
|
|
placeholder values, referenced by `imagePullSecrets` on both the dispatcher Deployment and
|
|
the committed worker Job template. It is a **second Secret holding the same Gitea token** as
|
|
`pm-agent-credentials`, because the two consumers need two formats: the application reads a
|
|
raw token from env, the kubelet reads a dockerconfigjson before any container starts. That
|
|
duplication is the established pattern here (the Keycloak client secret is mirrored the same
|
|
way, for the same reason) and it is documented in both `.example` files.
|
|
|
|
The operator's part is the value, and one scope: the Gitea token must now carry
|
|
**`read:package`** alongside `write:repository` and `write:issue`. Steps are in
|
|
`platform/pm-agent/pm-agent-registry.sops.yaml.example`, including the trap that
|
|
`.dockerconfigjson`'s registry key must be the bare host as the image reference spells it —
|
|
with a scheme or a port it matches nothing, the kubelet pulls anonymously, and the failure is
|
|
indistinguishable from the Secret being absent.
|
|
|
|
**The admission policy gained a seventh rule.** `imagePullSecrets` is a pod-spec field that
|
|
no container references, which is exactly why it is easy to leave unread — the same shape of
|
|
omission as `initContainers`, `ephemeralContainers`, `env` and `envFrom` before it. The
|
|
kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now
|
|
REQUIRES the committed entry and rule 7 forbids any other or any second one; both halves are
|
|
needed because a `foreach` over an absent list yields no result.
|
|
|
|
The leverage is small and worth saying so: the image is digest-pinned, so a different pull
|
|
credential cannot change *what* runs, and the only Secrets in this namespace are the agent's
|
|
own. It is pinned anyway, because "only the committed shape" is this policy's whole premise
|
|
and the rule costs four lines.
|
|
|
|
## What I verified
|
|
|
|
- **`kyverno test platform/kyverno/tests/pm-agent` → 33 passed, 0 failed** under the pinned
|
|
CLI (was 28 before this change). Three new fixtures, each a real path rather than a style
|
|
violation: `extra-pull-secret` (the committed entry PLUS a second one — rule 1's array
|
|
pattern is satisfied by the first element, which is why the count is asserted separately),
|
|
`wrong-pull-secret` (a different Secret in this namespace), and `no-pull-secret` (absent
|
|
entirely, refused by rule 1's pattern because rule 7's foreach cannot see it).
|
|
- **`PM_SELFTEST=1 pm-dispatch` → exit 0**, now also asserting that the committed Job
|
|
template names exactly `pm-agent-registry` — so the template and the policy cannot drift
|
|
apart without the image build failing.
|
|
- **`./ci/gate.sh` in the pinned toolbox, from a §0 worktree → exit 3 (UNDETERMINED)**, with
|
|
`gitea-settings` the only stage that did not run (no `SETTINGS_API_TOKEN` on this
|
|
workstation). Every other stage green, `spec-immutability` included. rc 3 is not a pass.
|
|
- **The gate caught me shipping the Secret in plaintext.** I wrote the placeholder
|
|
`pm-agent-registry.sops.yaml` and never ran `sops -e -i` on it. `no-plaintext-secrets`
|
|
refused it with both halves named — "has no top-level `sops:` block with a mac" and
|
|
"Secret value '.dockerconfigjson' is not SOPS ciphertext" — which is exactly the check
|
|
earning its place: the file looked finished, and the value was a placeholder, so nothing
|
|
about reading the diff would have made it obvious. Encrypted, and the stage is green.
|
|
- **The registry really does refuse anonymous clients**, measured before building this
|
|
(2026-10-05): the anonymous registry token's `access` claim is `null`, every `/v2/…`
|
|
path answers 401 whether it exists or not, and `/api/v1/packages/Home-Network` replies
|
|
`{"message":"token is required"}`. That is why this is a credential and not a posture
|
|
tweak.
|
|
|
|
## What I did NOT verify
|
|
|
|
- **That the pull now works.** Nothing has been pushed to that registry and the committed
|
|
digest is still the all-zero placeholder, so neither half of the registry fix has been
|
|
exercised end to end. The first real pull is what proves both at once — and if it fails, the
|
|
error names which half: `x509 … not valid for any names` is the resolution pin, `401` /
|
|
`unauthorized` is this credential.
|
|
- **That the `/etc/hosts` pin from #59 took effect.** The operator applied it; I could not
|
|
confirm the effect, because the dispatcher pod's retained events are all repeating
|
|
`BackOff` with no fresh `Pulling`, so the `x509` message still visible is the one from
|
|
before the play ran. Not evidence either way.
|
|
- **Whether `read:package` is sufficient** for a pull from a Gitea package of this type. It is
|
|
the documented read scope for packages; nothing here has exercised it, because no package
|
|
exists yet.
|
|
- **The dockerconfigjson placeholder is not a valid credential and is not meant to be.** Its
|
|
`auth` field is a `REPLACE-ME` string rather than base64, so a pull with it will fail at
|
|
authentication rather than be mistaken for working.
|
|
- **Anything about runtime.** Unchanged from the previous entries: the image has never been
|
|
built or pushed, no dispatcher has authenticated to Gitea, no issue has been triaged, no
|
|
worker Job has ever been created, and the admission policy — live, `Enforce`, now seven
|
|
rules — has still never admitted or refused a real Job.
|
|
|
|
**Addendum 2026-10-30 (rebased onto #81/#82/#83; see `decisions/2026-10-30-pm-agent-live.md`).**
|
|
Three statements above are now out of date, and one was wrong when written:
|
|
- *"Nothing has been pushed to that registry"* — no longer true. CI published agent-runner in
|
|
images run 340 (`sha-523cd9118f95` = `main`, digest `sha256:c4eba5b3…6eb42`), and this
|
|
branch now pins it.
|
|
- *"That the `/etc/hosts` pin from #59 took effect"* — now measured: all three prod nodes
|
|
carry the line, resolve the name to `192.168.2.20`, and get `401` with `ssl_verify_result=0`
|
|
from `https://gitea.home.keithjackson.dev/v2/` (see the new entry).
|
|
- *"Whether `read:package` is sufficient"* — the scope is necessary but not the whole answer:
|
|
org-package access is decided by account permission (team Packages unit, or org
|
|
visibility). Handled in the registry `.example`.
|
|
- **Wrong when written:** the scope list. Gitea 1.24.3 puts `GET /api/v1/user` — the
|
|
dispatcher's first call — behind the `user` scope category, so a token with only
|
|
`write:repository`, `write:issue` and `read:package` CrashLoops the dispatcher on startup.
|
|
The token needs `read:user` too.
|