Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5.5
|
7f0cbc7ba2 | ||
|
jakman85andClaude Opus 5
|
a7eadd4054 |
No files matched your search
+26
-138
@@ -60,21 +60,9 @@
|
||||
# entry's "What I did NOT verify" — that gap is stated, not papered over.
|
||||
#
|
||||
# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
|
||||
# `RENOVATE_TOKEN` — a Gitea access token for the bot account (`renovate-bot`) with FIVE scopes.
|
||||
# Each is required by a call Renovate 44.x makes against Gitea 1.24.3, and Gitea refuses a
|
||||
# token missing one with `403 token does not have at least one of required scope(s)`:
|
||||
# read:user GET /user (platform init: initPlatform)
|
||||
# read:organization GET /orgs/{owner}, /orgs/{o}/labels (initRepo -> isOrg, then org labels)
|
||||
# write:repository clone/push branches, /pulls, /statuses
|
||||
# write:issue /issues, /labels, /issues/{n}/labels (dependency dashboard, PR labels;
|
||||
# Gitea files all of these under the ISSUE scope, not repository)
|
||||
# read:package the Gitea container registry (RENOVATE_HOST_RULES below)
|
||||
# CORRECTED 2026-10-30. This list used to name only write:repository + read:user (+ read:package),
|
||||
# and the operator minted exactly that. Run 350 then failed at
|
||||
# `GET /api/v1/orgs/Home-Network 403 ... required=[read:organization], token scope=read:package,
|
||||
# write:repository,read:user`; write:issue would have been the next wall. The preflight below now
|
||||
# PROBES these scopes, so a short token fails in seconds with the missing scope named.
|
||||
# Set it as an Actions secret on this repository. Nothing in this repo mints,
|
||||
# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository`,
|
||||
# `read:user` and (since 2026-10-29) `read:package`. Set it as an Actions secret on this
|
||||
# repository. Nothing in this repo mints,
|
||||
# stores or references its value; it is consumed by name only.
|
||||
# Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
|
||||
# visible red is the correct state for "automation configured but not credentialed".
|
||||
@@ -128,37 +116,15 @@ concurrency:
|
||||
jobs:
|
||||
renovate:
|
||||
runs-on: ubuntu-latest
|
||||
# RENOVATE RUNS AS THE JOB CONTAINER, NOT THROUGH renovatebot/github-action. FIXED 2026-10-30.
|
||||
# That action does exactly one thing: `docker run ghcr.io/renovatebot/renovate:<version>`. On
|
||||
# this runner a job has NO Docker daemon, by decision (decisions/2026-10-29-image-pipeline.md:
|
||||
# act_runner sets container.docker_host to `-` when DOCKER_HOST is tcp://, and the DinD daemon
|
||||
# is privileged, so a job that could reach it would be root on the node). So every run of the
|
||||
# action died before Renovate started. Dispatch run 343 (2026-10-30), verbatim:
|
||||
# [command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** ...
|
||||
# failed to connect to the docker API at unix:///var/run/docker.sock; ... no such file or directory
|
||||
# ::error::Error: The process '/usr/bin/docker' failed with exit code 1
|
||||
# No auth, scope, config or network question was ever reached. Making the Renovate image the
|
||||
# job container needs no daemon: act starts it through the runner's own DinD exactly as it
|
||||
# starts the default ubuntu image, then `exec`s each step inside it.
|
||||
#
|
||||
# PINNED here (was the action's `renovate-version: 44.167.0`). renovate.json5 is validated
|
||||
# against this exact tag — keep them in step. Renovate's github-actions manager reads
|
||||
# `.gitea/workflows` and extracts `container.image`, so this pin is still self-maintained.
|
||||
#
|
||||
# The image runs as uid 12021 (`ubuntu`), and that is fine under act: the step command files
|
||||
# ($GITHUB_STEP_SUMMARY etc.) are created mode 0666 (gitea/act v0.261.2 pkg/runner/step.go),
|
||||
# HOME comes from the image env, and no step writes the workspace. There is deliberately NO
|
||||
# actions/checkout: Renovate clones the repo itself from RENOVATE_ENDPOINT, and as uid 12021
|
||||
# checkout could not `git init` the root-owned workspace volume anyway.
|
||||
container:
|
||||
image: ghcr.io/renovatebot/renovate:44.167.0
|
||||
# Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
|
||||
# well under a runner's patience. Without a timeout a wedged run holds the `renovate`
|
||||
# concurrency group and silently blocks every subsequent scheduled run.
|
||||
timeout-minutes: 40
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Preflight - the token must exist, and its absence must be LOUD
|
||||
shell: bash
|
||||
env:
|
||||
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
|
||||
run: |
|
||||
@@ -173,94 +139,33 @@ jobs:
|
||||
That is the failure mode that let renovate.json sit inert in this repo for months while
|
||||
seven files claimed "Renovate maintains this pin".
|
||||
|
||||
TO FIX: create a Gitea access token for the bot account (renovate-bot) with scopes
|
||||
read:user, read:organization, write:repository, write:issue and read:package (the
|
||||
header of .gitea/workflows/renovate.yaml says which call needs which), then add it to
|
||||
this repository as the Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
|
||||
TO FIX: create a Gitea access token for the bot account with scopes
|
||||
`write:repository`, `read:user` and `read:package` (the last one lets Renovate read
|
||||
this repo's own images in the Gitea registry), then add it to this repository as the
|
||||
Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
|
||||
MSG
|
||||
exit 1
|
||||
fi
|
||||
echo "RENOVATE_TOKEN is present (value not printed)."
|
||||
# Proves the step is running INSIDE the pinned Renovate image, not the default job image.
|
||||
echo "renovate $(renovate --version)"
|
||||
|
||||
- name: Preflight - the token has every scope Renovate needs
|
||||
shell: bash
|
||||
env:
|
||||
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
|
||||
# ADDED 2026-10-30. Renovate reports EVERY platform-init failure as the bare
|
||||
# "Authentication failure" (lib/modules/platform/gitea/index.ts initPlatform swallows the
|
||||
# HTTP error), and a later scope gap surfaces as "Repository has unknown error". Two
|
||||
# dispatches were spent learning one missing scope at a time. These probes are GETs only,
|
||||
# and they name the missing scope. The token never reaches argv: curl reads it from stdin
|
||||
# (`--config -`), so it is not in `ps` output or in the log.
|
||||
run: |
|
||||
set -uo pipefail
|
||||
GITEA=https://gitea.home.keithjackson.dev
|
||||
REPO=Home-Network/home-network-k3s-impl
|
||||
fail=0
|
||||
body=$(mktemp)
|
||||
probe() { # <scope it proves> <path> <auth: bearer|basic>
|
||||
local code
|
||||
if [ "$3" = basic ]; then
|
||||
code=$(printf 'user = "renovate:%s"\n' "$RENOVATE_TOKEN" \
|
||||
| curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
|
||||
else
|
||||
code=$(printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
|
||||
| curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
|
||||
fi
|
||||
if [ "$code" = 200 ]; then
|
||||
echo "ok $1 (GET ${2%%\?*} -> 200)"
|
||||
return
|
||||
fi
|
||||
fail=1
|
||||
echo "::error::scope $1: GET ${2%%\?*} -> $code"
|
||||
case "$code" in
|
||||
401) echo " 401 = Gitea rejected the token VALUE (wrong, expired or revoked)." ;;
|
||||
403) if grep -q 'Access denied' "$body"; then
|
||||
echo " 403 'Access denied' = a Cilium/Envoy policy denial, not Gitea."
|
||||
else
|
||||
echo " Gitea: $(jq -r '.message // empty' "$body" 2>/dev/null | head -c 300)"
|
||||
fi ;;
|
||||
*) echo " body: $(head -c 200 "$body")" ;;
|
||||
esac
|
||||
}
|
||||
probe read:user "/api/v1/user" bearer
|
||||
probe read:organization "/api/v1/orgs/${REPO%%/*}" bearer
|
||||
probe read:repository "/api/v1/repos/$REPO" bearer
|
||||
probe read:issue "/api/v1/repos/$REPO/labels?limit=1" bearer
|
||||
probe read:package "/v2/token?service=container_registry&scope=repository:home-network/toolbox:pull" basic
|
||||
# WRITE scopes cannot be proven by a GET. Gitea's scope-refusal message, however, lists
|
||||
# the token's scopes (names only, never the value): ask for an admin route this bot
|
||||
# cannot have, and read `token scope=` from the 403. Advisory if the format changes.
|
||||
printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
|
||||
| curl -sS --config - -o "$body" "$GITEA/api/v1/admin/orgs?limit=1" || true
|
||||
scopes=$(jq -r '.message // empty' "$body" 2>/dev/null | sed -n 's/.*token scope=//p')
|
||||
if [ -n "$scopes" ]; then
|
||||
echo "token scopes (from Gitea): $scopes"
|
||||
for want in write:repository write:issue; do
|
||||
case ",$scopes," in
|
||||
*",$want,"*|*",all,"*) echo "ok $want" ;;
|
||||
*) echo "::error::scope $want is missing (token scope=$scopes)"; fail=1 ;;
|
||||
esac
|
||||
done
|
||||
else
|
||||
echo "::warning::could not read the token's scope list; write:repository and write:issue are UNVERIFIED"
|
||||
fi
|
||||
rm -f "$body"
|
||||
if [ "$fail" -ne 0 ]; then
|
||||
echo "::error::RENOVATE_TOKEN failed the checks above (a 401 is the value, a 403 is a scope). Re-mint it for renovate-bot with read:user, read:organization, write:repository, write:issue, read:package and overwrite the secret."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Run Renovate
|
||||
shell: bash
|
||||
# The repo config is discovered by Renovate from the cloned repo (renovate.json5,
|
||||
# RENOVATE_REQUIRE_CONFIG=required). The action used to ALSO load it as the global config
|
||||
# via RENOVATE_CONFIG_FILE; that double-loading is gone. ci/lint.sh's `configurationFile:`
|
||||
# check therefore has nothing to resolve in this file, and passes.
|
||||
uses: renovatebot/[email protected]
|
||||
with:
|
||||
# Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
|
||||
# includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
|
||||
# This is the one dependency in the repo whose updater is its own updater.
|
||||
# MUST match the file's real name. #79 renamed it to renovate.json5 and left this line
|
||||
# saying `renovate.json`; every scheduled run from 2026-10-18 then died in ~1s with
|
||||
# ENOENT before Renovate started, and nothing in the gate noticed. ci/lint.sh now
|
||||
# asserts that every `configurationFile:` here names a file that exists.
|
||||
configurationFile: renovate.json5
|
||||
token: ${{ secrets.RENOVATE_TOKEN }}
|
||||
# PINNED 2026-10-29. Unset, this action version runs Renovate `39` (action.yml
|
||||
# `renovate-version` default at v41.0.9), i.e. 39.264.0, and renovate.json5 uses
|
||||
# `managerFilePatterns` (>= 40.2.0): under 39.x validation fails with five errors and
|
||||
# the run stops at CONFIG_VALIDATION. #79 validated against `latest`, not this.
|
||||
renovate-version: 44.167.0
|
||||
env:
|
||||
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
|
||||
# Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
|
||||
# header said were "set at runtime by the operator's renovate runner" — this is that
|
||||
# runner, so they are set here rather than in the repo config, which keeps renovate.json
|
||||
@@ -284,23 +189,6 @@ jobs:
|
||||
# token; it therefore needs scope `read:package` in addition to the ones listed in the
|
||||
# preflight above. Gitea ignores the username for token auth (services/auth/basic.go).
|
||||
RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]'
|
||||
# OPTIONAL, ADDED 2026-10-30. A github.com token (a fine-grained PAT with NO permissions,
|
||||
# public-repo read only) for lookups of GitHub-hosted dependencies. Without it Renovate
|
||||
# SKIPS them with `github-token-required` — measured in a local dry run: 17 deps, i.e.
|
||||
# every `uses: actions/*` here plus kubevirt/kubevirt, kubevirt/containerized-data-importer
|
||||
# and intel/intel-device-plugins-for-kubernetes. Unset is safe: the step below drops the
|
||||
# empty variable, and those deps simply stay unmaintained, with a WARN in the log.
|
||||
GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# An expression that evaluates to '' still EXPORTS the variable, as an empty string.
|
||||
# Drop those so an unset value means "Renovate's default", never "set to empty".
|
||||
for v in RENOVATE_DRY_RUN GITHUB_COM_TOKEN; do
|
||||
[ -n "${!v:-}" ] || unset "$v"
|
||||
done
|
||||
# Through the image's entrypoint, as `docker run` would have gone: it sources the
|
||||
# containerbase env and initialises the bundled tools before exec'ing Renovate.
|
||||
exec renovate-entrypoint.sh renovate
|
||||
|
||||
- name: Summary
|
||||
if: always()
|
||||
|
||||
-16
@@ -42,19 +42,3 @@ while IFS=: read -r wf _ value; do
|
||||
done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
|
||||
[ "$missing" -eq 0 ] || exit 1
|
||||
ok "workflow configurationFile references resolve"
|
||||
|
||||
# A workflow must not use an action that needs a Docker daemon: jobs on this runner get NONE, by
|
||||
# decision (decisions/2026-10-29-image-pipeline.md; act_runner maps docker_host to `-`). Such an
|
||||
# action is valid YAML and passes every other stage, then dies at runtime with
|
||||
# "failed to connect to the docker API at unix:///var/run/docker.sock". renovatebot/github-action
|
||||
# did exactly that on every Renovate run until 2026-10-30 (run 343). Run the tool's image as the
|
||||
# job `container:` instead (see .gitea/workflows/renovate.yaml). Container (`docker://`) actions
|
||||
# and Docker's own daemon-bound actions are rejected for the same reason. Comments are ignored.
|
||||
daemon_uses='uses:[[:space:]]*["'\'']?(docker://|renovatebot/github-action|docker/(build-push|setup-buildx|setup-qemu|login)-action)'
|
||||
bad=0
|
||||
while IFS= read -r hit; do
|
||||
err "${hit%%:*}: needs a Docker daemon, which jobs here do not get: $(echo "${hit#*:*:}" | sed -E 's/^[[:space:]]+//')"
|
||||
bad=1
|
||||
done < <(grep -HnE "^[^#]*${daemon_uses}" .gitea/workflows/*.y*ml 2>/dev/null || true)
|
||||
[ "$bad" -eq 0 ] || exit 1
|
||||
ok "no workflow uses a Docker-daemon-bound action"
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
date: 2026-10-06
|
||||
branch: feat/pm-agent-registry-pull-secret
|
||||
title: The kubelet's pull credential — the second half of making agent-runner pullable, and the seventh field the admission policy had not read
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The registry had two blockers. The first — the nodes resolving the registry hostname to the
|
||||
non-hairpinned WAN and failing TLS against the router's `CN=%D` certificate — was diagnosed
|
||||
and fixed by an `/etc/hosts` pin, merged in #59 and applied by the operator. This is the
|
||||
second: **the pull is unauthenticated, and this registry refuses unauthenticated clients
|
||||
outright.**
|
||||
|
||||
`pm-agent-registry` is a new `kubernetes.io/dockerconfigjson` Secret, SOPS-encrypted with
|
||||
placeholder values, referenced by `imagePullSecrets` on both the dispatcher Deployment and
|
||||
the committed worker Job template. It is a **second Secret holding the same Gitea token** as
|
||||
`pm-agent-credentials`, because the two consumers need two formats: the application reads a
|
||||
raw token from env, the kubelet reads a dockerconfigjson before any container starts. That
|
||||
duplication is the established pattern here (the Keycloak client secret is mirrored the same
|
||||
way, for the same reason) and it is documented in both `.example` files.
|
||||
|
||||
The operator's part is the value, and one scope: the Gitea token must now carry
|
||||
**`read:package`** alongside `write:repository` and `write:issue`. Steps are in
|
||||
`platform/pm-agent/pm-agent-registry.sops.yaml.example`, including the trap that
|
||||
`.dockerconfigjson`'s registry key must be the bare host as the image reference spells it —
|
||||
with a scheme or a port it matches nothing, the kubelet pulls anonymously, and the failure is
|
||||
indistinguishable from the Secret being absent.
|
||||
|
||||
**The admission policy gained a seventh rule.** `imagePullSecrets` is a pod-spec field that
|
||||
no container references, which is exactly why it is easy to leave unread — the same shape of
|
||||
omission as `initContainers`, `ephemeralContainers`, `env` and `envFrom` before it. The
|
||||
kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now
|
||||
REQUIRES the committed entry and rule 7 forbids any other or any second one; both halves are
|
||||
needed because a `foreach` over an absent list yields no result.
|
||||
|
||||
The leverage is small and worth saying so: the image is digest-pinned, so a different pull
|
||||
credential cannot change *what* runs, and the only Secrets in this namespace are the agent's
|
||||
own. It is pinned anyway, because "only the committed shape" is this policy's whole premise
|
||||
and the rule costs four lines.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **`kyverno test platform/kyverno/tests/pm-agent` → 33 passed, 0 failed** under the pinned
|
||||
CLI (was 28 before this change). Three new fixtures, each a real path rather than a style
|
||||
violation: `extra-pull-secret` (the committed entry PLUS a second one — rule 1's array
|
||||
pattern is satisfied by the first element, which is why the count is asserted separately),
|
||||
`wrong-pull-secret` (a different Secret in this namespace), and `no-pull-secret` (absent
|
||||
entirely, refused by rule 1's pattern because rule 7's foreach cannot see it).
|
||||
- **`PM_SELFTEST=1 pm-dispatch` → exit 0**, now also asserting that the committed Job
|
||||
template names exactly `pm-agent-registry` — so the template and the policy cannot drift
|
||||
apart without the image build failing.
|
||||
- **`./ci/gate.sh` in the pinned toolbox, from a §0 worktree → exit 3 (UNDETERMINED)**, with
|
||||
`gitea-settings` the only stage that did not run (no `SETTINGS_API_TOKEN` on this
|
||||
workstation). Every other stage green, `spec-immutability` included. rc 3 is not a pass.
|
||||
- **The gate caught me shipping the Secret in plaintext.** I wrote the placeholder
|
||||
`pm-agent-registry.sops.yaml` and never ran `sops -e -i` on it. `no-plaintext-secrets`
|
||||
refused it with both halves named — "has no top-level `sops:` block with a mac" and
|
||||
"Secret value '.dockerconfigjson' is not SOPS ciphertext" — which is exactly the check
|
||||
earning its place: the file looked finished, and the value was a placeholder, so nothing
|
||||
about reading the diff would have made it obvious. Encrypted, and the stage is green.
|
||||
- **The registry really does refuse anonymous clients**, measured before building this
|
||||
(2026-10-05): the anonymous registry token's `access` claim is `null`, every `/v2/…`
|
||||
path answers 401 whether it exists or not, and `/api/v1/packages/Home-Network` replies
|
||||
`{"message":"token is required"}`. That is why this is a credential and not a posture
|
||||
tweak.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That the pull now works.** Nothing has been pushed to that registry and the committed
|
||||
digest is still the all-zero placeholder, so neither half of the registry fix has been
|
||||
exercised end to end. The first real pull is what proves both at once — and if it fails, the
|
||||
error names which half: `x509 … not valid for any names` is the resolution pin, `401` /
|
||||
`unauthorized` is this credential.
|
||||
- **That the `/etc/hosts` pin from #59 took effect.** The operator applied it; I could not
|
||||
confirm the effect, because the dispatcher pod's retained events are all repeating
|
||||
`BackOff` with no fresh `Pulling`, so the `x509` message still visible is the one from
|
||||
before the play ran. Not evidence either way.
|
||||
- **Whether `read:package` is sufficient** for a pull from a Gitea package of this type. It is
|
||||
the documented read scope for packages; nothing here has exercised it, because no package
|
||||
exists yet.
|
||||
- **The dockerconfigjson placeholder is not a valid credential and is not meant to be.** Its
|
||||
`auth` field is a `REPLACE-ME` string rather than base64, so a pull with it will fail at
|
||||
authentication rather than be mistaken for working.
|
||||
- **Anything about runtime.** Unchanged from the previous entries: the image has never been
|
||||
built or pushed, no dispatcher has authenticated to Gitea, no issue has been triaged, no
|
||||
worker Job has ever been created, and the admission policy — live, `Enforce`, now seven
|
||||
rules — has still never admitted or refused a real Job.
|
||||
|
||||
**Addendum 2026-10-30 (rebased onto #81/#82/#83; see `decisions/2026-10-30-pm-agent-live.md`).**
|
||||
Three statements above are now out of date, and one was wrong when written:
|
||||
- *"Nothing has been pushed to that registry"* — no longer true. CI published agent-runner in
|
||||
images run 340 (`sha-523cd9118f95` = `main`, digest `sha256:c4eba5b3…6eb42`), and this
|
||||
branch now pins it.
|
||||
- *"That the `/etc/hosts` pin from #59 took effect"* — now measured: all three prod nodes
|
||||
carry the line, resolve the name to `192.168.2.20`, and get `401` with `ssl_verify_result=0`
|
||||
from `https://gitea.home.keithjackson.dev/v2/` (see the new entry).
|
||||
- *"Whether `read:package` is sufficient"* — the scope is necessary but not the whole answer:
|
||||
org-package access is decided by account permission (team Packages unit, or org
|
||||
visibility). Handled in the registry `.example`.
|
||||
- **Wrong when written:** the scope list. Gitea 1.24.3 puts `GET /api/v1/user` — the
|
||||
dispatcher's first call — behind the `user` scope category, so a token with only
|
||||
`write:repository`, `write:issue` and `read:package` CrashLoops the dispatcher on startup.
|
||||
The token needs `read:user` too.
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
date: 2026-10-30
|
||||
branch: feat/pm-agent-registry-pull-secret
|
||||
title: PM agent — #61 rebased onto the CI image pipeline, first published digest pinned, and the remaining path to live audited
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
PR #61 (`feat/pm-agent-registry-pull-secret`, the kubelet pull credential and admission rule
|
||||
7) was rebased onto `origin/main` after #81 (CI image pipeline), #82 and #83. Conflicts in
|
||||
`docs/plans/pm-agent-issue-to-pr.md`, `platform/pm-agent/README.md` and
|
||||
`runbooks/pm-agent.md` were resolved keeping both sides: #61's "pull credential now wired"
|
||||
and #81's "CI publishes, Renovate bumps". No manual build-and-push text came back.
|
||||
|
||||
On top of the rebase, as a second commit:
|
||||
|
||||
- **The first published digest is pinned**: `deployment.yaml` and `PM_WORKER_IMAGE` in
|
||||
`configmap.yaml` now name
|
||||
`gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42`
|
||||
(images run 340, tag `sha-523cd9118f95`, which is also `main`). The `# renovate-digest:`
|
||||
markers are unchanged and still sit on the line directly above each value, so later bumps
|
||||
stay Renovate's. Pinning by hand once removes the dependency on Renovate's first-ever run.
|
||||
- **The first run is dry, and scoped to a label**: `PM_DRY_RUN: "true"`,
|
||||
`PM_REQUIRE_LABEL: "agent"` (the plan's follow-up 3); `config-generation` bumped to `2`.
|
||||
Dry run is also a *correctness* requirement, not only caution: the pinned digest was
|
||||
built before this PR added `imagePullSecrets` to the worker Job template, which is baked
|
||||
into the image. Admission now requires that entry, so a worker Job built from the run-340
|
||||
image would be denied. Merging this PR touches `infra/agent/**`, so `images.yaml`
|
||||
re-publishes agent-runner; Renovate's bump to that digest is what makes workers
|
||||
admissible. Only then flip `PM_DRY_RUN` to `false`.
|
||||
- **The token scope list was wrong; it is fixed.** Gitea 1.24.3's `routers/api/v1/api.go`
|
||||
closes the `/user` route group with `tokenRequiresScopes(AccessTokenScopeCategoryUser)`,
|
||||
and the dispatcher's first call is `GET /user`. The documented scopes are now
|
||||
`read:user`, `write:repository`, `write:issue`, `read:package` everywhere (both
|
||||
`.example`s, README, runbook row).
|
||||
- **Org-package permission is documented separately from the scope.** Per
|
||||
`services/context/package.go` `determineAccessMode` (v1.24.3), a non-admin's access to an
|
||||
org-owned package is the highest Packages-unit mode across its org teams, else read if the
|
||||
org is visible to it. Home-Network is `public`, so it works today by visibility alone; the
|
||||
registry `.example` now has the operator create a `pm-agent-pull` team (no repositories,
|
||||
Packages = Read) so it does not depend on that.
|
||||
- **`sops` is toolbox-only**: both `.example`s now say so and give the in-place
|
||||
`./infra/devcontainer/dev.sh run sops <file>` edit (the same procedure
|
||||
`runbooks/vpn-remote-access.md` documents), which writes no plaintext file.
|
||||
- The runbook's Secret-key check used `-o jsonpath='{.data}'`, which prints values before the
|
||||
pipe trims them; it is now a `go-template` that emits key names only. Two new runbook
|
||||
rows: the `read:user` 403, and the admission denial from a pre-#61 digest.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **Registry digest**, read-only from the registry API (bearer token from `/v2/token`,
|
||||
`HEAD /v2/home-network/agent-runner/manifests/<tag>` with OCI and Docker accept headers):
|
||||
`main` and `sha-523cd9118f95` both answer `200`,
|
||||
`Docker-Content-Digest: sha256:c4eba5b3…6eb42`, `application/vnd.oci.image.manifest.v1+json`.
|
||||
Tags list: `[main, sha-523cd9118f95]`. The admission pattern
|
||||
`gitea.home.keithjackson.dev/home-network/agent-runner@sha256:*` matches the pinned value.
|
||||
- **#81's claim that the template's digest needs no tracking holds**: `pm-dispatch.build_job`
|
||||
sets `containers[0].image = cfg.worker_image` (from `PM_WORKER_IMAGE`) on every Job.
|
||||
Renovate's `renovate-digest` custom manager only matches `platform/pm-agent/*.yaml` and
|
||||
`versions.env`, so the template is not touched by it.
|
||||
- **Node path to the registry, read-only over SSH, all three prod nodes**: `/etc/hosts` line 9
|
||||
is `192.168.2.20 keycloak.home.keithjackson.dev gitea.home.keithjackson.dev`, nsswitch is
|
||||
`hosts: files dns`, and `curl https://gitea.home.keithjackson.dev/v2/` from the node answers
|
||||
`401` with `ssl_verify_result=0` and `remote_ip=192.168.2.20`. That is the host resolver,
|
||||
which is what containerd (a node process) uses; a pod's view would not have shown it.
|
||||
- **Network path from the namespace**: dispatcher and worker both use
|
||||
`PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000` (ConfigMap, Job template, and
|
||||
`pm-worker`'s hard allowlist, which refuses anything else); the git remote and the API base
|
||||
are both derived from it. So nothing goes through the Gateway and #83's
|
||||
`toEndpoints`-through-Gateway issue does not apply. Egress: `toEndpoints` gitea pods `:3000`
|
||||
(both CNPs), `kube-apiserver` (dispatcher only), `world:443` (the model). Ingress on the
|
||||
Gitea side: `gitea-allow-pm-agent-ingress` exists live and is `VALID True`.
|
||||
- **Live state (read-only, `--context k3s-prod`)**: namespace `pm-agent` exists; Deployment
|
||||
`0/1`, pod `ImagePullBackOff` for 24d on the all-zero digest; Secret `pm-agent-credentials`
|
||||
(2 keys) exists, `pm-agent-registry` does not yet (it ships in this PR); Flux Kustomization
|
||||
`pm-agent` Ready at `main@dfc432c`; ClusterPolicy `pm-agent-job-shape` Ready.
|
||||
- **Gitea state (read-only API)**: 1.24.3; org Home-Network `public`; no user `pm-agent`
|
||||
exists; teams are `Owners` and `registry-push`; collaborators are `jakman85` (owner) and
|
||||
`renovate-bot` (write); the repository has **no labels** and **no open issues**.
|
||||
- **Placeholders, without decrypting**: the registry ciphertext decodes to 155 bytes, exactly
|
||||
the length of the `.example`'s placeholder JSON. The credentials values decode to 57 and 39
|
||||
bytes — neither a 40-char Gitea token nor a ~100-char Claude OAuth token. Both files'
|
||||
`lastmodified` is the commit that created them as placeholders.
|
||||
- `kyverno test`, `PM_SELFTEST=1 pm-dispatch` and the gate: numbers in the PR description and
|
||||
the final handoff report for this run.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Any authenticated pull, poll or worker.** No `pm-agent` account or token exists, so the
|
||||
scope set has not been exercised end to end. The `read:user` requirement is read from
|
||||
Gitea's v1.24.3 source, not from a live 403.
|
||||
- **That the credentials ciphertext is a placeholder.** Lengths are strong evidence, not
|
||||
proof; I did not decrypt.
|
||||
- **That Renovate will open the post-merge bump.** Its token was being re-minted at the time
|
||||
of writing and it has never completed a run here. Renovate's `kubernetes` manager also
|
||||
scans `infra/agent/*.yaml` and `platform/**`; what it does with a digest-only reference
|
||||
with no tag (the template) is unmeasured.
|
||||
- **The dispatcher's `claude --print` call to `api.anthropic.com`** through `world:443`, and
|
||||
whether the run-340 image's dispatcher behaves as this tree's code does (it predates #61's
|
||||
selftest line, nothing else in `pm-dispatch.py`).
|
||||
- **The admission policy against a real Job.** Still never admitted or refused one live.
|
||||
@@ -1,108 +0,0 @@
|
||||
---
|
||||
date: 2026-10-30
|
||||
branch: fix/renovate-auth
|
||||
title: Renovate's token needs five scopes, not three; the workflow documented the wrong list, and now probes it
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**The fix is operator-side.** `RENOVATE_TOKEN` lacks `read:organization`, and it will next lack
|
||||
`write:issue`. The operator must re-mint it with five scopes:
|
||||
|
||||
- `read:user`
|
||||
- `read:organization`
|
||||
- `write:repository`
|
||||
- `write:issue`
|
||||
- `read:package`
|
||||
|
||||
The git half fixes the two things that sent the operator round this loop twice.
|
||||
|
||||
**`.gitea/workflows/renovate.yaml`**:
|
||||
- **The header's scope list is corrected**, with the call each scope serves. The old list named
|
||||
only `write:repository`, `read:user` and `read:package`, and the operator minted exactly that.
|
||||
The preflight's "TO FIX" text is corrected the same way.
|
||||
- **A new step, "Preflight - the token has every scope Renovate needs".**
|
||||
- It probes read-only GETs: `/user` for read:user, `/orgs/Home-Network` for read:organization,
|
||||
`/repos/{repo}` for read:repository, `/repos/{repo}/labels` for read:issue, and registry
|
||||
`/v2/token` for read:package.
|
||||
- It reads the token's own scope list out of Gitea's 403 from `/admin/orgs` (names only), and
|
||||
requires `write:repository` and `write:issue` in it.
|
||||
- It says what a failure means: 401 is the value; a Gitea 403 is a scope, and the probe prints
|
||||
Gitea's message; an "Access denied" 403 is a Cilium/Envoy denial.
|
||||
- The token is fed to curl on stdin (`--config -`), never in argv.
|
||||
|
||||
**`ci/lint.sh`**: a new guard rejects any non-comment `uses:` in `.gitea/workflows` that needs a
|
||||
Docker daemon: `docker://`, `renovatebot/github-action` and Docker's daemon-bound
|
||||
`docker/*-action`s. That is the class of bug fixed in #84.
|
||||
|
||||
### Root cause, with evidence
|
||||
|
||||
Run 350 (task 567, 06:49 UTC) **got further than 348 and 349.** Gitea's router log, filtered to
|
||||
the act-runner pod `10.0.0.38`:
|
||||
|
||||
06:49:31 GET /api/v1/user 200 OK
|
||||
06:49:31 GET /api/v1/version 200 OK
|
||||
06:49:31 GET /api/v1/repos/Home-Network/home-network-k3s-impl 200 OK
|
||||
06:49:31 GET /api/v1/orgs/Home-Network 403 Forbidden
|
||||
|
||||
There were no `/v2/` requests: Renovate stopped before extraction. From the job log, verbatim:
|
||||
|
||||
ERROR: Repository has unknown error (repository=Home-Network/home-network-k3s-impl)
|
||||
"url": "https://gitea.home.keithjackson.dev/api/v1/orgs/Home-Network",
|
||||
"message": "token does not have at least one of required scope(s), required=[read:organization], token scope=read:package,write:repository,read:user"
|
||||
"x-envoy-upstream-service-time": "27", "server": "envoy"
|
||||
INFO: Repository finished ... "result": "unknown-error", "exitCode": 8
|
||||
|
||||
**This is not Cilium.** The body is Gitea's JSON scope message, Gitea logged the request, and
|
||||
`x-envoy-upstream-service-time` shows that Envoy forwarded it upstream.
|
||||
|
||||
**The secret WAS updated.** Runs 348 and 349 got `GET /user 403` from `tokenRequiresScopes`. Run 350
|
||||
got `/user 200`, and its token scope list now includes `read:user`.
|
||||
|
||||
**Why `read:organization` is needed.** Renovate 44.167.0, `lib/modules/platform/gitea/index.ts`
|
||||
`initRepo`, calls `helper.isOrg` → `GET /orgs/{owner}` and rethrows on error. Gitea 1.24.3,
|
||||
`routers/api/v1/api.go`, puts `/orgs/{org}` behind
|
||||
`tokenRequiresScopes(AccessTokenScopeCategoryOrganization)`.
|
||||
|
||||
**Why `write:issue` is needed next.** In the same `api.go`, `/repos/{o}/{r}/issues…`, `/labels`
|
||||
and `/milestones` sit in a group with `tokenRequiresScopes(AccessTokenScopeCategoryIssue)`
|
||||
(lines 1466–1582). Renovate's gitea-helper uses `/issues`, `/issues/{n}/labels`, `/labels` and
|
||||
`/issues/{n}/comments`, for the dependency dashboard (`:dependencyDashboard` is extended) and for
|
||||
PR labels. A dry run only reads them; a scheduled run writes them.
|
||||
|
||||
The other causes the coordinator listed are ruled out:
|
||||
- **Schedule window and dry-run.** The run never reached branch processing.
|
||||
- **Repository config.** The run never reached the clone (`"cloned": undefined`).
|
||||
- **The Summary step.** It is not involved; the failure is Renovate's own exit 8.
|
||||
|
||||
## What I verified
|
||||
|
||||
- I read the whole run 350 job log (204 lines), and the router lines above. Runs 348 and 349 are
|
||||
as recorded in `decisions/2026-10-30-renovate-first-run.md`'s follow-up: `/user` got 403 from
|
||||
`tokenRequiresScopes`.
|
||||
- **The probe step, in the pinned image.** I extracted the step body from the YAML and ran it in
|
||||
`ghcr.io/renovatebot/renovate:44.167.0` (`curl` 8.5.0 and `jq` are present) with a deliberately
|
||||
**bogus** token. Every probe reported 401 with the "token VALUE" diagnosis, the scope-list read
|
||||
fell back to a warning, and the step exited 1.
|
||||
- **The scope-list parser.** Against run 350's exact message format it reports `write:issue`
|
||||
MISSING. Against a five-scope list it reports both OK. Against `all` it reports both OK.
|
||||
- **The lint guard.** It flags `renovatebot/github-action` in the pre-#84 `renovate.yaml`
|
||||
(dfc432c, line 152). It reports nothing on the current workflows, and it ignores commented
|
||||
lines. `bash -n ci/lint.sh` passes.
|
||||
- **Account facts, read-only.** `renovate-bot` is a collaborator with `write`, and the org is
|
||||
public.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The probe with a real token. UNVERIFIED.** I never had the token. In particular, I have not
|
||||
observed that `GET /api/v1/admin/orgs` gives renovate-bot's token a 403 with a `token scope=`
|
||||
list. The source says the scope check runs before `reqSiteAdmin`. If the message format
|
||||
differs, the step warns and does not fail.
|
||||
- **That five scopes are enough for a full non-dry run.** They cover every helper endpoint Renovate
|
||||
44.167.0 calls, mapped to Gitea 1.24.3's route categories by reading source. Branch pushes go
|
||||
over git-over-HTTPS with the token; I believe that needs write:repository, but I have not
|
||||
observed it.
|
||||
- **Registry reads as renovate-bot.** `read:package` on a public org is expected to work. The new
|
||||
probe is the first real test of it.
|
||||
- **Sensitive paths.** `.gitea/**` and `ci/**` are both sensitive. CI's `approval-policy` will block
|
||||
until the operator posts `/approve-override` as a PR review.
|
||||
@@ -1,113 +0,0 @@
|
||||
---
|
||||
date: 2026-10-30
|
||||
branch: fix/renovate-first-run
|
||||
title: Renovate never started because its action needs a Docker daemon jobs here do not get; it now runs as the job container
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**`.gitea/workflows/renovate.yaml`**: the job no longer uses `renovatebot/github-action`. The
|
||||
pinned Renovate image (`ghcr.io/renovatebot/renovate:44.167.0`, the same version the action was
|
||||
pinned to) is now the job's `container:`. A `run:` step calls `renovate-entrypoint.sh renovate`
|
||||
inside it. `actions/checkout` is gone, because Renovate clones the repo itself.
|
||||
`GITHUB_COM_TOKEN` is wired from a new **optional** secret, `RENOVATE_GITHUB_COM_TOKEN`. The step
|
||||
unsets it, and `RENOVATE_DRY_RUN`, when they evaluate to the empty string. The preflight also
|
||||
prints `renovate --version`, which proves the step ran inside the pinned image.
|
||||
|
||||
**`renovate.json5`**: one comment now names `container.image` as the version pin, in place of
|
||||
`renovate-version`. The config itself is unchanged.
|
||||
|
||||
### Why: the root cause
|
||||
|
||||
Every run of the action died before Renovate started. That includes the operator's dispatch,
|
||||
run 343 (task 555, 2026-10-30 06:14 UTC), and the 11 scheduled runs before it (312–322). From
|
||||
run 343's log:
|
||||
|
||||
RENOVATE_TOKEN is present (value not printed).
|
||||
[command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** --env RENOVATE_CONFIG_FILE=/github-action/renovate.json5 ...
|
||||
failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running: dial unix /var/run/docker.sock: connect: no such file or directory
|
||||
Error: The process '/usr/bin/docker' failed with exit code 1
|
||||
|
||||
The action is only a wrapper around `docker run`. On this runner a job deliberately gets no
|
||||
Docker daemon (`decisions/2026-10-29-image-pipeline.md`): act_runner maps `docker_host` to `-` when
|
||||
`DOCKER_HOST` is `tcp://`, and the DinD daemon is privileged. So this was a
|
||||
platform-compatibility failure. The run never reached authentication, token scope, config or the
|
||||
network. **The re-minted token is not implicated, and nothing in this log can say whether its
|
||||
scopes are right.**
|
||||
|
||||
A job `container:` needs no daemon. act creates it through the runner's DinD the same way it
|
||||
creates the default `ubuntu-latest` image, then `exec`s each step inside it. I checked this
|
||||
against gitea/act v0.261.2:
|
||||
|
||||
- The job container's entrypoint is `/bin/sleep` (`run_context.go`). The Renovate image has
|
||||
`/usr/bin/sleep`.
|
||||
- The step command files are written `0o666` (`step.go`), so the image's uid 12021 can write
|
||||
`$GITHUB_STEP_SUMMARY`.
|
||||
- `HOME` comes from the image env.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **The root cause.** I read all of run 343's job log (162 lines, job 0; job 1 is 404). It has one
|
||||
error, quoted above, and no Renovate output at all. Runs 312–322 are all `failure` from the
|
||||
`schedule` event.
|
||||
- **The config validates against the pinned tag.** `podman run ... ghcr.io/renovatebot/renovate:44.167.0
|
||||
renovate-config-validator /tmp/renovate.json5` printed `Config validated successfully`. It also
|
||||
printed the known migration diff for the top-level `docker` block.
|
||||
- **A dry run with the fix extracts and looks up dependencies.** This was a local reproduction. I
|
||||
made a clean snapshot of this branch's tracked files. I ran it in a podman container of the pinned
|
||||
image with entrypoint `/bin/sleep`, as the image's own uid 12021 with `HOME=/home/ubuntu`.
|
||||
Inside it I `exec`ed the workflow step's `run:` body, extracted verbatim from the YAML, with:
|
||||
- `RENOVATE_PLATFORM=local`, `RENOVATE_DRY_RUN=full` and `GITHUB_COM_TOKEN=` (empty, to
|
||||
exercise the unset);
|
||||
- a docker host rule for `gitea.home.keithjackson.dev` using the `gitea-admin` credential from
|
||||
`~/.git-credentials`, passed through env and never printed.
|
||||
|
||||
The run exited 0 and logged `Dependency extraction complete`:
|
||||
- flux 15 deps / 13 files
|
||||
- github-actions 24 / 4
|
||||
- kubernetes 131 / 67
|
||||
- kustomize 2 / 2
|
||||
- regex (both custom managers) 7 / 6
|
||||
- **total 179 deps / 92 files**
|
||||
|
||||
It found **42 updates**: 29 kubernetes, 6 flux, 2 kustomize, 1 github-actions and 4 regex
|
||||
`digest`. 7 of them are `major`.
|
||||
- **The `# renovate-digest:` lookups against the Gitea registry succeed.** All three
|
||||
`/v2/token?scope=repository:home-network/<img>:pull` requests succeeded, and so did
|
||||
`HEAD /v2/home-network/<img>/manifests/main`. They proposed `digest` updates for
|
||||
`conformance/versions.env`, `infra/agent/versions.env`, `platform/pm-agent/configmap.yaml` and
|
||||
`platform/pm-agent/deployment.yaml`, all of which carry the zero placeholder digest today. The
|
||||
`kubernetes` manager's copies of those images report `disabled`, which is the intended effect of
|
||||
the packageRule.
|
||||
- **The new pin is still maintained by Renovate.** The github-actions manager extracts
|
||||
`ghcr.io/renovatebot/renovate:44.167.0` from `renovate.yaml` as depType `container`, and proposes
|
||||
44.198.0.
|
||||
- **The `regex` vs `custom.regex` question from #73/#77 is closed.** Both custom managers produced
|
||||
dependencies.
|
||||
- **Missing GitHub token.** The local run logged `GitHub token is required for some dependencies`.
|
||||
That accounts for 17 deps skipped with `github-token-required`, which is why
|
||||
`RENOVATE_GITHUB_COM_TOKEN` is offered as an optional secret.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The workflow on the real runner. UNVERIFIED.** My reproduction ran with podman, not
|
||||
act_runner + DinD, and used `--platform=local`. The act behaviour (sleep entrypoint, `exec` as the
|
||||
image user, 0666 command files) comes from reading the gitea/act v0.261.2 source, not from a run.
|
||||
The first real evidence will be a `workflow_dispatch` after merge.
|
||||
- **The Gitea platform path and `RENOVATE_TOKEN`'s scopes. UNVERIFIED.** I have no access token, only
|
||||
the `gitea-admin` password, which Renovate's Gitea platform cannot use. So these were not
|
||||
exercised: platform init (`GET /user`, which needs `read:user`), repo clone, and the registry
|
||||
lookups using the **token** with `read:package` rather than a password.
|
||||
- **Behaviour inside the schedule window.** A dispatch outside 03:00–06:00 America/New_York
|
||||
(07:00–10:00 UTC during EDT) will extract and look up, but will log "not within schedule"
|
||||
instead of `DRY-RUN: Would create ...` lines. I did not change this.
|
||||
- **Renovate as uid 12021 doing artifact updates** (for example helm lockfiles) on a real,
|
||||
non-dry run. `--dry-run=full` does not run the tools.
|
||||
- **The self-bump.** Renovate will propose `ghcr.io/renovatebot/renovate` 44.167.0 → 44.198.0 as a
|
||||
minor, which the default rule auto-merges. `.gitea/**` is a sensitive path, so it will in fact
|
||||
wait for an administrator override. Nothing re-runs `renovate-config-validator` on that bump,
|
||||
and no gate check runs the validator.
|
||||
- **No alert for "Renovate red every day".** That gap is still open. This failure ran for 12 days
|
||||
without anyone noticing.
|
||||
- **Sensitive path.** `.gitea/**` is in `infra/gitea/approval-policy.json`, so CI's
|
||||
`approval-policy` stage will block until the operator posts `/approve-override` as a PR review.
|
||||
@@ -241,15 +241,18 @@ about runtime behaviour in this document is a design claim, not a measurement.
|
||||
`gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
|
||||
pin the Keycloak issuer already carries, applied by
|
||||
`ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`;
|
||||
(ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
|
||||
the credential decision now being local Gitea accounts with PATs, that means a
|
||||
`dockerconfigjson` for an account whose token carries `read:package`. Only then does the
|
||||
digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step.
|
||||
`.gitea/workflows/images.yaml` builds and verifies the image on rootless BuildKit
|
||||
(`platform/buildkit/`), publishes it on a push to `main`, and Renovate opens the one
|
||||
PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The
|
||||
same two blockers sit in front of `conformance-runner`, which shares the registry path and
|
||||
has never been pulled.
|
||||
(ii) the pull must be authenticated — **now wired**: `pm-agent-registry`
|
||||
(`dockerconfigjson`) is referenced by the Deployment and the committed worker Job
|
||||
template, and the admission policy pins `imagePullSecrets` to exactly that name. The
|
||||
operator supplies the value, from a Gitea token that also carries `read:package`. Only
|
||||
then does the digest matter. **Superseded 2026-10-29:** the build-and-push is no longer
|
||||
an operator step. `.gitea/workflows/images.yaml` builds and verifies the image on rootless
|
||||
BuildKit (`platform/buildkit/`) and publishes it on a push to `main`
|
||||
(`decisions/2026-10-29-image-pipeline.md`); the first published digest is pinned in
|
||||
`deployment.yaml` and `configmap.yaml`, and Renovate carries every later bump through the
|
||||
`# renovate-digest:` markers (`decisions/2026-10-30-pm-agent-live.md`). The same
|
||||
credential blocker sits in front of `conformance-runner`, which shares the registry path
|
||||
and has never been pulled.
|
||||
2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the
|
||||
bot's permission is `write`, not `admin`** — that measurement is the control that keeps
|
||||
the agent out of its own approval gate.
|
||||
|
||||
@@ -922,6 +922,8 @@ def selftest_template(path: str) -> None:
|
||||
assert pod.get("automountServiceAccountToken") is False, \
|
||||
"template must not mount a ServiceAccount token"
|
||||
assert pod.get("restartPolicy") == "Never", "a retried agent would double-push"
|
||||
assert [x.get("name") for x in pod.get("imagePullSecrets") or []] == ["pm-agent-registry"], \
|
||||
"the template must name exactly the committed pull secret (admission pins it)"
|
||||
assert len(pod.get("containers") or []) == 1, "exactly one container"
|
||||
c = pod["containers"][0]
|
||||
assert c.get("command") == ["/usr/local/bin/pm-worker"], "command must be the pinned entrypoint"
|
||||
|
||||
@@ -68,6 +68,13 @@ spec:
|
||||
# guarantee it cannot use one is for it not to have one.
|
||||
automountServiceAccountToken: false
|
||||
serviceAccountName: pm-agent-worker
|
||||
# The kubelet's pull credential, the same one the dispatcher uses. Pinned by the
|
||||
# admission policy to exactly this name: `imagePullSecrets` is a pod-spec field, so a
|
||||
# Job could otherwise name any Secret in the namespace and have the kubelet read it
|
||||
# before a container ever starts. The image is digest-pinned, so the leverage is small
|
||||
# — but "only the committed shape" is the rule this template exists to express.
|
||||
imagePullSecrets:
|
||||
- name: pm-agent-registry
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
|
||||
@@ -111,6 +111,12 @@ spec:
|
||||
=(hostNetwork): "false"
|
||||
=(hostPID): "false"
|
||||
=(hostIPC): "false"
|
||||
# The kubelet reads this Secret BEFORE any container starts, so it is part
|
||||
# of the shape even though no container references it. A pattern on an array
|
||||
# asserts "at least one element matches", which is enough to REQUIRE the
|
||||
# committed entry; rule 7 is what forbids a second one.
|
||||
imagePullSecrets:
|
||||
- name: pm-agent-registry
|
||||
containers:
|
||||
- image: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:*"
|
||||
command:
|
||||
@@ -336,6 +342,54 @@ spec:
|
||||
# dispatcher itself uses envFrom (it is a Deployment, not a Job, and not matched by this
|
||||
# policy); a worker must not.
|
||||
# =====================================================================================
|
||||
# =====================================================================================
|
||||
# Rule 7 — exactly one imagePullSecret, and it is the committed one.
|
||||
#
|
||||
# `imagePullSecrets` is a pod-spec field that no container references, which is precisely
|
||||
# why it is easy to leave unread — the same shape of omission as `initContainers` and
|
||||
# `env` before it. The kubelet reads the named Secret, in this namespace, before any
|
||||
# container starts. A Job naming a different one would have the kubelet read a Secret the
|
||||
# worker was never meant to touch.
|
||||
#
|
||||
# The leverage is genuinely small: the image is digest-pinned, so a different credential
|
||||
# cannot change WHAT runs, and the only Secrets in this namespace are the agent's own.
|
||||
# It is pinned anyway, because "only the committed shape" is the whole premise of this
|
||||
# policy, and because the cost of the rule is four lines while the cost of rediscovering
|
||||
# the omission is an incident.
|
||||
#
|
||||
# Rule 1's pattern REQUIRES the committed entry; this forbids any OTHER entry and any
|
||||
# second entry. A foreach over an absent list yields no result, which is why both halves
|
||||
# are needed.
|
||||
# =====================================================================================
|
||||
- name: worker-pull-secret-must-be-the-committed-one
|
||||
match:
|
||||
all:
|
||||
- resources:
|
||||
kinds:
|
||||
- Job
|
||||
namespaces:
|
||||
- pm-agent
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: pm-agent
|
||||
namespace: pm-agent
|
||||
validate:
|
||||
message: >-
|
||||
A worker Job may declare exactly ONE imagePullSecret, named `pm-agent-registry`.
|
||||
The kubelet reads that Secret before any container starts, so naming another one
|
||||
reaches a credential the worker was never meant to touch.
|
||||
foreach:
|
||||
- list: "request.object.spec.template.spec.imagePullSecrets"
|
||||
deny:
|
||||
conditions:
|
||||
any:
|
||||
- key: "{{ length(request.object.spec.template.spec.imagePullSecrets) }}"
|
||||
operator: NotEquals
|
||||
value: 1
|
||||
- key: "{{ element.name }}"
|
||||
operator: AnyNotIn
|
||||
value: ["pm-agent-registry"]
|
||||
|
||||
- name: worker-may-not-use-envfrom
|
||||
match:
|
||||
all:
|
||||
|
||||
@@ -56,6 +56,11 @@ results:
|
||||
kind: Job
|
||||
resources: [good-run, real-template]
|
||||
result: pass
|
||||
- policy: pm-agent-job-shape
|
||||
rule: worker-pull-secret-must-be-the-committed-one
|
||||
kind: Job
|
||||
resources: [good-run, real-template]
|
||||
result: pass
|
||||
|
||||
# ---------------- THE escalation, and the shape rule's other refusals ----------------
|
||||
# `evil-command` is the case the whole model exists for: right namespace, right creator,
|
||||
@@ -73,6 +78,14 @@ results:
|
||||
- host-namespaces
|
||||
- restart-onfailure
|
||||
- second-container
|
||||
- no-pull-secret
|
||||
result: fail
|
||||
|
||||
# ---------------- the pull secret: a second one, or a different one ----------------
|
||||
- policy: pm-agent-job-shape
|
||||
rule: worker-pull-secret-must-be-the-committed-one
|
||||
kind: Job
|
||||
resources: [extra-pull-secret, wrong-pull-secret]
|
||||
result: fail
|
||||
|
||||
# ---------------- args, and the second container ----------------
|
||||
|
||||
@@ -31,6 +31,8 @@ spec:
|
||||
restartPolicy: Never
|
||||
automountServiceAccountToken: false
|
||||
serviceAccountName: pm-agent-worker
|
||||
imagePullSecrets:
|
||||
- name: pm-agent-registry
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
@@ -95,6 +97,8 @@ spec:
|
||||
restartPolicy: Never
|
||||
automountServiceAccountToken: false
|
||||
serviceAccountName: pm-agent-worker
|
||||
imagePullSecrets:
|
||||
- name: pm-agent-registry
|
||||
containers:
|
||||
- name: worker
|
||||
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
@@ -492,3 +496,75 @@ spec:
|
||||
readOnlyRootFilesystem: true
|
||||
envFrom:
|
||||
- secretRef: { name: pm-agent-credentials }
|
||||
---
|
||||
# A SECOND pull secret beside the committed one. containers[0] is perfectly compliant, and
|
||||
# rule 1's array pattern is satisfied by the first element — which is why the count is
|
||||
# asserted separately.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: extra-pull-secret
|
||||
namespace: pm-agent
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
automountServiceAccountToken: false
|
||||
serviceAccountName: pm-agent-worker
|
||||
imagePullSecrets:
|
||||
- name: pm-agent-registry
|
||||
- name: pm-agent-credentials
|
||||
containers:
|
||||
- name: worker
|
||||
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["/usr/local/bin/pm-worker"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
---
|
||||
# A DIFFERENT pull secret — the kubelet would read a Secret the worker was never meant to
|
||||
# touch, before any container starts.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: wrong-pull-secret
|
||||
namespace: pm-agent
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
automountServiceAccountToken: false
|
||||
serviceAccountName: pm-agent-worker
|
||||
imagePullSecrets:
|
||||
- name: pm-agent-credentials
|
||||
containers:
|
||||
- name: worker
|
||||
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["/usr/local/bin/pm-worker"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
---
|
||||
# NO pull secret at all. Rule 7's foreach yields nothing over an absent list, so this is
|
||||
# rule 1's pattern to refuse — which is why the requirement lives in both.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: no-pull-secret
|
||||
namespace: pm-agent
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
automountServiceAccountToken: false
|
||||
serviceAccountName: pm-agent-worker
|
||||
containers:
|
||||
- name: worker
|
||||
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
|
||||
command: ["/usr/local/bin/pm-worker"]
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
+40
-21
@@ -46,15 +46,17 @@ waive the sensitive-path human review on its own pull request.
|
||||
Both are deliberate, and both fail **visibly** rather than silently:
|
||||
|
||||
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
|
||||
`PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
|
||||
publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
|
||||
`PM_WORKER_IMAGE` in `configmap.yaml` carry the digest of the **first** publish (images
|
||||
run 340, `sha-523cd9118f95`), pinned once on 2026-10-30 so going live did not wait on
|
||||
Renovate's first run; every later bump is Renovate's. That digest predates the worker
|
||||
template's `imagePullSecrets` entry, so worker Jobs built from it are refused by
|
||||
admission — which is why `PM_DRY_RUN` starts `true` (see `configmap.yaml`). `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
|
||||
(`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
|
||||
`infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
|
||||
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
|
||||
Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
|
||||
lines and moves them together in one pull request. Nobody builds-and-pushes this image from
|
||||
a workstation, and nobody pastes a digest. Until the first publish, and until that PR
|
||||
merges: `ImagePullBackOff`. The one operator prerequisite on the publishing side is the
|
||||
a workstation. The one operator prerequisite on the publishing side is the
|
||||
`REGISTRY_PUSH_TOKEN` Actions secret (see the header of `.gitea/workflows/images.yaml`);
|
||||
without it the publish job fails loudly rather than skipping.
|
||||
|
||||
@@ -78,21 +80,36 @@ Both are deliberate, and both fail **visibly** rather than silently:
|
||||
inventory, never typed on a node. Apply it scoped to that one task:
|
||||
`cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
|
||||
--tags hosts`.
|
||||
- **The pull is still unauthenticated** once the node reaches the right endpoint. An
|
||||
unauthenticated client gets nothing from this Gitea's registry, and there is no
|
||||
`imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
|
||||
(local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea
|
||||
account whose token carries `read:package`, referenced from the Deployment and the
|
||||
committed worker Job template — not built yet, because minting that token is yours.
|
||||
- **The pull must be authenticated** once the node reaches the right endpoint: an
|
||||
unauthenticated client gets nothing from this registry. **Now wired** —
|
||||
`pm-agent-registry` is a `dockerconfigjson` Secret referenced by both the Deployment and
|
||||
the committed worker Job template, and the admission policy pins it to exactly that
|
||||
name. What is left to you is the value: the Gitea token must carry **`read:package`** in
|
||||
addition to `read:user`, `write:repository` and `write:issue`, and the dockerconfigjson is built from
|
||||
it. Steps:
|
||||
[`pm-agent-registry.sops.yaml.example`](./pm-agent-registry.sops.yaml.example).
|
||||
|
||||
The same two apply to `conformance-runner`: same registry path, same publish-then-Renovate
|
||||
model, never pulled either.
|
||||
2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
|
||||
repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
|
||||
which Gitea account to create, which scopes, and why it must *not* be an administrator —
|
||||
is in [`pm-agent-credentials.sops.yaml.example`](./pm-agent-credentials.sops.yaml.example).
|
||||
Until then the dispatcher CrashLoops with `GET /user returned no login — is GITEA_TOKEN
|
||||
valid?`.
|
||||
2. **The credentials are placeholders — two Secrets, one token.** Both are encrypted to the
|
||||
repository's age recipient and both hold `REPLACE-ME-…` strings:
|
||||
|
||||
| Secret | type | read by | hand-off |
|
||||
|---|---|---|---|
|
||||
| `pm-agent-credentials` | Opaque | the application, as env | [`.example`](./pm-agent-credentials.sops.yaml.example) |
|
||||
| `pm-agent-registry` | dockerconfigjson | the **kubelet**, before any container starts | [`.example`](./pm-agent-registry.sops.yaml.example) |
|
||||
|
||||
They hold the **same** Gitea token in two formats, because a Secret cannot be both — the
|
||||
same one-value-in-two-files pattern the Keycloak client secret already uses here. So mint
|
||||
it once, with all four scopes: `read:package` (the pull), `read:user` (the dispatcher's
|
||||
startup `GET /user` — Gitea 1.24.3 scopes the whole `/user` group), `write:repository`
|
||||
(clone, branch push, pull request) and `write:issue` (issue comments). The account also
|
||||
needs package *permission*, not just the scope: a `pm-agent-pull` org team with Packages
|
||||
= Read and no repositories (steps in the registry `.example`). The first `.example`
|
||||
explains which account to create and why it must *not* be an administrator.
|
||||
|
||||
Until they are filled in: the pull fails `401`, and if you fix only the pull, the
|
||||
dispatcher CrashLoops with `GET /user returned no login — is GITEA_TOKEN valid?`.
|
||||
|
||||
Its Flux Kustomization is `wait: false` precisely because of those two states: a workload
|
||||
that is *expected* to be unhealthy for a while must never be able to gate anything.
|
||||
@@ -155,13 +172,14 @@ Job from a committed template. **The model's output is data, never a command.**
|
||||
| **cannot** | clone from or push to anywhere but `gitea.gitea.svc.cluster.local:3000`: `pm-worker` validates `PM_GITEA_URL` against a one-entry allowlist baked into the image. In-cluster consumers reach a service by its cluster-local name — a different path by design from how a client outside the cluster reaches it — so the external hostname is not an accepted value |
|
||||
| **cannot** | authenticate a `git push` **during the model session**: `GIT_ASKPASS` is unset for its duration, so the restriction is structural rather than a request in a prompt |
|
||||
| **cannot** | be created in any shape but the committed one — see the admission policy |
|
||||
| **cannot** | have the kubelet read any Secret but `pm-agent-registry` on its behalf: `imagePullSecrets` is pinned to exactly that one name |
|
||||
|
||||
### What is *not* defended, stated plainly
|
||||
|
||||
- **The worker's own credentials are inside the model's reach.** The session runs as the same
|
||||
uid in the same container as the wrapper, so it can read the environment, including its
|
||||
`GITEA_TOKEN` and `CLAUDE_CODE_OAUTH_TOKEN`. There is no isolation between them and none is
|
||||
claimed. What bounds it: the Gitea token is scoped `write:repository, write:issue` on **one
|
||||
claimed. What bounds it: the Gitea token is scoped `read:user, write:repository, write:issue, read:package` on **one
|
||||
repository** with no admin permission, and egress reaches only this Gitea and tcp/443.
|
||||
- **`world:443` is wider than `api.anthropic.com`.** Cilium can express the tighter rule
|
||||
(`toFQDNs`) but nothing in this repository uses FQDN policy and its prerequisite (Cilium's
|
||||
@@ -200,14 +218,15 @@ Two things answer it, and neither is sufficient alone:
|
||||
`command` with no args, that powerless SA, no mounted token, `restartPolicy: Never`,
|
||||
non-root with no privilege escalation and a read-only root filesystem, only the
|
||||
`workspace`/`tmp` **emptyDir** volumes, no init or ephemeral containers, only the
|
||||
committed env **names**, and no `envFrom`.
|
||||
committed env **names**, no `envFrom`, and exactly one `imagePullSecret` — the committed
|
||||
one.
|
||||
|
||||
Six rules, because pinning `command` only moves the injection to whatever adjacent field
|
||||
Seven rules, because pinning `command` only moves the injection to whatever adjacent field
|
||||
nobody enumerated — on the sibling policy that was `initContainers` (measured: a Job with a
|
||||
compliant `containers[0]` and a privileged init container was **admitted**, pass 3 fail 0),
|
||||
then `ephemeralContainers`, then `env`, then `envFrom`. All of them are closed here from the
|
||||
start, and `platform/kyverno/tests/pm-agent/` proves each refusal offline: **28 assertions,
|
||||
28 passing** under the pinned `kyverno` CLI. The worker's own input guards have their own
|
||||
start, and `platform/kyverno/tests/pm-agent/` proves each refusal offline: **33 assertions,
|
||||
33 passing** under the pinned `kyverno` CLI. The worker's own input guards have their own
|
||||
offline probe — `infra/agent/probe-validation.sh`, **17 refusals** — because a guard nothing
|
||||
exercises is a claim rather than a control.
|
||||
|
||||
|
||||
@@ -51,15 +51,20 @@ data:
|
||||
# the dispatcher and its workers share the committed Job template and the env-name
|
||||
# contract. A tag here is refused at startup (and would be DENIED by admission anyway).
|
||||
#
|
||||
# PLACEHOLDER UNTIL THE IMAGE IS PUBLISHED. The all-zero digest below is deliberately
|
||||
# impossible. CI builds, verifies and — on main — publishes agent-runner
|
||||
# (.gitea/workflows/images.yaml), and Renovate replaces this value with the digest the
|
||||
# REGISTRY returned for the image's `main` tag, in a pull request that also moves the
|
||||
# Deployment's image (same marker, same branch). Nobody pushes this image from a workstation.
|
||||
# Until the first publish the dispatcher starts, fails to pull, and says so — which is the
|
||||
# visible failure, not a silent one.
|
||||
# THE FIRST PUBLISHED DIGEST. CI builds, verifies and — on main — publishes agent-runner
|
||||
# (.gitea/workflows/images.yaml). This value is images run 340's publish (tag
|
||||
# sha-523cd9118f95, which `main` pointed at), read from the registry on 2026-10-30. From here
|
||||
# on Renovate replaces it with the digest the REGISTRY returns for the `main` tag, in a pull
|
||||
# request that also moves the Deployment's image (same marker, same branch). Nobody pushes
|
||||
# this image from a workstation.
|
||||
#
|
||||
# READ BEFORE TURNING PM_DRY_RUN OFF: this digest PREDATES the worker template's
|
||||
# `imagePullSecrets` entry (PR #61). Its baked template has none, and admission now REQUIRES
|
||||
# it — so a worker Job built from this image is DENIED. The merge that adds the entry
|
||||
# changes infra/agent/**, CI re-publishes agent-runner, and Renovate's bump to that digest
|
||||
# is what makes worker Jobs admissible. Until then, dry run is the only mode that works.
|
||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
|
||||
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000"
|
||||
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42"
|
||||
|
||||
# --- the model -------------------------------------------------------------
|
||||
# The PM only reads a thread and answers with one JSON object, so it holds no tools and
|
||||
@@ -104,13 +109,21 @@ data:
|
||||
# work somebody deliberately handed it while trust is being established, with no code
|
||||
# change and no redeploy of anything but this ConfigMap. The operator's call; the default
|
||||
# here honours the request as made.
|
||||
PM_REQUIRE_LABEL: ""
|
||||
#
|
||||
# SET TO `agent` FOR THE FIRST RUN (2026-10-30, docs/plans/pm-agent-issue-to-pr.md
|
||||
# follow-up 3). The label does not exist in the repository yet: create it in Gitea
|
||||
# (Issues -> Labels -> New Label, name `agent`) before labelling a test issue.
|
||||
PM_REQUIRE_LABEL: "agent"
|
||||
|
||||
# Dry run: the PM still reads threads, still asks the model and still posts its decision as
|
||||
# an issue comment — so the operator can read exactly what it WOULD have done — but creates
|
||||
# no worker Job and so changes no code. The honest way to watch this service for a day
|
||||
# before letting it write anything.
|
||||
PM_DRY_RUN: "false"
|
||||
#
|
||||
# "true" FOR THE FIRST RUN, and not only for caution: the pinned digest above cannot yet
|
||||
# create an admissible worker Job (see the PM_WORKER_IMAGE comment). Flip to "false" in the
|
||||
# same pull request as — or after — Renovate's first agent-runner digest bump.
|
||||
PM_DRY_RUN: "true"
|
||||
|
||||
# If the gate refuses a worker's change, the branch is NOT pushed by default and the issue
|
||||
# gets the gate log instead. Set to "true" to push anyway and open a red pull request —
|
||||
|
||||
@@ -48,9 +48,13 @@ spec:
|
||||
# It is a hand-maintained marker rather than a computed hash because Flux's
|
||||
# Kustomization here does not run a configmap generator; bump it in the same commit
|
||||
# as a change to configmap.yaml.
|
||||
pm-agent.homelab.keithjackson.dev/config-generation: "1"
|
||||
pm-agent.homelab.keithjackson.dev/config-generation: "2"
|
||||
spec:
|
||||
serviceAccountName: pm-agent
|
||||
# The kubelet's credential for pulling this image. Not the agent's Gitea credential —
|
||||
# see pm-agent-registry.sops.yaml.example for why they are two Secrets of one token.
|
||||
imagePullSecrets:
|
||||
- name: pm-agent-registry
|
||||
# The dispatcher DOES need its token: `kubectl create job` is its one privileged act,
|
||||
# and platform/pm-agent/rbac.yaml is what bounds it to that. Contrast the worker, whose
|
||||
# Job template sets this false because it needs no API access at all.
|
||||
@@ -69,10 +73,12 @@ spec:
|
||||
# disagree about the shape the admission policy enforces. Replaced together, in ONE
|
||||
# Renovate pull request: both lines carry the same marker, so Renovate treats them as
|
||||
# one dependency and bumps them in the same branch. The digest comes from CI's publish
|
||||
# of agent-runner's `main` tag (.gitea/workflows/images.yaml) — the all-zero
|
||||
# placeholder below is impossible on purpose (see configmap.yaml PM_WORKER_IMAGE).
|
||||
# of agent-runner's `main` tag (.gitea/workflows/images.yaml). The value below is the
|
||||
# FIRST publish (images run 340, tag sha-523cd9118f95 = main at the time), read from
|
||||
# the registry's manifest endpoint on 2026-10-30 and pinned by hand once so the
|
||||
# dispatcher does not wait on Renovate's first run; every later bump is Renovate's.
|
||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
|
||||
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
|
||||
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:c4eba5b3c508bf456a03391ac76f8275e7ea28a829da1450d99f6f6f25c6eb42
|
||||
imagePullPolicy: IfNotPresent
|
||||
command:
|
||||
- /usr/local/bin/pm-dispatch
|
||||
|
||||
@@ -35,6 +35,12 @@ resources:
|
||||
# the distance between naming a file in a comment and listing it under `resources:` is one
|
||||
# careless edit.
|
||||
- pm-agent-credentials.sops.yaml
|
||||
# The KUBELET's pull credential for agent-runner — a separate Secret from the one above
|
||||
# because it is a different consumer in a different format: dockerconfigjson, read before
|
||||
# any container starts. Same underlying Gitea token, which must also carry `read:package`.
|
||||
# Measured 2026-10-05: this registry refuses unauthenticated clients outright, so without
|
||||
# this the pull 401s. Also placeholders until the operator fills it in.
|
||||
- pm-agent-registry.sops.yaml
|
||||
- deployment.yaml
|
||||
# default-deny baseline + the two egress allow policies (gitea, kube-apiserver for the
|
||||
# dispatcher only, tcp/443 for the model). NO ingress allow exists, by design.
|
||||
|
||||
@@ -60,11 +60,21 @@
|
||||
# Keycloak service-account token therefore cannot authenticate to Gitea's API or to
|
||||
# git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that.
|
||||
#
|
||||
# Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT
|
||||
# `write:admin`, NOT `write:organization`, NOT `write:user`.
|
||||
# Scopes, exactly these four (one token serves both Secrets — see
|
||||
# pm-agent-registry.sops.yaml.example):
|
||||
# read:user GET /user, the dispatcher's startup identity check. Gitea 1.24.3
|
||||
# puts the whole /api/v1/user group behind the `user` scope
|
||||
# category (routers/api/v1/api.go), so without it the dispatcher
|
||||
# CrashLoops on its first call even with an otherwise-valid token.
|
||||
# write:repository the clone, the branch push, the pull-request create, and the
|
||||
# dispatcher's reads of pulls/branches/commits.
|
||||
# write:issue reading open issues and posting issue comments.
|
||||
# read:package the kubelet's image pull (pm-agent-registry).
|
||||
# Nothing else — in particular NOT `write:admin`, NOT `write:organization`, NOT
|
||||
# `write:user`. (Until 2026-10-30 this list said two scopes; `read:user` was missing
|
||||
# and `read:package` lived only in the registry template.)
|
||||
#
|
||||
# `write:repository` covers the clone, the branch push and the pull-request create;
|
||||
# `write:issue` covers reading and posting issue comments. (The repository's OTHER CI
|
||||
# (The repository's OTHER CI
|
||||
# token, `SETTINGS_API_TOKEN`, is a separate credential with `read:admin` — do not
|
||||
# conflate them, and do not give this service that one. The 2026-10-02 measurement on
|
||||
# PR #38 recorded in infra/gitea/approval-policy.json is about that token, not this one.)
|
||||
@@ -94,6 +104,17 @@
|
||||
#
|
||||
# (4) COPY, FILL, ENCRYPT, COMMIT:
|
||||
#
|
||||
# `sops` IS IN THE PINNED TOOLBOX, NOT ON THE HOST (measured 2026-10-30: absent from the
|
||||
# workstation's PATH; sops 3.9.4 and vi in the toolbox). The simplest path needs no
|
||||
# plaintext file at all — edit the committed, already-encrypted file in place, from a
|
||||
# worktree, the way runbooks/vpn-remote-access.md does for its key:
|
||||
#
|
||||
# ./infra/devcontainer/dev.sh run sops platform/pm-agent/pm-agent-credentials.sops.yaml
|
||||
#
|
||||
# It opens vi on the decrypted document; replace the REPLACE-ME value(s), save, quit, and
|
||||
# sops re-encrypts on write. If you take the cp route below instead, run the `sops -e -i`
|
||||
# line through `./infra/devcontainer/dev.sh run` as well, and delete any plaintext temp file.
|
||||
#
|
||||
# cp platform/pm-agent/pm-agent-credentials.sops.yaml.example \
|
||||
# platform/pm-agent/pm-agent-credentials.sops.yaml
|
||||
# $EDITOR platform/pm-agent/pm-agent-credentials.sops.yaml # the two real values
|
||||
@@ -120,7 +141,7 @@ metadata:
|
||||
type: Opaque
|
||||
stringData:
|
||||
# Gitea token for the dedicated, NON-ADMIN `pm-agent` account.
|
||||
# Scopes: write:repository, write:issue.
|
||||
# Scopes: read:user, write:repository, write:issue, read:package.
|
||||
gitea-token: REPLACE-ME-gitea-token
|
||||
# Output of `claude setup-token` on a subscription account. One year's validity.
|
||||
claude-code-oauth-token: REPLACE-ME-claude-code-oauth-token
|
||||
@@ -0,0 +1,39 @@
|
||||
# PLACEHOLDER VALUES — the kubelet cannot pull agent-runner until the operator replaces them.
|
||||
# Full hand-off steps: pm-agent-registry.sops.yaml.example (same directory).
|
||||
# SOPS-encrypted to the repository's age recipient; keys stay cleartext for reviewable diffs.
|
||||
#
|
||||
# This is the KUBELET's pull credential, not the agent's Gitea credential. The agent's is
|
||||
# pm-agent-credentials (Opaque, read as env by the application); this one is
|
||||
# dockerconfigjson and is read before any container starts. Same underlying Gitea token,
|
||||
# which must carry read:package as well as write:repository and write:issue.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: pm-agent-registry
|
||||
namespace: pm-agent
|
||||
labels:
|
||||
app.kubernetes.io/name: pm-agent
|
||||
app.kubernetes.io/part-of: pm-agent
|
||||
type: kubernetes.io/dockerconfigjson
|
||||
stringData:
|
||||
.dockerconfigjson: ENC[AES256_GCM,data:4OJS0P/oTcaXYuqlrc/936l+5/ZSqA9dlNMPx8HkDX9zmQ4q1qIumYmlXYMbcVm2c2byeYt0II0mGVLVVNOBjb5jgDB4vjyhSLN4HhoYBxgB+//hNcYe+uobmrdJTBJLEbd+lXamf1vfsEUTmLL7I4VQ2l/mjJbvqu8p2YNw25Ska9iO9mkxCzjOTfRdzMD0Pn1Z09fjo8GpYVo=,iv:AEysZLEqrybR+3l9kReHEeTbVFApzKPb8kAkibEYerE=,tag:TgE7T1zasye1bsTZ2BTFrA==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvdG1lQm16V0tJdWZXRXlU
|
||||
Z2R4RXVyTXd6VTVJUWtkbUVjZldnb1JFOWhVCjVMMDdDb3AwVkt3TzJzdjR0VnNs
|
||||
SFlncWg2cWlFTy9IZ1YxSC9Zd3ZMaEkKLS0tIEZFQWhIMW5zMURWYmlXaFg1ZjhD
|
||||
cG1Fb0V3VncrTm9SQ091a0J0Z3RURWMKpdzNVub0zH5S8nnCSCt2H6NcCqDbV24L
|
||||
Mf8K2eIfu/ixn2NfwoVVj2QO+n+3/P6PLphdoiKMrL+ms0+JqWFpkQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-10-06T06:01:43Z"
|
||||
mac: ENC[AES256_GCM,data:zwmmMbM9Me7wE8EOvrKD6hEMLVHNCxI85FHvMjtRcHHm+mtdACjofDNfpynFVc1xFMMchcoOfipxEfAAJcwbH0FtfLeZq6M+XNHAf81qIk3NJxrS5Kgi/9FXZJAAwhqo/eced89Uozq8ntwt+mgxOuyjMds5z4h/VfN7/RmsWHg=,iv:VlvAEqozM9VV50vVx7r5aq3BdaaLLubhHJyHt8SSgS0=,tag:4ji7UqfRfm9P+qJrezFpmw==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -0,0 +1,118 @@
|
||||
# =============================================================================
|
||||
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (`.yaml.example` is ignored by kustomize and is
|
||||
# never auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
|
||||
#
|
||||
# WHAT THIS IS: the credential the KUBELET uses to pull `agent-runner` from the in-cluster
|
||||
# Gitea registry. It is NOT the credential the agent uses to talk to Gitea — that one is
|
||||
# `pm-agent-credentials` and it is read by the application. Two Secrets, because they are
|
||||
# consumed by two different things in two different formats:
|
||||
#
|
||||
# pm-agent-credentials Opaque read by pm-dispatch / pm-worker as env vars
|
||||
# pm-agent-registry dockerconfigjson read by the KUBELET, before any container runs
|
||||
#
|
||||
# WHY A PULL CREDENTIAL IS NEEDED AT ALL. Measured 2026-10-05: an unauthenticated client
|
||||
# gets nothing from this Gitea's registry — it issues an anonymous token whose `access`
|
||||
# claim is `null`, and every path answers 401 whether it exists or not
|
||||
# (`/api/v1/packages/Home-Network` replies `{"message":"token is required"}`). So a pull
|
||||
# with no credential cannot succeed, and before this Secret there was no `imagePullSecret`
|
||||
# anywhere in the tree. See decisions/2026-10-05-pm-agent-registry-pull-prereq.md.
|
||||
#
|
||||
# THE OTHER HALF, ALREADY FIXED: the nodes used to resolve this registry's hostname to the
|
||||
# non-hairpinned WAN address and fail TLS against the router's `CN=%D` placeholder
|
||||
# certificate. That is the `/etc/hosts` pin in
|
||||
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml
|
||||
# (decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md). Both halves are needed; a
|
||||
# pull fails on whichever is missing.
|
||||
#
|
||||
# =============================================================================
|
||||
# HAND-OFF STEPS (operator)
|
||||
# =============================================================================
|
||||
#
|
||||
# (1) USE THE SAME `pm-agent` ACCOUNT the agent already has (created per
|
||||
# pm-agent-credentials.sops.yaml.example — the dedicated, NON-ADMIN one). A second
|
||||
# account would be a second thing to keep non-admin for no gain: this token reads one
|
||||
# package and nothing else.
|
||||
#
|
||||
# (2) ADD THE PACKAGE SCOPE. Gitea token scopes are fixed at creation, so this means
|
||||
# generating a token with all four scopes and using it in BOTH Secrets:
|
||||
#
|
||||
# read:package <- NEW, what the kubelet needs to pull
|
||||
# read:user <- the dispatcher's startup GET /user (Gitea 1.24.3 scopes it)
|
||||
# write:repository <- the agent's clone / branch push / pull-request create
|
||||
# write:issue <- the agent's issue comments
|
||||
#
|
||||
# THE SCOPE IS NOT THE PERMISSION. `read:package` only lets the token use whatever
|
||||
# package access the ACCOUNT has, and these packages belong to the ORGANISATION
|
||||
# (`home-network/…`), not to a repository. Gitea 1.24.3 decides org-package access in
|
||||
# services/context/package.go `determineAccessMode`: the highest Packages-unit mode of
|
||||
# any org team the user is in, else READ if the org is visible to that user. Home-Network
|
||||
# is `public` today (measured 2026-10-30), so a signed-in non-member already gets read —
|
||||
# but that is a side effect of a visibility setting nobody chose for this. Make it
|
||||
# explicit, and independent of visibility:
|
||||
#
|
||||
# Organisation Home-Network -> Teams -> New Team
|
||||
# name: pm-agent-pull permission: none (unit-level)
|
||||
# repositories: NONE (do not tick "all repositories")
|
||||
# units: Packages = Read; every other unit = No Access
|
||||
# then add `pm-agent` as its only member.
|
||||
#
|
||||
# That grants package read on the org and nothing on any repository; the repo `Write`
|
||||
# comes only from the collaborator entry in the credentials template.
|
||||
#
|
||||
# Gitea -> (as pm-agent) Settings -> Applications -> Generate New Token.
|
||||
# Put the SAME value in `pm-agent-credentials` (`gitea-token`) and in the
|
||||
# `.dockerconfigjson` below. One token, two consumers, two formats — the same
|
||||
# one-value-in-two-files pattern the Keycloak client secret already uses here, and for
|
||||
# the same reason: a Secret cannot be shared across formats.
|
||||
#
|
||||
# (3) BUILD THE DOCKERCONFIGJSON. `auth` is base64 of `<username>:<token>` — NOT a hash,
|
||||
# just encoding, so treat this file exactly as secret as the token itself:
|
||||
#
|
||||
# USER=pm-agent
|
||||
# TOKEN='<the token from step 2>'
|
||||
# AUTH=$(printf '%s:%s' "$USER" "$TOKEN" | base64 -w0)
|
||||
# cat > /tmp/dockerconfig.json <<JSON
|
||||
# {"auths":{"gitea.home.keithjackson.dev":{"username":"$USER","password":"$TOKEN","auth":"$AUTH"}}}
|
||||
# JSON
|
||||
#
|
||||
# THE REGISTRY KEY MUST BE THE HOST AS THE IMAGE REFERENCE SPELLS IT —
|
||||
# `gitea.home.keithjackson.dev`, no scheme, no port, no path. The kubelet matches the
|
||||
# pull host against these keys literally; `https://…` or a trailing `/v2/` silently
|
||||
# matches nothing and the pull is attempted anonymously, which 401s exactly as if this
|
||||
# Secret were absent.
|
||||
#
|
||||
# (4) COPY, FILL, ENCRYPT, COMMIT:
|
||||
#
|
||||
# `sops` IS IN THE PINNED TOOLBOX, NOT ON THE HOST (measured 2026-10-30: absent from the
|
||||
# workstation's PATH; sops 3.9.4 and vi in the toolbox). The simplest path needs no
|
||||
# plaintext file at all — edit the committed, already-encrypted file in place, from a
|
||||
# worktree, the way runbooks/vpn-remote-access.md does for its key:
|
||||
#
|
||||
# ./infra/devcontainer/dev.sh run sops platform/pm-agent/pm-agent-registry.sops.yaml
|
||||
#
|
||||
# It opens vi on the decrypted document; replace the REPLACE-ME value(s), save, quit, and
|
||||
# sops re-encrypts on write. If you take the cp route below instead, run the `sops -e -i`
|
||||
# line through `./infra/devcontainer/dev.sh run` as well, and delete any plaintext temp file.
|
||||
#
|
||||
# cp platform/pm-agent/pm-agent-registry.sops.yaml.example \
|
||||
# platform/pm-agent/pm-agent-registry.sops.yaml
|
||||
# # paste the one-line JSON from step 3 as the .dockerconfigjson value
|
||||
# sops -e -i platform/pm-agent/pm-agent-registry.sops.yaml
|
||||
#
|
||||
# The root `.sops.yaml` rule matches this path and encrypts `stringData`, so the diff
|
||||
# stays reviewable. NEVER `sops -d` it into a terminal or a transcript (CLAUDE.md §6).
|
||||
# It is already listed under `resources:` in platform/pm-agent/kustomization.yaml.
|
||||
#
|
||||
# (5) ROTATION. Rotating the Gitea token means updating BOTH Secrets, because it is one
|
||||
# token. The dispatcher must then be restarted to re-read its env; worker Jobs and the
|
||||
# kubelet pick the new value up on their own.
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: pm-agent-registry
|
||||
namespace: pm-agent
|
||||
type: kubernetes.io/dockerconfigjson
|
||||
stringData:
|
||||
# One line. Registry key = the host exactly as the image reference spells it.
|
||||
.dockerconfigjson: '{"auths":{"gitea.home.keithjackson.dev":{"username":"pm-agent","password":"REPLACE-ME-the-same-gitea-token","auth":"REPLACE-ME-base64-of-username:token"}}}'
|
||||
+3
-4
@@ -29,11 +29,10 @@
|
||||
// `managerFilePatterns` key ("Custom Manager contains disallowed fields", "Invalid configuration
|
||||
// option: flux.managerFilePatterns", ...), because 39.x only knows `fileMatch`; the rename landed
|
||||
// in 40.2.0. So even after the workflow found this file, every run would have stopped at
|
||||
// CONFIG_VALIDATION. renovate.yaml now pins the exact release this file is validated against —
|
||||
// since 2026-10-30 as the job's `container.image` tag (the action is gone: it needed a Docker
|
||||
// daemon jobs here do not get). VALIDATE WITH THAT SAME TAG:
|
||||
// CONFIG_VALIDATION. renovate.yaml now pins `renovate-version` to the exact release this file is
|
||||
// validated against. VALIDATE WITH THAT SAME TAG:
|
||||
// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
|
||||
// ghcr.io/renovatebot/renovate:<container.image tag in renovate.yaml> \
|
||||
// ghcr.io/renovatebot/renovate:<renovate-version from renovate.yaml> \
|
||||
// renovate-config-validator /tmp/renovate.json5
|
||||
// (Not `--strict`: that also fails on a pending migration of the pre-existing top-level
|
||||
// `"docker": {"enabled": true}` block at the bottom, which is a warning, not an error.)
|
||||
|
||||
@@ -67,9 +67,11 @@ Work through these in order; the first three are by far the most likely.
|
||||
| Symptom in the log | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
|
||||
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been published; the committed digest is the deliberate placeholder | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
|
||||
| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
|
||||
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
||||
| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
|
||||
| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
|
||||
| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets |
|
||||
| a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
|
||||
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push |
|
||||
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
|
||||
| `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
|
||||
|
||||
Reference in new issue
Block a user