Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
dbdb215651 |
No files matched your search
@@ -115,59 +115,14 @@ CI. That is a real coverage gap, not a phase.
|
||||
| variable | `REQUIRE_CLUSTER_TESTS` | `'true'` (Phase 1+) makes unverified cluster-needing touched reqs a hard block |
|
||||
| secret | `CONFORMANCE_TRIGGER_KUBECONFIG` | narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) |
|
||||
|
||||
## How images are built
|
||||
|
||||
This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
|
||||
built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
|
||||
verified and published by [`images.yaml`](images.yaml), never from a workstation
|
||||
(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh).
|
||||
|
||||
**The builder is rootless BuildKit** ([`platform/buildkit/`](../../platform/buildkit/README.md)),
|
||||
reached over mTLS with a pinned `buildx` — no job here touches a Docker daemon, privileged or
|
||||
otherwise (operator decision on PR #81). The client certificate reaches jobs as the Actions
|
||||
secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT`, `BUILDKIT_CLIENT_KEY`.
|
||||
|
||||
**Verification is a build target.** Each Containerfile ends in a `verify` stage that runs every
|
||||
in-image assertion at build time as the non-root uid with `RUN --network=none` (tool presence and
|
||||
pinned versions, `PM_SELFTEST=1 pm-dispatch`, the pm-worker probe, the `conformance-clone`
|
||||
refusals, no writable path outside the workspace). The pipeline builds `<image>-verify` before it
|
||||
builds, let alone pushes, the image. Build args come from each image's own
|
||||
`build-image.sh --print-build-args`, which is also the local podman path — so the recipe and the
|
||||
proof exist once.
|
||||
|
||||
| Trigger | What happens |
|
||||
|---|---|
|
||||
| pull request touching an image's inputs | build + verify only. Nothing is pushed and **no registry credential is referenced** |
|
||||
| push to `main` touching an image's inputs | build + verify + publish to `gitea.home.keithjackson.dev/home-network/<image>` as `sha-<commit>` (immutable) and `main` (moving); the **registry-returned** digest is in the job summary |
|
||||
| `workflow_dispatch` | the same as a push when run on `main` (all images by default); verify-only elsewhere |
|
||||
|
||||
Only images whose inputs changed are built. The publish job **never writes to git**: Renovate
|
||||
follows each image's `main` tag through a `# renovate-digest:` marker next to every digest
|
||||
reference (`platform/pm-agent/`, `infra/agent/versions.env`, `conformance/versions.env`) and
|
||||
opens the bump PR, which goes through the gate like anything else. A PR opened with the Actions
|
||||
token would not trigger the required gate, which is why it is Renovate's job and not this one's.
|
||||
|
||||
Credential: the repository secret **`REGISTRY_PUSH_TOKEN`** (scope `write:package`, owned by an
|
||||
account with package write on the org). The automatic Actions token cannot push in Gitea 1.24.3
|
||||
— the reasoning, with source citations, is in the header of `images.yaml`. Retention:
|
||||
[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
|
||||
action, keeps every digest git pins plus the newest tags.
|
||||
|
||||
Cost: the job shares the single act_runner slot with the gate, but the build itself runs in the
|
||||
`buildkitd` pod, not in DinD, so the runner's disk and memory are untouched. Until
|
||||
`platform/buildkit` is deployed and its client certificate copied
|
||||
(`infra/gitea/buildkit-client-secret.sh`), the job fails loudly at its builder preflight.
|
||||
|
||||
## Files
|
||||
|
||||
```
|
||||
.gitea/workflows/
|
||||
merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision)
|
||||
images.yaml build/verify/publish this repo's container images (see above)
|
||||
scripts/
|
||||
install-tools.sh installs the pinned tools per stage (from ci/versions.env)
|
||||
decide.sh the R-TEST-03 merge decision over job results
|
||||
images.sh the image pipeline's driver (what to build, base, push, digest)
|
||||
README.md this file
|
||||
```
|
||||
|
||||
|
||||
@@ -1,185 +0,0 @@
|
||||
# Container images — built, verified and published by CI, never from a workstation.
|
||||
#
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# WHY THIS FILE EXISTS (operator rule, 2026-10-29)
|
||||
# ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
# "Container image creation, hosting and lifecycle must be part of CI/CD and the codebase, never a
|
||||
# one-off build/push from a workstation." Before this workflow, three images existed only as
|
||||
# workstation builds and NOTHING had ever been pushed to the Gitea registry: the Home-Network org's
|
||||
# package list was empty, platform/pm-agent/ pinned an all-zero digest and sat in ImagePullBackOff,
|
||||
# and conformance/versions.env carried an empty digest. Both build scripts said "publishing is an
|
||||
# operator action" — a step with no log, no recorded inputs and no verification anyone could read.
|
||||
#
|
||||
# WHAT IT DOES — the logic is in scripts/images.sh; this file only wires triggers and secrets.
|
||||
# pull_request -> build + VERIFY only. No push, and no registry credential is referenced.
|
||||
# push to main -> build + verify + PUBLISH to gitea.home.keithjackson.dev/home-network/<image>
|
||||
# dispatch -> same as a push when run on main; verify-only on any other ref.
|
||||
# Images: toolbox (infra/devcontainer), agent-runner (infra/agent, FROM toolbox) and
|
||||
# conformance-runner (conformance/incluster). Only images whose inputs changed are built.
|
||||
#
|
||||
# THE DIGEST LIFECYCLE — this workflow NEVER writes to git.
|
||||
# 1. publish pushes `sha-<commit>` (immutable) and `main` (moving), and prints the digest the
|
||||
# REGISTRY returns in the job summary.
|
||||
# 2. Renovate (renovate.yaml) follows the `main` tag of each image through the
|
||||
# `# renovate-digest:` markers next to every digest reference, and opens the bump PR.
|
||||
# 3. That PR goes through the gate like any other change, and Flux deploys it.
|
||||
# The bump PR is Renovate's and not this job's because a PR opened with the Actions token does not
|
||||
# trigger workflows, so it could never pass the REQUIRED gate check; Renovate's token is a real
|
||||
# user's.
|
||||
#
|
||||
# THE PUSH CREDENTIAL — `REGISTRY_PUSH_TOKEN`, a repository Actions secret, referenced BY NAME.
|
||||
# The automatic Actions token CANNOT be used. Gitea 1.24.3 source, verified:
|
||||
# * services/auth/basic.go (Basic.Verify) turns a task token into the pseudo-user
|
||||
# `gitea-actions` (ActionsUserID -2) and sets NO token scope;
|
||||
# * routers/api/packages/container/container.go (Authenticate, the /v2/token exchange every
|
||||
# docker client uses) then requires read:package / write:package / all scope and answers 401;
|
||||
# * services/context/package.go (determineAccessMode) has `// TODO: ActionUser permission check`
|
||||
# and no branch for that user, so even past login it would get at most READ on a public org.
|
||||
# So: a personal access token with scope `write:package`, minted by an account that has package
|
||||
# WRITE on the Home-Network org (org owner, or a team with the Packages unit at write). A dedicated
|
||||
# bot account in a packages-only team is the least-privilege choice. The username sent with it is
|
||||
# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
|
||||
# is optional and only labels the login.
|
||||
#
|
||||
# THE BUILDER — rootless BuildKit (operator decision on PR #81, 2026-10-29).
|
||||
# The act runner's job containers get NO Docker daemon (act_runner 0.2.11 forces docker_host "-"
|
||||
# for a tcp:// DinD, and gitea/act strips every bind when valid_volumes is empty — measured on
|
||||
# this PR's first run), and handing them the privileged DinD daemon would be root on the node for
|
||||
# anyone who can push a branch. So images are built by platform/buildkit/: a ROOTLESS buildkitd
|
||||
# (not privileged; uid 1000 in a user namespace), reached over mTLS at
|
||||
# buildkitd.buildkit.svc.cluster.local:1234 with a pinned `buildx` (remote driver). Its client
|
||||
# certificate reaches jobs as three Actions secrets — BUILDKIT_CA, BUILDKIT_CLIENT_CERT,
|
||||
# BUILDKIT_CLIENT_KEY — copied from cert-manager by infra/gitea/buildkit-client-secret.sh.
|
||||
# Pull requests get the client certificate (they must build) but never a registry credential.
|
||||
#
|
||||
# UNTIL platform/buildkit IS DEPLOYED (Flux applies it only after merge) THIS JOB FAILS, LOUDLY,
|
||||
# at its builder preflight. A skipped Gitea job reports `success`, so it is red on purpose.
|
||||
#
|
||||
# VERIFICATION is each Containerfile's `verify` stage, built before anything is pushed.
|
||||
#
|
||||
name: images
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main]
|
||||
paths:
|
||||
- infra/devcontainer/Containerfile
|
||||
- infra/devcontainer/versions.env
|
||||
- infra/devcontainer/build-image.sh
|
||||
- infra/agent/**
|
||||
- conformance/incluster/Containerfile
|
||||
- conformance/incluster/conformance-clone
|
||||
- conformance/incluster/build-image.sh
|
||||
- .gitea/workflows/images.yaml
|
||||
- .gitea/workflows/scripts/images.sh
|
||||
- platform/buildkit/versions.env
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- infra/devcontainer/Containerfile
|
||||
- infra/devcontainer/versions.env
|
||||
- infra/devcontainer/build-image.sh
|
||||
- infra/agent/**
|
||||
- conformance/incluster/Containerfile
|
||||
- conformance/incluster/conformance-clone
|
||||
- conformance/incluster/build-image.sh
|
||||
- .gitea/workflows/images.yaml
|
||||
- .gitea/workflows/scripts/images.sh
|
||||
- platform/buildkit/versions.env
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
images:
|
||||
description: "Which images: 'all', or a space-separated subset of: toolbox agent-runner conformance-runner"
|
||||
required: false
|
||||
default: "all"
|
||||
|
||||
concurrency:
|
||||
# Never cancel a publish half-way: a pushed `sha-` tag with no matching `main` move is
|
||||
# recoverable (a rerun skips what is already published) but pointless to provoke.
|
||||
group: images-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
images:
|
||||
runs-on: ubuntu-latest
|
||||
# Measured on the PR that introduced this file — see its decisions entry. Generous because a
|
||||
# cold DinD store downloads the Fedora base and every pinned tool on each run.
|
||||
timeout-minutes: 75
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0 # the per-image change detection diffs against the base commit
|
||||
|
||||
# ── pull requests: build + verify, NO credential ───────────────────────────────────────
|
||||
- name: Build and verify (pull request — nothing is pushed)
|
||||
if: github.event_name == 'pull_request'
|
||||
env:
|
||||
MODE: verify
|
||||
IMAGES: auto
|
||||
PR_BASE_REF: ${{ github.base_ref }}
|
||||
# The BUILDER's client certificate (platform/buildkit). Not a registry credential:
|
||||
# a pull request can build, never push.
|
||||
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
|
||||
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
|
||||
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
|
||||
export BASE_SHA
|
||||
./.gitea/workflows/scripts/images.sh
|
||||
|
||||
# ── non-main dispatch: verify only ──────────────────────────────────────────────────────
|
||||
- name: Build and verify (dispatch off main — nothing is pushed)
|
||||
if: github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main'
|
||||
env:
|
||||
MODE: verify
|
||||
IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
|
||||
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
|
||||
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
|
||||
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
|
||||
run: ./.gitea/workflows/scripts/images.sh
|
||||
|
||||
# ── main: publish ───────────────────────────────────────────────────────────────────────
|
||||
- name: Preflight - the push token must exist, and its absence must be LOUD
|
||||
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
|
||||
env:
|
||||
REGISTRY_PUSH_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${REGISTRY_PUSH_TOKEN:-}" ]; then
|
||||
echo "::error::REGISTRY_PUSH_TOKEN is not set on this repository." >&2
|
||||
cat >&2 <<'MSG'
|
||||
|
||||
Images cannot be published without a registry credential, and this step fails rather
|
||||
than skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would
|
||||
make "nothing was published" look exactly like "published". That is how this repo went
|
||||
a month with images that existed only on one workstation.
|
||||
|
||||
WHY NOT THE BUILT-IN ACTIONS TOKEN: Gitea 1.24.3 gives it no package scope at the
|
||||
registry's /v2/token exchange (routers/api/packages/container/container.go,
|
||||
Authenticate) — see the header of .gitea/workflows/images.yaml.
|
||||
|
||||
TO FIX: as an account with package WRITE on the Home-Network org (ideally a dedicated
|
||||
bot in a packages-only team), create an access token with scope `write:package` and
|
||||
nothing else, then add it to this repository as the Actions secret
|
||||
REGISTRY_PUSH_TOKEN. Optionally set the Actions variable REGISTRY_PUSH_USER to that
|
||||
account's name. Do not commit the token anywhere. Then re-run this workflow
|
||||
(workflow_dispatch on main, images: all).
|
||||
MSG
|
||||
exit 1
|
||||
fi
|
||||
echo "REGISTRY_PUSH_TOKEN is present (value not printed)."
|
||||
|
||||
- name: Build, verify and publish (main)
|
||||
if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
|
||||
env:
|
||||
MODE: publish
|
||||
IMAGES: ${{ github.event_name == 'workflow_dispatch' && (inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images) || 'auto' }}
|
||||
BASE_SHA: ${{ github.event.before }}
|
||||
REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
|
||||
REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
|
||||
BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
|
||||
BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
|
||||
BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
|
||||
run: ./.gitea/workflows/scripts/images.sh
|
||||
@@ -60,9 +60,8 @@
|
||||
# entry's "What I did NOT verify" — that gap is stated, not papered over.
|
||||
#
|
||||
# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
|
||||
# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository`,
|
||||
# `read:user` and (since 2026-10-29) `read:package`. Set it as an Actions secret on this
|
||||
# repository. Nothing in this repo mints,
|
||||
# `RENOVATE_TOKEN` — a Gitea access token for the bot account, scopes `write:repository` and
|
||||
# `read:user`. Set it as an Actions secret on this repository. Nothing in this repo mints,
|
||||
# stores or references its value; it is consumed by name only.
|
||||
# Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
|
||||
# visible red is the correct state for "automation configured but not credentialed".
|
||||
@@ -75,26 +74,12 @@ name: renovate
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# 08:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` with
|
||||
# `"timezone": "America/New_York"`, and THAT inner schedule is authoritative — Renovate re-checks
|
||||
# it and DECLINES TO OPEN PRs outside it. So this cron must fire INSIDE the local window, not
|
||||
# merely near it.
|
||||
#
|
||||
# WAS `15 3 * * *`, WHICH NEVER WOULD HAVE WORKED (fixed 2026-10-18, before the first scheduled
|
||||
# run). Gitea Actions cron is UTC, so 03:15 UTC is 23:15 the PREVIOUS EVENING in New York --
|
||||
# 3h45m BEFORE the window opens, not "slightly ahead" of it as the original comment claimed.
|
||||
# Every nightly run would have started, logged "not within schedule", opened zero PRs and
|
||||
# exited SUCCESSFULLY. A green tick forever, and no updates ever: the exact "failure mode is a
|
||||
# green tick" shape this repo keeps getting burned by, and indistinguishable from "nothing
|
||||
# needed updating".
|
||||
#
|
||||
# THE WINDOW IN UTC MOVES WITH US DST, and a cron cannot follow it:
|
||||
# EDT (UTC-4, Mar-Nov): local 03:00-06:00 == 07:00-10:00 UTC
|
||||
# EST (UTC-5, Nov-Mar): local 03:00-06:00 == 08:00-11:00 UTC
|
||||
# 08:15 UTC is the only quarter-hour comfortably inside BOTH (04:15 EDT / 03:15 EST), so the
|
||||
# schedule keeps working across both switchovers without an edit. Do not "tidy" this back to an
|
||||
# early-morning-UTC value without redoing this arithmetic.
|
||||
- cron: "15 8 * * *"
|
||||
# 03:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` in
|
||||
# America/New_York, and that inner schedule is the authoritative one — Renovate re-checks it and
|
||||
# will decline to open PRs outside it. The two are deliberately both present: this cron decides
|
||||
# when the process RUNS, renovate.json decides when it may ACT. Keeping the cron slightly ahead
|
||||
# of the repo window means a run is always in flight when the window opens.
|
||||
- cron: "15 3 * * *"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
dry_run:
|
||||
@@ -140,9 +125,8 @@ jobs:
|
||||
seven files claimed "Renovate maintains this pin".
|
||||
|
||||
TO FIX: create a Gitea access token for the bot account with scopes
|
||||
`write:repository`, `read:user` and `read:package` (the last one lets Renovate read
|
||||
this repo's own images in the Gitea registry), then add it to this repository as the
|
||||
Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
|
||||
`write:repository` and `read:user`, then add it to this repository as the Actions
|
||||
secret RENOVATE_TOKEN. Do not commit it anywhere.
|
||||
MSG
|
||||
exit 1
|
||||
fi
|
||||
@@ -154,17 +138,8 @@ jobs:
|
||||
# Pinned, and Renovate maintains this pin itself: renovate.json's enabledManagers
|
||||
# includes `github-actions`, and `helpers:pinGitHubActionDigests` is in its `extends`.
|
||||
# This is the one dependency in the repo whose updater is its own updater.
|
||||
# MUST match the file's real name. #79 renamed it to renovate.json5 and left this line
|
||||
# saying `renovate.json`; every scheduled run from 2026-10-18 then died in ~1s with
|
||||
# ENOENT before Renovate started, and nothing in the gate noticed. ci/lint.sh now
|
||||
# asserts that every `configurationFile:` here names a file that exists.
|
||||
configurationFile: renovate.json5
|
||||
configurationFile: renovate.json
|
||||
token: ${{ secrets.RENOVATE_TOKEN }}
|
||||
# PINNED 2026-10-29. Unset, this action version runs Renovate `39` (action.yml
|
||||
# `renovate-version` default at v41.0.9), i.e. 39.264.0, and renovate.json5 uses
|
||||
# `managerFilePatterns` (>= 40.2.0): under 39.x validation fails with five errors and
|
||||
# the run stops at CONFIG_VALIDATION. #79 validated against `latest`, not this.
|
||||
renovate-version: 44.167.0
|
||||
env:
|
||||
# Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
|
||||
# header said were "set at runtime by the operator's renovate runner" — this is that
|
||||
@@ -181,14 +156,6 @@ jobs:
|
||||
RENOVATE_REQUIRE_CONFIG: "required"
|
||||
RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
|
||||
LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
|
||||
# ADDED 2026-10-29: registry read access for this repo's OWN images (toolbox,
|
||||
# agent-runner, conformance-runner — published by .gitea/workflows/images.yaml and
|
||||
# followed through `# renovate-digest:` markers). The Gitea registry answers 401 to
|
||||
# anonymous reads, and Renovate scopes the platform token to hostType `gitea`, which a
|
||||
# docker lookup never matches — so without this rule every digest lookup fails. Same
|
||||
# token; it therefore needs scope `read:package` in addition to the ones listed in the
|
||||
# preflight above. Gitea ignores the username for token auth (services/auth/basic.go).
|
||||
RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]'
|
||||
|
||||
- name: Summary
|
||||
if: always()
|
||||
|
||||
@@ -1,373 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# .gitea/workflows/scripts/images.sh — the image pipeline's driver (images.yaml calls this).
|
||||
#
|
||||
# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
|
||||
#
|
||||
# toolbox infra/devcontainer/ the pinned dev/gate toolchain
|
||||
# agent-runner infra/agent/ FROM toolbox; the PM dispatcher + worker
|
||||
# conformance-runner conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
|
||||
#
|
||||
# THE BUILDER (operator decision on PR #81): a ROOTLESS BuildKit daemon, platform/buildkit/,
|
||||
# reached over mTLS. No CI job may reach a privileged daemon, so there is no `docker build` and no
|
||||
# `docker run` here: this script drives buildkitd with a pinned `buildx` (remote driver) and
|
||||
# nothing else. The client certificate arrives as Actions secrets (BUILDKIT_CA,
|
||||
# BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY; infra/gitea/buildkit-client-secret.sh puts them
|
||||
# there). If the daemon is unreachable this FAILS LOUDLY — it never skips.
|
||||
#
|
||||
# VERIFICATION IS A BUILD TARGET. Each Containerfile ends in a `verify` stage that runs every
|
||||
# in-image assertion at build time (as the non-root uid, with RUN --network=none); this script
|
||||
# builds `<image>-verify` and only then the image. Build args come from each image's own
|
||||
# `build-image.sh --print-build-args`, the same list the local podman path uses.
|
||||
#
|
||||
# MODES (env MODE):
|
||||
# verify build the verify targets, never push, hold no registry credential. Pull requests.
|
||||
# publish verify, then build + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
|
||||
# `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
|
||||
# it). The digest recorded is the one the REGISTRY returns, read back after the push.
|
||||
#
|
||||
# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
|
||||
# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
|
||||
# file, platform/buildkit/versions.env) builds everything in verify mode, and nothing extra in
|
||||
# publish mode: re-publishing unchanged inputs would only mint new digests and churn Renovate PRs.
|
||||
#
|
||||
# THE agent-runner BASE:
|
||||
# * verify mode -> FROM the toolbox built in this run, linked with a bake `target:` context —
|
||||
# a pull request holds no registry credential and that toolbox exists in no registry. If the
|
||||
# toolbox's inputs did not change it is built anyway, as the base.
|
||||
# * publish mode, toolbox published in this run -> FROM the digest the registry just returned.
|
||||
# * publish mode otherwise -> AGENT_BASE_IMAGE from infra/agent/versions.env (the published
|
||||
# toolbox digest Renovate keeps current). If that is still the all-zero placeholder the
|
||||
# toolbox has never been published, so it is built and published first (the bootstrap).
|
||||
#
|
||||
# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
|
||||
# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
|
||||
# infra/agent/versions.env (agent-runner input -> ONE more agent-runner build, whose own bump
|
||||
# lands in platform/pm-agent/ and stops). The worker Job template is deliberately NOT tracked:
|
||||
# it is baked INTO agent-runner, so bumping it would rebuild agent-runner, mint a new digest,
|
||||
# bump it again, forever. pm-dispatch overwrites that field at runtime anyway.
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
||||
cd "${REPO_ROOT}"
|
||||
|
||||
# shellcheck disable=SC1091
|
||||
BK_HOST_OVERRIDE="${BUILDKIT_HOST:-}"; BK_NAME_OVERRIDE="${BUILDKIT_SERVERNAME:-}"
|
||||
. ./platform/buildkit/versions.env
|
||||
BUILDKIT_HOST="${BK_HOST_OVERRIDE:-${BUILDKIT_HOST}}"
|
||||
BUILDKIT_SERVERNAME="${BK_NAME_OVERRIDE:-${BUILDKIT_SERVERNAME}}"
|
||||
|
||||
MODE="${MODE:-verify}"
|
||||
IMAGES="${IMAGES:-auto}"
|
||||
REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
|
||||
NAMESPACE="${NAMESPACE:-home-network}"
|
||||
HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
|
||||
BASE_SHA="${BASE_SHA:-}"
|
||||
SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"
|
||||
ZERO_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
|
||||
|
||||
ALL_IMAGES=(toolbox agent-runner conformance-runner)
|
||||
SHA_TAG="sha-${HEAD_SHA:0:12}"
|
||||
MOVING_TAG="main"
|
||||
|
||||
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
info() { printf ' %s\n' "$*"; }
|
||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
|
||||
yn() { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
|
||||
|
||||
case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
|
||||
for t in git jq curl sha256sum; do command -v "$t" >/dev/null || die "the job image lacks '$t'"; done
|
||||
|
||||
# --- per-image facts ------------------------------------------------------------------------
|
||||
# Inputs: a change under any of these rebuilds that image. conformance-runner takes its pins from
|
||||
# infra/devcontainer/versions.env. conformance/versions.env is NOT an input (it only RECORDS the
|
||||
# published digest), and must not become one: Renovate writes it after every publish -> a loop.
|
||||
inputs() {
|
||||
case "$1" in
|
||||
toolbox) echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
|
||||
agent-runner) echo "infra/agent/" ;;
|
||||
conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
|
||||
esac
|
||||
}
|
||||
context_of() { case "$1" in toolbox) echo infra/devcontainer ;; agent-runner) echo infra/agent ;; conformance-runner) echo conformance/incluster ;; esac; }
|
||||
PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh platform/buildkit/versions.env"
|
||||
|
||||
declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
|
||||
|
||||
changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
|
||||
[ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
|
||||
}
|
||||
|
||||
select_images() {
|
||||
local i
|
||||
if [ "${IMAGES}" != "auto" ]; then
|
||||
for i in ${IMAGES}; do
|
||||
case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
|
||||
done
|
||||
return
|
||||
fi
|
||||
if [ -z "${BASE_SHA}" ] || [[ "${BASE_SHA}" =~ ^0+$ ]] || ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
|
||||
info "no usable diff base ('${BASE_SHA}'): building everything"
|
||||
for i in "${ALL_IMAGES[@]}"; do WANT[$i]=1; NOTE[$i]="no diff base"; done
|
||||
return
|
||||
fi
|
||||
info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
|
||||
for i in "${ALL_IMAGES[@]}"; do
|
||||
# shellcheck disable=SC2046
|
||||
if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
|
||||
done
|
||||
# shellcheck disable=SC2086
|
||||
if [ "${MODE}" = verify ] && changed_any ${PIPELINE_FILES}; then
|
||||
for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || { WANT[$i]=1; NOTE[$i]="pipeline changed"; }; done
|
||||
fi
|
||||
}
|
||||
|
||||
# --- the builder: pinned buildx + mTLS to rootless buildkitd ---------------------------------
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "${WORK}"' EXIT # client key + registry auth live only here, only this long
|
||||
export BUILDX_CONFIG="${WORK}/buildx" DOCKER_CONFIG="${WORK}/docker"
|
||||
# Single-manifest images: the admission policies, Renovate and registry-retention.sh all reason
|
||||
# about one manifest per tag. (Provenance attestations would turn every push into an index.)
|
||||
export BUILDX_NO_DEFAULT_ATTESTATIONS=1
|
||||
mkdir -p "${BUILDX_CONFIG}" "${DOCKER_CONFIG}" "${WORK}/certs"
|
||||
echo '{}' > "${DOCKER_CONFIG}/config.json"
|
||||
|
||||
setup_builder() {
|
||||
log "Builder: rootless buildkitd at ${BUILDKIT_HOST} (mTLS)"
|
||||
BUILDX="${BUILDX_BIN:-}"
|
||||
if [ -z "${BUILDX}" ]; then
|
||||
BUILDX="${WORK}/buildx-bin"
|
||||
curl -fsSL --retry 3 -o "${BUILDX}" \
|
||||
"https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64" \
|
||||
|| die "could not download buildx v${BUILDX_VERSION}"
|
||||
echo "${BUILDX_SHA256} ${BUILDX}" | sha256sum -c --quiet - \
|
||||
|| die "buildx v${BUILDX_VERSION} does not match the sha256 pinned in platform/buildkit/versions.env — refusing to run it"
|
||||
chmod +x "${BUILDX}"
|
||||
fi
|
||||
info "client: $("${BUILDX}" version)"
|
||||
|
||||
local missing=()
|
||||
[ -n "${BUILDKIT_CA:-}" ] || missing+=(BUILDKIT_CA)
|
||||
[ -n "${BUILDKIT_CLIENT_CERT:-}" ] || missing+=(BUILDKIT_CLIENT_CERT)
|
||||
[ -n "${BUILDKIT_CLIENT_KEY:-}" ] || missing+=(BUILDKIT_CLIENT_KEY)
|
||||
[ "${#missing[@]}" -eq 0 ] || die "the builder's client certificate is missing: ${missing[*]} not set.
|
||||
These Actions secrets are copied from the cert-manager Secret buildkit/buildkit-client-ci by
|
||||
infra/gitea/buildkit-client-secret.sh (an operator action, after platform/buildkit is
|
||||
deployed). Without them nothing can be built — this fails rather than skipping ON PURPOSE."
|
||||
umask 077
|
||||
printf '%s\n' "${BUILDKIT_CA}" > "${WORK}/certs/ca.pem"
|
||||
printf '%s\n' "${BUILDKIT_CLIENT_CERT}" > "${WORK}/certs/cert.pem"
|
||||
printf '%s\n' "${BUILDKIT_CLIENT_KEY}" > "${WORK}/certs/key.pem"
|
||||
umask 022
|
||||
|
||||
"${BUILDX}" create --name ci-remote --driver remote \
|
||||
--driver-opt "cacert=${WORK}/certs/ca.pem,cert=${WORK}/certs/cert.pem,key=${WORK}/certs/key.pem,servername=${BUILDKIT_SERVERNAME}" \
|
||||
"${BUILDKIT_HOST}" >/dev/null
|
||||
if ! out="$(timeout 90 "${BUILDX}" inspect --bootstrap --builder ci-remote 2>&1)"; then
|
||||
printf '%s\n' "${out}" >&2
|
||||
die "buildkitd is UNREACHABLE at ${BUILDKIT_HOST} — nothing can be built, and this job fails
|
||||
rather than skipping. In order of likelihood:
|
||||
1. platform/buildkit is not deployed yet. Flux applies it only after it merges to main
|
||||
(Kustomization 'buildkit'); a pull request that ADDS it cannot reach it.
|
||||
2. the BUILDKIT_* secrets are stale (the client cert was renewed) or from another CA:
|
||||
re-run infra/gitea/buildkit-client-secret.sh.
|
||||
3. the NetworkPolicy (platform/buildkit/networkpolicy.yaml) does not admit this runner."
|
||||
fi
|
||||
printf '%s\n' "${out}" | grep -E '^(Name|Driver|Status|BuildKit version|Platforms):' | sed 's/^/ /' || true
|
||||
}
|
||||
|
||||
# KEY=VALUE lines from an image's build-image.sh --print-build-args -> a JSON object.
|
||||
build_args_json() {
|
||||
local ctx; ctx="$(context_of "$1")"
|
||||
"./${ctx}/build-image.sh" --print-build-args | jq -Rn '[inputs | select(length>0) | capture("^(?<k>[^=]+)=(?<v>.*)$")] | map({(.k): .v}) | add'
|
||||
}
|
||||
|
||||
# bake_run <bakefile> <targets...>
|
||||
bake_run() {
|
||||
local f="$1"; shift
|
||||
"${BUILDX}" bake --builder ci-remote --progress plain -f "$f" \
|
||||
--metadata-file "${WORK}/meta.json" "$@"
|
||||
}
|
||||
|
||||
# Write a bake file with a verify target and a ship target for one image.
|
||||
# $1 image, $2 agent base mode: "" | "link" | <image ref>
|
||||
write_bake() {
|
||||
local img="$1" base="${2:-}" ctx args f
|
||||
ctx="$(context_of "$img")"
|
||||
args="$(build_args_json "$img")"
|
||||
f="${WORK}/bake-${img}.json"
|
||||
local contexts='{}'
|
||||
if [ "${img}" = agent-runner ]; then
|
||||
if [ "${base}" = link ]; then
|
||||
# FROM ${AGENT_BASE_IMAGE} resolves to the named context below, i.e. the toolbox target
|
||||
# of this same bake file — built here, never pushed anywhere.
|
||||
args="$(jq -c '. + {AGENT_BASE_IMAGE: "toolbox-in-this-run"}' <<<"${args}")"
|
||||
contexts='{"toolbox-in-this-run": "target:toolbox"}'
|
||||
else
|
||||
args="$(jq -c --arg b "${base}" '. + {AGENT_BASE_IMAGE: $b}' <<<"${args}")"
|
||||
fi
|
||||
fi
|
||||
jq -n --arg img "$img" --arg ctx "$ctx" --argjson args "$args" --argjson contexts "$contexts" \
|
||||
--arg tbargs "$( [ "${base}" = link ] && build_args_json toolbox || echo '{}')" \
|
||||
--arg rev "${HEAD_SHA}" --arg repo "${REGISTRY}/${NAMESPACE}/${img}" \
|
||||
--arg sha "${SHA_TAG}" --arg moving "${MOVING_TAG}" --arg mode "${MODE}" '
|
||||
{target: (
|
||||
{
|
||||
($img + "-verify"): {context: $ctx, dockerfile: "Containerfile", target: "verify",
|
||||
args: $args, contexts: $contexts, output: ["type=cacheonly"]},
|
||||
($img): ({context: $ctx, dockerfile: "Containerfile", target: $img, args: $args,
|
||||
contexts: $contexts,
|
||||
labels: {"org.opencontainers.image.revision": $rev}}
|
||||
+ (if $mode == "publish"
|
||||
then {tags: [($repo + ":" + $sha), ($repo + ":" + $moving)], output: ["type=image,push=true"]}
|
||||
else {output: ["type=cacheonly"]} end))
|
||||
}
|
||||
+ (if ($contexts | has("toolbox-in-this-run"))
|
||||
then {toolbox: {context: "infra/devcontainer", dockerfile: "Containerfile", target: "toolbox",
|
||||
args: ($tbargs | fromjson), output: ["type=cacheonly"]}}
|
||||
else {} end)
|
||||
)}' > "$f"
|
||||
echo "$f"
|
||||
}
|
||||
|
||||
# --- registry helpers (publish mode only) ----------------------------------------------------
|
||||
REG_TOKEN=""
|
||||
registry_token() {
|
||||
[ -n "${REG_TOKEN}" ] && return 0
|
||||
REG_TOKEN="$(curl -fsS --config <(printf 'user = "%s:%s"\n' "${REGISTRY_USER}" "${REGISTRY_TOKEN}") \
|
||||
"https://${REGISTRY}/v2/token?service=container_registry&scope=*" | jq -r '.token // empty')" \
|
||||
|| die "could not obtain a registry bearer token from https://${REGISTRY}/v2/token"
|
||||
[ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
|
||||
}
|
||||
# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
|
||||
# Any other answer is fatal: "cannot tell" must never be read as "not published", or the
|
||||
# immutable-tag guard would be bypassed by an auth hiccup.
|
||||
registry_digest() {
|
||||
registry_token
|
||||
local hdrs code digest
|
||||
hdrs="$(curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "${REG_TOKEN}") \
|
||||
-H 'Accept: application/vnd.oci.image.manifest.v1+json' \
|
||||
-H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
|
||||
-H 'Accept: application/vnd.oci.image.index.v1+json' \
|
||||
-H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
|
||||
"https://${REGISTRY}/v2/${NAMESPACE}/$1/manifests/$2" | tr -d '\r')" \
|
||||
|| die "registry unreachable while reading $1:$2"
|
||||
code="$(awk 'NR==1{print $2}' <<<"${hdrs}")"
|
||||
digest="$(awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' <<<"${hdrs}" | tail -1)"
|
||||
case "${code}" in
|
||||
200) [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] || die "registry answered 200 for $1:$2 with no usable digest"
|
||||
echo "${digest}" ;;
|
||||
404) ;;
|
||||
*) die "registry answered HTTP ${code} for $1:$2 — cannot tell whether it exists" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# build_one <image> [agent base] — verify, then (publish) build + push + read back.
|
||||
build_one() {
|
||||
local img="$1" base="${2:-}" t0 f pushed got existing
|
||||
t0=$(date +%s)
|
||||
f="$(write_bake "$img" "$base")"
|
||||
log "VERIFY ${img} (Containerfile target: verify, on rootless BuildKit)"
|
||||
bake_run "$f" "${img}-verify"
|
||||
BUILT[$img]=1
|
||||
if [ "${MODE}" = publish ]; then
|
||||
existing="$(registry_digest "${img}" "${SHA_TAG}")"
|
||||
# Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
|
||||
# silently change what `sha-<commit>` means after something may already have pinned it.
|
||||
[ -z "${existing}" ] || die "${REGISTRY}/${NAMESPACE}/${img}:${SHA_TAG} already exists (${existing}); tags are immutable"
|
||||
log "BUILD + PUSH ${img} -> ${REGISTRY}/${NAMESPACE}/${img}:{${SHA_TAG},${MOVING_TAG}}"
|
||||
bake_run "$f" "${img}"
|
||||
pushed="$(jq -r --arg t "$img" '.[$t]["containerimage.digest"] // empty' "${WORK}/meta.json")"
|
||||
# The AUTHORITATIVE digest is what the registry serves, read back — not what the client
|
||||
# reported. They should agree; if they do not, nothing downstream may trust either.
|
||||
got="$(registry_digest "${img}" "${SHA_TAG}")"
|
||||
[ -n "${got}" ] || die "pushed ${img}:${SHA_TAG} but the registry does not serve it"
|
||||
[ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] || die "${img}: client reported ${pushed} but the registry serves ${got}"
|
||||
[ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
|
||||
|| die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
|
||||
DIGEST[$img]="${got}"
|
||||
info "${img}: registry digest ${got}"
|
||||
fi
|
||||
SECS[$img]=$(( $(date +%s) - t0 ))
|
||||
info "${img}: done in ${SECS[$img]}s"
|
||||
}
|
||||
|
||||
# --- main ---------------------------------------------------------------------------------
|
||||
log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
|
||||
select_images
|
||||
|
||||
if [ "${MODE}" = publish ]; then
|
||||
: "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
|
||||
REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
|
||||
# buildx (and through it buildkitd, which does the push) reads registry auth from this file.
|
||||
jq -n --arg r "${REGISTRY}" --arg a "$(printf '%s:%s' "${REGISTRY_USER}" "${REGISTRY_TOKEN}" | base64 -w0)" \
|
||||
'{auths: {($r): {auth: $a}}}' > "${DOCKER_CONFIG}/config.json"
|
||||
chmod 600 "${DOCKER_CONFIG}/config.json"
|
||||
registry_token
|
||||
# Publish idempotence: an image already published at THIS commit is not rebuilt. That is what
|
||||
# makes a re-run after a half-finished publish safe — the immutable tag is never pushed twice.
|
||||
for i in "${ALL_IMAGES[@]}"; do
|
||||
[ -n "${WANT[$i]:-}" ] || continue
|
||||
d="$(registry_digest "${i}" "${SHA_TAG}")"
|
||||
if [ -n "${d}" ]; then
|
||||
info "${i}: already published at ${SHA_TAG} (${d}); not rebuilding"
|
||||
unset "WANT[$i]"; DIGEST[$i]="${d}"; NOTE[$i]="already published at this commit"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# Decide the agent-runner base (see header).
|
||||
AGENT_BASE=""
|
||||
if [ -n "${WANT[agent-runner]:-}" ]; then
|
||||
PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
|
||||
if [ "${MODE}" = verify ]; then
|
||||
AGENT_BASE=link
|
||||
[ -n "${WANT[toolbox]:-}" ] || { WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"; }
|
||||
elif [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
|
||||
case "${PINNED_BASE}" in
|
||||
*@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
|
||||
*@sha256:*) AGENT_BASE="${PINNED_BASE}" ;;
|
||||
*) die "infra/agent/versions.env AGENT_BASE_IMAGE='${PINNED_BASE}' is not digest-pinned; CI will not build on a floating base" ;;
|
||||
esac
|
||||
fi
|
||||
fi
|
||||
|
||||
for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip) (${NOTE[$i]:-inputs unchanged})"; done
|
||||
if [ -z "${WANT[toolbox]:-}${WANT[agent-runner]:-}${WANT[conformance-runner]:-}" ]; then
|
||||
info "nothing to build"
|
||||
else
|
||||
setup_builder
|
||||
fi
|
||||
|
||||
if [ -n "${WANT[toolbox]:-}" ]; then build_one toolbox; fi
|
||||
if [ -n "${WANT[agent-runner]:-}" ]; then
|
||||
if [ -z "${AGENT_BASE}" ]; then
|
||||
AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
|
||||
fi
|
||||
info "agent-runner base: $( [ "${AGENT_BASE}" = link ] && echo "the toolbox built in this run (bake target: context)" || echo "${AGENT_BASE}")"
|
||||
build_one agent-runner "${AGENT_BASE}"
|
||||
fi
|
||||
if [ -n "${WANT[conformance-runner]:-}" ]; then build_one conformance-runner; fi
|
||||
|
||||
# --- summary ------------------------------------------------------------------------------
|
||||
{
|
||||
echo "## Images (${MODE})"
|
||||
echo
|
||||
echo "commit \`${HEAD_SHA}\` — builder: rootless BuildKit \`${BUILDKIT_HOST}\` (mTLS)"
|
||||
echo
|
||||
echo "| image | verify target built | seconds | published digest (registry-returned) | note |"
|
||||
echo "|---|---|---|---|---|"
|
||||
for i in "${ALL_IMAGES[@]}"; do
|
||||
d="${DIGEST[$i]:-}"
|
||||
echo "| \`${i}\` | $(yn "${BUILT[$i]:-}" yes no) | ${SECS[$i]:--} | $(yn "${d}" "\`${REGISTRY}/${NAMESPACE}/${i}@${d}\`" -) | ${NOTE[$i]:-inputs unchanged} |"
|
||||
done
|
||||
echo
|
||||
if [ "${MODE}" = publish ]; then
|
||||
echo "Tags pushed: \`${SHA_TAG}\` (immutable) and \`${MOVING_TAG}\` (moving)."
|
||||
echo "Nothing here writes to git. Renovate opens the digest-bump PR on its next run."
|
||||
else
|
||||
echo "Verify mode: nothing was pushed and no registry credential was present."
|
||||
fi
|
||||
} >>"${SUMMARY}"
|
||||
log "done"
|
||||
@@ -73,7 +73,7 @@ spec:
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: '0'
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
- name: FTLCONF_webserver_api_password
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
||||
+1
-32
@@ -143,39 +143,8 @@ env:
|
||||
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
|
||||
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
|
||||
# every client that takes the ADVERTISED connection cannot reach the server at all.
|
||||
#
|
||||
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
|
||||
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
|
||||
#
|
||||
# dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
|
||||
# dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
|
||||
# dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
|
||||
#
|
||||
# dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
|
||||
# bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
|
||||
# * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
|
||||
# single lookup, which is maximum exposure to any momentary gap; while
|
||||
# * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
|
||||
# upstream (standard forwarder behaviour when the forward target does not answer) it latches
|
||||
# the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
|
||||
# The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
|
||||
# live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
|
||||
# flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
|
||||
# That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
|
||||
#
|
||||
# 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
|
||||
# re-election (both seconds), short enough that a genuine VIP change still propagates inside a
|
||||
# minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
|
||||
# propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
|
||||
# LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
|
||||
# the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
|
||||
# cache expires, the router falls back, and the public answer latches again. That is a real
|
||||
# outage, and the fix for it is PiHole availability, not DNS caching.
|
||||
#
|
||||
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
|
||||
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
|
||||
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
|
||||
# carries the full rationale and the measurement that found it.
|
||||
|
||||
-15
@@ -27,18 +27,3 @@ fi
|
||||
|
||||
yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
|
||||
ok "yaml/lint passed (${#files[@]} files)"
|
||||
|
||||
# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
|
||||
# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
|
||||
# while the gate stayed green. Valid YAML, so yamllint cannot see it.
|
||||
missing=0
|
||||
while IFS=: read -r wf _ value; do
|
||||
value="${value#*configurationFile:}"
|
||||
value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')"
|
||||
if [ -n "$value" ] && [ ! -e "$value" ]; then
|
||||
err "${wf}: configurationFile '${value}' does not exist in the repo"
|
||||
missing=1
|
||||
fi
|
||||
done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
|
||||
[ "$missing" -eq 0 ] || exit 1
|
||||
ok "workflow configurationFile references resolve"
|
||||
@@ -1,21 +0,0 @@
|
||||
---
|
||||
# Flux Kustomization -> ./platform/buildkit — rootless BuildKit, the image builder CI uses
|
||||
# (.gitea/workflows/images.yaml). Its own Kustomization so a stuck Certificate or a failing
|
||||
# builder can never stall any other tier (Flux health-checks Certificates under wait:true).
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: buildkit
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: true
|
||||
dependsOn:
|
||||
- name: platform # cert-manager CRDs (Issuer/Certificate) come from its HelmRelease
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/buildkit
|
||||
@@ -98,5 +98,4 @@ resources:
|
||||
# deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
|
||||
# See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
|
||||
- kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform)
|
||||
- buildkit.yaml # Flux Kustomization -> ./platform/buildkit (dependsOn platform)
|
||||
- cluster-vars.yaml # overlay knob surface (base defaults)
|
||||
@@ -98,10 +98,7 @@ RUN set -eux; \
|
||||
|
||||
# --- runtime -----------------------------------------------------------------
|
||||
ARG FEDORA_VERSION=44
|
||||
# Stage `conformance-runner` is the image that ships; stage `verify` (end of file) is a TEST stage
|
||||
# built FROM it that only runs assertions and is never pushed. build-image.sh and the CI
|
||||
# pipeline build `--target verify` to prove it and `--target conformance-runner` for the image.
|
||||
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION} AS conformance-runner
|
||||
FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
|
||||
|
||||
ARG CHAINSAW_VERSION=
|
||||
ARG KUBECTL_VERSION=
|
||||
@@ -201,86 +198,3 @@ WORKDIR /workspace
|
||||
# prepend something the policy cannot see. `bash` is the interactive fallback for a human
|
||||
# debugging the image locally.
|
||||
CMD ["/bin/bash"]
|
||||
|
||||
# =============================================================================
|
||||
# verify — a TEST stage, never shipped. Every assertion conformance/incluster/build-image.sh used
|
||||
# to make with `run --user 65532 --read-only --tmpfs /workspace` now runs here, at build time,
|
||||
# because CI builds on rootless BuildKit where no container runtime exists.
|
||||
#
|
||||
# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE:
|
||||
# * uid 65532 ............ exact (USER inherited from the conformance-runner stage).
|
||||
# * no network ........... exact: RUN --network=none, and the first check proves it.
|
||||
# * --read-only rootfs ... NOT reproducible (a RUN step always has a writable layer).
|
||||
# Approximated by permission: everything uid 65532 can write must lie
|
||||
# under /workspace, /tmp or /var/tmp. In the Job only /workspace (and
|
||||
# /results) are writable — the image points HOME and TMPDIR there — and
|
||||
# readOnlyRootFilesystem is enforced by
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml.
|
||||
# =============================================================================
|
||||
FROM conformance-runner AS verify
|
||||
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
|
||||
echo "--- no network (the build-time stand-in for the Job's isolation) ---"
|
||||
if getent hosts gitea.home.keithjackson.dev >/dev/null 2>&1 || timeout 3 bash -c '</dev/tcp/1.1.1.1/443' 2>/dev/null; then
|
||||
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
|
||||
fi
|
||||
echo "confirmed: no network"
|
||||
echo "--- id ---"; id
|
||||
echo "--- HOME ---"; echo "HOME=${HOME}"
|
||||
echo "--- bash ---"; bash --version | head -1
|
||||
echo "--- python3 ---"; python3 --version
|
||||
echo "--- git ---"; git --version
|
||||
echo "--- kubectl ---"; kubectl version --client
|
||||
echo "--- chainsaw ---"; chainsaw version
|
||||
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
|
||||
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
|
||||
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
|
||||
if [ -w /usr/local/bin/conformance-clone ]; then
|
||||
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
|
||||
fi
|
||||
echo "confirmed: not writable by uid $(id -u)"
|
||||
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check that only
|
||||
# greps stderr would pass a build where the guard had stopped rejecting.
|
||||
expect_fail() {
|
||||
local label="$1"; shift
|
||||
local out rc=0
|
||||
out="$(env "$@" 2>&1)" || rc=$?
|
||||
echo "--- ${label} ---"
|
||||
printf "%s\n" "${out}" | head -2
|
||||
if [ "${rc}" -eq 0 ]; then
|
||||
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
|
||||
fi
|
||||
echo "(rejected, exit ${rc})"
|
||||
}
|
||||
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
|
||||
conformance-clone --layer l2
|
||||
expect_fail "conformance-clone rejects a missing SHA" \
|
||||
conformance-clone
|
||||
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
|
||||
CONFORMANCE_GIT_SHA=main conformance-clone
|
||||
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
|
||||
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
|
||||
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
|
||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||
HOME=/etc conformance-clone
|
||||
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
|
||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
|
||||
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
|
||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||
CONFORMANCE_SRC=/etc/src conformance-clone
|
||||
echo "--- the root filesystem is not writable by the runner (expected: a failure) ---"
|
||||
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
|
||||
echo "confirmed: / is not writable"
|
||||
echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
|
||||
stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
|
||||
| grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
|
||||
if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi
|
||||
echo "confirmed: no writable path outside /workspace, /tmp, /var/tmp"
|
||||
echo "--- HOME is writable (the workspace emptyDir) ---"
|
||||
touch "${HOME}/.probe" || { echo "UNEXPECTED: ${HOME} not writable"; exit 1; }
|
||||
echo "confirmed: ${HOME} is writable"
|
||||
echo "--- uid is non-zero ---"
|
||||
[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
|
||||
echo "confirmed: running as uid $(id -u), not root"
|
||||
echo "CONFORMANCE-RUNNER VERIFIED"
|
||||
VERIFY
|
||||
@@ -61,15 +61,10 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
|
||||
> digest-pinned to the conformance-runner image, `command` exactly
|
||||
> `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
|
||||
> `env`/`envFrom` were the same shape of hole and are closed in the same policy.
|
||||
- `build-image.sh` — builds the above and **verifies the toolchain by running it**: the
|
||||
Containerfile's `verify` stage executes every tool and every `conformance-clone` refusal as
|
||||
uid 65532 with no network, at build time. It **never pushes**, and neither does anyone from a
|
||||
workstation: CI (`.gitea/workflows/images.yaml`) builds the same Containerfile and `verify`
|
||||
stage on rootless BuildKit (`platform/buildkit/`), and on a push to `main` publishes the
|
||||
verified image under `sha-<commit>` and `main`. Renovate then moves
|
||||
`CONFORMANCE_RUNNER_IMAGE_DIGEST` in `conformance/versions.env` onto the registry-returned
|
||||
digest in a pull request (that file is a sensitive path, so the bump needs an
|
||||
administrator's `/approve-override`).
|
||||
- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
|
||||
image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
|
||||
prints the exact `podman push --digestfile …` command for the operator instead. See
|
||||
`conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
|
||||
`--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
|
||||
and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
|
||||
default verify stays offline so an air-gapped build is fully checked, not half-checked).
|
||||
|
||||
@@ -1,49 +1,43 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
|
||||
# image and prove it works. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
|
||||
# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
|
||||
#
|
||||
# ./build-image.sh # build the verify target, then the image
|
||||
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
|
||||
# ./build-image.sh # build + verify + print the digest and push command
|
||||
# ./build-image.sh --no-verify # build only (skip the in-image tool proof)
|
||||
# ./build-image.sh --verify-clone # ALSO really clone this repo with conformance-clone.
|
||||
# # Opt-in because it needs the Gitea host reachable; the
|
||||
# # default verify is offline so an air-gapped build is
|
||||
# # still fully checked rather than half-checked.
|
||||
# ENGINE=docker ./build-image.sh # build with docker instead of podman
|
||||
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
|
||||
#
|
||||
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
|
||||
# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
|
||||
# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
|
||||
# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
|
||||
# in the job summary; Renovate then opens the pull request that moves conformance/versions.env
|
||||
# onto that digest. The proof that the image works is the Containerfile's `verify` stage, which
|
||||
# this script (podman locally) and CI both build.
|
||||
# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
|
||||
# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
|
||||
# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
|
||||
# non-root uid the policy demands, and prints the exact `podman push` line for a human.
|
||||
#
|
||||
# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
|
||||
# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
|
||||
# pins the toolbox gate runs with. Nothing in this file invents a version.
|
||||
#
|
||||
# NO DIGEST IS PRINTED. A manifest digest does not exist until a registry holds the manifest,
|
||||
# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
|
||||
# push time (.gitea/workflows/scripts/images.sh) and Renovate carries it into
|
||||
# conformance/versions.env.
|
||||
# THE DIGEST CAVEAT — read it before pasting a digest into anything.
|
||||
# The digest printed here is the LOCAL manifest digest. It is normally identical to the
|
||||
# registry digest (podman pushes the manifest it already has rather than recompressing),
|
||||
# but that is a property of the push, not a guarantee. So the push command below uses
|
||||
# `--digestfile`: the digest the registry actually returns is the authoritative one, and
|
||||
# that is the one that goes into conformance/versions.env.
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
|
||||
ENGINE="${ENGINE:-podman}"
|
||||
TAG_OVERRIDE="${IMAGE_TAG:-}"
|
||||
|
||||
VERIFY=1
|
||||
VERIFY_CLONE=0
|
||||
PRINT_ARGS=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--no-verify) VERIFY=0 ;;
|
||||
--print-build-args) PRINT_ARGS=1 ;;
|
||||
--verify-clone) VERIFY_CLONE=1 ;;
|
||||
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "unknown argument: $arg" >&2; exit 2 ;;
|
||||
@@ -54,6 +48,8 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
info() { printf ' %s\n' "$*"; }
|
||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
||||
|
||||
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
||||
|
||||
# --- pins: sourced, never invented -------------------------------------------
|
||||
TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
|
||||
[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
|
||||
@@ -63,21 +59,14 @@ source "${TOOLCHAIN_ENV}"
|
||||
: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
|
||||
: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
|
||||
|
||||
# THE build args, in ONE place; `--print-build-args` hands them to the CI pipeline, which builds
|
||||
# this Containerfile on rootless BuildKit.
|
||||
BUILD_ARGS=("CHAINSAW_VERSION=${CHAINSAW_VERSION}" "KUBECTL_VERSION=${KUBECTL_VERSION}" "FEDORA_VERSION=${FEDORA_VERSION}")
|
||||
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
|
||||
|
||||
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
||||
|
||||
# --- image name: fixed by the admission policy -------------------------------
|
||||
# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
|
||||
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
|
||||
# is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
|
||||
IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
|
||||
# The tag is a local handle only. CI re-tags what it publishes (git SHA + `main`), and
|
||||
# admission ignores tags entirely: only the digest is ever referenced from a Job.
|
||||
IMAGE_TAG="${TAG_OVERRIDE:-chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}}"
|
||||
# The tag is build convenience only (a human-readable handle for the push). Admission
|
||||
# ignores it: only the digest is ever referenced from a Job.
|
||||
IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
|
||||
IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
|
||||
|
||||
log "Building ${IMAGE_REF}"
|
||||
@@ -87,31 +76,86 @@ info " CHAINSAW ${CHAINSAW_VERSION}"
|
||||
info " KUBECTL ${KUBECTL_VERSION}"
|
||||
info " FEDORA ${FEDORA_VERSION}"
|
||||
|
||||
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
|
||||
LABEL_ARGS=()
|
||||
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
|
||||
|
||||
ARG_FLAGS=()
|
||||
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
|
||||
"${ENGINE}" build \
|
||||
--file "${HERE}/Containerfile" \
|
||||
--tag "${IMAGE_REF}" \
|
||||
--build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
|
||||
--build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
|
||||
--build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
|
||||
"${HERE}"
|
||||
|
||||
# --- verify: the point of the whole exercise ---------------------------------
|
||||
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So the proof is that
|
||||
# the tools EXECUTE, as the NON-ROOT uid, with no network, that conformance-clone is not writable
|
||||
# by that uid and refuses every hostile input, and that nothing outside /workspace is writable.
|
||||
# That proof is the Containerfile's `verify` stage, built here and in CI alike (CI has no
|
||||
# container runtime — rootless BuildKit only). See the stage's header for what a build step can
|
||||
# and cannot reproduce of the Job's shape. A build that merely succeeded is not evidence.
|
||||
# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
|
||||
# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
|
||||
# filesystem — the shape the admission policy forces. A pass here is the only evidence
|
||||
# this image is usable; a build that merely succeeded is not.
|
||||
if [ "${VERIFY}" -eq 1 ]; then
|
||||
log "Verifying the toolchain (Containerfile target: verify)"
|
||||
"${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
|
||||
log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
|
||||
"${ENGINE}" run --rm \
|
||||
--user 65532:65532 \
|
||||
--read-only \
|
||||
--tmpfs /workspace:rw,mode=1777 \
|
||||
--entrypoint "" \
|
||||
"${IMAGE_REF}" \
|
||||
/bin/bash -euo pipefail -c '
|
||||
echo "--- id ---"; id
|
||||
echo "--- HOME ---"; echo "HOME=${HOME}"
|
||||
echo "--- bash ---"; bash --version | head -1
|
||||
echo "--- python3 ---"; python3 --version
|
||||
echo "--- git ---"; git --version
|
||||
echo "--- kubectl ---"; kubectl version --client
|
||||
echo "--- chainsaw ---"; chainsaw version
|
||||
echo "--- baked versions ---"; cat /etc/conformance-runner-versions
|
||||
echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
|
||||
stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
|
||||
if [ -w /usr/local/bin/conformance-clone ]; then
|
||||
echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
|
||||
fi
|
||||
echo "confirmed: not writable by uid $(id -u)"
|
||||
# Each of these MUST fail. Assert the non-zero exit as well as the message: a check
|
||||
# that only greps stderr would pass a build where the guard had stopped rejecting.
|
||||
# (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
|
||||
# expected failure — without it the verify dies at the first negative case.)
|
||||
expect_fail() {
|
||||
local label="$1"; shift
|
||||
local out rc=0
|
||||
out="$(env "$@" 2>&1)" || rc=$?
|
||||
echo "--- ${label} ---"
|
||||
printf "%s\n" "${out}" | head -2
|
||||
if [ "${rc}" -eq 0 ]; then
|
||||
echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
|
||||
fi
|
||||
echo "(rejected, exit ${rc})"
|
||||
}
|
||||
expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
|
||||
conformance-clone --layer l2
|
||||
expect_fail "conformance-clone rejects a missing SHA" \
|
||||
conformance-clone
|
||||
expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
|
||||
CONFORMANCE_GIT_SHA=main conformance-clone
|
||||
expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
|
||||
CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
|
||||
expect_fail "conformance-clone rejects a HOME off the workspace volume" \
|
||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||
HOME=/etc conformance-clone
|
||||
expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
|
||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||
CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
|
||||
expect_fail "conformance-clone refuses a clone target off the workspace volume" \
|
||||
CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
|
||||
CONFORMANCE_SRC=/etc/src conformance-clone
|
||||
echo "--- rootfs is read-only (expected: a failure) ---"
|
||||
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
|
||||
echo "confirmed: / is read-only"
|
||||
echo "--- HOME is writable (the workspace emptyDir) ---"
|
||||
touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
|
||||
echo "--- uid is non-zero ---"
|
||||
[ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
|
||||
'
|
||||
log "Toolchain verified"
|
||||
fi
|
||||
|
||||
"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target conformance-runner \
|
||||
--file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
|
||||
|
||||
# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
|
||||
# LOCAL ONLY: this one `run`s the image, which needs a container engine, so CI does not do it.
|
||||
# The negative cases above prove conformance-clone refuses bad input. This proves the
|
||||
# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
|
||||
# rev-parse assertion is only an unexercised line of shell.
|
||||
@@ -140,12 +184,46 @@ if [ "${VERIFY_CLONE}" -eq 1 ]; then
|
||||
log "conformance-clone verified against the real repository"
|
||||
fi
|
||||
|
||||
# --- result ------------------------------------------------------------------
|
||||
# --- digest ------------------------------------------------------------------
|
||||
DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
|
||||
IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
|
||||
SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
|
||||
|
||||
log "Built ${IMAGE_REF}"
|
||||
info "image id ${IMAGE_ID}"
|
||||
info "size $(( SIZE / 1024 / 1024 )) MiB"
|
||||
info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml) and"
|
||||
info "Renovate moves conformance/versions.env onto the registry-returned digest."
|
||||
info "local manifest digest ${DIGEST}"
|
||||
info "image id ${IMAGE_ID}"
|
||||
info "size $(( SIZE / 1024 / 1024 )) MiB"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
================================================================================
|
||||
NOT PUSHED. Publishing is an operator action — this script holds no registry
|
||||
credential and will never acquire one.
|
||||
|
||||
OPERATOR: run these two commands, then paste the digest from the digestfile into
|
||||
conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
|
||||
|
||||
${ENGINE} login gitea.home.keithjackson.dev
|
||||
|
||||
${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
|
||||
${IMAGE_REF}
|
||||
|
||||
Then:
|
||||
|
||||
cat /tmp/conformance-runner.digest
|
||||
# -> sha256:.... THIS is the authoritative digest, not the local one above.
|
||||
|
||||
# conformance/versions.env
|
||||
CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
|
||||
|
||||
WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
|
||||
is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
|
||||
${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
|
||||
and so is any other repository path.
|
||||
|
||||
TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
|
||||
\`${IMAGE_REPO}@sha256:*\` to be replaced with the
|
||||
LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
|
||||
That is a separate, human-reviewed PR against the security boundary.
|
||||
================================================================================
|
||||
EOF
|
||||
@@ -29,8 +29,7 @@
|
||||
# THEN (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
|
||||
# infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
|
||||
# lockstep rule), the Containerfile takes it as a build ARG, installs it from the
|
||||
# pinned prometheus release and records it for provenance, build-image.sh (the one
|
||||
# recipe `dev.sh build` and CI both use) passes it, and the
|
||||
# pinned prometheus release and records it for provenance, dev.sh passes it, and the
|
||||
# CI runner installs it for the l0 stage;
|
||||
# (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
|
||||
# evidence only if the engine matches the one prod evaluates rules with;
|
||||
@@ -247,12 +246,10 @@ check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
|
||||
"infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
|
||||
"that provenance line is how an operator tells a stale image from a current one")
|
||||
|
||||
# The build recipe moved from dev.sh into build-image.sh on 2026-10-29, so that `dev.sh build`
|
||||
# and the CI image pipeline (.gitea/workflows/images.yaml) share ONE recipe. Check the recipe.
|
||||
BUILDSH = ROOT / "infra/devcontainer/build-image.sh"
|
||||
check(BUILDSH.is_file() and '"PROMTOOL_VERSION=$PROMTOOL_VERSION"' in BUILDSH.read_text(),
|
||||
"infra/devcontainer/build-image.sh does not pass PROMTOOL_VERSION in its BUILD_ARGS — the ARG "
|
||||
"would be empty and the build would fetch a bogus URL")
|
||||
DEVSH = ROOT / "infra/devcontainer/dev.sh"
|
||||
check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
|
||||
"infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
|
||||
"empty and the build would fetch a bogus URL")
|
||||
|
||||
INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
|
||||
if INSTALL.is_file():
|
||||
|
||||
@@ -445,21 +445,8 @@ join = re.search(r"--server https://\{\{\s*k3s_api_host", K3S_TEXT)
|
||||
check(join is not None,
|
||||
"roles/k3s_server does not join secondaries with `--server https://{{ k3s_api_host }}` "
|
||||
"— without a join target the servers are separate single-node clusters")
|
||||
# LINE-BOUNDED ON PURPOSE -- DO NOT "SIMPLIFY" BACK TO `.*` + re.S.
|
||||
# This was `--server https://.*?\n(?:.*\n){0,10}?\s*when:...` with re.S, which made every `.`
|
||||
# match newlines too. That is nested unbounded repetition (a `{0,10}` of a quantifier that can each
|
||||
# swallow the whole document), so a FAILED or late match backtracks exponentially. It cost
|
||||
# **19m54s of CPU in CI** (measured 2026-10-18 via `docker top` on the live job container) and made
|
||||
# the whole merge gate take ~28 minutes instead of ~2 on any change touching `infra/**` -- because
|
||||
# `ci/touched-requirements.sh` maps `infra/*` to R-ENV-01, which pulls in this whole-repo check.
|
||||
# It was latent for as long as the regex existed and only crossed the threshold when
|
||||
# roles/k3s_server grew by ~100 lines (the kubelet graceful-shutdown config, PR #71): K3S_TEXT is
|
||||
# every *.yml in that role concatenated, so the role growing is the input growing.
|
||||
# `[^\n]*` cannot cross a line, so the `{0,10}` is a genuine ten-line window with nothing to
|
||||
# backtrack into. Measured on the same input: 0.2 ms, same verdict (match found).
|
||||
join_when = re.search(
|
||||
r"--server https://[^\n]*\n(?:[^\n]*\n){0,10}?[ \t]*when:[ \t]*k3s_server_mode[ \t]*==[ \t]*'secondary'",
|
||||
K3S_TEXT)
|
||||
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
|
||||
K3S_TEXT, re.S)
|
||||
check(join_when is not None,
|
||||
"the join install task is not gated on k3s_server_mode == 'secondary' — the founder "
|
||||
"would also try to join itself")
|
||||
|
||||
@@ -42,19 +42,6 @@
|
||||
# whose every solver is an automated challenge type (dns01 / http01) — the
|
||||
# same code path serves issuance AND renewal, so a solver a human must
|
||||
# complete would defeat both;
|
||||
# (5b) the ONE exception to (5) — WORKLOAD mTLS, added 2026-10-29 — is a Certificate
|
||||
# annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`. The spec's TLS
|
||||
# requirements are about EXTERNALLY REACHABLE services (R-TLS-01; docs/08), and
|
||||
# ADR-0006 says outright that "mTLS between workloads is a separate concern … not
|
||||
# covered here". A public ACME CA cannot issue for a `*.svc.cluster.local` name or a
|
||||
# client-auth certificate at all. The annotation is NOT trusted: an exempt
|
||||
# Certificate must PROVE it is workload-internal — its Secret is served by NO Gateway
|
||||
# listener, every DNS name / commonName is cluster-internal (a bare label or
|
||||
# `*.svc[.cluster.local]`), it carries no IP/URI/email SANs, and its issuer is a
|
||||
# NAMESPACED Issuer in its own namespace whose spec is `selfSigned` or a `ca` backed
|
||||
# by a Secret that a tracked, likewise-exempt Certificate produces. Issuance and
|
||||
# renewal stay unattended (cert-manager renews CA-issued certs itself), and (9)
|
||||
# still applies. Every exemption is printed. First user: platform/buildkit.
|
||||
# (6) each dns01 solver's provider credential is referenced BY NAME and that
|
||||
# Secret is delivered SOPS-encrypted from git via a kustomization — the
|
||||
# operator's one-time token hand-off (docs/08 OQ-4), never a per-cert step;
|
||||
@@ -278,7 +265,6 @@ CERT_SECRETS: dict[tuple[str, str], str] = { # (namespace, secretName)
|
||||
for rel, d in CERTS
|
||||
}
|
||||
listeners_checked = 0
|
||||
GATEWAY_SECRETS: set[tuple[str, str]] = set() # every (namespace, Secret) a listener serves
|
||||
for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
|
||||
gw_ns = ns_of(gw)
|
||||
for lis in ((gw.get("spec") or {}).get("listeners") or []):
|
||||
@@ -294,7 +280,6 @@ for rel, gw in of_kind("Gateway", "gateway.networking.k8s.io/"):
|
||||
for r in refs:
|
||||
kind = str(r.get("kind", "Secret"))
|
||||
key = (str(r.get("namespace") or gw_ns), str(r.get("name")))
|
||||
GATEWAY_SECRETS.add(key)
|
||||
check(kind == "Secret",
|
||||
f"{rel}: listener {lis.get('name')!r} certificateRef kind is {kind!r}; only "
|
||||
"Secret refs can be traced to a cert-manager Certificate")
|
||||
@@ -339,61 +324,8 @@ CM_NS = ns_of(cm_releases[0][1], "cert-manager") if cm_releases else "cert-manag
|
||||
ISSUER_INDEX = {(d.get("kind"), name_of(d)): (rel, d) for rel, d in ISSUERS}
|
||||
solver_creds: list[tuple[str, str, str, str]] = [] # (issuer, ns, secretName, ref field)
|
||||
issuers_verified: set[str] = set()
|
||||
|
||||
# (5b) workload-mTLS exemption — see the header. Proven, never taken on trust.
|
||||
SCOPE_ANN = "homelab.keithjackson.dev/tls-scope"
|
||||
INTERNAL_NAME = re.compile(r"^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$")
|
||||
NS_ISSUERS = {(ns_of(d), name_of(d)): (rel, d) for rel, d in of_kind("Issuer", "cert-manager.io/")}
|
||||
EXEMPT_SECRETS = {(ns_of(d), str((d.get("spec") or {}).get("secretName", "")))
|
||||
for _, d in CERTS
|
||||
if ((d.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN) == "workload-mtls"}
|
||||
exempted: list[str] = []
|
||||
|
||||
def workload_mtls_ok(rel: str, cert: dict) -> bool:
|
||||
"""True if `cert` claims the workload-mTLS scope AND proves it; records failures."""
|
||||
ann = ((cert.get("metadata") or {}).get("annotations") or {}).get(SCOPE_ANN)
|
||||
if ann is None:
|
||||
return False
|
||||
cns, cname = ns_of(cert), name_of(cert)
|
||||
spec = cert.get("spec") or {}
|
||||
ok = True
|
||||
def need(cond, msg):
|
||||
nonlocal ok
|
||||
check(cond, f"{rel}: Certificate {cns}/{cname} claims {SCOPE_ANN}={ann!r} but {msg}")
|
||||
ok = ok and bool(cond)
|
||||
need(ann == "workload-mtls", "the only accepted value is 'workload-mtls'")
|
||||
need((cns, str(spec.get("secretName", ""))) not in GATEWAY_SECRETS,
|
||||
"its Secret is served by a Gateway listener — that is external TLS, which must be ACME (R-TLS-01)")
|
||||
names = [str(n) for n in (spec.get("dnsNames") or [])]
|
||||
if spec.get("commonName"):
|
||||
names.append(str(spec["commonName"]))
|
||||
for n in names:
|
||||
internal = INTERNAL_NAME.match(n) and ("." not in n or n.endswith(".svc") or n.endswith(".svc.cluster.local"))
|
||||
need(bool(internal), f"name {n!r} is not cluster-internal (bare label or *.svc[.cluster.local])")
|
||||
for f in ("ipAddresses", "uris", "emailAddresses"):
|
||||
need(not spec.get(f), f"it carries {f}, which a workload-internal certificate has no use for")
|
||||
ref = spec.get("issuerRef") or {}
|
||||
need(str(ref.get("kind", "Issuer")) == "Issuer",
|
||||
"its issuer is not a NAMESPACED Issuer — a ClusterIssuer could sign for any namespace")
|
||||
hit = NS_ISSUERS.get((cns, str(ref.get("name", ""))))
|
||||
need(hit is not None, f"its Issuer {ref.get('name')!r} is not tracked in namespace {cns!r}")
|
||||
if hit is not None:
|
||||
ispec = hit[1].get("spec") or {}
|
||||
kinds = sorted(ispec)
|
||||
need(kinds in (["selfSigned"], ["ca"]),
|
||||
f"its Issuer is {kinds} — only selfSigned or ca are accepted for workload mTLS")
|
||||
if kinds == ["ca"]:
|
||||
need((cns, str((ispec.get("ca") or {}).get("secretName", ""))) in EXEMPT_SECRETS,
|
||||
"its CA Issuer's Secret is not produced by a tracked workload-mtls Certificate "
|
||||
"in the same namespace — the CA itself would have to be minted by hand")
|
||||
if ok:
|
||||
exempted.append(f"{cns}/{cname}")
|
||||
return ok
|
||||
|
||||
for rel, cert in CERTS:
|
||||
spec = cert.get("spec") or {}
|
||||
if workload_mtls_ok(rel, cert):
|
||||
continue # (5b): proven workload-internal; (9) below still applies
|
||||
ref = spec.get("issuerRef") or {}
|
||||
kind = str(ref.get("kind", "Issuer"))
|
||||
per_env = resolve_per_env(str(ref.get("name", "")))
|
||||
@@ -447,8 +379,6 @@ for rel, cert in CERTS:
|
||||
solver_creds.append((iname, want_ns, n, f"{provider}.{field}"))
|
||||
|
||||
check(len(issuers_verified) > 0, "no issuer was reachable from any Certificate — (5) proved nothing")
|
||||
if exempted:
|
||||
print(f" (5b) workload-mTLS certificates exempted from ACME after proving it: {', '.join(exempted)}")
|
||||
|
||||
# (6) the solver credential is delivered, SOPS-encrypted, from git — never per-cert by hand.
|
||||
check(len(solver_creds) > 0,
|
||||
|
||||
+21
-18
@@ -40,29 +40,32 @@ PYTHON_MIN="3.9"
|
||||
# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*
|
||||
# Any other path, and any TAG reference, is DENIED at admission. Do not "tidy" these values.
|
||||
CONFORMANCE_RUNNER_IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
|
||||
# Local build handle only. Admission never sees the tag; a Job must reference the digest.
|
||||
# CI publishes under `sha-<commit>` and `main` instead (.gitea/workflows/images.yaml).
|
||||
# Build handle only. Admission never sees the tag; a Job must reference the digest.
|
||||
CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
|
||||
#
|
||||
# ############################################################################
|
||||
# # THE DIGEST IS WRITTEN BY RENOVATE, NOT BY HAND. #
|
||||
# # DIGEST DELIBERATELY EMPTY — THE IMAGE HAS NOT BEEN PUBLISHED YET. #
|
||||
# ############################################################################
|
||||
# CI builds and verifies the image (non-root uid 65532, read-only rootfs, the
|
||||
# conformance-clone refusals) and, on a push to main, publishes it and prints the digest the
|
||||
# REGISTRY returned (.gitea/workflows/images.yaml). Renovate follows the image's `main` tag
|
||||
# through the marker below and opens the pull request that moves this value. Nobody pushes
|
||||
# this image from a workstation, and nobody pastes a digest here.
|
||||
# The image has been BUILT and its toolchain verified locally (non-root uid 65532,
|
||||
# read-only rootfs: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4, chainsaw 0.2.12,
|
||||
# and `./run.sh --layer l2 --no-cluster` completing with rc 0). It has NOT been pushed:
|
||||
# pushing needs a registry credential, and publishing the artifact a fail-closed security
|
||||
# boundary then trusts is an OPERATOR action, not an agent's.
|
||||
#
|
||||
# ALL-ZERO UNTIL THE FIRST PUBLISH. Before 2026-10-29 this was EMPTY, so that a consumer
|
||||
# rendering the Job would fail visibly. It is now the same impossible all-zero digest
|
||||
# platform/pm-agent/ uses, because Renovate can only replace a digest that is there to match.
|
||||
# A consumer MUST treat the all-zero value as "not published" (no consumer exists yet: the
|
||||
# L2/L3 trigger is unwired, docs/plans/l2-l3-conformance-ci-wiring.md). Note also that
|
||||
# chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the
|
||||
# `sha256:` prefix this value carries — the trigger must strip it.
|
||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main
|
||||
CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
|
||||
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}"
|
||||
# OPERATOR, to complete the pin:
|
||||
# 1. ./conformance/incluster/build-image.sh # rebuild, verify, print the commands
|
||||
# 2. podman login gitea.home.keithjackson.dev
|
||||
# 3. podman push --digestfile /tmp/conformance-runner.digest \
|
||||
# "${CONFORMANCE_RUNNER_IMAGE_REPO}:${CONFORMANCE_RUNNER_IMAGE_TAG}"
|
||||
# 4. Set CONFORMANCE_RUNNER_IMAGE_DIGEST below to `cat /tmp/conformance-runner.digest`
|
||||
# — the digest the REGISTRY returned, not the local manifest digest the build printed.
|
||||
# 5. Land steps 4 in a PR (this file is GitOps input, not a live edit).
|
||||
#
|
||||
# UNTIL THEN: this is empty on purpose, NOT a placeholder to be guessed at. An empty value
|
||||
# makes any consumer that renders the Job fail visibly; a fake or stale digest would instead
|
||||
# produce an ImagePullBackOff that looks like a cluster fault. Do not invent one.
|
||||
CONFORMANCE_RUNNER_IMAGE_DIGEST=""
|
||||
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_DIGEST:+${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}}"
|
||||
#
|
||||
# AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
|
||||
# its `@sha256:*` wildcard to be replaced with the LITERAL digest once the image exists —
|
||||
|
||||
@@ -1,106 +0,0 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: feat/graceful-cluster-shutdown
|
||||
title: "Graceful node shutdown was never configured, and scaling a StatefulSet deleted its PVC"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
All three nodes had to be powered off to move them onto a UPS. There was **no procedure for it**:
|
||||
`infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev cluster, so
|
||||
the whole sequence was improvised live. Two defects came out of doing it.
|
||||
|
||||
**1. Kubelet graceful node shutdown was unconfigured on every prod node.** No `shutdownGracePeriod`
|
||||
anywhere in `infra/ansible/`, so `shutdown -h` stopped k3s, containerd went with it, and every pod
|
||||
was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25
|
||||
attached Longhorn volumes. `roles/k3s_server` now writes `kubelet-arg` into
|
||||
`/etc/rancher/k3s/config.yaml` with `shutdown-grace-period=90s` and
|
||||
`shutdown-grace-period-critical-pods=20s`.
|
||||
|
||||
The half that is easy to miss and silently defeats the other half: **graceful node shutdown is
|
||||
enforced by logind, not by the kubelet.** The kubelet holds an inhibitor *delay* lock and systemd
|
||||
waits at most `InhibitDelayMaxSec` — **default 5 seconds** — before proceeding regardless. A grace
|
||||
period above that cap is truncated with no error, so the feature looks configured and still
|
||||
hard-kills pods. The role now ships a `logind.conf.d` drop-in setting it to 120, and **asserts**
|
||||
`k3s_logind_inhibit_delay_max_sec >= k3s_kubelet_shutdown_grace_period` rather than documenting the
|
||||
relationship, because the failure is invisible.
|
||||
|
||||
**2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume.** `monitoring/loki` carries
|
||||
`persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}`, so the quiesce's
|
||||
`scale --replicas=0` destroyed `storage-loki-0` outright. Loki stores with `type: filesystem`, so
|
||||
that PVC was the only copy of its log data. This is the defect in this change set that actually
|
||||
destroyed something, and it was mine.
|
||||
|
||||
New `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`) encodes the sequence that
|
||||
worked, and **refuses to scale any StatefulSet whose `whenScaled` is `Delete`** — it leaves it
|
||||
running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect
|
||||
`degraded`. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed
|
||||
from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS)
|
||||
rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it.
|
||||
|
||||
`runbooks/cluster-maintenance-shutdown.md` (new, registered in `runbooks/kustomization.yaml`) is the
|
||||
operator page: shutdown order, the TrueNAS-after-nodes constraint, power-on order, the
|
||||
un-hibernate-Postgres step Flux will not do for you, and a "what looks broken but is fine" section
|
||||
separating the `4/19`-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the
|
||||
self-recovering `keycloak-operator` from the one thing that genuinely needs a human (a stuck
|
||||
`oauth2-proxy` replica).
|
||||
|
||||
## What I verified
|
||||
|
||||
* **Graceful shutdown was genuinely absent**, not merely undocumented: `grep -rn
|
||||
'shutdownGracePeriod|graceful-node-shutdown'` over `infra/ansible/` returns nothing, and no
|
||||
`logind` or `InhibitDelay` setting exists anywhere in the tree.
|
||||
* **The whole shutdown/restart cycle was executed end to end on prod 2026-10-17** and the cluster
|
||||
came back: 3/3 nodes Ready with etcd `healthz` ok, 19/19 Flux Kustomizations Ready,
|
||||
`platform-config`'s `lastAppliedRevision` equal to the GitRepository artifact revision, all 3
|
||||
CNPG clusters `healthy` with primaries elected, the UniFi VM `Running`, PiHole resolving
|
||||
(`home.keithjackson.dev` → `192.168.2.20` via `192.168.2.23`), and the Gateway serving
|
||||
`200/200/401` for apex/gitea/plex.
|
||||
* **The PVC-deletion mechanism is confirmed, not inferred:** `kubectl get sts loki -o
|
||||
jsonpath={.spec.persistentVolumeClaimRetentionPolicy}` returns
|
||||
`{"whenDeleted":"Delete","whenScaled":"Delete"}`, the PVC was `NotFound` after the cycle, and the
|
||||
Longhorn volume count had gone 25 → 24.
|
||||
* **The damage was repaired and verified:** `scale sts loki --replicas=1` (restoring the replica
|
||||
count the chart declares) recreated `storage-loki-0` as a fresh 50Gi Bound PVC, `loki-0` reached
|
||||
`2/2 Running`, and the volume count returned to 25.
|
||||
* **The guard works against the live cluster**, tested both ways: `sts_deletes_pvc_on_scale
|
||||
monitoring loki` returns true (would be skipped) and `pihole pihole` returns false (safe to
|
||||
scale).
|
||||
* **The script's read-only paths run clean on prod:** `status` reports correctly, and the discovery
|
||||
function returns the 13 workloads currently holding a PVC.
|
||||
* `bash -n` clean; the two edited Ansible YAML files parse.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **The kubelet change has NEVER BEEN APPLIED OR TESTED.** This is the significant gap. I did not
|
||||
run the role, did not restart k3s, and have **not** observed a node actually drain on
|
||||
`shutdown -h`. Everything about it is reasoning from kubelet/systemd semantics plus the measured
|
||||
absence of the setting. Applying it needs a **rolling** k3s restart (`serial: 1` — kube-vip keeps
|
||||
the API up, etcd holds at 2/3), which is an operator action (CLAUDE.md §5). **Verify by draining
|
||||
one node for real:** apply, then `shutdown -h now` on a single node and watch whether pods
|
||||
terminate gracefully instead of vanishing.
|
||||
* **Whether 90s/20s are the right numbers.** Chosen to comfortably cover a Postgres fast shutdown
|
||||
and a Longhorn flush; not measured against either. The budget is spent on every reboot, so if a
|
||||
node takes noticeably longer to go down, this is why.
|
||||
* **Whether k3s accepts `kubelet-arg` in `config.yaml` as written.** The file already carries
|
||||
`kube-apiserver-arg` as a list and `kubelet-arg` is the documented sibling, but I did not render
|
||||
or apply it, so a typo would surface only at k3s restart — and a bad `config.yaml` is exactly the
|
||||
kind of thing that makes k3s fail to come up. **Review this one before applying.**
|
||||
* **The `assert` task's `regex_replace` on the grace period** assumes a plain `90s`-style value; a
|
||||
value expressed as `1m30s` would parse to the nonsense integer `130`. Fine for the committed
|
||||
defaults, wrong if someone writes a compound duration.
|
||||
* **`resume`'s Flux-reconcile wait has never run**, and neither has `quiesce` as a script — the
|
||||
2026-10-17 cycle was done with the equivalent commands by hand, so the script encodes a sequence
|
||||
that worked but has not itself been executed end to end. Only `status` and the discovery and guard
|
||||
functions have run.
|
||||
* **Loki's lost log window is gone for good.** Up to 72h of logs (its `retention_period`), destroyed
|
||||
by my scale-down. Nothing recovers it; Velero's schedules cover the namespace but the PVC was
|
||||
deleted, not corrupted, and I did not attempt a restore because the data self-expires anyway.
|
||||
* **No other `whenScaled: Delete` StatefulSet was audited.** The guard now catches them generically,
|
||||
but I did not enumerate which other workloads carry the policy, so I cannot say loki was the only
|
||||
one at risk. The runbook gives the one-liner to list them.
|
||||
* **TrueNAS and the router are entirely outside this change.** TrueNAS is not in the Ansible
|
||||
inventory and refuses SSH, so its shutdown stays a web-UI step with no automation and no check.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1
|
||||
offline subset only.
|
||||
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
|
||||
@@ -1,311 +0,0 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: feat/keycloak-passkeys
|
||||
title: Passkeys declared for the homelab realm — and the realm import will not apply them
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`platform/keycloak/keycloak-realm-import.yaml` now declares passkey (WebAuthn Passwordless)
|
||||
login for the `homelab` realm, in two parts:
|
||||
|
||||
1. **The WebAuthn Passwordless policy.** Ten `webAuthnPolicyPasswordless*` fields on the realm:
|
||||
RP ID `${BASE_DOMAIN}`, RP entity name `Homelab SSO`, signature algorithms `[ES256, RS256]`,
|
||||
`requireResidentKey: "Yes"`, `userVerificationRequirement: required`,
|
||||
`authenticatorAttachment: "not specified"`, `attestationConveyancePreference: none`,
|
||||
`acceptableAaguids: []`, `createTimeout: 300`, `avoidSameAuthenticatorRegister: true`.
|
||||
Deliberately the **passwordless** family only; the two-factor `webAuthnPolicy*` family is left
|
||||
at Keycloak's defaults because nothing in this realm uses `webauthn-authenticator`.
|
||||
|
||||
2. **The browser flow.** `browser-mfa` keeps its alias and stays bound as `browserFlow`; its
|
||||
interior is restructured so a passkey is an **ALTERNATIVE** beside the password form:
|
||||
|
||||
```
|
||||
browser-mfa (top level, unchanged)
|
||||
browser-mfa-forms ALTERNATIVE
|
||||
auth-username-form REQUIRED <- was auth-username-password-form
|
||||
browser-mfa-passkey-or-password REQUIRED <- NEW sub-flow
|
||||
webauthn-authenticator-passwordless ALTERNATIVE <- NEW: the passkey
|
||||
browser-mfa-password-and-otp ALTERNATIVE <- NEW sub-flow
|
||||
auth-password-form REQUIRED <- was half of the combined form
|
||||
browser-mfa-conditional-otp CONDITIONAL (unchanged, relocated intact)
|
||||
```
|
||||
|
||||
The design decisions, with their reasoning, are written into the manifest comments in this repo's
|
||||
usual density. The ones that matter:
|
||||
|
||||
* **Additive, not replacing. Enrolment optional, not forced.** This IdP gates Grafana, Gitea,
|
||||
Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the \*arr stack *and* `kubectl` via the
|
||||
`kubernetes` OIDC client. A mandatory passkey flow that misfires is a total lock-out from the
|
||||
platform and the cluster API at once, leaving only the Keycloak bootstrap admin and the
|
||||
break-glass cert kubeconfig. The gain from forcing it (one human stops typing a password) does
|
||||
not price that risk.
|
||||
* **No `requiredActions` block was added, on purpose.** Declaring one would have been the obvious
|
||||
way to add `webauthn-register-passwordless`, and it would have been actively dangerous — see
|
||||
"What I verified".
|
||||
* **RP ID is `${BASE_DOMAIN}`, not `keycloak.${BASE_DOMAIN}`.** A passkey's RP ID is fixed at
|
||||
enrolment and can never be re-keyed, so it must not be bound to a name that can move. This realm
|
||||
already declares an `auth.${BASE_DOMAIN}` redirect URI, and this repo already owns a reconciler
|
||||
built because a hostname move silently failed to reach the realm. Residual risk, unavoidable:
|
||||
changing `BASE_DOMAIN` itself invalidates every enrolled passkey.
|
||||
* **A passkey login is not also asked for TOTP.** `browser-mfa-conditional-otp` hangs off the
|
||||
password branch only. The justification is `userVerificationRequirement: required` — the
|
||||
ceremony already proves possession plus a verified human, phishing-resistantly and origin-bound.
|
||||
Stacking a replayable shared secret on top of that buys friction. The manifest names the exact
|
||||
one-line edit to reverse this decision.
|
||||
* **Login becomes two screens** (username, then password) for every user, passkey or not, because
|
||||
the server must know who is logging in before it can decide whether a passkey exists. Accepted
|
||||
cost; it is the enabling change.
|
||||
* **Usernameless login deliberately not used**, although Keycloak 26.0.6 supports it — see "What I
|
||||
verified" for the measured reason it would have made every login open on a passkey prompt.
|
||||
|
||||
`platform/keycloak/README.md` gains a `## Passkeys` section: the decision table, the fact that
|
||||
this does not reach the live realm, the two operator paths that would apply it, and the four-step
|
||||
browser verification nobody has run.
|
||||
|
||||
No secret, key, credential or AAGUID was minted or committed. No file under `clusters/**`,
|
||||
`conformance/**`, `ci/**`, `platform/kyverno/**`, `**/rbac.yaml` or `**/*.sops.yaml` was touched,
|
||||
so this should not trip `ci/approval-policy.sh`'s sensitive-path review requirement.
|
||||
|
||||
## What I verified
|
||||
|
||||
**The live realm import has already completed and will never re-run.** Measured on prod,
|
||||
2026-10-17:
|
||||
|
||||
```
|
||||
$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
|
||||
{"conditions":[{"message":"","status":"True","type":"Done"},
|
||||
{"message":"","status":"False","type":"Started"},
|
||||
{"message":"","status":"False","type":"HasErrors"}]}
|
||||
```
|
||||
|
||||
`Done=True` / `Started=False`, exactly the one-shot state `platform/keycloak/README.md` already
|
||||
documents for client secrets and client URLs. **So the change in this PR has zero effect on how
|
||||
anyone logs in today**, and the client-secret/client-URL reconciler does not and cannot close the
|
||||
gap: its hard endpoint allow-list makes `/authentication` unreachable by design, and realm-level
|
||||
policy fields are not client fields.
|
||||
|
||||
**The CRD accepts every field I used.** Measured against the live CRD (not the upstream docs):
|
||||
`kubectl --context k3s-prod get crd keycloakrealmimports.k8s.keycloak.org` lists all twelve
|
||||
`webAuthnPolicyPasswordless*` properties with the types I wrote (`RequireResidentKey`,
|
||||
`RpEntityName`, `RpId`, `UserVerificationRequirement`, `AttestationConveyancePreference`,
|
||||
`AuthenticatorAttachment` as `string`; `SignatureAlgorithms`, `AcceptableAaguids`, `ExtraOrigins`
|
||||
as `array/string`; `CreateTimeout` as `integer`; `AvoidSameAuthenticatorRegister` as `boolean`),
|
||||
plus `authenticationFlows`, `authenticatorConfig` and `requiredActions`. The `spec.realm` schema
|
||||
has **no** `x-kubernetes-preserve-unknown-fields`, so a misspelled field would have been pruned
|
||||
silently rather than rejected — which is why this was checked rather than assumed.
|
||||
|
||||
**Running Keycloak is 26.0.6.** `quay.io/keycloak/keycloak:26.0.6` on `keycloak-0`/`keycloak-1`,
|
||||
operator `quay.io/keycloak/keycloak-operator:26.0.6`. Every source claim below is read from the
|
||||
`26.0.6` tag, not from memory or from docs.
|
||||
|
||||
**Every provider ID is quoted from upstream source, not guessed.**
|
||||
|
||||
| String I used | Where it is defined at tag 26.0.6 |
|
||||
|---|---|
|
||||
| `webauthn-authenticator-passwordless` | `WebAuthnPasswordlessAuthenticatorFactory.PROVIDER_ID`, display "WebAuthn Passwordless Authenticator" |
|
||||
| `auth-username-form` | `UsernameFormFactory.PROVIDER_ID`, display "Username Form" |
|
||||
| `auth-password-form` | `PasswordFormFactory.PROVIDER_ID`, display "Password Form" |
|
||||
| `webauthn-register-passwordless` | `WebAuthnPasswordlessRegisterFactory.PROVIDER_ID` (referenced in comments only) |
|
||||
|
||||
**`requiredActions` is REPLACE, not merge — measured, and it is why I did not declare one.**
|
||||
`model/storage-private/.../DefaultExportImportManager.java` lines 327-338:
|
||||
|
||||
```java
|
||||
if (rep.getRequiredActions() != null) {
|
||||
for (RequiredActionProviderRepresentation action : rep.getRequiredActions()) { ... }
|
||||
DefaultRequiredActions.addDeleteAccountAction(newRealm);
|
||||
} else {
|
||||
DefaultRequiredActions.addActions(newRealm);
|
||||
}
|
||||
```
|
||||
|
||||
Declaring one required action therefore skips the entire default set. `RealmManager` is no
|
||||
backstop: `setupRequiredActions` is `if (realm.getRequiredActionProvidersStream().count() == 0)`,
|
||||
and the count is already non-zero by then (`RepresentationToModel.importRealm` runs at
|
||||
`RealmManager.java:585`, the backstop at `:612`). A realm built that way would have had the passkey
|
||||
action plus `delete_account` and nothing else — **no `CONFIGURE_TOTP`**, which would leave
|
||||
`auth-otp-form` demanding a TOTP that no platform-admin could ever enrol: a hard lock-out
|
||||
delivered by a change whose purpose was to *add* a login method.
|
||||
|
||||
**And declaring it would have bought nothing.** `DefaultRequiredActions.addWebAuthnPasswordlessRegisterAction`
|
||||
registers `webauthn-register-passwordless` with `setEnabled(true)`, `setDefaultAction(false)`,
|
||||
priority 80, gated on `Profile.Feature.WEB_AUTHN` — which is `Type.DEFAULT` in
|
||||
`common/.../Profile.java:74`, i.e. on unless explicitly disabled. `enabled=true` /
|
||||
`defaultAction=false` *is* the additive-and-optional posture this change wants.
|
||||
|
||||
**`authenticationFlows` has the opposite semantics, which is why declaring flows is safe.**
|
||||
`importAuthenticationFlows` adds the declared flows, then backfills each unbound binding
|
||||
(browser, registration, direct grant, reset credentials, client auth, first broker login, docker
|
||||
auth) from the built-in defaults individually (`DefaultExportImportManager.java` ~1320-1410).
|
||||
|
||||
**The graceful degradation to password is measured from the flow engine, not assumed.** For a user
|
||||
with no passkey: `WebAuthnPasswordlessAuthenticator.requiresUser()` returns `false`, so
|
||||
`DefaultAuthenticationFlow.processSingleFlowExecutionModel` takes the `else` branch (`daf:456-462`),
|
||||
whose four conditions all hold — a user is set (from `auth-username-form`), `configuredFor` is
|
||||
false, `WebAuthnPasswordlessAuthenticatorFactory.isUserSetupAllowed()` is `false`, and
|
||||
`WebAuthnAuthenticator implements Authenticator, CredentialValidator<...>` — so it raises
|
||||
`CREDENTIAL_SETUP_REQUIRED`. The ALTERNATIVE loop (`daf:289-302`) catches exactly that, marks the
|
||||
execution `ATTEMPTED` and proceeds to the next alternative, with the upstream comment *"the user,
|
||||
since he has alternatives, should be able to go to another alternative and continue"*. Hence: no
|
||||
passkey, no passkey screen, straight to the password form.
|
||||
|
||||
**Why usernameless was rejected, measured.** The `configuredFor` guard sits inside the
|
||||
`requiresUser()` branch. With `requiresUser()` false and no user yet in context, that guard is
|
||||
skipped and the engine just calls `authenticate(context)` — so a passkey execution placed before
|
||||
the username form is offered to *everyone*, including users who will never own one.
|
||||
|
||||
**The policy sentinels are read from the shipped code, and one of my initial assumptions was
|
||||
wrong.** `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED = "not specified"` is the
|
||||
realm-representation sentinel — but `WebAuthnConstants.OPTION_NOT_SPECIFIED = ""` (empty string) is
|
||||
the WebAuthn-API-level one. I checked both because they disagree. The login theme's
|
||||
`webauthnRegister.js` at 26.0.6 settles the behaviour:
|
||||
|
||||
```js
|
||||
if (input.requireResidentKey !== 'not specified') {
|
||||
if (input.requireResidentKey === 'Yes') { authenticatorSelection.requireResidentKey = true; }
|
||||
else { authenticatorSelection.requireResidentKey = false; }
|
||||
}
|
||||
...
|
||||
if (input.createTimeout !== 0) { publicKey.timeout = input.createTimeout * 1000; }
|
||||
rp: {id: input.rpId, name: input.rpEntityName}
|
||||
```
|
||||
|
||||
So `"Yes"` is a literal string compare (any other non-sentinel value means *false*), `"not
|
||||
specified"` is compared literally for four fields, and `createTimeout` is **seconds** with `0`
|
||||
meaning "send no timeout". It also shows 26.0.6 emits only the legacy `requireResidentKey` member,
|
||||
never `residentKey: "required"` — the client maps that per WebAuthn L2, but the policy cannot
|
||||
express `residentKey: "preferred"` at all.
|
||||
|
||||
`[ES256, RS256]` is exactly `WebAuthnPolicy.DEFAULT_POLICY` /
|
||||
`Constants.DEFAULT_WEBAUTHN_POLICY_SIGNATURE_ALGORITHMS` — declared explicitly, not changed.
|
||||
`getWebAuthnPolicyPasswordless` was read field by field to confirm `[]` for `acceptableAaguids`
|
||||
means "no restriction" and that `RpEntityName` must be non-empty or the **update** path
|
||||
(`DefaultExportImportManager.java:842`) drops the whole policy silently.
|
||||
|
||||
**The manifest parses and renders.** `yaml.safe_load` round-trips the file and the flow graph reads
|
||||
back exactly as designed (five flows, the executions and priorities above, `browserFlow:
|
||||
browser-mfa`, `requiredActions` absent). `python3 ci/render-keycloak-client-urls.py --check` →
|
||||
`client-urls.yaml is in sync`, rc 0 (no client URL changed, so no regeneration was needed).
|
||||
`yamllint -c ci/yamllint.yaml platform/keycloak/keycloak-realm-import.yaml` → rc 0; no line exceeds
|
||||
the 160-column ceiling.
|
||||
|
||||
**`browser-mfa` keeps its alias deliberately.** `conformance/tests/l2-behavioral/keycloak-config-as-code`
|
||||
asserts `kubectl get keycloakrealmimport homelab -o jsonpath={.spec.realm.browserFlow} | grep -q
|
||||
'browser-mfa'`. Renaming the top-level flow would have failed that test.
|
||||
|
||||
**Envsubst safety.** `platform-config`'s Flux Kustomization declares
|
||||
`postBuild.substituteFrom: cluster-vars` (`clusters/base/platform-config.yaml:26-29`), so this file
|
||||
is substituted. The only brace token I added is `${BASE_DOMAIN}`, which is declared at
|
||||
`clusters/base/cluster-vars.yaml:42` (`home.keithjackson.dev`). Every comment I wrote avoids braces
|
||||
entirely — the RP-ID discussion says `keycloak.<BASE_DOMAIN>` with angle brackets precisely so an
|
||||
unparseable token cannot fail the whole Kustomization.
|
||||
|
||||
**Gate result: rc 3, zero failures, `gitea-settings` the only stage the gate counts as not-run** —
|
||||
the documented honest best local outcome. Run in the pinned toolbox from this worktree with all
|
||||
three §3 workarounds (`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main
|
||||
checkout's `.git` and the sibling spec repo with `SPEC_REPO=/home-network-k3s`):
|
||||
|
||||
```
|
||||
[ ok ] stage spec-immutability: pass <- NOT skipped; both shim mounts worked, R-TEST-06 covered
|
||||
[ ok ] stage decisions-log: pass <- archive frozen, 48 entries well-formed
|
||||
[ ok ] stage lint: pass
|
||||
[ ok ] stage l0: pass <- incl. flux-postbuild-substitution + keycloak-client-secret-reconciler
|
||||
[ ok ] stage render-golden: pass
|
||||
[ ok ] stage l1: pass
|
||||
[ ok ] stage approval-policy: pass <- no sensitive path touched, so no non-author review needed
|
||||
[skip] stage gitea-settings: SKIPPED <- needs SETTINGS_API_TOKEN, CI-only; expected
|
||||
[skip] stage conformance: SKIPPED <- cluster-gated L2/L3; expected
|
||||
MERGE GATE: UNDETERMINED — Stages that did not run: gitea-settings
|
||||
```
|
||||
|
||||
No lint finding and no check failure names either file I changed. Touched requirements resolved to
|
||||
`R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06`; coverage 51/51, 0 gaps.
|
||||
|
||||
**A note on how that number was obtained, because the first attempt was worthless.** My first gate
|
||||
run returned the same rc 3 and the same seven passes — against a tree that did not contain this
|
||||
change at all. The gate's `tracked_files()` reads **git**, and my edits were still uncommitted, so
|
||||
every stage gated `origin/main`. The tell was in the log and I nearly missed it: the
|
||||
touched-requirement subset reported UNDETERMINED with *"check out the feature branch you mean to
|
||||
gate (not the base branch)"*, and `decisions-log` counted **47** entries rather than 48. **A green
|
||||
gate run on an uncommitted worktree is a gate run on the base branch.** The numbers above are from
|
||||
the second run, after committing and rebasing.
|
||||
|
||||
**The base moved under me mid-run.** `origin/main` advanced from `9a41a8e` to `1a0b208` (PR #70,
|
||||
oauth2-proxy OIDC startup race) while I was working. This branch is rebased onto `1a0b208`, so the
|
||||
gate above ran on the base CI will actually merge into. The rebase was clean — PR #70 touched
|
||||
`platform/oauth2-proxy/helmrelease.yaml` and its own decisions entry, disjoint from these files.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
**This change is unapplied. It changes nothing about the live realm, and I did not attempt to make
|
||||
it.** `Done=True` / `Started=False` was measured, not inferred, and I ran no mutating `kubectl`,
|
||||
no `flux reconcile`, no admin-API call and no `kcadm`. Whether the operator's eventual
|
||||
admin-console reconstruction of this flow matches what is declared here is, by construction,
|
||||
unverifiable from git.
|
||||
|
||||
**I never enrolled a passkey and never pointed a browser at any of this.** Nothing here has been
|
||||
exercised by a real WebAuthn ceremony on any device. Specifically unverified:
|
||||
|
||||
* **That the RP ID works.** `home.keithjackson.dev` is a registrable-domain suffix of
|
||||
`keycloak.home.keithjackson.dev`, so it *should* satisfy the browser's check — but I verified
|
||||
that by reading the WebAuthn rule, not by watching a browser accept it. A wrong RP ID fails
|
||||
**silently, in the browser, before Keycloak is involved**, with no server-side log line. If
|
||||
enrolment produces no OS prompt and no error, this is the first suspect.
|
||||
* **That `"not specified"` is the right literal for `authenticatorAttachment` in a realm import.**
|
||||
I traced it to `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED` and to the theme's string
|
||||
compare, but I have not seen a realm import accept it. Omitting the field would default to the
|
||||
same value; if an import ever rejects the literal, drop the line.
|
||||
* **That `createTimeout: 300` behaves.** Seconds-vs-milliseconds is read from the theme JS. The
|
||||
claim that 300 is long enough for the cross-device QR/hybrid flow, and that Chrome clamps rather
|
||||
than rejects it, is reasoning about browser behaviour, not a measurement.
|
||||
* **Whether a passkey user can still reach the password form.** I argued it from
|
||||
`createAuthenticationSelectionList` producing a "Try another way" option, and I noted that for a
|
||||
user holding both credentials the *first* screen is chosen by the user's own credential ordering
|
||||
rather than by my priorities. Neither was observed.
|
||||
* **The two-screen login regression for ordinary users.** `auth-username-form` +
|
||||
`auth-password-form` replacing `auth-username-password-form` is a visible change for every user
|
||||
including non-passkey family accounts, and nobody has logged in through it.
|
||||
* **That conditional OTP for platform-admins still fires on the password branch.** The sub-flow and
|
||||
its `conditional-otp-platform-admin` config are byte-identical and merely relocated one level
|
||||
deeper, but "identical YAML" is not "still enforced at runtime". This is the highest-value
|
||||
regression check and it is untested. The previous breakage of this exact step
|
||||
(`condUserRole` vs a non-existent property key) 403'd every platform-admin login.
|
||||
* **That a platform-admin authenticating by passkey is not challenged for TOTP.** That is the
|
||||
intended behaviour of the shape I wrote; it is unobserved, and it is also a deliberate
|
||||
relaxation of R-IDP-04 as previously implemented, which the operator may want to reverse.
|
||||
|
||||
**Whether the realm import reconciles into an existing realm: measured as NO for the completed
|
||||
one-shot, but I did not test a re-import.** I did not delete and recreate the
|
||||
`KeycloakRealmImport`, so I cannot say what a re-run would do to the existing `homelab` realm —
|
||||
whether it would merge the new policy and flows, duplicate flow aliases, or fail. The 2026-09-06
|
||||
total SSO outage documented in `platform/keycloak/README.md` is the reason I did not experiment.
|
||||
**Treat the re-import path in the README as untested advice, not a runbook.**
|
||||
|
||||
**Nothing was proved about the live realm's current state.** I did not read the realm's existing
|
||||
required actions, its existing WebAuthn policies, or its live flow executions — doing so needs the
|
||||
admin credential and I did not decrypt or use it. So I cannot confirm that the live realm's
|
||||
required-action set actually contains `webauthn-register-passwordless` today; I can only show that
|
||||
Keycloak 26.0.6's defaults register it and that this repo's import never declared any required
|
||||
actions to suppress them. **That inference has not been checked against the realm.**
|
||||
|
||||
**L2/L3 conformance did not run** (no cluster from the gate container; `vars.CLUSTER_AVAILABLE` is
|
||||
unset in CI anyway), so the existing `keycloak-config-as-code` behavioural test did not re-execute
|
||||
against this change. I reasoned about its assertion by reading it.
|
||||
|
||||
**No new conformance test was added.** Nothing in `conformance/` asserts the passkey policy or the
|
||||
new flow shape, so a future edit can silently remove either and no gate stage will notice. That is
|
||||
a real gap this PR leaves open; closing it would touch `conformance/**`, which is a sensitive path
|
||||
needing non-author review.
|
||||
|
||||
**The quoted gate run predates this file's own gate-result paragraph.** It gated commit `207f5db`;
|
||||
the gate numbers and the rebase note above were then written into this entry, so the pushed tree
|
||||
differs from the gated one by this handoff's prose only — no manifest, no check, no generated file.
|
||||
`decisions-log` is the only stage that reads this file, and it validates structure (frontmatter,
|
||||
the three headings, non-stub bodies), which is unchanged. I did not re-run the full gate for a
|
||||
prose edit; CI will gate the pushed commit.
|
||||
|
||||
**`gitea-settings` did not run locally** — it needs `SETTINGS_API_TOKEN`, which exists only in CI.
|
||||
I did not re-measure branch protection, the runner, or `required_approvals` against the Gitea API
|
||||
this session; the claims in `CLAUDE.md` §2 are taken on trust here and are explicitly flagged there
|
||||
as needing re-measurement.
|
||||
@@ -1,108 +0,0 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: fix/coredns-ha
|
||||
title: "coredns ran at one replica, so any node drain was a cluster-wide DNS outage"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**MEASURED on prod 2026-10-18: `kube-system/coredns` had `replicas: 1`.** In-cluster DNS was a
|
||||
single pod on a single node, so draining or rebooting that node was a **cluster-wide in-cluster DNS
|
||||
outage** until the pod rescheduled and became ready — every Service lookup, every CNPG replica
|
||||
finding its primary, every operator addressing the API by name. It was invisible because `1/1
|
||||
Ready` looks perfectly healthy.
|
||||
|
||||
It also became urgent rather than latent: the scheduled reboot automation on
|
||||
`feat/automated-upgrades` drains and reboots one node at a time, unattended, overnight — turning a
|
||||
single point of failure into a recurring one.
|
||||
|
||||
New `platform/coredns/hpa.yaml`: a `HorizontalPodAutoscaler` on the coredns Deployment with
|
||||
`minReplicas: 2`, `maxReplicas: 3`, registered in the existing `platform/coredns/kustomization.yaml`
|
||||
alongside the split-horizon ConfigMap already there.
|
||||
|
||||
**Why an HPA and not `replicas: 2` — this is the whole decision.** k3s owns the coredns Deployment
|
||||
and we cannot durably edit it: it is applied by k3s's own deploy controller from a bundled manifest
|
||||
(`AddOn/coredns`, source `/data/k3s/server/manifests/coredns.yaml`, `managedFields` managers
|
||||
`deploy@k3s-prod-1` and `k3s`, plus an `objectset.rio.cattle.io/hash` label). k3s re-stages its
|
||||
bundled manifests on startup, so a `replicas: 2` written there — or a `kubectl scale` — is reverted
|
||||
on the next k3s restart or upgrade. Two alternatives were weighed and rejected:
|
||||
|
||||
* **`coredns.yaml.skip` + our own templated copy** (the pattern `roles/kube_vip` already uses for a
|
||||
file in `server/manifests`). Deterministic and never flaps, but it takes ownership of coredns away
|
||||
from k3s permanently — k3s upgrades would stop updating it and the version would need refreshing
|
||||
by hand forever. Too high a price for a replica count.
|
||||
* **A Flux-managed patch of the Deployment.** Rejected outright: Flux and k3s's deploy controller
|
||||
would both own one object and fight continuously — the exact shape of the `default-deny`
|
||||
NetworkPolicy drift loop this cluster already suffered (~2500 events/10min).
|
||||
|
||||
An HPA avoids both: a separate object k3s does not own, driving the `/scale` subresource rather than
|
||||
the Deployment spec, so there is no shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA
|
||||
sees it below `minReplicas` and scales back within one reconcile.
|
||||
|
||||
It is a **floor-holder, not an autoscaler**: the CPU metric exists only because an `autoscaling/v2`
|
||||
HPA needs one to be valid. `scaleDown.stabilizationWindowSeconds: 600` because DNS is the worst
|
||||
thing in the cluster to flap; `scaleUp` is immediate, since that is the path that restores redundancy
|
||||
after a k3s restart.
|
||||
|
||||
**Immediate action taken outside this PR:** `kubectl -n kube-system scale deploy coredns
|
||||
--replicas=2` was run at the operator's explicit request, so the cluster has two replicas **now**
|
||||
rather than after merge. That hand scale is what this PR makes durable.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **The defect:** `kubectl -n kube-system get deploy coredns` → `replicas=1 available=1`.
|
||||
* **k3s ownership:** the `AddOn/coredns` object exists with source
|
||||
`/data/k3s/server/manifests/coredns.yaml`; the Deployment has no `ownerReferences` but carries
|
||||
`objectset.rio.cattle.io/hash` and `managedFields` managers `deploy@k3s-prod-1` / `k3s`. So it is
|
||||
k3s-managed, not Flux-managed — independently corroborated by another agent, which confirmed
|
||||
`platform/coredns/` contains **only** the `coredns-custom` ConfigMap and does not touch the
|
||||
Deployment or its replica count.
|
||||
* **The hand scale works and holds:** coredns went to `2/2 Running` on two different nodes, and
|
||||
in-cluster DNS kept answering throughout (`nslookup kubernetes.default.svc.cluster.local
|
||||
10.43.0.10` → `10.43.0.1`). Still `spec.replicas=2 ready=2` after ~2 minutes, with the AddOn
|
||||
checksum **unchanged** — so k3s's deploy controller does not revert on a drift timer; it
|
||||
re-applies on manifest change and on restart.
|
||||
* **The HPA's preconditions hold:** coredns ships `requests: {cpu: 100m, memory: 70Mi}` (so a
|
||||
Resource/Utilization metric is meaningful), and metrics-server is installed as a k3s addon and
|
||||
verified serving — `kubectl top pod -l k8s-app=kube-dns` returns **5m and 4m** against that 100m
|
||||
request, i.e. ~5% utilisation, confirming the 80% target will never be approached and the HPA will
|
||||
sit at `minReplicas`.
|
||||
* **The kustomization edit is additive:** `git diff --stat` is `1 file changed, 4 insertions(+)`,
|
||||
and `- coredns` was already registered in `platform/kustomization.yaml`, so no change was needed
|
||||
there.
|
||||
* All three touched/added YAML files parse.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **The HPA has never been applied.** I did not reconcile it, so I have **not** observed it create,
|
||||
reach `ScalingActive=True`, resolve its metric against real coredns data, or hold `minReplicas`.
|
||||
Applying is an operator action (CLAUDE.md §5). The failure mode to watch for is
|
||||
`ScalingActive=False` with `FailedGetResourceMetric` — the HPA would then be present and inert,
|
||||
which looks identical to working because the hand scale already left 2 replicas in place. **Check
|
||||
`kubectl -n kube-system describe hpa coredns` after merge**, not just the replica count.
|
||||
* **The central claim that k3s reverts `replicas` on restart is REASONED, NOT OBSERVED.** It rests
|
||||
on k3s re-staging bundled manifests at startup plus the measured ownership evidence above. I did
|
||||
**not** restart k3s to watch the revert happen — doing so is a rolling control-plane restart, an
|
||||
operator action. If k3s in fact preserves a modified replica count, this HPA is unnecessary
|
||||
(though harmless).
|
||||
* **I did not confirm the HPA wins the race against k3s's deploy controller after a restart.** Both
|
||||
write the scale; the expectation is the HPA re-asserts within a reconcile, leaving a brief window
|
||||
at 1 replica. Neither the window's length nor the outcome has been measured.
|
||||
* **I nearly shipped a destructive change.** My first write of
|
||||
`platform/coredns/kustomization.yaml` replaced the file wholesale, listing only `hpa.yaml` — which
|
||||
would have pruned the existing `coredns-custom` ConfigMap and **broken in-cluster split-horizon
|
||||
DNS (R-DNS-01/03)**. It was caught by a guard assertion before commit and restored from `HEAD`, and
|
||||
the diff is now additive. Recorded because the near-miss is the useful part: `platform/coredns/`
|
||||
already existed and I assumed it did not. **No conformance check would obviously have caught the
|
||||
removal** — it is valid YAML and a legitimate-looking prune.
|
||||
* **Whether 2 is enough.** Three nodes, and coredns' anti-affinity only *prefers* spreading, so both
|
||||
replicas could in principle land on one node and a single drain would still take DNS down. I did
|
||||
not add a `topologySpreadConstraint` or a PodDisruptionBudget — both arguably belong here and
|
||||
neither is in this change.
|
||||
* **No alert fires if coredns drops to 1 replica**, so a future silent regression (k3s upgrade
|
||||
changing the manifest, the HPA going inert) would not be reported. That gap is not closed here.
|
||||
* **No conformance test asserts this**, so a future edit could remove the HPA with no gate stage
|
||||
noticing.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
|
||||
subset only.
|
||||
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
|
||||
@@ -1,76 +0,0 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: fix/ha-check-catastrophic-regex
|
||||
title: "One regex made the merge gate take 28 minutes instead of 2"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The merge gate started taking **~28 minutes** instead of its usual **<2 minutes**, on every change
|
||||
touching `infra/**`. One regex was responsible for essentially all of it.
|
||||
|
||||
`conformance/tests/l0-static/ha-control-plane-as-code/check.sh` contained:
|
||||
|
||||
```python
|
||||
join_when = re.search(r"--server https://.*?\n(?:.*\n){0,10}?\s*when:\s*k3s_server_mode\s*==\s*'secondary'",
|
||||
K3S_TEXT, re.S)
|
||||
```
|
||||
|
||||
With `re.S`, every `.` also matches newlines, so `.*?` and the nested `(?:.*\n){0,10}?` can each
|
||||
consume the entire document. That is **nested unbounded repetition** — ten levels of a quantifier
|
||||
that can each swallow everything — and a failed or late match backtracks exponentially.
|
||||
|
||||
Replaced `.` with `[^\n]*` and dropped `re.S`, so the `{0,10}` is a genuine ten-line window with
|
||||
nothing to backtrack into. Same intent, same verdict, no DOTALL.
|
||||
|
||||
**Why it surfaced now, and it was mine.** `K3S_TEXT` is every `*.yml` in `roles/k3s_server`
|
||||
concatenated. The regex was latent for as long as it existed; **PR #71 added ~100 lines to that role
|
||||
(the kubelet graceful-shutdown config), which grew the input past the point where exponential
|
||||
backtracking stopped being survivable.** Nothing about the check changed — my change grew its input.
|
||||
|
||||
**Why only `infra/**` changes were slow.** `ci/touched-requirements.sh:268` maps `infra/*` to
|
||||
`R-ENV-01`, and the resulting L0 run is `--include-scope whole-repo --requirements R-ENV-01`, which
|
||||
pulls in this whole-repo check. Doc-only and `platform/**` changes never selected it, which is why
|
||||
#69 (1m55s) and #72 (1m53s) stayed fast while #71 (27m44s), the #73 merge (27m46s) and #75 all ran
|
||||
~28 minutes.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **The hot spot, by measurement rather than reading.** `docker top` on the live CI job container for
|
||||
run #299 showed `python3 -` with **19:54 of CPU time** under
|
||||
`ha-control-plane-as-code/check.sh`, inside
|
||||
`conformance/run.sh --layer l0 --include-scope whole-repo --requirements R-ENV-01`.
|
||||
* **Reproduced locally**: the check did not finish within a 100-second budget on an unmodified
|
||||
checkout.
|
||||
* **Localised to the exact line** by extracting the inline Python and running it under
|
||||
`faulthandler.dump_traceback_later(25)`, which named `re/__init__.py:177 in search` called from
|
||||
the `join_when` line — not inferred from reading the patterns.
|
||||
* **The pattern is unique in the repo**: `grep -rlnE "\{0,[0-9]+\}\?" conformance/tests/l0-static/*/check.sh`
|
||||
returns only this file, so there is no sibling with the same hazard.
|
||||
* **The fix is fast AND verdict-preserving**, tested against the real `K3S_TEXT` (22,179 chars, 403
|
||||
lines): the new regex matches (`YES`) in **0.2 ms**. Sanity-checked that the thing it looks for
|
||||
genuinely exists — one `--server https://` and three `k3s_server_mode == 'secondary'` occurrences.
|
||||
* **The whole check now runs in 0.199s, rc 0, 115 assertions**, with the same success message as
|
||||
before.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **I did not re-time the full gate in CI after the fix.** The expectation is that `infra/**` changes
|
||||
return to the ~2-minute range, but that is arithmetic from the 19m54s figure, not a measured
|
||||
post-fix run. The PR's own CI run is the first real datapoint — read it.
|
||||
* **I did not audit the other 23 L0 checks for different performance pathologies.** This one was
|
||||
found because it was actively burning CPU in a live job; a check that is merely slow (not
|
||||
exponential) would not have stood out, and `ci/gate.sh` has **no per-stage timing** at all
|
||||
(`grep -nE "SECONDS|date \+|elapsed"` finds nothing), so there is no data on where the remaining
|
||||
time goes.
|
||||
* **Nothing prevents this class of bug returning.** No check asserts gate stage runtime, and no lint
|
||||
forbids nested quantifiers under `re.S`. A timing budget per stage, or a `re` lint, would close it;
|
||||
neither is written here.
|
||||
* **I did not verify my claim about *why* only `infra/**` was affected beyond the path mapping and
|
||||
the observed durations.** The correlation is strong (5 runs, two clear clusters) and the live
|
||||
process list confirms the `--requirements R-ENV-01` scope, but I did not read
|
||||
`conformance/run.sh`'s scope-selection logic end to end.
|
||||
* The regex rewrite assumes the `when:` clause is indented with spaces or tabs only (`[ \t]*`). A
|
||||
file using some other whitespace before `when:` would now fail to match where the old DOTALL
|
||||
version would have succeeded. Unlikely in YAML, and the current input matches, but it is a
|
||||
narrowing.
|
||||
@@ -1,82 +0,0 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: fix/pihole-local-ttl
|
||||
title: "The split-horizon answer was uncacheable while the wrong answer cached for 5 minutes"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator reported DNS "flapping between the cluster ingress and the router (public IP)".
|
||||
PiHole was not at fault — both replicas answer correctly and consistently. The bug is an
|
||||
**asymmetry in cacheability** that we created.
|
||||
|
||||
Measured on prod 2026-10-18:
|
||||
|
||||
```
|
||||
dig home.keithjackson.dev @192.168.2.23 (pihole) -> 192.168.2.20 TTL=0
|
||||
dig home.keithjackson.dev @1.1.1.1 (public) -> wan.home.keithjackson.dev TTL=300
|
||||
dig home.keithjackson.dev @192.168.2.1 (router) -> wan.... TTL=153 -> 148 -> 143
|
||||
```
|
||||
|
||||
dnsmasq answers `address=` entries with **TTL 0** unless told otherwise. The SOHO router forwards
|
||||
LAN queries to PiHole and **caches** the result — the decrementing TTL above is the proof it caches
|
||||
rather than forwarding live. So:
|
||||
|
||||
* the **correct** answer is **uncacheable** (TTL 0): the router must re-query PiHole on *every*
|
||||
lookup, which is maximum exposure to any momentary gap;
|
||||
* the **wrong** answer is **cacheable for 300s**: the instant the router falls back to its upstream
|
||||
(standard forwarder behaviour when the forward target does not answer) it latches the public WAN
|
||||
address and serves it to **every LAN client** for five minutes.
|
||||
|
||||
So **one sub-second PiHole gap becomes five minutes of the whole house resolving to the WAN IP** —
|
||||
then it expires, the router re-queries PiHole, gets TTL 0 again, and is immediately re-exposed.
|
||||
That asymmetry, not the gap itself, is why this is visible as flapping rather than a blip.
|
||||
|
||||
Fix: `local-ttl=60` prepended to `FTLCONF_misc_dnsmasq_lines` in `catalog/pihole.yaml`, so the
|
||||
private answer is cacheable and a short gap is invisible to clients. `apps/wave-0/pihole.yaml`
|
||||
regenerated with the canonical render command.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **PiHole is NOT the fault.** Both replicas, queried directly by pod IP, answer `192.168.2.20`;
|
||||
the VIP returned it 6/6; this workstation 8/8. DHCP hands out only `192.168.2.23`.
|
||||
* **The router caches rather than forwards live** — three successive queries returned TTL 153, 148,
|
||||
143 (decrementing), which a pure forwarder could not do.
|
||||
* **The TTL asymmetry**, as quoted above: 0 from PiHole, 300 from public DNS.
|
||||
* **The router does answer DNS and returns the public chain** (`wan.home.keithjackson.dev` → the WAN
|
||||
address), so a fallback always has a wrong answer available.
|
||||
* **`local-ttl` is the right directive**: dnsmasq applies it to replies made from configuration,
|
||||
`/etc/hosts` and DHCP leases, which covers all five `address=` names with one directive. It is a
|
||||
standalone option, not a sub-field of `address=`, so prepending it to the `;`-separated
|
||||
`FTLCONF_misc_dnsmasq_lines` list is syntactically correct.
|
||||
* **The render is clean and scoped**: `catalog/render.py --all catalog -o apps --wave-layout
|
||||
--no-image-automation` (the canonical command `ci/render-golden.sh` asserts against, and
|
||||
deliberately **not** `--emit-waves`) changed exactly two files — `catalog/pihole.yaml` and
|
||||
`apps/wave-0/pihole.yaml` — and the rendered diff is the single `local-ttl=60;` prefix.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **UNAPPLIED.** Nothing has been reconciled, so I have not observed PiHole serving a non-zero TTL,
|
||||
nor the router caching the private answer, nor the flap stopping. **Verify after merge with
|
||||
`dig home.keithjackson.dev @192.168.2.23` (expect `TTL=60`, not 0) and then the same query at the
|
||||
router (expect `192.168.2.20` with a decrementing TTL, not `wan.…`).**
|
||||
* **I did not confirm that PiHole/FTL accepts `local-ttl` via `FTLCONF_misc_dnsmasq_lines`.** The
|
||||
directive is valid dnsmasq and the field is a passthrough, but `catalog/pihole.yaml`'s own header
|
||||
records that FTL has previously written a bogus line into `dnsmasq.conf` and reported "0 invalid"
|
||||
— so a malformed passthrough fails *silently*. If the TTL is still 0 after merge, that is the
|
||||
cause, not this reasoning. Check the pod's `/etc/pihole/dnsmasq.conf`.
|
||||
* **60s is chosen, not derived.** Long enough for a pod restart and a lease re-election (both
|
||||
seconds); it does **nothing** for an outage longer than the TTL, where the cache expires and the
|
||||
public answer latches again. That is a real availability problem, not a caching one.
|
||||
* **This does not remove the trigger.** `192.168.2.23` is served by **two** LoadBalancer Services
|
||||
(`pihole-dns-tcp`, `pihole-dns-udp`) with **two independently-elected L2 announcement leases**, so
|
||||
two nodes can ARP for the same address at once — they sat on different nodes earlier the same day
|
||||
(tcp on prod-2, udp on prod-1). The same pattern applies to crafty (`.22`) and unifi (`.21`).
|
||||
Merging each pair into one mixed-protocol Service would remove the class; the operator has parked
|
||||
that decision, and it is deliberately **not** in this change.
|
||||
* **Client-side bypasses are unexamined.** A device with its own resolver, or a browser with
|
||||
DNS-over-HTTPS, bypasses PiHole entirely and always gets the public answer; this fix cannot help
|
||||
there. Only this one workstation's resolver chain was inspected.
|
||||
* **No alert fires if the split-horizon answer regresses to the public address.** Nothing probes
|
||||
`home.keithjackson.dev` from a LAN vantage point and compares it against `${GATEWAY_VIP}`, so the
|
||||
next recurrence would again be found by a human noticing. That gap is not closed here.
|
||||
@@ -1,78 +0,0 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: fix/renovate-cron-window
|
||||
title: "The Renovate cron fired outside Renovate's own schedule, so it would never have opened a PR"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`.gitea/workflows/renovate.yaml` shipped (PR #73) with `cron: "15 3 * * *"`. **That schedule could
|
||||
never have produced a single PR.**
|
||||
|
||||
Gitea Actions cron is **UTC**. `renovate.json` carries `"schedule": ["after 3am and before 6am"]`
|
||||
with `"timezone": "America/New_York"`, and that inner schedule is **authoritative** — Renovate
|
||||
re-checks it and declines to open PRs outside it. So:
|
||||
|
||||
```
|
||||
cron 03:15 UTC -> 23:15 EDT, the PREVIOUS EVENING in New York
|
||||
window -> 03:00-06:00 local
|
||||
inside? -> NO (3h45m before the window opens)
|
||||
```
|
||||
|
||||
The original comment claimed the cron was "slightly ahead of the repo window ... so a run is always
|
||||
in flight when the window opens". It is 3h45m *behind*, on the previous day. Every nightly run would
|
||||
have started, logged "not within schedule", opened **zero** PRs and exited **successfully** — a
|
||||
green tick forever, indistinguishable from "nothing needed updating". That is the same
|
||||
"failure-mode-is-a-green-tick" shape this repo keeps hitting, and it would have hidden the ~30
|
||||
pending updates PR #73 was written to surface.
|
||||
|
||||
Changed to `cron: "15 8 * * *"`, chosen to survive DST rather than to be tidy. The window's UTC
|
||||
position moves:
|
||||
|
||||
| | local window | in UTC |
|
||||
|---|---|---|
|
||||
| EDT (UTC-4, Mar–Nov) | 03:00–06:00 | 07:00–10:00 |
|
||||
| EST (UTC-5, Nov–Mar) | 03:00–06:00 | 08:00–11:00 |
|
||||
|
||||
`08:15 UTC` is inside **both** (04:15 EDT / 03:15 EST), so no edit is needed at either switchover.
|
||||
A cron cannot follow DST, so this is the only way to get it right once.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **The defect, by arithmetic rather than assertion:** `datetime(2026,10,18,03,15, UTC)` converted to
|
||||
`America/New_York` via `zoneinfo` is `2026-10-17 23:15 EDT`; `3 <= hour < 6` is **False**.
|
||||
* **The fix, at both DST extremes:** `08:15 UTC` → `04:15 EDT` in July and `03:15 EST` in December,
|
||||
both inside `03:00–06:00`. Computed, not reasoned.
|
||||
* **The inner schedule really is authoritative and really is set:** `renovate.json` on `main` carries
|
||||
`timezone = America/New_York` and `schedule = ['after 3am and before 6am']`.
|
||||
* **`RENOVATE_TOKEN` now exists** as a repository Actions secret (listed by name via the Gitea API
|
||||
alongside `SETTINGS_API_TOKEN`), so the workflow's fail-hard preflight will pass. It did not exist
|
||||
when PR #73 was authored.
|
||||
* The edited workflow parses as YAML.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **Renovate has still never executed.** This fixes *when* it runs, not *whether it works*. No run
|
||||
has happened, so nothing here proves it extracts dependencies, authenticates to Gitea, or opens a
|
||||
usable PR.
|
||||
* **`enabledManagers` may silently disable the custom managers — UNRESOLVED and the likelier
|
||||
remaining bug.** `renovate.json` lists `enabledManagers: [..., 'regex', ...]` and defines
|
||||
`customManagers` (correct modern key; `regexManagers` is absent, which is right). But modern
|
||||
Renovate names the custom manager **`custom.regex`** in `enabledManagers`, not bare `regex`. If
|
||||
the bare name is invalid or ignored, every inline `# renovate:` marker in the tree does nothing —
|
||||
including the `external-snapshotter` `newTag` pin PR #73 flagged as probably inert. **I did not
|
||||
change it**, because guessing at a config key that could fail validation and take the whole run
|
||||
down is worse than leaving a question that one dry run answers definitively. The `dry_run: true`
|
||||
dispatch log settles it: look for a per-manager dependency-extraction count, and for any config
|
||||
validation warning naming `regex`.
|
||||
* **Whether a dry run inside vs outside the window differs usefully.** Renovate honours its schedule
|
||||
even in dry-run, so a manual dispatch outside 03:00–06:00 local will also log "not within
|
||||
schedule". That makes the first dispatch's output less informative than it looks; to see what
|
||||
Renovate *would* do, either dispatch inside the local window or override the schedule for that
|
||||
run. Not tested either way.
|
||||
* **No alert fires if the schedule stops firing.** PR #73 recorded this gap and it is still open: a
|
||||
Renovate workflow that silently stops running (disabled Actions, expired token, deleted secret)
|
||||
produces no signal at all. Closing it needs a check on last-successful-run age, which nothing
|
||||
does.
|
||||
* No conformance test asserts the cron falls inside `renovate.json`'s window, so this exact bug can
|
||||
return. A static check comparing the two is feasible and is not written.
|
||||
@@ -1,78 +0,0 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: fix/renovate-invalid-config
|
||||
title: "renovate.json was invalid, so the first manual run errored out"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator ran the Renovate workflow manually and it **failed**. The cause is not the workflow:
|
||||
**`renovate.json` was never a valid Renovate config.**
|
||||
|
||||
It carried five pseudo-comment keys — `_comment`, `_kubernetes_comment`, `_unmanageable_comment`,
|
||||
`_ignorePaths_comment`, `_docker_comment` — because JSON has no comments. Renovate **rejects unknown
|
||||
top-level options**, so the entire config failed validation. Proven with the official validator
|
||||
rather than inferred:
|
||||
|
||||
```
|
||||
$ podman run --rm -v "$PWD/renovate.json:/tmp/renovate.json:ro,Z" \
|
||||
ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json
|
||||
ERROR: Found errors in configuration
|
||||
Invalid configuration option: _comment
|
||||
Invalid configuration option: _docker_comment
|
||||
Invalid configuration option: _ignorePaths_comment
|
||||
Invalid configuration option: _kubernetes_comment
|
||||
Invalid configuration option: _unmanageable_comment
|
||||
```
|
||||
|
||||
This is the second independent reason Renovate had never worked, on top of PR #77's cron firing
|
||||
outside `renovate.json`'s own schedule window. Either alone was sufficient to produce "nothing ever
|
||||
happens"; #73 shipped both.
|
||||
|
||||
**Converted `renovate.json` → `renovate.json5`.** JSON5 permits real `//` comments, and Renovate
|
||||
discovers `renovate.json5` natively — nothing in `.gitea/` or `ci/` pins the old filename (only
|
||||
prose inside comments), and no `RENOVATE_CONFIG_FILE` is set. Every word of the original rationale
|
||||
is preserved as comments positioned next to what it explains, which a flattened `description`
|
||||
string would have lost.
|
||||
|
||||
A header on the new file records the failure, the validator command, and the instruction to run it
|
||||
before committing — this class of bug is invisible to the repo's own gate.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **The exact failure**, from the official validator, quoted above. Not read off a log, not inferred
|
||||
from the workflow.
|
||||
* **The fix validates**: the same validator against `renovate.json5` returns
|
||||
`INFO: Config validated successfully against 1 file(s)`.
|
||||
* **Nothing was dropped in the conversion**, checked programmatically against `git show
|
||||
HEAD:renovate.json`: 18 non-comment keys before, 18 after, **no missing keys, no extra keys, and
|
||||
no key whose value changed**. `docker: {enabled: true}` confirmed still present — the validator
|
||||
prints a normalisation diff that mentions it, which is its own migration suggestion, not a
|
||||
removal.
|
||||
* **74 comment lines survive** in the new file, so the rationale is intact.
|
||||
* **`RENOVATE_TOKEN` exists** as a repository Actions secret (listed by name via the Gitea API), so
|
||||
the workflow's fail-hard preflight is satisfied.
|
||||
* **A suspicion was DISPROVED:** `enabledManagers` containing bare `regex` was thought to need to be
|
||||
`custom.regex`, which would have silently disabled every inline `# renovate:` marker. The
|
||||
validator accepts `regex`, so that concern was unfounded and nothing here changes it.
|
||||
* Nothing functionally references `renovate.json` by name — `grep -rn "renovate\.json" .gitea/ ci/
|
||||
conformance/` returns only comment prose.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **Renovate has still never completed a run.** This removes one of two blockers; PR #77 removes the
|
||||
other. Neither is merged, so **no successful Renovate run has been observed** and I cannot say the
|
||||
bot works — only that these two specific defects are fixed.
|
||||
* **I did not re-run the workflow after the fix.** The honest test is a `workflow_dispatch` with
|
||||
`dry_run: true` once both this and #77 are merged. Expect ~30 dependencies behind, rate-limited to
|
||||
`prHourlyLimit: 2` / `prConcurrentLimit: 5`.
|
||||
* **Whether the managers actually extract anything.** A valid config that matches no files is still
|
||||
a no-op. The dry-run log's per-manager dependency counts are the only thing that proves the
|
||||
`managerFilePatterns` are right, and I have not seen them.
|
||||
* **Whether `renovate.json5` is honoured by the pinned Renovate version in the workflow.** JSON5
|
||||
discovery is long-standing and the validator accepted the file, but the workflow's image tag was
|
||||
not cross-checked against the version that added it.
|
||||
* **Nothing in the gate catches an invalid Renovate config.** This bug would have recurred silently.
|
||||
A one-line L0 check shelling out to `renovate-config-validator` would close it; it is not written
|
||||
here, and `conformance/**` is a sensitive path.
|
||||
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
|
||||
@@ -1,374 +0,0 @@
|
||||
---
|
||||
date: 2026-10-29
|
||||
branch: feat/image-pipeline
|
||||
title: "Images are built, verified and published by CI on rootless BuildKit; Renovate carries the digests into git"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Operator rule (2026-10-29): container image creation, hosting and lifecycle belong to CI/CD and
|
||||
the codebase, never to a one-off workstation build/push. Before this branch the Home-Network org's
|
||||
package list was **empty**, `platform/pm-agent/` pinned an all-zero digest (ImagePullBackOff), and
|
||||
both `build-image.sh` scripts said "publishing is an operator action".
|
||||
|
||||
### The builder: rootless BuildKit — DECISION MADE (operator, on PR #81)
|
||||
|
||||
**The problem it answers, measured on this PR's first CI run (images run 325, failed in 6 s):** a
|
||||
CI job on this runner gets **no Docker daemon at all**, for two reasons, both in the deployed
|
||||
versions' source:
|
||||
- act_runner v0.2.11 `internal/app/cmd/daemon.go` sets `container.docker_host` to `-` ("never
|
||||
mount into jobs") whenever `DOCKER_HOST` is `tcp://`, which it is here (the DinD sidecar);
|
||||
- gitea/act v0.261.2 `pkg/container/docker_run.go` `sanitizeConfig` drops **every** bind,
|
||||
socket included, when `valid_volumes` is empty, which it is (the runner has no config file).
|
||||
|
||||
Giving jobs the DinD daemon would have made every branch push root on the node, because that
|
||||
daemon is `privileged: true`.
|
||||
|
||||
**The decision:** no job may reach a privileged daemon, and images are built by a **rootless
|
||||
BuildKit** daemon over mTLS. The runner (`apps/wave-1/gitea-actions.yaml`, DOCKER_HOST,
|
||||
valid_volumes) is **untouched**.
|
||||
|
||||
- **`platform/buildkit/`** — a new hand-managed component in its own namespace `buildkit`. The
|
||||
runner manifest is renderer-generated (`catalog/render.py` EXTRA_UNITS), and the builder has no
|
||||
business sitting next to a privileged container. Flux reconciles it as the new Kustomization
|
||||
`buildkit` (`clusters/base/buildkit.yaml`, `dependsOn: platform`).
|
||||
- **Deployment:** `moby/buildkit:v0.34.1-rootless@sha256:f68752a5…0c4f89`, pinned by index
|
||||
digest. The securityContext, every line measured (see *What I verified*):
|
||||
- `privileged: false`;
|
||||
- uid/gid 1000, `runAsNonRoot`;
|
||||
- `capabilities: drop ALL, add SETUID, SETGID`;
|
||||
- `allowPrivilegeEscalation: true`;
|
||||
- `seccompProfile: Unconfined` and `appArmorProfile: Unconfined`;
|
||||
- `readOnlyRootFilesystem: true`, with five emptyDirs.
|
||||
|
||||
Other settings:
|
||||
- `--oci-worker-no-process-sandbox`;
|
||||
- no ServiceAccount token;
|
||||
- memory limit 4Gi, which covers build steps too;
|
||||
- cache emptyDir with `sizeLimit: 6Gi`, and GC capped at 4 GB in `buildkitd.toml`
|
||||
(ConfigMap).
|
||||
- **mTLS via cert-manager** (`certificates.yaml`): a self-signed CA used only here, the server
|
||||
cert `buildkitd-tls`, and the client cert `buildkit-client-ci`. buildkitd verifies every
|
||||
client against that CA.
|
||||
- **Service:** ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`.
|
||||
- **NetworkPolicies:**
|
||||
- default-deny;
|
||||
- ingress on 1234 **only from the act-runner pod** (`gitea-actions`,
|
||||
`app.kubernetes.io/name=act-runner`);
|
||||
- egress: DNS, world :443/:80 for downloads, and the `ingress` entity (the Gateway) for the
|
||||
registry push.
|
||||
- `versions.env`: the CI-side client, `buildx` v0.38.1 with the sha256 from its
|
||||
`checksums.txt`, plus the address and certificate name CI dials.
|
||||
- **How DinD job containers appear to Cilium (measured, read-only).**
|
||||
`iptables -t nat -S POSTROUTING` in the dind container shows
|
||||
`-A POSTROUTING -s 172.18.0.0/16 ! -o br-… -j MASQUERADE` (and the same for docker0). So a job's
|
||||
connection leaves with the **act-runner pod's IP, and therefore that pod's Cilium identity**.
|
||||
That makes "only the act-runner pod" expressible and correct, but it cannot tell one job from
|
||||
another, or a job from act_runner itself. Per-job authentication is the client certificate.
|
||||
- **Admission: no Kyverno exception needed.**
|
||||
- Every ClusterPolicy actually applied (`platform-config/kustomization.yaml`; live:
|
||||
`conformance-trigger-job-shape`, `label-cdi-worker-pods`, `pm-agent-job-shape`,
|
||||
`require-auth-mode-label`, `require-default-deny-baseline`, `require-egress-class-label`,
|
||||
`restrict-host-namespaces`, `secret-provenance`, `validate-default-deny-shape`) matches only
|
||||
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`, or only its own
|
||||
namespace in the job-shape policies' case. `buildkit` carries no such label.
|
||||
- There are no namespaced Kyverno Policies and no cluster-wide PSA default (both measured).
|
||||
- The namespace is deliberately left without a PSA label, because PSS `baseline` forbids
|
||||
`Unconfined` seccomp and AppArmor.
|
||||
- `validate-default-deny-shape` does apply to the namespace's default-deny NetworkPolicy, and
|
||||
its shape (both directions) satisfies it.
|
||||
- **Unprivileged user namespaces on the nodes (measured, read-only, via Ansible `sysctl`).** On
|
||||
all three Debian 13.7 nodes, kernel `7.1.8+deb13-amd64`:
|
||||
- `kernel.unprivileged_userns_clone = 1`;
|
||||
- `user.max_user_namespaces = 127206`;
|
||||
- no `kernel.apparmor_restrict_unprivileged_userns` (an Ubuntu-only key);
|
||||
- AppArmor enabled (`Y`).
|
||||
|
||||
So this is a satisfied prerequisite, not an open one.
|
||||
- **The TLS conformance check needed a narrow, self-proving exception** (`conformance/**`, so this
|
||||
changes what green means; flagged for the reviewer).
|
||||
- The first gate run failed L0 `tls-lifecycle-automated` clause (5): "every Certificate's
|
||||
issuer must be ACME". That is broader than the spec it serves:
|
||||
- R-TLS-01 and docs/08 concern *externally reachable* services;
|
||||
- ADR-0006 says "mTLS between workloads is a separate concern … not covered here";
|
||||
- a public CA cannot issue for a `*.svc.cluster.local` name or a client-auth certificate.
|
||||
- The new clause (5b) exempts a Certificate annotated
|
||||
`homelab.keithjackson.dev/tls-scope: workload-mtls` **only if it proves it**:
|
||||
- no Gateway listener serves its Secret;
|
||||
- every DNS name and the commonName are a bare label or `*.svc[.cluster.local]`;
|
||||
- it has no IP, URI or email SANs;
|
||||
- its issuer is a namespaced `selfSigned`/`ca` Issuer in its own namespace;
|
||||
- a `ca` issuer's Secret is produced by a likewise-exempt tracked Certificate.
|
||||
- Exemptions are printed, and clause (9), the renewal arithmetic, still applies.
|
||||
- **Negative-tested:** a public DNS name, and a Gateway serving `buildkitd-tls`, each fail the
|
||||
check with a specific message.
|
||||
- **`infra/gitea/buildkit-client-secret.sh`** (operator action): copies `ca.crt`, `tls.crt` and
|
||||
`tls.key` from the Secret `buildkit/buildkit-client-ci` into the Actions secrets `BUILDKIT_CA`,
|
||||
`BUILDKIT_CLIENT_CERT` and `BUILDKIT_CLIENT_KEY`. Before writing anything it checks that:
|
||||
- the cert verifies against the CA;
|
||||
- the cert is a client-auth cert;
|
||||
- the key matches the cert.
|
||||
|
||||
It never prints a value. `--check` reports whether the stored copy is older than the current
|
||||
certificate.
|
||||
|
||||
### Verification moved into the Containerfiles
|
||||
|
||||
`docker run` cannot exist on BuildKit, so each Containerfile now names its shipped stage after the
|
||||
image (`toolbox`, `agent-runner`, `conformance-runner`) and ends in a **`verify` test stage**,
|
||||
`FROM <image> AS verify`. That stage runs every previous assertion at build time, as the image's
|
||||
non-root uid (65532 for agent-runner and conformance-runner), in `RUN --network=none`. It is never
|
||||
pushed.
|
||||
- **Every existing assertion is preserved**, and the toolbox gained one it never had: every tool
|
||||
on PATH, and five tools report their pinned versions.
|
||||
- `probe-validation.sh` is `COPY`ed into the verify stage only, so it never ships.
|
||||
- **Bug fixed on the way:** in the old `docker run` block, `cmd | grep -q … && echo ok` and
|
||||
`[ … ] && echo` could never fail the script under `set -e` + `pipefail`. The "entrypoints refuse
|
||||
arguments", "not root", "no API key baked in" and "HOME writable" checks asserted nothing. They
|
||||
are real assertions now.
|
||||
- **What a build step cannot reproduce, and how it is approximated** (also written in each stage's
|
||||
header):
|
||||
- **uid:** exact.
|
||||
- **`--network none`:** exact, and each stage *proves* it. Its first check fails if DNS or a
|
||||
TCP connection works. A negative control without `--network=none` fails that check.
|
||||
- **`--read-only` rootfs:** not reproducible, because a RUN step always has a writable layer.
|
||||
It is approximated by permission:
|
||||
- `touch /should-fail` must fail;
|
||||
- a `find / -xdev -writable ! -type l` inventory, run as the uid, must lie entirely under
|
||||
`/workspace`, `/tmp` or `/var/tmp`. `! -type l` was needed: Fedora's `/usr/tmp` symlink
|
||||
to `/var/tmp` tripped it.
|
||||
|
||||
The real control remains the pods' `readOnlyRootFilesystem`, which the admission policies
|
||||
enforce.
|
||||
- **`--tmpfs /workspace`:** approximated by the image's own `/workspace`, owned by 65532, which
|
||||
is exactly what the pod's emptyDir replaces.
|
||||
|
||||
The opt-in `--verify-clone` still `run`s the image, so it remains local-only.
|
||||
- **`build-image.sh` (all three):**
|
||||
- verify is `build --target verify` and the image is `build --target <image>`;
|
||||
- a new `--print-build-args` prints the pins as `KEY=VALUE`, which is what CI consumes. The
|
||||
arg list therefore exists once;
|
||||
- the local podman path still works.
|
||||
- The L0 check `alert-rules-unit-tested` now looks for `"PROMTOOL_VERSION=$PROMTOOL_VERSION"` in
|
||||
the toolbox's build-arg list.
|
||||
|
||||
### The pipeline (`.gitea/workflows/images.yaml` + `scripts/images.sh`)
|
||||
|
||||
- **The builder.** The driver downloads the pinned `buildx` and refuses a checksum mismatch. It
|
||||
writes the client cert from the `BUILDKIT_*` secrets to a 0600 temp dir that is removed on exit,
|
||||
creates a `remote` builder, and runs `buildx inspect --bootstrap`.
|
||||
- **It fails loudly, never skips.** It stops with a clear message if:
|
||||
- the secrets are absent;
|
||||
- buildkitd is unreachable. The message lists the causes in order: not deployed yet (Flux
|
||||
applies it only after merge), stale certificate, NetworkPolicy.
|
||||
- **Pull request:** `buildx bake <image>-verify` for changed images. Nothing is pushed and no
|
||||
registry credential is referenced. The `BUILDKIT_*` builder secrets *are* given to PR jobs,
|
||||
because they must build.
|
||||
- agent-runner is built FROM the toolbox built **in the same run** through a bake `target:`
|
||||
context. That toolbox exists in no registry, so the toolbox is built as its base even when
|
||||
its own inputs did not change.
|
||||
- **Push to main / dispatch on main:** for each image, the verify target first, and only then the
|
||||
image itself with `push=true`. Tags are `sha-<12 hex>` (immutable) and `main` (moving).
|
||||
- The digest is read back from the registry and compared with the client's metadata.
|
||||
- agent-runner builds FROM the toolbox digest just pushed, or else from the pinned
|
||||
`AGENT_BASE_IMAGE` (the zero placeholder bootstraps).
|
||||
- `BUILDX_NO_DEFAULT_ATTESTATIONS=1` keeps one manifest per tag, which is what admission,
|
||||
Renovate and the retention script assume.
|
||||
- **Unchanged from the first iteration:** change detection, publish idempotence,
|
||||
`REGISTRY_PUSH_TOKEN` and its preflight, the Renovate markers, and `registry-retention.sh`. The
|
||||
DinD prune step is gone, because nothing is built in DinD now.
|
||||
|
||||
### Unchanged from the first iteration (still true)
|
||||
|
||||
- **Digest references** carry `# renovate-digest: image=<repo> tag=main` markers:
|
||||
- `platform/pm-agent/deployment.yaml` and `configmap.yaml`;
|
||||
- `infra/agent/versions.env` `AGENT_BASE_IMAGE`, now the published toolbox by digest, zero
|
||||
until the first publish;
|
||||
- `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE_DIGEST`, which went from `""` to zero.
|
||||
|
||||
The admission policies are untouched.
|
||||
- **`renovate.json5`**:
|
||||
- the `custom.regex` marker manager;
|
||||
- `enabledManagers` `regex` → `custom.regex`;
|
||||
- the `kubernetes` manager disabled for these three repos;
|
||||
- `ignorePaths` narrowed so `conformance/versions.env` is visible.
|
||||
|
||||
`renovate.yaml` pins `renovate-version: 44.167.0` and adds a docker `RENOVATE_HOST_RULES` entry
|
||||
using `RENOVATE_TOKEN`, which now needs `read:package`.
|
||||
- **`infra/gitea/registry-retention.sh`** (dry run by default; operator action); the
|
||||
`conformance-trigger-kubeconfig.sh` precondition fix; and the docs (pm-agent README, runbook,
|
||||
plan, conformance README, devcontainer README, workflows README). The docs are updated again
|
||||
for BuildKit, and there is a new `platform/buildkit/README.md`.
|
||||
|
||||
### Decisions and pushback
|
||||
|
||||
- **The push credential is a PAT secret, `REGISTRY_PUSH_TOKEN`; the Actions token cannot push.**
|
||||
Checked against Gitea **v1.24.3** source:
|
||||
- `services/auth/basic.go` `Basic.Verify` turns a task token into the pseudo-user
|
||||
`gitea-actions` (ID −2) and sets no token scope;
|
||||
- `routers/api/packages/container/container.go` `Authenticate` (the `/v2/token` exchange every
|
||||
docker client performs) requires `read:package`/`write:package`/`all` scope, so it answers
|
||||
401. The Actions token cannot even *pull*;
|
||||
- `services/context/package.go` `determineAccessMode` has `// TODO: ActionUser permission check`
|
||||
and no branch for that user. Even past login it would get at most read on a public org;
|
||||
- `models/repo/repo_unit.go` `ActionsConfig` has no token-permissions setting in 1.24.3.
|
||||
|
||||
The workflow fails loudly when the secret is absent, copying the `RENOVATE_TOKEN` preflight
|
||||
pattern. With BuildKit the credential reaches the daemon through the buildx session (a
|
||||
0600 `config.json` in the job's temp dir), never through the cluster.
|
||||
- **`regex` vs `custom.regex`: both work. Switched to the canonical `custom.regex`.** In Renovate:
|
||||
- `lib/config/migrations/custom/enabled-managers-migration.ts` rewrites `regex` →
|
||||
`custom.regex`;
|
||||
- that migration runs on every repository run (`lib/workers/repository/init/merge.ts` →
|
||||
`migrateAndValidate`), not only in the validator;
|
||||
- `lib/modules/manager/index.ts` `getEnabledManagersList` accepts either spelling;
|
||||
- `custom.regex` exists since 37.51.1 (renovatebot/renovate#24079).
|
||||
- **The open question hid a bigger bug: the Renovate config on main is invalid for the Renovate
|
||||
the workflow actually runs.**
|
||||
- `renovatebot/[email protected]` defaults `renovate-version` to `39`
|
||||
(`action.yml`, `src/docker.ts`), which is 39.264.0.
|
||||
- The validator used in #79 was `ghcr.io/renovatebot/renovate` with no tag, i.e. `latest`,
|
||||
44.167.0.
|
||||
- **Measured:** `renovate:39.264.0 renovate-config-validator` on main's `renovate.json5` gives
|
||||
`ERROR: Found errors in configuration` with five errors, all `managerFilePatterns`.
|
||||
`fileMatch` was renamed in 40.2.0.
|
||||
- So once #80 fixes the filename, every run would still die at `CONFIG_VALIDATION`.
|
||||
- Fixed by pinning `renovate-version: 44.167.0`. This branch's config validates clean on it.
|
||||
The only `--strict` complaint is a pending migration of the pre-existing top-level
|
||||
`"docker": {"enabled": true}` block, which I left alone.
|
||||
- **Unworkable as briefed: tracking the worker Job template's digest.**
|
||||
- `infra/agent/worker-job-template.yaml` is `COPY`ed into agent-runner. A Renovate bump there
|
||||
changes agent-runner's inputs, CI publishes a new digest, Renovate bumps it again, and so on
|
||||
forever.
|
||||
- `pm-dispatch` overwrites that field from `PM_WORKER_IMAGE` at runtime
|
||||
(`pm-dispatch.py:649`), so nothing is lost by leaving it all-zero. It is commented as
|
||||
deliberately untracked.
|
||||
- For the same loop reason, `conformance/versions.env` is **not** an input of the
|
||||
conformance-runner build, and a pipeline-only change does not republish on main.
|
||||
- **Changed from the brief: retention is a script, not a Gitea cleanup rule.**
|
||||
- Gitea 1.24.3 has **no API** for cleanup rules (none in `routers/api/v1/api.go`; web
|
||||
session+CSRF forms only, in `routers/web/web.go`). So a rule cannot be created, or
|
||||
drift-asserted, as code. **`ci/gitea-settings.sh` was therefore not extended.** A cheaper
|
||||
proxy (count versions per image) would need `read:package` on `SETTINGS_API_TOKEN`, which
|
||||
it lacks.
|
||||
- A built-in rule is also **unsafe here**. Gitea stores a manifest as a file of the TAG
|
||||
version it was pushed under (`routers/api/packages/container/manifest.go`
|
||||
`createPackageAndVersion`), and resolves a by-digest pull from those files
|
||||
(`models/packages/container/search.go`). Deleting the tag a deployed digest was pushed under
|
||||
breaks the `@sha256:` pull, and the rule's "newest N" knows nothing about what git pins. The
|
||||
script keeps every digest git pins.
|
||||
- **Auto-merge.** Digest bumps of these images auto-merge on a green gate (the existing
|
||||
`digest` rule). The exception is the conformance-runner bump: `conformance/versions.env` is a
|
||||
sensitive path, so it waits for an administrator's `/approve-override` every time.
|
||||
|
||||
## What I verified
|
||||
|
||||
**Proved locally, against the exact pinned images:**
|
||||
|
||||
- **Rootless buildkitd, configured as the manifest configures it.**
|
||||
- Setup: the pinned `moby/buildkit:v0.34.1-rootless` digest, under rootless podman, with
|
||||
`--cap-drop ALL --cap-add SETUID --cap-add SETGID`, seccomp unconfined and `--read-only`;
|
||||
tmpfs at the same five paths; the ConfigMap's `buildkitd.toml`; mTLS on :1234.
|
||||
- The image's `/etc/subuid` was overridden with `user:1:999` and `user:1001:64535`, because
|
||||
rootless podman maps only 65,537 uids. rootlesskit's real `newuidmap` user-namespace path
|
||||
still ran.
|
||||
- It started, `buildctl debug workers` listed the OCI worker, and `debug workers -v` showed the
|
||||
GC config applied (max 4.29 GB, reserved 1.07 GB).
|
||||
- **The securityContext, item by item.** Each requirement was measured by taking it away:
|
||||
- **default seccomp:** daemon starts, every RUN step fails → `Unconfined` is required;
|
||||
- **no `SETUID`/`SETGID`:** rootlesskit dies with
|
||||
`newuidmap … operation not permitted` → both capabilities are required;
|
||||
- **`--read-only` rootfs:** works once the five emptyDirs exist. Its only failure was the
|
||||
probe client's `~/.docker`, which is now an emptyDir;
|
||||
- **minimal caps + Unconfined seccomp:** works, and a RUN step executes.
|
||||
- **The full CI path, in the actual CI job image.** Setup: `ghcr.io/catthehacker/ubuntu:act-22.04`
|
||||
running `.gitea/workflows/scripts/images.sh` in verify mode. It downloaded and sha256-checked
|
||||
buildx v0.38.1, connected over mTLS, and bake-built all three `verify` targets from an empty
|
||||
cache. rc 0, 1m32s total:
|
||||
- `TOOLBOX VERIFIED` (42 s);
|
||||
- `AGENT-RUNNER VERIFIED` (26 s): selftest OK, probe "20 refusal(s), 0 failure(s)", and built
|
||||
FROM the in-run toolbox through the `target:` context, with the log showing
|
||||
`[agent-runner-verify toolbox 1/6]` and `base=toolbox-in-this-run`;
|
||||
- `CONFORMANCE-RUNNER VERIFIED` (21 s): all seven refusals;
|
||||
- every stage printed `confirmed: no network`.
|
||||
- **mTLS rejects what it should.** Tested with a real BuildKit API call (`buildctl debug workers`):
|
||||
- no client cert → `remote error: tls: certificate required`;
|
||||
- a client cert from another CA → rejected the same way;
|
||||
- the valid cert → accepted.
|
||||
|
||||
Through the driver:
|
||||
- another CA's client cert → rc 1;
|
||||
- the wrong CA for the server → rc 1;
|
||||
- an unreachable host → rc 1 in about 20 s, with the explanatory message;
|
||||
- missing secrets → rc 1, naming the three secrets.
|
||||
- **The local podman path**, `build-image.sh` for all three (rc 0).
|
||||
- Negative control: a verify stage expecting a different promtool version printed
|
||||
`DRIFT promtool want 9.9.9` and failed the build.
|
||||
- Negative control: the `--network=none` check fails when the flag is removed.
|
||||
- **`buildkit-client-secret.sh`'s sanity checks**, run offline on local certs. They tell apart:
|
||||
- a good client cert;
|
||||
- a server cert (not client-auth);
|
||||
- a mismatched key;
|
||||
- a cert from another CA.
|
||||
|
||||
**Measured live, read-only:**
|
||||
- node user namespaces, Debian 13.7 and AppArmor (via an Ansible `sysctl`/`cat`);
|
||||
- DinD's MASQUERADE rules;
|
||||
- the live ClusterPolicies, namespaced Policies, PSA labels, and cert-manager v1.21.3 with its
|
||||
ClusterIssuers;
|
||||
- the act-runner pod labels.
|
||||
|
||||
**Renovate (unchanged since the first iteration):**
|
||||
- the validator with 44.167.0: exit 0;
|
||||
- `extract` finds all four markers;
|
||||
- the `kubernetes` duplicates are `skipReason: disabled`;
|
||||
- 39.264.0 rejects `main`'s config.
|
||||
|
||||
**Local gate, CI results:** see the PR body. They are recorded there because this entry is part of
|
||||
the tree being gated.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **platform/buildkit has never run in the cluster.** It can't until it merges: agents may not
|
||||
`kubectl apply`, and Flux reconciles only `main`. So this PR's own `images` job **cannot build
|
||||
and is expected to fail at the builder preflight**, which is the designed loud failure. The
|
||||
following are therefore all unproven in-cluster:
|
||||
- kubelet admitting the pod;
|
||||
- containerd honouring `appArmorProfile: Unconfined`;
|
||||
- the five emptyDirs;
|
||||
- cert-manager issuing the three certs;
|
||||
- the CiliumNetworkPolicies.
|
||||
- **AppArmor `Unconfined` is reasoned, not measured.** My test host runs SELinux, so the AppArmor
|
||||
requirement comes from upstream's example plus the nodes having AppArmor enabled. If the pod
|
||||
were to work under `RuntimeDefault` AppArmor, that line could be tightened; I did not test it.
|
||||
- **The local userns mapping is narrower than the cluster's.** Locally it was 65,535 ids across
|
||||
two ranges; in the cluster it is the image's own `100000:65536` against real host subuids. That
|
||||
is the normal case, but it is the cluster's first test.
|
||||
- **The NetworkPolicies are untested.** In particular:
|
||||
- that `toEntities: [ingress]` admits the push to the Gateway VIP. The comment explains the
|
||||
reasoning, but it is unmeasured;
|
||||
- that world :443/:80 covers every download the builds make. A dnf mirror on another port
|
||||
would fail a build visibly.
|
||||
- **No push, anywhere.** The whole publish path is unexercised:
|
||||
- `REGISTRY_PUSH_TOKEN` reaching buildkitd through the buildx session;
|
||||
- `push=true`;
|
||||
- the metadata digest compared with the registry read-back;
|
||||
- agent-runner FROM `toolbox@sha256:`;
|
||||
- the immutable-tag guard;
|
||||
- idempotence.
|
||||
|
||||
So are the 15 s Envoy route-timeout risk on blob finalisation, and the Gitea PVC (10Gi, shared
|
||||
with repositories) absorbing ~2 GB per toolbox publish until retention runs.
|
||||
- **Node disk under real builds.** The cache emptyDir (6Gi limit, 4 GB GC cap) sits on nodes that
|
||||
had 7–9 GB free. It is untested under concurrent pressure.
|
||||
- **Certificate rotation is designed, not exercised.** The server cert renews 30 days early but
|
||||
buildkitd only reads it at start; the client copy needs the script re-run.
|
||||
- **`--oci-worker-no-process-sandbox` is an accepted risk, not a mitigated one.** A build step can
|
||||
ptrace the daemon and read its TLS key and cache. Anyone with the client cert, which means every
|
||||
PR job, can run build steps.
|
||||
- **Renovate has still never completed a run.** It is blocked on #80, on `RENOVATE_TOKEN` gaining
|
||||
`read:package`, and on this branch. The `[email protected]` + 44.x image pairing is
|
||||
untested.
|
||||
- **Pods cannot pull these images until PR #61** (the kubelet pull secret) merges.
|
||||
- **`registry-retention.sh --apply` has never run**; its keep logic has only run against an empty
|
||||
registry.
|
||||
@@ -1,57 +0,0 @@
|
||||
---
|
||||
date: 2026-10-29
|
||||
branch: fix/renovate-config-filename
|
||||
title: Renovate has failed every run since #79 — the workflow still named renovate.json
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**`.gitea/workflows/renovate.yaml`** — `configurationFile: renovate.json` → `renovate.json5`.
|
||||
|
||||
PR #79 (`fix/renovate-invalid-config`) renamed the config to `renovate.json5` and did not update
|
||||
the one line in the workflow that names it. Measured 2026-10-29 from the Gitea Actions log of run
|
||||
322 (task 513, schedule, 08:15 UTC):
|
||||
|
||||
Error: ENOENT: no such file or directory, stat '/workspace/Home-Network/home-network-k3s-impl/renovate.json'
|
||||
|
||||
Runs 315–322 (2026-10-22 → 10-29, every one listed) are all `failure` on the `schedule` event,
|
||||
each failing in about one second. Renovate never started. `RENOVATE_TOKEN` **does** exist as a
|
||||
repository Actions secret (listed by name via the API; value not read), so the token preflight
|
||||
passes and this filename is the only blocker observed.
|
||||
|
||||
**`ci/lint.sh`** — a new assertion after yamllint: every non-comment `configurationFile:` key in
|
||||
`.gitea/workflows/*.y*ml` must name a path that exists. This is valid YAML, so yamllint cannot see
|
||||
it, and the gate stayed green through 11 days of failures.
|
||||
|
||||
**Second blocker, same PR:** the workflow did not pin a Renovate version, so
|
||||
`renovatebot/[email protected]` ran its default, Renovate **39** (`action.yml`:
|
||||
`renovate-version` `default: '39'`, read at that tag). `renovate.json5` uses `managerFilePatterns`,
|
||||
which needs >= 40.2.0. Under 39.264.0 the validator rejects the config with five errors. This was
|
||||
found and reproduced by the image-pipeline worker (PR #81). #79 had validated against `latest`. Pinned
|
||||
`renovate-version: 44.167.0`, the version the config validates clean against per that worker.
|
||||
|
||||
## What I verified
|
||||
|
||||
- The root cause, from the run log quoted above, and that `origin/main` has `renovate.json5` and
|
||||
no `renovate.json` (`git ls-tree`).
|
||||
- The new guard, both ways, run standalone: on this branch it prints
|
||||
`workflow configurationFile references resolve`, rc 0. With the line reverted to `renovate.json`
|
||||
it prints `configurationFile 'renovate.json' does not exist in the repo`, rc 1. My first version
|
||||
also matched its own explanatory comment in the workflow and failed. That is why the grep is
|
||||
anchored to non-comment lines.
|
||||
- Toolbox gate: result and rc are reported in the PR, not predicted here.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The 39.x rejection and the 44.167.0 pass were not reproduced by me.** I confirmed only the
|
||||
action's default (`'39'`). The validation results are the PR #81 worker's.
|
||||
- **Renovate has still never completed a run.** This removes the observed blocker. Others may
|
||||
follow it. The test is a `workflow_dispatch` with `dry_run: true` after merge, and its log is
|
||||
the first evidence that Renovate authenticates, extracts dependencies, and honours `.json5`.
|
||||
- **The `enabledManagers: ["regex"]` vs `custom.regex` question** (#73, #77) is still open. The
|
||||
same dry-run log answers it: look for the regex/custom manager's dependency-extraction count.
|
||||
- **No alert for "Renovate keeps failing".** The guard prevents this specific cause from recurring.
|
||||
Nothing notices a scheduled workflow that is red every day. That gap stays open.
|
||||
- **Sensitive path.** `.gitea/**` and `ci/**` are both in `approval-policy.json`. CI will block
|
||||
on `approval-policy` until the operator posts `/approve-override` as a review. The local gate
|
||||
reports that stage as N/A.
|
||||
@@ -244,12 +244,10 @@ about runtime behaviour in this document is a design claim, not a measurement.
|
||||
(ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
|
||||
the credential decision now being local Gitea accounts with PATs, that means a
|
||||
`dockerconfigjson` for an account whose token carries `read:package`. Only then does the
|
||||
digest matter. **Superseded 2026-10-29:** the build-and-push is no longer an operator step.
|
||||
`.gitea/workflows/images.yaml` builds and verifies the image on rootless BuildKit
|
||||
(`platform/buildkit/`), publishes it on a push to `main`, and Renovate opens the one
|
||||
PR that replaces both placeholder digests (`decisions/2026-10-29-image-pipeline.md`). The
|
||||
same two blockers sit in front of `conformance-runner`, which shares the registry path and
|
||||
has never been pulled.
|
||||
digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
|
||||
placeholder digests. Expect the build itself to need iteration too: the nodejs module
|
||||
stream and the npm install have never run against this base. The same two blockers sit in
|
||||
front of `conformance-runner`, which shares the registry path and has never been pulled.
|
||||
2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the
|
||||
bot's permission is `write`, not `admin`** — that measurement is the control that keeps
|
||||
the agent out of its own approval gate.
|
||||
|
||||
@@ -50,10 +50,7 @@
|
||||
# default is deliberately EMPTY so a build that forgets it fails loudly instead of
|
||||
# resolving some other image named `agent-base`.
|
||||
ARG AGENT_BASE_IMAGE=
|
||||
# Stage `agent-runner` is the image that ships; stage `verify` (end of file) is a TEST stage built
|
||||
# FROM it that only runs assertions and is never pushed. build-image.sh and the CI pipeline
|
||||
# build `--target verify` to prove the image and `--target agent-runner` for the image itself.
|
||||
FROM ${AGENT_BASE_IMAGE} AS agent-runner
|
||||
FROM ${AGENT_BASE_IMAGE}
|
||||
|
||||
# Re-declared AFTER the FROM: an ARG from before the first FROM is not in scope in the
|
||||
# build stage, so the provenance layer below would otherwise record an empty base.
|
||||
@@ -211,80 +208,3 @@ WORKDIR /workspace
|
||||
# paths above; an ENTRYPOINT here would either be ignored or prepend something the policy
|
||||
# cannot see. `bash` is the interactive fallback for a human debugging the image locally.
|
||||
CMD ["/bin/bash"]
|
||||
|
||||
# =============================================================================
|
||||
# verify — a TEST stage, never shipped. Every assertion infra/agent/build-image.sh used to make
|
||||
# with `run --user 65532 --read-only --tmpfs /workspace --tmpfs /tmp --network none` now runs
|
||||
# here, at build time, because CI builds on rootless BuildKit where no container runtime exists.
|
||||
#
|
||||
# WHAT A BUILD STEP CAN AND CANNOT REPRODUCE, stated so nobody over-reads a green verify:
|
||||
# * uid 65532 ............ exact: USER is inherited from the agent-runner stage.
|
||||
# * --network none ....... exact: RUN --network=none, and the first check proves it.
|
||||
# * --read-only rootfs ... NOT reproducible: a RUN step always has a writable layer. It is
|
||||
# APPROXIMATED by permission: as uid 65532, every path the process can
|
||||
# write is inventoried and must lie under /workspace, /tmp or /var/tmp
|
||||
# (the pod mounts emptyDirs on the first two; /var/tmp is read-only
|
||||
# there, and TMPDIR points into /workspace so nothing should use it).
|
||||
# The pod's readOnlyRootFilesystem is enforced by admission
|
||||
# (platform/kyverno/admission/pm-agent-job-shape.yaml), not by this.
|
||||
# * --tmpfs /workspace ... approximated by the image's own /workspace (owned 65532), which is
|
||||
# what the pod's emptyDir replaces.
|
||||
# The probe script is COPYed into THIS stage only, so it never ships in the image.
|
||||
# =============================================================================
|
||||
FROM agent-runner AS verify
|
||||
COPY probe-validation.sh /opt/verify/probe-validation.sh
|
||||
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
|
||||
echo "--- no network (the build-time stand-in for run --network none) ---"
|
||||
if getent hosts registry.npmjs.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
|
||||
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
|
||||
fi
|
||||
echo " confirmed: no network"
|
||||
echo "--- id ---"; id
|
||||
echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}"
|
||||
echo "--- baked versions ---"; cat /etc/agent-runner-versions
|
||||
echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---"
|
||||
for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do
|
||||
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
|
||||
else printf " MISSING %s\n" "$t"; missing=1; fi
|
||||
done
|
||||
[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; }
|
||||
echo "--- python can import the renderer dependency ---"
|
||||
python3 -c "import yaml; print('PyYAML', yaml.__version__)"
|
||||
echo "--- claude ---"; claude --version
|
||||
echo "--- the two entrypoints refuse arguments ---"
|
||||
# Each must REFUSE. (The old docker-run version wrote `cmd | grep -q ... && echo`, which under
|
||||
# `set -e` + pipefail could never fail the script; these are real assertions.)
|
||||
for e in pm-worker pm-dispatch; do
|
||||
out="$(/usr/local/bin/$e --anything 2>&1 || true)"
|
||||
grep -q "takes no arguments" <<<"$out" || { echo "UNEXPECTED: $e accepted an argument: $out"; exit 1; }
|
||||
echo " $e: rejected"
|
||||
done
|
||||
echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---"
|
||||
PM_SELFTEST=1 /usr/local/bin/pm-dispatch
|
||||
echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---"
|
||||
for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do
|
||||
stat -c " %n %U:%G %a" "$f"
|
||||
if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi
|
||||
done
|
||||
echo " confirmed: not writable by uid $(id -u)"
|
||||
echo "--- the root filesystem is not writable by the runtime uid (expected: a failure) ---"
|
||||
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: / is writable by uid $(id -u)"; exit 1; fi
|
||||
echo " confirmed: / is not writable"
|
||||
echo "--- everything uid $(id -u) CAN write lies under /workspace, /tmp or /var/tmp ---"
|
||||
stray="$(find / -xdev \( -path /proc -o -path /sys -o -path /dev \) -prune -o -writable ! -type l -print 2>/dev/null \
|
||||
| grep -vE '^/(workspace|tmp|var/tmp)(/|$)' || true)"
|
||||
if [ -n "${stray}" ]; then echo "UNEXPECTED: writable outside the allowed paths:"; printf ' %s\n' ${stray} | head -20; exit 1; fi
|
||||
echo " confirmed: no writable path outside /workspace, /tmp, /var/tmp"
|
||||
echo "--- the two writable paths really are writable ---"
|
||||
touch "${HOME}/.probe" /tmp/.probe || { echo "UNEXPECTED: ${HOME} or /tmp not writable"; exit 1; }
|
||||
echo " confirmed: ${HOME} and /tmp are writable"
|
||||
echo "--- uid is non-zero ---"
|
||||
[ "$(id -u)" -ne 0 ] || { echo "UNEXPECTED: running as root"; exit 1; }
|
||||
echo " confirmed: uid $(id -u), not root"
|
||||
echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---"
|
||||
[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] || { echo "UNEXPECTED: an Anthropic key is baked in"; exit 1; }
|
||||
echo " confirmed: neither is set"
|
||||
echo "--- pm-worker's input validation, with the interpreter and script that run in the cluster ---"
|
||||
PM_WORKER_PATH=/usr/local/bin/pm-worker /bin/bash /opt/verify/probe-validation.sh
|
||||
echo "AGENT-RUNNER VERIFIED"
|
||||
VERIFY
|
||||
+140
-86
@@ -1,36 +1,26 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# infra/agent/build-image.sh — build the pinned `agent-runner` image and prove it works.
|
||||
# infra/agent/build-image.sh — build the pinned `agent-runner` image and print its digest.
|
||||
#
|
||||
# ./build-image.sh # build the verify target, then the image
|
||||
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
|
||||
# ./build-image.sh # build + verify + print the digest and push command
|
||||
# ./build-image.sh --no-verify # build only (skip the in-image proof)
|
||||
# ENGINE=docker ./build-image.sh # build with docker instead of podman
|
||||
# AGENT_BASE_IMAGE=home-k3s-dev:local ./build-image.sh
|
||||
# # build on a LOCAL toolbox instead of the pinned one
|
||||
# # (`infra/devcontainer/dev.sh build` makes that tag)
|
||||
# IMAGE_TAG=foo ./build-image.sh # override the local tag (CI uses this)
|
||||
#
|
||||
# THIS SCRIPT NEVER PUSHES — and nobody pushes this image from a workstation either.
|
||||
# Publishing is done by CI: .gitea/workflows/images.yaml builds the SAME Containerfile on
|
||||
# rootless BuildKit (platform/buildkit/), taking its build args from `--print-build-args`, and
|
||||
# on a push to `main` pushes it to the Gitea registry and records the registry-returned digest
|
||||
# in the job summary. Renovate then opens the pull request that moves platform/pm-agent/ onto
|
||||
# that digest. This script is the LOCAL path (podman by default); its proof — the
|
||||
# Containerfile's `verify` stage — is the same one CI runs.
|
||||
# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is not
|
||||
# an agent's to hold, and the pushed digest is what a fail-closed admission policy then
|
||||
# trusts. So this builds locally, proves the toolchain and both entrypoints work INSIDE the
|
||||
# image as the non-root uid with the read-only root filesystem the policy forces, and prints
|
||||
# the exact push line for a human.
|
||||
#
|
||||
# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the pins this image adds
|
||||
# (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds the
|
||||
# platform toolchain the base already carries. Nothing in this file invents a version.
|
||||
# VERSIONS ARE NOT DEFINED HERE. infra/agent/versions.env holds the four pins this image
|
||||
# adds (base image, node, Claude Code CLI, model) and infra/devcontainer/versions.env holds
|
||||
# the platform toolchain the base already carries. Nothing in this file invents a version.
|
||||
#
|
||||
# THE BASE. AGENT_BASE_IMAGE in versions.env is the PUBLISHED toolbox, pinned by digest and
|
||||
# kept current by Renovate. An AGENT_BASE_IMAGE in the environment overrides it — CI passes
|
||||
# the toolbox it built in the same run, and a developer can pass `home-k3s-dev:local`.
|
||||
#
|
||||
# NO `run` STEP. Verification used to `run` the image as uid 65532 with a read-only rootfs.
|
||||
# CI has no container runtime (rootless BuildKit only), so the assertions moved into the
|
||||
# Containerfile's `verify` stage and this script builds that target. `--print-build-args`
|
||||
# prints the pins as KEY=VALUE for the CI pipeline; AGENT_BASE_IMAGE is the caller's choice.
|
||||
# THE DIGEST CAVEAT — read it before pasting a digest into anything. The digest printed here
|
||||
# is the LOCAL manifest digest. It is normally identical to the registry digest, but that is
|
||||
# a property of the push rather than a guarantee, so the push command below uses
|
||||
# `--digestfile`: the digest the REGISTRY returns is the authoritative one, and that is the
|
||||
# one that goes into platform/pm-agent/configmap.yaml.
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
@@ -39,11 +29,9 @@ REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
|
||||
ENGINE="${ENGINE:-podman}"
|
||||
|
||||
VERIFY=1
|
||||
PRINT_ARGS=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--no-verify) VERIFY=0 ;;
|
||||
--print-build-args) PRINT_ARGS=1 ;;
|
||||
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "unknown argument: $arg" >&2; exit 2 ;;
|
||||
esac
|
||||
@@ -53,14 +41,13 @@ log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
info() { printf ' %s\n' "$*"; }
|
||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
||||
|
||||
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
||||
|
||||
# --- pins: sourced, never invented -------------------------------------------
|
||||
AGENT_ENV="${HERE}/versions.env"
|
||||
[ -f "${AGENT_ENV}" ] || die "missing ${AGENT_ENV} — the pins live there, not here"
|
||||
BASE_OVERRIDE="${AGENT_BASE_IMAGE:-}"
|
||||
TAG_OVERRIDE="${IMAGE_TAG:-}"
|
||||
# shellcheck disable=SC1090
|
||||
source "${AGENT_ENV}"
|
||||
[ -z "${BASE_OVERRIDE}" ] || AGENT_BASE_IMAGE="${BASE_OVERRIDE}"
|
||||
: "${AGENT_BASE_IMAGE:?AGENT_BASE_IMAGE not set in ${AGENT_ENV}}"
|
||||
: "${NODEJS_MAJOR:?NODEJS_MAJOR not set in ${AGENT_ENV}}"
|
||||
: "${CLAUDE_CODE_VERSION:?CLAUDE_CODE_VERSION not set in ${AGENT_ENV}}"
|
||||
@@ -68,39 +55,16 @@ source "${AGENT_ENV}"
|
||||
: "${AGENT_MODEL:?AGENT_MODEL not set in ${AGENT_ENV}}"
|
||||
: "${AGENT_IMAGE_REPO:?AGENT_IMAGE_REPO not set in ${AGENT_ENV}}"
|
||||
|
||||
# THE build args, in ONE place; `--print-build-args` hands them to CI (which builds on rootless
|
||||
# BuildKit and supplies its own base reference). AGENT_BASE_IMAGE is deliberately NOT in this
|
||||
# list: the caller decides the base.
|
||||
BUILD_ARGS=(
|
||||
"NODEJS_MAJOR=${NODEJS_MAJOR}" "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}"
|
||||
"CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" "AGENT_MODEL=${AGENT_MODEL}"
|
||||
)
|
||||
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
|
||||
|
||||
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
||||
|
||||
# --- the base must exist, locally or in the registry ---------------------------
|
||||
# Deliberately NOT built on demand. The toolbox is its own reviewed artefact with its own
|
||||
# pins, and silently building it here would mean this script could change the toolchain a
|
||||
# worker gates with as a side effect of rebuilding the agent image.
|
||||
case "${AGENT_BASE_IMAGE}" in
|
||||
*@sha256:0000000000000000000000000000000000000000000000000000000000000000)
|
||||
die "AGENT_BASE_IMAGE is still the all-zero placeholder in ${AGENT_ENV}: the toolbox has
|
||||
not been published yet. Either let CI publish it (.gitea/workflows/images.yaml) and
|
||||
merge Renovate's digest bump, or build on a local toolbox:
|
||||
${REPO_ROOT}/infra/devcontainer/dev.sh build
|
||||
AGENT_BASE_IMAGE=home-k3s-dev:local $0" ;;
|
||||
esac
|
||||
if ! "${ENGINE}" image inspect "${AGENT_BASE_IMAGE}" >/dev/null 2>&1; then
|
||||
info "base ${AGENT_BASE_IMAGE} is not local; pulling it"
|
||||
"${ENGINE}" pull "${AGENT_BASE_IMAGE}" >/dev/null \
|
||||
|| die "base image '${AGENT_BASE_IMAGE}' is neither local nor pullable (registry login?).
|
||||
To build on a local toolbox instead:
|
||||
${REPO_ROOT}/infra/devcontainer/dev.sh build
|
||||
AGENT_BASE_IMAGE=home-k3s-dev:local $0"
|
||||
# --- the base must already exist ---------------------------------------------
|
||||
# Deliberately NOT built on demand. The toolbox build is its own reviewed artefact with its
|
||||
# own pins, and silently building it here would mean this script could change the toolchain
|
||||
# a worker gates with as a side effect of rebuilding the agent image.
|
||||
if ! "${ENGINE}" image exists "${AGENT_BASE_IMAGE}" 2>/dev/null; then
|
||||
die "base image '${AGENT_BASE_IMAGE}' is not present. Build the pinned toolbox first:
|
||||
${REPO_ROOT}/infra/devcontainer/dev.sh build"
|
||||
fi
|
||||
|
||||
IMAGE_TAG="${TAG_OVERRIDE:-claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}}"
|
||||
IMAGE_TAG="claude${CLAUDE_CODE_VERSION}-node${NODEJS_MAJOR}"
|
||||
IMAGE_REF="${AGENT_IMAGE_REPO}:${IMAGE_TAG}"
|
||||
|
||||
log "Building ${IMAGE_REF}"
|
||||
@@ -111,36 +75,126 @@ info " node ${NODEJS_MAJOR}"
|
||||
info " claude ${CLAUDE_CODE_VERSION} (floor ${CLAUDE_CODE_MIN_VERSION})"
|
||||
info " model ${AGENT_MODEL}"
|
||||
|
||||
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
|
||||
LABEL_ARGS=()
|
||||
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
|
||||
ARG_FLAGS=(--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}")
|
||||
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
|
||||
"${ENGINE}" build \
|
||||
--file "${HERE}/Containerfile" \
|
||||
--tag "${IMAGE_REF}" \
|
||||
--build-arg "AGENT_BASE_IMAGE=${AGENT_BASE_IMAGE}" \
|
||||
--build-arg "NODEJS_MAJOR=${NODEJS_MAJOR}" \
|
||||
--build-arg "CLAUDE_CODE_VERSION=${CLAUDE_CODE_VERSION}" \
|
||||
--build-arg "CLAUDE_CODE_MIN_VERSION=${CLAUDE_CODE_MIN_VERSION}" \
|
||||
--build-arg "AGENT_MODEL=${AGENT_MODEL}" \
|
||||
"${HERE}"
|
||||
|
||||
# --- verify: the point of the whole exercise ---------------------------------
|
||||
# A build that merely succeeded is not evidence this image is usable. The proof is the
|
||||
# Containerfile's `verify` stage: as the NON-ROOT uid, with no network, the toolchain executes,
|
||||
# both entrypoints refuse arguments, the dispatcher selftest passes, the entrypoints are not
|
||||
# writable, nothing outside /workspace and /tmp is writable, and pm-worker's input guards
|
||||
# (probe-validation.sh) refuse every hostile input. It is a build target rather than a
|
||||
# `run` so the SAME proof runs here and in CI, where no container runtime exists — see the
|
||||
# stage's header for what a build step can and cannot reproduce of the pod's shape.
|
||||
# A build that merely succeeded is not evidence this image is usable. Prove the tools
|
||||
# EXECUTE, as the NON-ROOT uid, with a READ-ONLY root filesystem and only the two volumes
|
||||
# the admission policy admits — the exact shape a worker Job runs in. Anything that works
|
||||
# here but not there is a 3am discovery.
|
||||
if [ "${VERIFY}" -eq 1 ]; then
|
||||
log "Verifying (Containerfile target: verify)"
|
||||
"${ENGINE}" build "${ARG_FLAGS[@]}" --target verify --file "${HERE}/Containerfile" "${HERE}"
|
||||
log "Verifying inside the image (uid 65532, read-only rootfs, workspace+tmp only)"
|
||||
"${ENGINE}" run --rm \
|
||||
--user 65532:65532 \
|
||||
--read-only \
|
||||
--tmpfs /workspace:rw,mode=1777 \
|
||||
--tmpfs /tmp:rw,mode=1777 \
|
||||
--network none \
|
||||
--entrypoint "" \
|
||||
"${IMAGE_REF}" \
|
||||
/bin/bash -euo pipefail -c '
|
||||
echo "--- id ---"; id
|
||||
echo "--- HOME ---"; echo "HOME=${HOME} CLAUDE_CONFIG_DIR=${CLAUDE_CONFIG_DIR}"
|
||||
echo "--- baked versions ---"; cat /etc/agent-runner-versions
|
||||
echo "--- the gate toolchain the worker needs (CLAUDE.md §3) ---"
|
||||
for t in git python3 bash yamllint check-jsonschema kyverno promtool chainsaw flux sops age kubectl jq diff; do
|
||||
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
|
||||
else printf " MISSING %s\n" "$t"; missing=1; fi
|
||||
done
|
||||
[ -z "${missing:-}" ] || { echo "the gate cannot run in this image"; exit 1; }
|
||||
echo "--- python can import the renderer dependency ---"
|
||||
python3 -c "import yaml; print(\"PyYAML\", yaml.__version__)"
|
||||
echo "--- claude ---"; claude --version
|
||||
echo "--- the two entrypoints refuse arguments ---"
|
||||
/usr/local/bin/pm-worker --anything 2>&1 | grep -q "takes no arguments" && echo " pm-worker: rejected"
|
||||
/usr/local/bin/pm-dispatch --anything 2>&1 | grep -q "takes no arguments" && echo " pm-dispatch: rejected"
|
||||
echo "--- the dispatcher selftest (branch vocabulary, result envelopes, Job template, persona) ---"
|
||||
PM_SELFTEST=1 /usr/local/bin/pm-dispatch
|
||||
echo "--- the entrypoints are root-owned and NOT writable by the runtime uid ---"
|
||||
for f in /usr/local/bin/pm-worker /usr/local/bin/pm-dispatch /etc/pm-agent/worker-job-template.yaml; do
|
||||
stat -c " %n %U:%G %a" "$f"
|
||||
if [ -w "$f" ]; then echo "UNEXPECTED: uid $(id -u) can write $f"; exit 1; fi
|
||||
done
|
||||
echo " confirmed: not writable by uid $(id -u)"
|
||||
echo "--- rootfs is read-only (expected: a failure) ---"
|
||||
if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
|
||||
echo " confirmed: / is read-only"
|
||||
echo "--- the two writable paths really are writable ---"
|
||||
touch "${HOME}/.probe" /tmp/.probe && echo " confirmed: ${HOME} and /tmp are writable"
|
||||
echo "--- uid is non-zero ---"
|
||||
[ "$(id -u)" -ne 0 ] && echo " confirmed: uid $(id -u), not root"
|
||||
echo "--- no Anthropic API key is baked into the image (it would outrank the subscription token) ---"
|
||||
[ -z "${ANTHROPIC_API_KEY:-}${ANTHROPIC_AUTH_TOKEN:-}" ] && echo " confirmed: neither is set"
|
||||
'
|
||||
# The worker's input guards, exercised in the image, with the interpreter and the script
|
||||
# that will actually run in the cluster. The probe is mounted read-only rather than baked
|
||||
# in: it is a build-time check, not something a worker Job has any business carrying.
|
||||
log "Probing pm-worker's input validation inside the image"
|
||||
"${ENGINE}" run --rm \
|
||||
--user 65532:65532 \
|
||||
--read-only \
|
||||
--tmpfs /workspace:rw,mode=1777 \
|
||||
--tmpfs /tmp:rw,mode=1777 \
|
||||
--network none \
|
||||
--volume "${HERE}/probe-validation.sh:/probe-validation.sh:ro,Z" \
|
||||
--entrypoint "" \
|
||||
"${IMAGE_REF}" \
|
||||
/bin/bash /probe-validation.sh
|
||||
|
||||
log "Image verified"
|
||||
fi
|
||||
|
||||
"${ENGINE}" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target agent-runner \
|
||||
--file "${HERE}/Containerfile" --tag "${IMAGE_REF}" "${HERE}"
|
||||
|
||||
# --- result ------------------------------------------------------------------
|
||||
# No digest is printed: a manifest digest does not exist until a registry has the manifest,
|
||||
# and the only digest anything may pin is the one the REGISTRY returns. CI captures that at
|
||||
# push time and prints it in the job summary (.gitea/workflows/scripts/images.sh).
|
||||
# --- digest ------------------------------------------------------------------
|
||||
DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
|
||||
SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
|
||||
|
||||
log "Built ${IMAGE_REF}"
|
||||
info "image id $("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
|
||||
info "size $(( SIZE / 1024 / 1024 )) MiB"
|
||||
info "Not pushed. CI publishes this image from main (.gitea/workflows/images.yaml); see"
|
||||
info "platform/pm-agent/README.md for how a new digest reaches the cluster."
|
||||
info "local manifest digest ${DIGEST}"
|
||||
info "size $(( SIZE / 1024 / 1024 )) MiB"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
================================================================================
|
||||
NOT PUSHED. Publishing is an operator action — this script holds no registry
|
||||
credential and will never acquire one.
|
||||
|
||||
OPERATOR: run these, then put the digest from the digestfile into
|
||||
platform/pm-agent/configmap.yaml (PM_WORKER_IMAGE) via a pull request:
|
||||
|
||||
${ENGINE} login gitea.home.keithjackson.dev
|
||||
|
||||
${ENGINE} push --digestfile /tmp/agent-runner.digest ${IMAGE_REF}
|
||||
|
||||
cat /tmp/agent-runner.digest
|
||||
# -> sha256:.... THIS is the authoritative digest, not the local one above.
|
||||
|
||||
# platform/pm-agent/configmap.yaml
|
||||
PM_WORKER_IMAGE: "${AGENT_IMAGE_REPO}@\$(cat /tmp/agent-runner.digest)"
|
||||
|
||||
WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/pm-agent-job-shape.yaml is a
|
||||
fail-closed Enforce policy. It DENIES any worker Job whose image is not
|
||||
${AGENT_IMAGE_REPO}@sha256:<digest> — a tag reference is denied,
|
||||
and so is any other repository path. The dispatcher also refuses to start if
|
||||
PM_WORKER_IMAGE carries no digest, so the failure names its own cause.
|
||||
|
||||
THE SAME IMAGE RUNS THE DISPATCHER. platform/pm-agent/deployment.yaml pins it too, so a
|
||||
new digest is a two-line change in one pull request: the Deployment's image and
|
||||
PM_WORKER_IMAGE. They must be the same digest — the dispatcher and its workers share the
|
||||
committed Job template and the env-name contract, and a mismatched pair would disagree
|
||||
about it.
|
||||
|
||||
TIGHTEN THE BOUNDARY AFTERWARDS: the policy admits the wildcard
|
||||
\`${AGENT_IMAGE_REPO}@sha256:*\`, so anyone who can push to that registry path can publish
|
||||
a different digest. Replacing the wildcard with the LITERAL digest closes that gap; it is a
|
||||
one-line change, reviewed in a pull request, which is exactly where a change to the
|
||||
security boundary belongs.
|
||||
================================================================================
|
||||
EOF
|
||||
@@ -16,15 +16,13 @@
|
||||
# scrubbed environment and fake credentials, and if a guard ever stopped firing, the case
|
||||
# would fail here rather than by really cloning something.
|
||||
#
|
||||
# Run offline, no cluster, no Gitea, no model. The Containerfile's `verify` stage runs it inside
|
||||
# the built image as well (as uid 65532, no network), where the interpreter and the script are
|
||||
# the ones that will actually run in the cluster — locally via build-image.sh and in CI.
|
||||
# Run offline, no cluster, no Gitea, no model. infra/agent/build-image.sh runs it inside the
|
||||
# built image as well, where the interpreter and the script are the ones that will actually
|
||||
# run in the cluster.
|
||||
###############################################################################
|
||||
set -uo pipefail
|
||||
|
||||
# `:-$0` keeps this safe if it is ever fed over stdin (`bash -s`), where BASH_SOURCE is empty
|
||||
# and `set -u` would abort on it.
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)"
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
WORKER="${PM_WORKER_PATH:-${HERE}/pm-worker}"
|
||||
[ -r "$WORKER" ] || { echo "cannot read ${WORKER}" >&2; exit 2; }
|
||||
|
||||
|
||||
@@ -9,16 +9,10 @@
|
||||
# truth and, the first time the two drifted, the worker's "green" would stop meaning the
|
||||
# gate's green. So this file pins ONLY what the toolbox does not carry.
|
||||
|
||||
# The toolbox image the runner is layered on: the PUBLISHED toolbox, by digest. CI builds and
|
||||
# publishes it (.gitea/workflows/images.yaml) and Renovate moves this line onto each new digest
|
||||
# of its `main` tag via the marker below, in a pull request — nobody edits it by hand.
|
||||
#
|
||||
# ALL-ZERO UNTIL THE FIRST PUBLISH. The first run of the image pipeline on main sees this
|
||||
# placeholder, builds and publishes the toolbox first, and builds agent-runner FROM that fresh
|
||||
# digest. For a LOCAL build on a local toolbox, override it:
|
||||
# infra/devcontainer/dev.sh build && AGENT_BASE_IMAGE=home-k3s-dev:local infra/agent/build-image.sh
|
||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
|
||||
AGENT_BASE_IMAGE=gitea.home.keithjackson.dev/home-network/toolbox@sha256:0000000000000000000000000000000000000000000000000000000000000000
|
||||
# The toolbox image the runner is layered on. Built by `infra/devcontainer/dev.sh build`
|
||||
# (tag `home-k3s-dev:local`); infra/agent/build-image.sh asserts it exists rather than
|
||||
# silently building a different base.
|
||||
AGENT_BASE_IMAGE=home-k3s-dev:local
|
||||
|
||||
# Node.js — the runtime for the Claude Code CLI, installed from the Fedora base's module
|
||||
# stream. A major version, not a patch: the CLI supports the active LTS line and the
|
||||
|
||||
@@ -80,12 +80,6 @@ spec:
|
||||
# Replaced by the dispatcher with the digest-pinned image. A tag here would be
|
||||
# DENIED by admission; the dispatcher refuses a non-digest PM_WORKER_IMAGE at
|
||||
# startup so the error names the real cause.
|
||||
#
|
||||
# DELIBERATELY ALL-ZERO, AND DELIBERATELY NOT TRACKED BY RENOVATE. This file is
|
||||
# baked INTO agent-runner, so it cannot carry agent-runner's own digest: bumping it
|
||||
# would change the image's inputs, CI would publish a new digest, Renovate would bump
|
||||
# it again — forever. pm-dispatch overwrites this field from PM_WORKER_IMAGE
|
||||
# (platform/pm-agent/configmap.yaml, which IS tracked) on every Job it creates.
|
||||
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
|
||||
imagePullPolicy: IfNotPresent
|
||||
# The pinned entrypoint, with NO args. Args on a pinned command are
|
||||
|
||||
@@ -107,31 +107,3 @@ k3s_base_domain: "home.keithjackson.dev"
|
||||
|
||||
# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
|
||||
k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
|
||||
|
||||
# --- Graceful node shutdown (kubelet) ----------------------------------------------------------
|
||||
# MEASURED 2026-10-17, during the UPS move: graceful node shutdown was NOT configured on any prod
|
||||
# node, so `shutdown -h` did NOT drain pods. systemd stops k3s, containerd goes with it, and every
|
||||
# pod is SIGKILLed where it stands -- including 9 CloudNativePG instances mid-checkpoint, the UniFi
|
||||
# VM, and 25 attached Longhorn volumes. The whole-cluster shutdown that day had to quiesce
|
||||
# everything BY HAND first (runbooks/cluster-maintenance-shutdown.md), and the one volume that did
|
||||
# not finish detaching came back `degraded` and had to rebuild.
|
||||
#
|
||||
# With these set, the kubelet takes a systemd inhibitor lock and, on shutdown, evicts ordinary pods
|
||||
# first and critical ones last, so a plain `shutdown -h`/reboot drains the node. That makes the
|
||||
# manual scale-down in the runbook a belt-and-braces step for a PLANNED whole-cluster shutdown
|
||||
# rather than the only thing standing between a reboot and a hard kill.
|
||||
#
|
||||
# 90s/20s, not larger: the budget is spent on every reboot and node upgrade, and the kubelet waits
|
||||
# the FULL ordinary period before it starts on critical pods. Postgres does a fast shutdown in
|
||||
# well under 90s here, and Longhorn needs only enough time to flush and detach.
|
||||
k3s_kubelet_shutdown_grace_period: "90s"
|
||||
k3s_kubelet_shutdown_grace_period_critical_pods: "20s"
|
||||
|
||||
# *** THE HALF THAT IS EASY TO MISS, AND SILENTLY BREAKS THE OTHER HALF. ***
|
||||
# Graceful node shutdown works by the kubelet holding a logind INHIBITOR DELAY lock. systemd caps
|
||||
# how long any inhibitor may delay shutdown at `InhibitDelayMaxSec`, which DEFAULTS TO 5 SECONDS.
|
||||
# If that cap is below the grace period above, systemd proceeds anyway and the drain is truncated
|
||||
# to ~5s -- i.e. the feature appears configured, reports no error, and still hard-kills pods. So
|
||||
# this MUST be >= k3s_kubelet_shutdown_grace_period; the role asserts it rather than trusting it.
|
||||
# Set a little above the kubelet's total (90s + 20s) to leave headroom.
|
||||
k3s_logind_inhibit_delay_max_sec: 120
|
||||
@@ -15,12 +15,3 @@
|
||||
port: "{{ k3s_api_port }}"
|
||||
timeout: 180
|
||||
listen: Restart k3s
|
||||
|
||||
# logind re-reads InhibitDelayMaxSec only on restart. Restarting logind does NOT log users out or
|
||||
# disturb k3s -- it manages login sessions and inhibitor locks, not workloads. The kubelet picks up
|
||||
# the new cap when it next takes its lock, so no k3s restart is needed for THIS half.
|
||||
- name: Restart logind
|
||||
ansible.builtin.systemd_service:
|
||||
name: systemd-logind
|
||||
state: restarted
|
||||
listen: Restart logind
|
||||
@@ -86,16 +86,6 @@
|
||||
# k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
|
||||
# BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
|
||||
# config.yaml is unchanged and the Restart handler does not fire.
|
||||
# ── Graceful node shutdown ──────────────────────────────────────────────────────────────
|
||||
# Without these the kubelet takes no systemd inhibitor lock, so `shutdown -h` stops
|
||||
# containerd and SIGKILLs every pod where it stands -- MEASURED on all three prod nodes
|
||||
# 2026-10-17 (defaults/main.yml has the detail). The companion logind drop-in below raises
|
||||
# InhibitDelayMaxSec, WITHOUT WHICH systemd caps the drain at ~5s and these values are
|
||||
# quietly ignored.
|
||||
kubelet-arg:
|
||||
- "shutdown-grace-period={{ k3s_kubelet_shutdown_grace_period }}"
|
||||
- "shutdown-grace-period-critical-pods={{ k3s_kubelet_shutdown_grace_period_critical_pods }}"
|
||||
|
||||
kube-apiserver-arg:
|
||||
- "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
|
||||
- "oidc-client-id=kubernetes"
|
||||
@@ -193,46 +183,3 @@
|
||||
host: "127.0.0.1"
|
||||
port: "{{ k3s_api_port }}"
|
||||
timeout: 180
|
||||
|
||||
# ── Graceful node shutdown: the systemd half — tag `shutdown` ──────────────────────────────────
|
||||
# The kubelet's shutdown-grace-period is ENFORCED BY logind, not by the kubelet: the kubelet holds
|
||||
# an inhibitor DELAY lock, and systemd will wait at most `InhibitDelayMaxSec` for it (default 5s)
|
||||
# before proceeding regardless. A grace period above that cap is therefore silently truncated --
|
||||
# the feature looks configured, logs no error, and still hard-kills pods. Assert the relationship
|
||||
# instead of documenting it, because the failure is invisible.
|
||||
- name: Assert the logind inhibitor cap can actually cover the kubelet grace period
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- (k3s_logind_inhibit_delay_max_sec | int) >= (k3s_kubelet_shutdown_grace_period | regex_replace('[^0-9]', '') | int)
|
||||
fail_msg: >-
|
||||
k3s_logind_inhibit_delay_max_sec ({{ k3s_logind_inhibit_delay_max_sec }}s) is below
|
||||
k3s_kubelet_shutdown_grace_period ({{ k3s_kubelet_shutdown_grace_period }}). systemd would
|
||||
stop waiting for the kubelet's inhibitor lock before the drain finished, so pods would be
|
||||
SIGKILLed anyway and graceful node shutdown would be configured but ineffective.
|
||||
quiet: true
|
||||
tags: [shutdown]
|
||||
|
||||
- name: Ensure /etc/systemd/logind.conf.d exists
|
||||
ansible.builtin.file:
|
||||
path: /etc/systemd/logind.conf.d
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
tags: [shutdown]
|
||||
|
||||
# A drop-in, not an edit of logind.conf: the distro file stays pristine and `state: absent` here
|
||||
# would fully revert the change.
|
||||
- name: Raise the logind inhibitor delay cap so the kubelet can finish draining
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/systemd/logind.conf.d/10-k3s-graceful-shutdown.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# Managed by Ansible (roles/k3s_server). See defaults/main.yml.
|
||||
# MUST stay >= kubelet shutdown-grace-period, or the drain is truncated to this value.
|
||||
[Login]
|
||||
InhibitDelayMaxSec={{ k3s_logind_inhibit_delay_max_sec }}
|
||||
notify: Restart logind
|
||||
tags: [shutdown]
|
||||
@@ -8,10 +8,7 @@
|
||||
# packages are retained as harmless disk-image utilities. Fedora base is pinned for
|
||||
# reproducibility.
|
||||
ARG FEDORA_VERSION=44
|
||||
# Stage `toolbox` is the image that ships. Stage `verify` (end of file) is a TEST stage: it is
|
||||
# built FROM the finished image and only runs assertions, and it is never pushed. Build with
|
||||
# `--target toolbox` for the image and `--target verify` to prove it (build-image.sh does both).
|
||||
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS toolbox
|
||||
FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION}
|
||||
|
||||
ARG OPENTOFU_VERSION
|
||||
ARG AGE_VERSION
|
||||
@@ -98,48 +95,3 @@ RUN { echo "opentofu=${OPENTOFU_VERSION}"; echo "age=${AGE_VERSION}"; echo "sops
|
||||
|
||||
WORKDIR /work
|
||||
CMD ["/bin/bash"]
|
||||
|
||||
# =============================================================================
|
||||
# verify — a TEST stage, never shipped. It runs every assertion against the finished image AT
|
||||
# BUILD TIME, so CI needs no container runtime to prove the image: the image pipeline builds
|
||||
# this target on rootless BuildKit (platform/buildkit/), where `docker run` does not exist.
|
||||
# `RUN --network=none` reproduces the old `run --network none` exactly, and the first check
|
||||
# below proves it (it fails if any network is reachable).
|
||||
# =============================================================================
|
||||
FROM toolbox AS verify
|
||||
ARG PROMTOOL_VERSION
|
||||
ARG KUBECTL_VERSION
|
||||
ARG CHAINSAW_VERSION
|
||||
ARG KYVERNO_VERSION
|
||||
ARG SOPS_VERSION
|
||||
RUN --network=none /bin/bash -euo pipefail <<'VERIFY'
|
||||
echo "--- no network (the build-time stand-in for run --network none) ---"
|
||||
if getent hosts registry.fedoraproject.org >/dev/null 2>&1 || curl -sS --max-time 3 -o /dev/null https://1.1.1.1 2>/dev/null; then
|
||||
echo "UNEXPECTED: network reachable in the verify stage"; exit 1
|
||||
fi
|
||||
echo " confirmed: no network"
|
||||
echo "--- baked versions ---"; cat /etc/dev-toolchain-versions
|
||||
echo "--- every tool is on PATH ---"
|
||||
for t in tofu age age-keygen sops flux kubectl helm kyverno kyverno-json chainsaw yq \
|
||||
yamllint check-jsonschema promtool ansible-playbook git make jq diff python3 \
|
||||
wg qrencode; do
|
||||
if command -v "$t" >/dev/null 2>&1; then printf " ok %s\n" "$t";
|
||||
else printf " MISSING %s\n" "$t"; missing=1; fi
|
||||
done
|
||||
[ -z "${missing:-}" ] || { echo "toolbox is missing tools"; exit 1; }
|
||||
echo "--- version-sensitive tools report their pins ---"
|
||||
: "${PROMTOOL_VERSION:?}" "${KUBECTL_VERSION:?}" "${CHAINSAW_VERSION:?}" "${KYVERNO_VERSION:?}" "${SOPS_VERSION:?}"
|
||||
check() { # name pin actual-output
|
||||
if printf "%s" "$3" | grep -qF "$2"; then printf " ok %-10s %s\n" "$1" "$2";
|
||||
else printf " DRIFT %-10s want %s, got: %s\n" "$1" "$2" "$(printf "%s" "$3" | head -1)"; drift=1; fi
|
||||
}
|
||||
check promtool "${PROMTOOL_VERSION}" "$(promtool --version 2>&1)"
|
||||
check kubectl "${KUBECTL_VERSION}" "$(kubectl version --client 2>&1)"
|
||||
check chainsaw "${CHAINSAW_VERSION}" "$(chainsaw version 2>&1)"
|
||||
check kyverno "${KYVERNO_VERSION}" "$(kyverno version 2>&1)"
|
||||
check sops "${SOPS_VERSION}" "$(sops --version --disable-version-check 2>&1 || sops --version 2>&1)"
|
||||
[ -z "${drift:-}" ] || { echo "a tool does not report its pinned version"; exit 1; }
|
||||
echo "--- python can import what the renderer and the gate need ---"
|
||||
python3 -c "import yaml; print(' PyYAML', yaml.__version__)"
|
||||
echo "TOOLBOX VERIFIED"
|
||||
VERIFY
|
||||
@@ -34,24 +34,6 @@ cd infra/devcontainer
|
||||
```
|
||||
`ENGINE=docker ./dev.sh` to use docker instead of rootless podman.
|
||||
|
||||
### The published toolbox
|
||||
|
||||
`./dev.sh build` delegates to [`build-image.sh`](build-image.sh). The recipe is the
|
||||
`Containerfile` plus that script's build-arg list, and the proof is the Containerfile's `verify`
|
||||
stage (every tool on `PATH`, the version-sensitive ones report their `versions.env` pins, no
|
||||
network). CI builds the same Containerfile on rootless BuildKit with the args from
|
||||
`build-image.sh --print-build-args`, and, on a push to `main` that changes
|
||||
`Containerfile`, `versions.env` or `build-image.sh`, **publishes** the result as
|
||||
`gitea.home.keithjackson.dev/home-network/toolbox` (tags `sha-<commit>` and `main`) —
|
||||
see [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). `agent-runner` is
|
||||
built `FROM` that published digest (`infra/agent/versions.env`, kept current by Renovate).
|
||||
Nobody pushes this image from a workstation. To use the published one locally instead of
|
||||
building, after `podman login gitea.home.keithjackson.dev` (a token with `read:package`):
|
||||
|
||||
```bash
|
||||
IMAGE=gitea.home.keithjackson.dev/home-network/toolbox:main ./dev.sh run make -C ci gate
|
||||
```
|
||||
|
||||
## Choosing a cluster — `CLUSTER=dev|prod`
|
||||
|
||||
```bash
|
||||
|
||||
@@ -1,84 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
###############################################################################
|
||||
# infra/devcontainer/build-image.sh — build the pinned dev `toolbox` image and prove it works.
|
||||
#
|
||||
# ./build-image.sh # build + verify (tag: home-k3s-dev:local)
|
||||
# ./build-image.sh --no-verify # build only
|
||||
# IMAGE=foo:bar ./build-image.sh # build under another local name
|
||||
# ENGINE=docker ./build-image.sh # build with docker instead of podman
|
||||
# ./build-image.sh --print-build-args # print KEY=VALUE build args (what CI consumes)
|
||||
#
|
||||
# ONE BUILD DEFINITION, TWO CALLERS. `dev.sh build` calls this for local development. The CI
|
||||
# image pipeline (.gitea/workflows/images.yaml -> .gitea/workflows/scripts/images.sh) builds the
|
||||
# SAME Containerfile on rootless BuildKit (platform/buildkit/) — no container engine exists in a
|
||||
# CI job — taking its build args from `--print-build-args`, and PUBLISHES the result as
|
||||
# gitea.home.keithjackson.dev/home-network/toolbox. The Containerfile (including its `verify`
|
||||
# test stage) and the arg list here are the one recipe both use.
|
||||
#
|
||||
# THIS SCRIPT NEVER PUSHES. Publishing is CI's job (images.yaml, on push to main), not a
|
||||
# workstation's: an image that exists only because someone ran `podman push` from a laptop has
|
||||
# no build log, no recorded inputs and no verification anyone else can read.
|
||||
#
|
||||
# VERSIONS ARE NOT DEFINED HERE. infra/devcontainer/versions.env is the single source of truth.
|
||||
###############################################################################
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENGINE="${ENGINE:-podman}"
|
||||
IMAGE="${IMAGE:-home-k3s-dev:local}"
|
||||
|
||||
VERIFY=1
|
||||
PRINT_ARGS=0
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--no-verify) VERIFY=0 ;;
|
||||
--print-build-args) PRINT_ARGS=1 ;;
|
||||
-h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "unknown argument: $arg" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
log() { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
|
||||
die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
|
||||
|
||||
# shellcheck disable=SC1091
|
||||
source "${HERE}/versions.env"
|
||||
|
||||
# THE build args, in ONE place. `--print-build-args` hands exactly this list to the CI pipeline
|
||||
# (.gitea/workflows/scripts/images.sh), which builds on rootless BuildKit rather than calling
|
||||
# "$ENGINE build" — so CI and a local build cannot drift apart on a pin.
|
||||
BUILD_ARGS=(
|
||||
"FEDORA_VERSION=$FEDORA_VERSION" "OPENTOFU_VERSION=$OPENTOFU_VERSION" "AGE_VERSION=$AGE_VERSION"
|
||||
"SOPS_VERSION=$SOPS_VERSION" "FLUX_VERSION=$FLUX_VERSION" "KUBECTL_VERSION=$KUBECTL_VERSION"
|
||||
"HELM_VERSION=$HELM_VERSION" "KYVERNO_VERSION=$KYVERNO_VERSION"
|
||||
"KYVERNO_JSON_VERSION=$KYVERNO_JSON_VERSION" "CHAINSAW_VERSION=$CHAINSAW_VERSION"
|
||||
"YAMLLINT_VERSION=$YAMLLINT_VERSION" "YQ_VERSION=$YQ_VERSION"
|
||||
"CHECK_JSONSCHEMA_VERSION=$CHECK_JSONSCHEMA_VERSION" "PROMTOOL_VERSION=$PROMTOOL_VERSION"
|
||||
)
|
||||
if [ "${PRINT_ARGS}" -eq 1 ]; then printf '%s\n' "${BUILD_ARGS[@]}"; exit 0; fi
|
||||
|
||||
command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
|
||||
|
||||
ARG_FLAGS=()
|
||||
for a in "${BUILD_ARGS[@]}"; do ARG_FLAGS+=(--build-arg "$a"); done
|
||||
# Provenance: CI passes the commit it is building, so a pulled image names its source tree.
|
||||
LABEL_ARGS=()
|
||||
[ -z "${BUILD_REVISION:-}" ] || LABEL_ARGS+=(--label "org.opencontainers.image.revision=${BUILD_REVISION}")
|
||||
|
||||
# --- verify --------------------------------------------------------------------
|
||||
# The assertions live in the Containerfile's `verify` stage (every tool on PATH, the
|
||||
# version-sensitive ones report their versions.env pins — promtool above all, because
|
||||
# ci/versions.env asserts a promtool/Prometheus lockstep — and no network), so the SAME proof
|
||||
# runs here under podman and in CI under rootless BuildKit. A failing assertion fails the build.
|
||||
if [ "${VERIFY}" -eq 1 ]; then
|
||||
log "Verifying (Containerfile target: verify)"
|
||||
"$ENGINE" build "${ARG_FLAGS[@]}" --target verify -f "${HERE}/Containerfile" "${HERE}"
|
||||
log "Toolbox verified"
|
||||
fi
|
||||
|
||||
log "Building ${IMAGE}"
|
||||
"$ENGINE" build "${LABEL_ARGS[@]}" "${ARG_FLAGS[@]}" --target toolbox \
|
||||
-t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
|
||||
|
||||
log "Built ${IMAGE}"
|
||||
printf ' image id %s\n' "$("$ENGINE" image inspect --format '{{.Id}}' "$IMAGE")"
|
||||
@@ -72,10 +72,22 @@ esac
|
||||
CLUSTER_KUBECONFIG="${REPO_ROOT}/infra/ansible/.cache/${CLUSTER}.kubeconfig"
|
||||
|
||||
build() {
|
||||
# The build recipe lives in build-image.sh so that CI (.gitea/workflows/images.yaml), which
|
||||
# builds and PUBLISHES this image, and a local `dev.sh build` cannot drift apart. --no-verify
|
||||
# keeps the local loop as fast as it always was; CI runs the verification.
|
||||
IMAGE="$IMAGE" ENGINE="$ENGINE" "${HERE}/build-image.sh" --no-verify
|
||||
"$ENGINE" build \
|
||||
--build-arg FEDORA_VERSION="$FEDORA_VERSION" \
|
||||
--build-arg OPENTOFU_VERSION="$OPENTOFU_VERSION" \
|
||||
--build-arg AGE_VERSION="$AGE_VERSION" \
|
||||
--build-arg SOPS_VERSION="$SOPS_VERSION" \
|
||||
--build-arg FLUX_VERSION="$FLUX_VERSION" \
|
||||
--build-arg KUBECTL_VERSION="$KUBECTL_VERSION" \
|
||||
--build-arg HELM_VERSION="$HELM_VERSION" \
|
||||
--build-arg KYVERNO_VERSION="$KYVERNO_VERSION" \
|
||||
--build-arg KYVERNO_JSON_VERSION="$KYVERNO_JSON_VERSION" \
|
||||
--build-arg CHAINSAW_VERSION="$CHAINSAW_VERSION" \
|
||||
--build-arg YAMLLINT_VERSION="$YAMLLINT_VERSION" \
|
||||
--build-arg YQ_VERSION="$YQ_VERSION" \
|
||||
--build-arg CHECK_JSONSCHEMA_VERSION="$CHECK_JSONSCHEMA_VERSION" \
|
||||
--build-arg PROMTOOL_VERSION="$PROMTOOL_VERSION" \
|
||||
-t "$IMAGE" -f "${HERE}/Containerfile" "${HERE}"
|
||||
}
|
||||
|
||||
run() {
|
||||
|
||||
@@ -1,104 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Copy the image builder's CLIENT certificate from the cluster into the three repository Actions
|
||||
# secrets the image pipeline reads: BUILDKIT_CA, BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY.
|
||||
#
|
||||
# ./infra/gitea/buildkit-client-secret.sh # copy and verify
|
||||
# ./infra/gitea/buildkit-client-secret.sh --check # compare only; changes nothing
|
||||
#
|
||||
# RUNNING IT IS AN OPERATOR ACTION. It writes repository secrets.
|
||||
#
|
||||
# WHERE THE CERTIFICATE COMES FROM. platform/buildkit/certificates.yaml has cert-manager issue
|
||||
# `buildkit-client-ci` from a CA that exists only for the builder; buildkitd admits a connection
|
||||
# only with a certificate signed by that CA. CI jobs hold NO cluster credential (R-SEC-02), so they
|
||||
# cannot read the Secret themselves: this script is the one bridge, run by a human who has both
|
||||
# a kube context and the Gitea credential.
|
||||
#
|
||||
# WHEN TO RUN IT
|
||||
# 1. once, after Flux has applied platform/buildkit (Kustomization `buildkit` Ready, and
|
||||
# `kubectl -n buildkit get certificate buildkit-client-ci` READY=True);
|
||||
# 2. after every renewal of that certificate (duration 1y, renewed 30 days early). Until then
|
||||
# the image pipeline fails LOUDLY at its builder preflight — it never skips.
|
||||
# `--check` tells you whether the copy is stale.
|
||||
#
|
||||
# NOTHING SECRET IS PRINTED. The key goes from `kubectl get secret` into the Gitea secrets API
|
||||
# through shell variables and nowhere else. The script prints only names, sizes and the
|
||||
# certificate's subject and expiry (public data).
|
||||
#
|
||||
# NEEDS: kubectl with a context for the cluster (default k3s-prod), jq, curl, openssl, base64,
|
||||
# and the Gitea credential `git` already has stored for the host — the same requirements as
|
||||
# infra/gitea/conformance-trigger-kubeconfig.sh, run from the same place.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
NS=buildkit; SECRET=buildkit-client-ci; KCTX="${KCTX:-k3s-prod}"
|
||||
CHECK=0
|
||||
for a in "$@"; do
|
||||
case "$a" in
|
||||
--check) CHECK=1 ;;
|
||||
-h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "unknown argument: $a" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
die() { echo "ERROR: $*" >&2; exit 1; }
|
||||
for t in kubectl jq curl openssl base64 git; do command -v "$t" >/dev/null || die "missing required tool: $t"; done
|
||||
|
||||
echo "==> reading Secret ${NS}/${SECRET} (context ${KCTX}; values not shown)"
|
||||
js="$(kubectl --context "$KCTX" -n "$NS" get secret "$SECRET" -o json)" \
|
||||
|| die "cannot read ${NS}/${SECRET} — is platform/buildkit deployed and the Certificate Ready?"
|
||||
CA="$(jq -r '.data["ca.crt"] // empty' <<<"$js" | base64 -d)"
|
||||
CRT="$(jq -r '.data["tls.crt"] // empty' <<<"$js" | base64 -d)"
|
||||
KEY="$(jq -r '.data["tls.key"] // empty' <<<"$js" | base64 -d)"
|
||||
unset js
|
||||
[ -n "$CA" ] && [ -n "$CRT" ] && [ -n "$KEY" ] || die "${NS}/${SECRET} lacks ca.crt/tls.crt/tls.key"
|
||||
|
||||
# Sanity before anything is written: the leaf is a CLIENT cert issued by this CA, and the key
|
||||
# belongs to it. A wrong pairing would otherwise surface only as a TLS failure in CI.
|
||||
openssl verify -CAfile <(printf '%s\n' "$CA") <(printf '%s\n' "$CRT") >/dev/null 2>&1 \
|
||||
|| die "tls.crt does not verify against ca.crt"
|
||||
openssl x509 -noout -ext extendedKeyUsage <<<"$CRT" 2>/dev/null | grep -q "TLS Web Client Authentication" \
|
||||
|| die "tls.crt is not a client-auth certificate"
|
||||
[ "$(openssl x509 -noout -pubkey <<<"$CRT" | openssl sha256)" = "$(openssl pkey -pubout <<<"$KEY" 2>/dev/null | openssl sha256)" ] \
|
||||
|| die "tls.key does not match tls.crt"
|
||||
echo " $(openssl x509 -noout -subject <<<"$CRT") / $(openssl x509 -noout -enddate <<<"$CRT")"
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"; unset creds
|
||||
[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for $HOST"
|
||||
echo "==> authenticating to Gitea as '${U}' (secret not shown)"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets"
|
||||
|
||||
if [ "$CHECK" -eq 1 ]; then
|
||||
# Gitea never returns a secret's value, only its name and creation time. "Stale" therefore
|
||||
# means: the secret was created before the certificate's current notBefore.
|
||||
names="$(api "$API" | jq -c '[.[]? | {name, created_at}]')"
|
||||
nb="$(date -u -d "$(openssl x509 -noout -startdate <<<"$CRT" | cut -d= -f2)" +%s)"
|
||||
rc=0
|
||||
for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do
|
||||
c="$(jq -r --arg n "$n" '.[] | select(.name==$n) | .created_at' <<<"$names")"
|
||||
if [ -z "$c" ]; then echo " MISSING $n"; rc=1
|
||||
elif [ "$(date -u -d "$c" +%s)" -lt "$nb" ]; then echo " STALE $n (stored $c, certificate re-issued since)"; rc=1
|
||||
else echo " current $n (stored $c)"; fi
|
||||
done
|
||||
exit "$rc"
|
||||
fi
|
||||
|
||||
put() { # name value
|
||||
local code
|
||||
code="$(api -o /dev/null -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
|
||||
-d "$(jq -n --arg d "$2" '{data:$d}')" "${API}/$1")"
|
||||
[ "$code" = 201 ] || [ "$code" = 204 ] || die "PUT $1 failed: HTTP ${code}"
|
||||
echo " stored $1 (${#2} bytes, value not shown)"
|
||||
}
|
||||
echo "==> storing repository Actions secrets"
|
||||
put BUILDKIT_CA "$CA"
|
||||
put BUILDKIT_CLIENT_CERT "$CRT"
|
||||
put BUILDKIT_CLIENT_KEY "$KEY"
|
||||
unset CA CRT KEY
|
||||
|
||||
# VERIFY BY READING BACK (names only — Gitea never returns values). curl -sS exits 0 on an HTTP
|
||||
# error, which is how set-ci-token.sh once claimed a store that never happened.
|
||||
names="$(api "$API" | jq -r '[.[]?.name] | join(" ")')"
|
||||
for n in BUILDKIT_CA BUILDKIT_CLIENT_CERT BUILDKIT_CLIENT_KEY; do
|
||||
grep -qw "$n" <<<"$names" || die "stored $n but it is NOT listed (${names}) — refusing to claim success"
|
||||
done
|
||||
echo "==> stored and VERIFIED present. Re-run the images workflow (workflow_dispatch) to use them."
|
||||
@@ -252,16 +252,11 @@ step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secr
|
||||
|
||||
# R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
|
||||
# all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
|
||||
# EVALUATED, not grepped: the line is `CONFORMANCE_RUNNER_IMAGE="${..._REPO}@${..._DIGEST}"`, so
|
||||
# its literal text never contains 64 hex digits and the grep this replaced could never pass. The
|
||||
# all-zero digest is the "not published yet" placeholder (Renovate replaces it after CI's first
|
||||
# publish, .gitea/workflows/images.yaml) and must not count as pinned.
|
||||
VERSIONS="${REPO_ROOT}/conformance/versions.env"
|
||||
runner_img="$( (set +u; . "$VERSIONS" >/dev/null 2>&1; printf '%s' "${CONFORMANCE_RUNNER_IMAGE:-}") )"
|
||||
if [[ "$runner_img" =~ @sha256:[0-9a-f]{64}$ ]] && [[ ! "$runner_img" =~ @sha256:0{64}$ ]]; then
|
||||
pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env (${runner_img##*@})"
|
||||
if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
|
||||
pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
|
||||
else
|
||||
fail_arm "conformance/versions.env has no PUBLISHED digest for CONFORMANCE_RUNNER_IMAGE (got '${runner_img}') — the image pipeline has not published it yet, or Renovate's bump is unmerged (plan §3 item 1)"
|
||||
fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
|
||||
fi
|
||||
|
||||
# R2: the Job template is the real thing, not the skeleton: the admitted image AND an
|
||||
|
||||
@@ -1,130 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Registry retention for this repo's OWN images (toolbox, agent-runner, conformance-runner):
|
||||
# keep what is deployed and what is recent, delete the rest. DRY RUN unless --apply.
|
||||
#
|
||||
# ./infra/gitea/registry-retention.sh # show what would be deleted
|
||||
# ./infra/gitea/registry-retention.sh --apply # delete it
|
||||
# KEEP=15 ./infra/gitea/registry-retention.sh # keep the 15 newest sha- tags (default 10)
|
||||
#
|
||||
# RUNNING IT IS AN OPERATOR ACTION. It deletes registry content, so it is not wired into CI and
|
||||
# no agent runs it with --apply.
|
||||
#
|
||||
# WHY A SCRIPT, AND NOT GITEA'S BUILT-IN "CLEANUP RULE" — measured against Gitea 1.24.3's source:
|
||||
#
|
||||
# 1. There is NO API for cleanup rules. routers/api/v1/api.go has no such route; the rules live
|
||||
# only behind the web UI's session+CSRF forms (routers/web/web.go, org settings
|
||||
# `/packages/rules/add`). A rule therefore cannot be created, read or drift-checked as code —
|
||||
# which is also why ci/gitea-settings.sh cannot assert one exists.
|
||||
# 2. A built-in rule would DELETE WHAT IS DEPLOYED. Every reference to these images is
|
||||
# digest-only (`<repo>@sha256:...`, required by admission). Gitea stores a manifest as a
|
||||
# file OF THE TAG VERSION it was pushed under (routers/api/packages/container/manifest.go
|
||||
# createPackageAndVersion), and resolves a by-digest pull by searching those files
|
||||
# (models/packages/container/search.go). So deleting the tag a digest was pushed under makes
|
||||
# that digest unpullable — MANIFEST_UNKNOWN, i.e. ImagePullBackOff on the next reschedule.
|
||||
# The built-in rule keeps "the newest N" and a name pattern, and knows nothing about which
|
||||
# digest git currently pins. Renovate's bump PRs can lag, so "newest N" is not a guarantee.
|
||||
#
|
||||
# WHAT IS KEPT, per image:
|
||||
# * the `main` and `latest` tags;
|
||||
# * the KEEP newest `sha-<commit>` tags (by creation time);
|
||||
# * EVERY tag whose manifest digest is referenced anywhere in the tracked tree at REF
|
||||
# (default origin/main) — i.e. whatever Flux is deploying, and the toolbox digest
|
||||
# agent-runner builds FROM;
|
||||
# * every version whose NAME is a digest (`sha256:...`): those are index children, and Gitea's
|
||||
# own cleanup handles them (services/packages/container/cleanup.go ShouldBeSkipped).
|
||||
# Everything else is deleted. Gitea's `cleanup_packages` cron then reclaims the unreferenced
|
||||
# blobs (default: daily, blobs older than 24h) — space comes back the day after, not instantly.
|
||||
#
|
||||
# THE STORAGE IT PROTECTS. Packages live on the SAME 10Gi Longhorn PVC as every repository and
|
||||
# Gitea's database (catalog/gitea.yaml). The toolbox is ~1.4 GB uncompressed and every main push
|
||||
# that touches its inputs adds a complete, non-reproducible copy. Unpruned, the registry is what
|
||||
# fills the volume the git host lives on.
|
||||
#
|
||||
# CREDENTIAL: the one `git` has stored for the host (as the other scripts here). It needs
|
||||
# `read:package` to list and `write:package` to --apply, and the account needs package write on
|
||||
# the org. The secret is never printed.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; NS=home-network
|
||||
IMAGES=(toolbox agent-runner conformance-runner)
|
||||
KEEP="${KEEP:-10}"
|
||||
REF="${REF:-origin/main}"
|
||||
APPLY=0
|
||||
for a in "$@"; do
|
||||
case "$a" in
|
||||
--apply) APPLY=1 ;;
|
||||
-h|--help) sed -n '2,/^set -euo/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) echo "unknown argument: $a" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
[[ "$KEEP" =~ ^[0-9]+$ ]] && [ "$KEEP" -ge 1 ] || { echo "KEEP must be a positive integer" >&2; exit 2; }
|
||||
command -v jq >/dev/null || { echo "jq is required" >&2; exit 1; }
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
|
||||
git -C "$REPO_ROOT" fetch -q origin main 2>/dev/null || echo " (could not fetch origin/main; using the local ${REF})" >&2
|
||||
git -C "$REPO_ROOT" rev-parse -q --verify "${REF}^{commit}" >/dev/null || { echo "no such ref: ${REF}" >&2; exit 1; }
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
|
||||
echo "authenticating as '${U}' (secret not shown)"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
REG_TOKEN="$(api "https://${HOST}/v2/token?service=container_registry" | jq -r '.token // empty')"
|
||||
[ -n "$REG_TOKEN" ] || { echo "registry token exchange failed (does the credential have read:package?)" >&2; exit 1; }
|
||||
manifest_digest() { # <image> <tag> -> digest, or nothing
|
||||
curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "$REG_TOKEN") \
|
||||
-H 'Accept: application/vnd.oci.image.manifest.v1+json' \
|
||||
-H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
|
||||
-H 'Accept: application/vnd.oci.image.index.v1+json' \
|
||||
-H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
|
||||
"https://${HOST}/v2/${NS}/$1/manifests/$2" \
|
||||
| tr -d '\r' | awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' | tail -1
|
||||
}
|
||||
|
||||
TOTAL_DEL=0
|
||||
for img in "${IMAGES[@]}"; do
|
||||
echo
|
||||
echo "==> ${NS}/${img}"
|
||||
# Digests git pins at REF. The marker convention puts the digest on the line after
|
||||
# `# renovate-digest: image=<repo>`; the plain `<repo>@sha256:` grep catches any other use.
|
||||
repo="${HOST}/${NS}/${img}"
|
||||
pinned="$( { git -C "$REPO_ROOT" grep -h -A1 -F "renovate-digest: image=${repo} " "$REF" -- . 2>/dev/null || true
|
||||
git -C "$REPO_ROOT" grep -h -F "${repo}@sha256:" "$REF" -- . 2>/dev/null || true; } \
|
||||
| grep -oE 'sha256:[0-9a-f]{64}' | grep -v 'sha256:0\{64\}' | sort -u || true)"
|
||||
echo " pinned at ${REF}: $(wc -w <<<"$pinned") digest(s)"
|
||||
|
||||
versions="[]"; page=1
|
||||
while :; do
|
||||
chunk="$(api "https://${HOST}/api/v1/packages/${OWNER}?type=container&q=${img}&limit=50&page=${page}")"
|
||||
jq -e 'type=="array"' >/dev/null <<<"$chunk" || { echo " package list failed: $(head -c 200 <<<"$chunk")" >&2; exit 1; }
|
||||
[ "$(jq length <<<"$chunk")" -gt 0 ] || break
|
||||
versions="$(jq -c --argjson a "$versions" --arg n "$img" '$a + [ .[] | select(.name==$n) | {version, created_at} ]' <<<"$chunk")"
|
||||
page=$((page + 1))
|
||||
done
|
||||
echo " versions: $(jq length <<<"$versions")"
|
||||
|
||||
keep_sha="$(jq -r '[ .[] | select(.version|startswith("sha-")) ] | sort_by(.created_at) | reverse | .[:'"$KEEP"'][] .version' <<<"$versions")"
|
||||
while IFS= read -r v; do
|
||||
[ -n "$v" ] || continue
|
||||
case "$v" in
|
||||
main|latest) echo " keep ${v} (moving tag)"; continue ;;
|
||||
sha256:*) echo " keep ${v} (digest-named version; Gitea's cleanup owns those)"; continue ;;
|
||||
esac
|
||||
if grep -qxF "$v" <<<"$keep_sha"; then echo " keep ${v} (among the ${KEEP} newest)"; continue; fi
|
||||
d="$(manifest_digest "$img" "$v")"
|
||||
if [ -z "$d" ]; then echo " keep ${v} (could not read its digest — never delete what cannot be checked)"; continue; fi
|
||||
if grep -qxF "$d" <<<"$pinned"; then echo " keep ${v} (${d} is pinned at ${REF})"; continue; fi
|
||||
if [ "$APPLY" -eq 1 ]; then
|
||||
code="$(api -o /dev/null -w '%{http_code}' -X DELETE \
|
||||
"https://${HOST}/api/v1/packages/${OWNER}/container/${img}/$(jq -rn --arg v "$v" '$v|@uri')")"
|
||||
if [ "$code" = 204 ]; then echo " DELETED ${v} (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1));
|
||||
else echo " FAILED ${v} HTTP ${code}" >&2; fi
|
||||
else
|
||||
echo " would delete ${v} (${d})"; TOTAL_DEL=$((TOTAL_DEL + 1))
|
||||
fi
|
||||
done < <(jq -r '.[].version' <<<"$versions")
|
||||
done
|
||||
|
||||
echo
|
||||
if [ "$APPLY" -eq 1 ]; then echo "deleted ${TOTAL_DEL} version(s). Blobs are reclaimed by Gitea's cleanup_packages cron."
|
||||
else echo "DRY RUN: ${TOTAL_DEL} version(s) would be deleted. Re-run with --apply to delete them."; fi
|
||||
@@ -1,265 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =================================================================================================
|
||||
# quiesce.sh — bring the prod cluster to a safe standstill before powering nodes off, and bring it
|
||||
# back afterwards. Companion to runbooks/cluster-maintenance-shutdown.md.
|
||||
# =================================================================================================
|
||||
# WHY THIS EXISTS. On 2026-10-17 all three nodes had to be powered off to move them onto a UPS, and
|
||||
# there was NO procedure for it: `infra/cluster.sh stop` is `virsh`-based and only ever applied to
|
||||
# the decommissioned dev cluster. The whole sequence was improvised live. This script is that
|
||||
# sequence, so the next person does not improvise it.
|
||||
#
|
||||
# WHY IT IS NOT JUST `shutdown -h`. Graceful node shutdown was unconfigured on every prod node that
|
||||
# day, so systemd stopped containerd and SIGKILLed every pod where it stood. roles/k3s_server now
|
||||
# sets shutdown-grace-period (+ the logind InhibitDelayMaxSec cap that actually enforces it), which
|
||||
# makes a bare reboot safe. This script is still the right tool for a PLANNED whole-cluster
|
||||
# shutdown, for three reasons the kubelet's drain does not cover:
|
||||
# 1. The UniFi VM needs an ACPI guest shutdown, not an eviction.
|
||||
# 2. CloudNativePG wants hibernation, not pod deletion -- and hibernation is IMPERATIVE state
|
||||
# that Flux will not set or clear for you.
|
||||
# 3. Draining three nodes one at a time makes pods RESCHEDULE onto the survivors, churning
|
||||
# Longhorn replicas for no reason when the whole cluster is going down anyway. Scaling in
|
||||
# place avoids all of it.
|
||||
#
|
||||
# SAFE TO RE-RUN. Every step is idempotent and `resume` is the exact inverse of `quiesce`.
|
||||
#
|
||||
# WHAT IT DELIBERATELY DOES NOT DO: power anything off. Node poweroff is an operator action
|
||||
# (CLAUDE.md §5) and needs the become password; the runbook gives the ansible one-liner. It also
|
||||
# does not touch TrueNAS (not in the inventory, SSH refused -- web UI only) or the router.
|
||||
set -euo pipefail
|
||||
|
||||
CONTEXT="${CONTEXT:-k3s-prod}"
|
||||
KUBECTL=(kubectl --context "$CONTEXT" --request-timeout=30s)
|
||||
TIMEOUT="${TIMEOUT:-600}"
|
||||
|
||||
# The three CloudNativePG clusters, as namespace/name. Hibernation is per-Cluster.
|
||||
CNPG=(keycloak/keycloak-db nextcloud-db/nextcloud-db semaphore-db/semaphore-db)
|
||||
|
||||
log() { printf '\033[34m[quiesce]\033[0m %s\n' "$*"; }
|
||||
ok() { printf '\033[32m[ ok ]\033[0m %s\n' "$*"; }
|
||||
warn() { printf '\033[33m[warn]\033[0m %s\n' "$*"; }
|
||||
die() { printf '\033[31m[fail]\033[0m %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
need() { command -v "$1" >/dev/null || die "$1 not on PATH — run this inside ./infra/devcontainer/dev.sh"; }
|
||||
|
||||
# ---------------------------------------------------------------- discovery
|
||||
# Every Deployment/StatefulSet whose pods mount a PVC — Longhorn OR the TrueNAS NFS bulk tier.
|
||||
# Computed from the LIVE cluster rather than hardcoded, so a newly onboarded service is covered
|
||||
# without anyone remembering to add it here. PROTECTED namespaces are excluded because scaling
|
||||
# them down would remove the very machinery that performs the detach (longhorn-system), keep the
|
||||
# API reachable (kube-system), or finish the hibernation and the VM shutdown (cnpg-system,
|
||||
# kubevirt).
|
||||
pvc_workloads() {
|
||||
"${KUBECTL[@]}" get pods -A -o json | python3 -c '
|
||||
import json,sys
|
||||
PROTECT={"longhorn-system","kube-system","cilium-secrets","cnpg-system","kubevirt","flux-system","cdi"}
|
||||
out={}
|
||||
for p in json.load(sys.stdin)["items"]:
|
||||
ns=p["metadata"]["namespace"]
|
||||
if ns in PROTECT: continue
|
||||
if not any(v.get("persistentVolumeClaim") for v in (p["spec"].get("volumes") or [])): continue
|
||||
ow=(p["metadata"].get("ownerReferences") or [])
|
||||
if not ow: continue
|
||||
k,n=ow[0]["kind"],ow[0]["name"]
|
||||
if k=="ReplicaSet": k,n="deployment",n.rsplit("-",1)[0]
|
||||
elif k=="StatefulSet": k="statefulset"
|
||||
else: continue # Jobs, and CNPG Clusters, are handled separately
|
||||
out[(ns,k,n)]=1
|
||||
for ns,k,n in sorted(out): print(ns,k,n)
|
||||
'
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- the PVC-deletion guard
|
||||
# *** DO NOT REMOVE THIS CHECK. IT EXISTS BECAUSE SCALING A StatefulSet DESTROYED DATA. ***
|
||||
# A StatefulSet may carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and then
|
||||
# scaling it to 0 DELETES ITS PVCs -- and with them the Longhorn volumes. MEASURED 2026-10-17: the
|
||||
# quiesce for the UPS move scaled `monitoring/loki` (whenScaled: Delete) to 0 and its
|
||||
# `storage-loki-0` PVC and volume were deleted outright. Loki stores with `type: filesystem`, so
|
||||
# that PVC was the only copy; 72h retention bounded the loss to a rolling window, but nothing about
|
||||
# the mechanism is bounded -- the same action on a Delete-policy StatefulSet holding real state
|
||||
# would be unrecoverable. The volume count going 25 -> 24 was the only visible symptom, and it was
|
||||
# first misread as a miscount.
|
||||
#
|
||||
# So: such a StatefulSet is NEVER scaled. It is left RUNNING and its volume hard-detaches at
|
||||
# poweroff, which costs a replica rebuild on restart (degraded -> healthy, automatic). A rebuild is
|
||||
# an enormously better outcome than a deletion, and that is the whole trade being made here.
|
||||
sts_deletes_pvc_on_scale() { # ns name -> 0 if scaling it would delete its PVCs
|
||||
local policy
|
||||
policy=$("${KUBECTL[@]}" -n "$1" get statefulset "$2" \
|
||||
-o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy.whenScaled}' 2>/dev/null || true)
|
||||
[ "$policy" = "Delete" ]
|
||||
}
|
||||
|
||||
wait_for() { # wait_for <description> <shell-condition>
|
||||
local desc="$1" cond="$2" waited=0
|
||||
log "waiting: $desc (up to ${TIMEOUT}s)"
|
||||
until eval "$cond"; do
|
||||
(( waited += 10 )); (( waited >= TIMEOUT )) && { warn "TIMED OUT waiting: $desc"; return 1; }
|
||||
sleep 10
|
||||
done
|
||||
ok "$desc"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- quiesce
|
||||
do_quiesce() {
|
||||
need kubectl; need python3
|
||||
|
||||
# Refuse to start if Longhorn is already unhealthy. Powering off on top of a degraded volume is
|
||||
# how one replica becomes zero; better to stop here and let it rebuild first.
|
||||
local unhealthy
|
||||
unhealthy=$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io \
|
||||
-o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)
|
||||
[ "$unhealthy" = "0" ] || die "$unhealthy Longhorn volume(s) are not healthy — fix or wait for rebuild BEFORE powering off"
|
||||
ok "Longhorn: all volumes healthy"
|
||||
|
||||
# 1. Flux first, or it reverts every scale-down below within its reconcile interval.
|
||||
log "suspending Flux Kustomizations"
|
||||
local k
|
||||
for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
|
||||
"${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":true}}' >/dev/null
|
||||
done
|
||||
ok "all Kustomizations suspended"
|
||||
|
||||
# 2. The VM needs an ACPI shutdown of the GUEST. KubeVirt does that on running:false; a plain
|
||||
# pod eviction would be a power cut to the VM.
|
||||
if "${KUBECTL[@]}" -n unifi get vm unifi >/dev/null 2>&1; then
|
||||
log "ACPI-stopping the UniFi VM"
|
||||
"${KUBECTL[@]}" -n unifi patch vm unifi --type=merge -p '{"spec":{"running":false}}' >/dev/null
|
||||
fi
|
||||
|
||||
# 3. Postgres: hibernation is a clean smart shutdown that KEEPS the PVCs. CNPG does one instance
|
||||
# at a time, primary last-ish, so this is not instant.
|
||||
log "hibernating CloudNativePG clusters"
|
||||
local c ns name
|
||||
for c in "${CNPG[@]}"; do
|
||||
ns="${c%%/*}"; name="${c##*/}"
|
||||
"${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
|
||||
cnpg.io/hibernation=on --overwrite >/dev/null 2>&1 || warn "could not hibernate $c"
|
||||
done
|
||||
|
||||
# 4. Everything holding a PVC, scaled in place. NFS consumers first so nothing holds a hard
|
||||
# TrueNAS mount; PiHole last because it is the LAN resolver.
|
||||
log "scaling down PVC-holding workloads"
|
||||
local line ns2 kind nm deferred=() skipped_delete_policy=()
|
||||
while read -r ns2 kind nm; do
|
||||
[ -z "${ns2:-}" ] && continue
|
||||
if [ "$kind" = "statefulset" ] && sts_deletes_pvc_on_scale "$ns2" "$nm"; then
|
||||
warn "SKIPPING $ns2/$nm — whenScaled: Delete would DELETE its PVC (see the guard above)."
|
||||
warn " Left running; its volume hard-detaches at poweroff and rebuilds on restart."
|
||||
skipped_delete_policy+=("$ns2/$nm")
|
||||
continue
|
||||
fi
|
||||
if [ "$ns2" = "pihole" ]; then deferred+=("$ns2 $kind $nm"); continue; fi
|
||||
"${KUBECTL[@]}" -n "$ns2" scale "$kind" "$nm" --replicas=0 >/dev/null 2>&1 \
|
||||
&& printf ' scaled 0 %s/%s %s\n' "$ns2" "$kind" "$nm" \
|
||||
|| warn "could not scale $ns2/$kind/$nm"
|
||||
done < <(pvc_workloads)
|
||||
for line in "${deferred[@]:-}"; do
|
||||
[ -z "${line:-}" ] && continue
|
||||
# shellcheck disable=SC2086
|
||||
set -- $line
|
||||
"${KUBECTL[@]}" -n "$1" scale "$2" "$3" --replicas=0 >/dev/null 2>&1 \
|
||||
&& printf ' scaled 0 %s/%s %s (LAN DNS — expect name resolution to stop)\n' "$1" "$2" "$3"
|
||||
done
|
||||
|
||||
# 5. The only assertion that matters. An attached volume at poweroff comes back degraded and
|
||||
# rebuilds; this is what we are buying with all of the above.
|
||||
wait_for "UniFi VMI gone" \
|
||||
'[ "$('"${KUBECTL[*]}"' -n unifi get vmi --no-headers 2>/dev/null | wc -l)" = "0" ]' || true
|
||||
wait_for "all Postgres instances stopped" \
|
||||
'[ "$('"${KUBECTL[*]}"' get pods -A --no-headers 2>/dev/null | grep -cE "(keycloak|nextcloud|semaphore)-db-[0-9]")" = "0" ]' || true
|
||||
wait_for "all Longhorn volumes detached" \
|
||||
'[ "$('"${KUBECTL[*]}"' -n longhorn-system get volumes.longhorn.io -o jsonpath="{range .items[*]}{.status.state}{\"\n\"}{end}" 2>/dev/null | grep -c attached)" = "0" ]' || \
|
||||
warn "some volumes are STILL ATTACHED — they will come back degraded and rebuild; acceptable, but note which"
|
||||
|
||||
if [ "${#skipped_delete_policy[@]}" -gt 0 ]; then
|
||||
echo
|
||||
warn "NOT scaled down (whenScaled: Delete — scaling them would delete their PVCs):"
|
||||
printf ' %s\n' "${skipped_delete_policy[@]}"
|
||||
warn "Expect these volumes to come back 'degraded' and rebuild. That is correct and automatic."
|
||||
fi
|
||||
echo
|
||||
ok "QUIESCED. Safe to power off."
|
||||
cat <<'NEXT'
|
||||
|
||||
Next, from inside the devcontainer (it holds the become password):
|
||||
cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
|
||||
-m shell -a 'shutdown -h +1 "maintenance"'
|
||||
|
||||
THEN, in this order and not before:
|
||||
1. TrueNAS (192.168.2.30) — web UI only; not in the ansible inventory and SSH is refused.
|
||||
2. Router / switch last.
|
||||
NEXT
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- resume
|
||||
do_resume() {
|
||||
need kubectl
|
||||
|
||||
# THE STEP THAT IS ALWAYS FORGOTTEN. Hibernation is imperative state: Flux neither sets nor
|
||||
# clears it, so resuming Flux alone leaves all three databases down and every app that needs
|
||||
# one looking broken for reasons that point nowhere near Postgres.
|
||||
log "un-hibernating CloudNativePG clusters"
|
||||
local c ns name
|
||||
for c in "${CNPG[@]}"; do
|
||||
ns="${c%%/*}"; name="${c##*/}"
|
||||
"${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
|
||||
cnpg.io/hibernation=off --overwrite >/dev/null 2>&1 || warn "could not un-hibernate $c"
|
||||
done
|
||||
|
||||
# Flux restores every replica count AND the VM's running:true from git, so nothing here needs to
|
||||
# undo the scale-downs by hand.
|
||||
log "resuming Flux Kustomizations"
|
||||
local k
|
||||
for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
|
||||
"${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":false}}' >/dev/null
|
||||
done
|
||||
ok "all Kustomizations resumed"
|
||||
|
||||
wait_for "Flux fully reconciled" \
|
||||
'[ "$('"${KUBECTL[*]}"' -n flux-system get kustomizations --no-headers 2>/dev/null | grep -vc True)" = "0" ]' || \
|
||||
warn "some Kustomizations are not Ready — the graph serialises behind platform-config, so give it a few minutes before worrying"
|
||||
|
||||
echo
|
||||
ok "RESUMED."
|
||||
cat <<'NEXT'
|
||||
|
||||
Known cold-start race, not a fault in this script: oauth2-proxy performs OIDC discovery at
|
||||
startup and exits 1 on the 503 it gets while Keycloak is still coming up, then sits in
|
||||
exponential CrashLoopBackOff. If a replica is stuck, delete the pod once:
|
||||
kubectl --context k3s-prod -n oauth2-proxy delete pod <name>
|
||||
NEXT
|
||||
}
|
||||
|
||||
do_status() {
|
||||
need kubectl
|
||||
printf 'nodes: %s Ready\n' "$("${KUBECTL[@]}" get nodes --no-headers | grep -c ' Ready')"
|
||||
printf 'flux: %s/%s Ready\n' \
|
||||
"$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | grep -c True)" \
|
||||
"$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | wc -l)"
|
||||
printf 'longhorn: %s attached, %s detached, %s not healthy\n' \
|
||||
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c attached)" \
|
||||
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c detached)" \
|
||||
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)"
|
||||
printf 'postgres: %s instance pod(s)\n' \
|
||||
"$("${KUBECTL[@]}" get pods -A --no-headers | grep -cE '(keycloak|nextcloud|semaphore)-db-[0-9]' || true)"
|
||||
printf 'unifi VM: %s VMI\n' "$("${KUBECTL[@]}" -n unifi get vmi --no-headers 2>/dev/null | wc -l)"
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
quiesce) do_quiesce ;;
|
||||
resume) do_resume ;;
|
||||
status) do_status ;;
|
||||
*) cat >&2 <<USAGE
|
||||
usage: $0 {quiesce|resume|status}
|
||||
|
||||
quiesce suspend Flux, ACPI-stop the UniFi VM, hibernate Postgres, scale down every
|
||||
PVC-holding workload, and wait for all Longhorn volumes to detach.
|
||||
Does NOT power anything off — see runbooks/cluster-maintenance-shutdown.md.
|
||||
resume the exact inverse: un-hibernate Postgres, resume Flux, wait for reconcile.
|
||||
status one-screen readout of nodes / Flux / Longhorn / Postgres / VM.
|
||||
|
||||
Run it inside the pinned toolbox, which supplies kubectl and the context:
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
|
||||
USAGE
|
||||
exit 2 ;;
|
||||
esac
|
||||
@@ -1,74 +0,0 @@
|
||||
# platform/buildkit — rootless BuildKit, the CI image builder
|
||||
|
||||
The builder behind [`.gitea/workflows/images.yaml`](../../.gitea/workflows/images.yaml). It
|
||||
builds `toolbox`, `agent-runner` and `conformance-runner`, and on `main` pushes them to the
|
||||
Gitea registry. Flux reconciles it as the Kustomization `buildkit`
|
||||
([`clusters/base/buildkit.yaml`](../../clusters/base/buildkit.yaml), `dependsOn: platform`).
|
||||
|
||||
## Why it exists (operator decision on PR #81, 2026-10-29)
|
||||
|
||||
CI jobs run as containers on the act runner's privileged DinD sidecar, and **cannot reach any
|
||||
Docker daemon**. act_runner 0.2.11 forces `docker_host: "-"` for a `tcp://` DinD, and gitea/act
|
||||
strips every bind mount when `valid_volumes` is empty. Measured on PR #81's first run. Giving jobs
|
||||
that daemon would make every branch push root on the node. So the rule is: **no job may reach a
|
||||
privileged daemon.** Builds go to this rootless daemon over mTLS instead.
|
||||
|
||||
## Shape
|
||||
|
||||
| Piece | What it is |
|
||||
|---|---|
|
||||
| `deployment.yaml` | `moby/buildkit:v0.34.1-rootless`, pinned by digest. Not privileged: it runs as uid 1000, which rootlesskit maps to uid 0 inside a user namespace only. The `securityContext` is commented line by line with how each item was measured |
|
||||
| `certificates.yaml` | cert-manager: a self-signed CA used only here, plus the server cert `buildkitd-tls` and the CI client cert `buildkit-client-ci` |
|
||||
| `service.yaml` | ClusterIP `buildkitd.buildkit.svc.cluster.local:1234`, mTLS only. Never on the Gateway |
|
||||
| `networkpolicy.yaml` | Default-deny, plus two narrow allows. **Ingress** is only from the act-runner pod: a job's traffic is masqueraded to that pod's IP by DinD (measured), so that pod is the identity Cilium sees. **Egress** is DNS, world :443/:80 for downloads, and the Gateway (`ingress` entity) for the registry push |
|
||||
| `configmap.yaml` | `buildkitd.toml`: cache GC capped at 4 GB, because the cache is an `emptyDir` on node disk |
|
||||
| `versions.env` | The CI side: pinned `buildx` (with its sha256) and the address and certificate name CI dials. Not a Kubernetes resource |
|
||||
|
||||
## Admission
|
||||
|
||||
**No Kyverno exception is needed.**
|
||||
- Every ClusterPolicy that is actually applied (the list in
|
||||
[`platform-config/kustomization.yaml`](../../platform-config/kustomization.yaml)) matches only
|
||||
namespaces labelled `homelab.keithjackson.dev/app-namespace: "true"`. This namespace, like
|
||||
`gitea-actions`, does not carry that label.
|
||||
- The job-shape policies (`pm-agent-job-shape`, `conformance-trigger-job-shape`) match only their
|
||||
own namespaces.
|
||||
- There is no cluster-wide Pod Security default. This namespace is deliberately left without a
|
||||
`pod-security` label, because PSS `baseline` forbids the `Unconfined` seccomp and AppArmor
|
||||
profiles that rootless BuildKit needs.
|
||||
|
||||
## Prerequisites (operator, in order)
|
||||
|
||||
1. **Unprivileged user namespaces on the nodes.** Already true. Measured 2026-10-29 on all three
|
||||
Debian 13.7 nodes (kernel 7.1.8): `kernel.unprivileged_userns_clone = 1`,
|
||||
`user.max_user_namespaces = 127206`, and no Ubuntu-style AppArmor userns restriction.
|
||||
2. **Merge.** Flux applies this directory. Check that the Kustomization `buildkit` is Ready and
|
||||
that all three Certificates in namespace `buildkit` are READY.
|
||||
3. **Copy the client certificate to CI.** Run
|
||||
[`infra/gitea/buildkit-client-secret.sh`](../../infra/gitea/buildkit-client-secret.sh).
|
||||
Re-run it yearly, after renewal; `--check` reports whether the copy is stale.
|
||||
4. **Restart buildkitd after the server certificate renews.** It reads the certificate at start,
|
||||
and cert-manager renews it 30 days before expiry, so a restart inside that window is needed.
|
||||
|
||||
Until step 3 is done, the `images` workflow fails at its builder preflight with a message naming
|
||||
the cause. It never skips.
|
||||
|
||||
## What a build step can do
|
||||
|
||||
The daemon runs with `--oci-worker-no-process-sandbox`, so build steps share its PID namespace.
|
||||
buildkitd warns about this itself at startup.
|
||||
|
||||
- **A build step CAN** signal or ptrace the daemon and read what it can read, including this
|
||||
pod's TLS key and the build cache.
|
||||
- **A build step CANNOT** reach the node. Everything runs as uid 1000 in a user namespace,
|
||||
unprivileged, with no host mounts.
|
||||
- **Who can run one:** anyone holding the client certificate, i.e. every CI job, including pull
|
||||
requests.
|
||||
|
||||
## Local equivalent
|
||||
|
||||
The pinned image runs under rootless podman with the same restrictions:
|
||||
`--cap-drop ALL --cap-add SETUID --cap-add SETGID --security-opt seccomp=unconfined --read-only`,
|
||||
plus tmpfs at the same five paths. That is how the `securityContext` was measured. Rootless podman
|
||||
maps only 65,537 uids, so the image's `/etc/subuid` (`user:100000:65536`) must be overridden with
|
||||
two ranges that fit: `user:1:999` and `user:1001:64535`.
|
||||
@@ -1,115 +0,0 @@
|
||||
---
|
||||
# mTLS for buildkitd, issued by cert-manager from a CA that exists ONLY for this purpose.
|
||||
#
|
||||
# buildkit-selfsigned (Issuer) --signs--> buildkit-ca (CA Certificate, 10y)
|
||||
# buildkit-ca (Issuer) --signs--> buildkitd-tls (server, the daemon presents it)
|
||||
# --signs--> buildkit-client-ci (client, CI presents it)
|
||||
#
|
||||
# buildkitd verifies EVERY client certificate against this CA (--tlscacert), so holding a client
|
||||
# certificate signed by it is what "may build" means. Nothing else in the cluster trusts this CA,
|
||||
# and it trusts nothing else.
|
||||
#
|
||||
# THE CLIENT CERTIFICATE REACHES CI AS ACTIONS SECRETS, not through the cluster: CI jobs hold no
|
||||
# cluster credential (R-SEC-02). infra/gitea/buildkit-client-secret.sh copies ca.crt/tls.crt/
|
||||
# tls.key from the Secret below into the repository secrets BUILDKIT_CA, BUILDKIT_CLIENT_CERT and
|
||||
# BUILDKIT_CLIENT_KEY without printing them. Re-run it after every renewal (see renewBefore).
|
||||
#
|
||||
# WHY A PRIVATE CA AND NOT LET'S ENCRYPT. The platform's TLS rule (R-TLS-01/02, docs/08,
|
||||
# ADR-0006: ACME DNS-01) is about EXTERNALLY REACHABLE services; ADR-0006 itself says "mTLS between
|
||||
# workloads is a separate concern … not covered here", and a public CA cannot issue for a
|
||||
# `*.svc.cluster.local` name or a client-auth certificate anyway. buildkitd is ClusterIP-only. Each
|
||||
# Certificate below is therefore annotated `homelab.keithjackson.dev/tls-scope: workload-mtls`,
|
||||
# which the L0 check tls-lifecycle-automated accepts ONLY if the certificate proves it: no
|
||||
# Gateway serves its Secret, every name is cluster-internal, and the issuer is a namespaced
|
||||
# selfSigned/CA Issuer whose CA is itself issued here. Issuance and renewal stay unattended.
|
||||
#
|
||||
# ROTATION. cert-manager renews both leaves `renewBefore` their expiry and rewrites the Secrets.
|
||||
# * server: buildkitd reads its certificate at START, so the pod must restart inside the 30-day
|
||||
# renewal window (a node reboot by kured, a Deployment rollout, or a manual restart).
|
||||
# * client: the Actions secrets are a COPY; re-run the script. Until then CI fails loudly at
|
||||
# its TLS preflight (.gitea/workflows/scripts/images.sh), never silently.
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Issuer
|
||||
metadata:
|
||||
name: buildkit-selfsigned
|
||||
namespace: buildkit
|
||||
spec:
|
||||
selfSigned: {}
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: buildkit-ca
|
||||
namespace: buildkit
|
||||
annotations:
|
||||
homelab.keithjackson.dev/tls-scope: workload-mtls # see the header: R-TLS-01 is about external TLS
|
||||
spec:
|
||||
isCA: true
|
||||
commonName: buildkit-ca
|
||||
secretName: buildkit-ca
|
||||
duration: 87600h # 10 years: rotating the CA means re-issuing everything and re-copying
|
||||
privateKey:
|
||||
algorithm: ECDSA
|
||||
size: 256
|
||||
issuerRef:
|
||||
name: buildkit-selfsigned
|
||||
kind: Issuer
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Issuer
|
||||
metadata:
|
||||
name: buildkit-ca
|
||||
namespace: buildkit
|
||||
spec:
|
||||
ca:
|
||||
secretName: buildkit-ca
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: buildkitd-tls
|
||||
namespace: buildkit
|
||||
annotations:
|
||||
homelab.keithjackson.dev/tls-scope: workload-mtls
|
||||
spec:
|
||||
secretName: buildkitd-tls
|
||||
commonName: buildkitd.buildkit.svc.cluster.local
|
||||
dnsNames:
|
||||
- buildkitd.buildkit.svc.cluster.local # what CI dials (platform/buildkit/versions.env)
|
||||
- buildkitd.buildkit.svc
|
||||
- buildkitd
|
||||
duration: 8760h
|
||||
renewBefore: 720h
|
||||
usages:
|
||||
- server auth
|
||||
- digital signature
|
||||
privateKey:
|
||||
algorithm: ECDSA
|
||||
size: 256
|
||||
rotationPolicy: Always
|
||||
issuerRef:
|
||||
name: buildkit-ca
|
||||
kind: Issuer
|
||||
---
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: buildkit-client-ci
|
||||
namespace: buildkit
|
||||
annotations:
|
||||
homelab.keithjackson.dev/tls-scope: workload-mtls
|
||||
spec:
|
||||
secretName: buildkit-client-ci
|
||||
commonName: gitea-actions-image-pipeline
|
||||
duration: 8760h
|
||||
renewBefore: 720h
|
||||
usages:
|
||||
- client auth
|
||||
- digital signature
|
||||
privateKey:
|
||||
algorithm: ECDSA
|
||||
size: 256
|
||||
rotationPolicy: Always
|
||||
issuerRef:
|
||||
name: buildkit-ca
|
||||
kind: Issuer
|
||||
@@ -1,24 +0,0 @@
|
||||
---
|
||||
# buildkitd.toml. Validated against the pinned image (v0.34.1-rootless): `buildctl debug workers
|
||||
# -v` reports "Maximum used space: 4.29GB / Reserved space: 1.07GB" for the default policy.
|
||||
#
|
||||
# WHY 4GB. The cache lives in an emptyDir on the NODE's disk (deployment.yaml), and the nodes had
|
||||
# 7-9 GB free when this was written. The largest image (toolbox) is ~1.4 GB uncompressed; the
|
||||
# three images share their Fedora base. The emptyDir's sizeLimit sits above this cap, so GC
|
||||
# trims the cache well before the kubelet would evict the pod for exceeding it.
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: buildkitd-config
|
||||
namespace: buildkit
|
||||
labels:
|
||||
app.kubernetes.io/name: buildkitd
|
||||
data:
|
||||
buildkitd.toml: |
|
||||
[worker.oci]
|
||||
enabled = true
|
||||
gc = true
|
||||
maxUsedSpace = "4GB"
|
||||
reservedSpace = "1GB"
|
||||
[worker.containerd]
|
||||
enabled = false
|
||||
@@ -1,142 +0,0 @@
|
||||
---
|
||||
# buildkitd, ROOTLESS. The builder behind .gitea/workflows/images.yaml; see README.md.
|
||||
#
|
||||
# THE SECURITY CONTEXT IS THE BOUNDARY, AND EVERY LINE OF IT WAS MEASURED (2026-10-29, the pinned
|
||||
# image under podman with the same restrictions; decisions/2026-10-29-image-pipeline.md):
|
||||
# * privileged: false — never. This is the whole point of the operator's decision on PR #81:
|
||||
# the alternative was handing CI the privileged DinD daemon, i.e. root on the node.
|
||||
# * runAsUser/Group 1000 — the image's `user`; rootlesskit maps it to uid 0 INSIDE a user
|
||||
# namespace only. The nodes allow that (measured: kernel.unprivileged_userns_clone = 1,
|
||||
# user.max_user_namespaces = 127206 on all three Debian 13 nodes).
|
||||
# * capabilities: drop ALL, add SETUID + SETGID — the image's newuidmap/newgidmap are NOT
|
||||
# setuid; they carry FILE capabilities, which need SETUID/SETGID in the bounding set.
|
||||
# Measured: with drop ALL and nothing added, rootlesskit fails "newuidmap ... operation not
|
||||
# permitted" and the daemon never starts.
|
||||
# * allowPrivilegeEscalation: true — required for the same reason: no_new_privs (what `false`
|
||||
# sets) stops a file capability from ever taking effect. It does NOT grant privileges the
|
||||
# bounding set above does not contain.
|
||||
# * seccompProfile: Unconfined — measured: with the default seccomp profile the daemon starts
|
||||
# but every RUN step fails (the runtime cannot create the build step's namespaces/mounts).
|
||||
# * appArmorProfile: Unconfined — upstream's documented requirement (buildkit
|
||||
# examples/kubernetes, *.rootless.yaml); the nodes run AppArmor (enabled=Y) and containerd's
|
||||
# default profile denies mount. NOT measured here: the test host runs SELinux, not AppArmor.
|
||||
# * readOnlyRootFilesystem: true — measured to work with the five emptyDirs below.
|
||||
#
|
||||
# --oci-worker-no-process-sandbox: build steps share the daemon's PID namespace instead of getting
|
||||
# their own (creating one needs an unmasked /proc, i.e. more privilege). Consequence, stated by
|
||||
# buildkitd itself at startup: a build step can signal or ptrace the daemon and read what the
|
||||
# daemon can read — including this pod's TLS key and the build cache. So a build step is trusted
|
||||
# at the level of "may impersonate this builder", which is the level anyone holding the client
|
||||
# certificate already has. It can NOT reach the node: everything still runs as uid 1000 in a user
|
||||
# namespace, unprivileged, with no host mounts.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: buildkitd
|
||||
namespace: buildkit
|
||||
labels:
|
||||
app.kubernetes.io/name: buildkitd
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate # one builder, one cache; never two daemons fighting over a node's disk
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: buildkitd
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: buildkitd
|
||||
spec:
|
||||
automountServiceAccountToken: false # the builder needs no Kubernetes API at all
|
||||
enableServiceLinks: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000 # makes the 0440 TLS Secret below readable by uid 1000, and nobody else
|
||||
containers:
|
||||
- name: buildkitd
|
||||
# Pinned by digest (the multi-arch index). The kubernetes manager in renovate.json5
|
||||
# tracks platform/**, so Renovate proposes bumps; the CI client (buildx, pinned in
|
||||
# versions.env) talks to any recent daemon.
|
||||
image: docker.io/moby/buildkit:v0.34.1-rootless@sha256:f68752a5a94acfcdad2d07867b7075601a421638e17fc5f28efc1c814d0c4f89
|
||||
args:
|
||||
- --addr
|
||||
- unix:///run/user/1000/buildkit/buildkitd.sock # local only: the probes below
|
||||
- --addr
|
||||
- tcp://0.0.0.0:1234 # CI, mTLS only
|
||||
- --tlscacert
|
||||
- /certs/ca.crt # client certs are VERIFIED against this CA: no cert, no build
|
||||
- --tlscert
|
||||
- /certs/tls.crt
|
||||
- --tlskey
|
||||
- /certs/tls.key
|
||||
- --config
|
||||
- /etc/buildkit/buildkitd.toml
|
||||
- --oci-worker-no-process-sandbox
|
||||
ports:
|
||||
- name: buildkit
|
||||
containerPort: 1234
|
||||
protocol: TCP
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["buildctl", "debug", "workers"]
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 30
|
||||
livenessProbe:
|
||||
exec:
|
||||
command: ["buildctl", "debug", "workers"]
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 60
|
||||
timeoutSeconds: 10
|
||||
securityContext:
|
||||
privileged: false
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
allowPrivilegeEscalation: true
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
add: ["SETUID", "SETGID"]
|
||||
seccompProfile:
|
||||
type: Unconfined
|
||||
appArmorProfile:
|
||||
type: Unconfined
|
||||
resources:
|
||||
# Build steps (dnf, npm, pip) run INSIDE this container's cgroup, so the limit covers
|
||||
# the daemon and the build together.
|
||||
requests:
|
||||
cpu: 250m
|
||||
memory: 512Mi
|
||||
ephemeral-storage: 2Gi
|
||||
limits:
|
||||
memory: 4Gi
|
||||
volumeMounts:
|
||||
- { name: certs, mountPath: /certs, readOnly: true }
|
||||
- { name: config, mountPath: /etc/buildkit, readOnly: true }
|
||||
- { name: cache, mountPath: /home/user/.local/share/buildkit }
|
||||
- { name: run, mountPath: /run/user }
|
||||
- { name: tmpdir, mountPath: /home/user/.local/tmp } # the image's TMPDIR
|
||||
- { name: tmp, mountPath: /tmp }
|
||||
- { name: clicfg, mountPath: /home/user/.docker } # buildctl (the probes) writes here
|
||||
volumes:
|
||||
- name: certs
|
||||
secret:
|
||||
secretName: buildkitd-tls # cert-manager: ca.crt + tls.crt + tls.key
|
||||
defaultMode: 0440
|
||||
- name: config
|
||||
configMap:
|
||||
name: buildkitd-config
|
||||
- name: cache
|
||||
emptyDir:
|
||||
sizeLimit: 6Gi # GC caps use at 4GB (configmap.yaml); this is the backstop
|
||||
- name: run
|
||||
emptyDir: {}
|
||||
- name: tmpdir
|
||||
emptyDir: {}
|
||||
- name: tmp
|
||||
emptyDir: {}
|
||||
- name: clicfg
|
||||
emptyDir: {}
|
||||
@@ -1,13 +0,0 @@
|
||||
---
|
||||
# platform/buildkit — rootless BuildKit, the CI image builder. Reconciled by the Flux
|
||||
# Kustomization `buildkit` (clusters/base/buildkit.yaml). See README.md.
|
||||
# versions.env is NOT a resource: it pins the CI-side client (buildx) and the address CI dials.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- certificates.yaml
|
||||
- configmap.yaml
|
||||
- deployment.yaml
|
||||
- service.yaml
|
||||
- networkpolicy.yaml
|
||||
@@ -1,26 +0,0 @@
|
||||
---
|
||||
# buildkit — the image builder for CI (.gitea/workflows/images.yaml). Operator decision on PR #81
|
||||
# (2026-10-29): images are built by a ROOTLESS BuildKit daemon reached over mTLS, and NO CI job
|
||||
# may reach a privileged daemon. See README.md for the whole design.
|
||||
#
|
||||
# A NAMESPACE OF ITS OWN, deliberately NOT `gitea-actions`:
|
||||
# * The act-runner namespace exists to hold a PRIVILEGED DinD sidecar. Putting the builder
|
||||
# beside it would put a TLS server key next to a privileged container for no benefit.
|
||||
# * Its NetworkPolicies, Secrets and quota then belong to one component, and the ingress rule
|
||||
# ("only the act-runner pod may connect") can be written from the outside.
|
||||
#
|
||||
# NO `app-namespace` LABEL, like gitea-actions: this is CI plumbing, not a user-facing app. That
|
||||
# is also why NO Kyverno exception is needed — every applied ClusterPolicy
|
||||
# (platform-config/kustomization.yaml) matches only namespaces labelled
|
||||
# homelab.keithjackson.dev/app-namespace: "true", and none of the job-shape policies match here.
|
||||
#
|
||||
# NO pod-security.kubernetes.io LABEL, on purpose. Rootless BuildKit needs seccomp AND AppArmor
|
||||
# `Unconfined` (measured — see deployment.yaml), which PSS `baseline` forbids; there is no
|
||||
# cluster-wide PSA default here, so an unlabelled namespace admits it. It is NOT privileged and
|
||||
# the deployment's securityContext is the boundary — read it before relaxing anything.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: buildkit
|
||||
labels:
|
||||
homelab.keithjackson.dev/ci: buildkit
|
||||
@@ -1,88 +0,0 @@
|
||||
---
|
||||
# Default-deny for the namespace, then two narrow allows.
|
||||
#
|
||||
# WHO MAY CONNECT — AND WHY THE ACT-RUNNER POD IS THE RIGHT SOURCE (measured 2026-10-29).
|
||||
# CI jobs do not run as pods. act_runner starts each job as a container on its DinD sidecar, on a
|
||||
# docker bridge network INSIDE the act-runner pod's network namespace, and dockerd masquerades it:
|
||||
# kubectl -n gitea-actions exec deploy/act-runner -c dind -- iptables -t nat -S POSTROUTING
|
||||
# -A POSTROUTING -s 172.18.0.0/16 ! -o br-c0f13d041ad6 -j MASQUERADE
|
||||
# -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
|
||||
# So a job's connection leaves with the act-runner POD's IP and therefore Cilium's identity for
|
||||
# that pod (namespace gitea-actions, app.kubernetes.io/name=act-runner). That is exactly what the
|
||||
# ingress rule names. It cannot distinguish one job from another, or a job from act_runner
|
||||
# itself — per-job authentication is the mTLS client certificate, not this policy. What the
|
||||
# policy adds is that NOTHING ELSE in the cluster can even open the port.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny
|
||||
namespace: buildkit
|
||||
labels:
|
||||
homelab.keithjackson.dev/baseline: default-deny
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress: []
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: buildkitd-allow-ci-ingress
|
||||
namespace: buildkit
|
||||
labels:
|
||||
app.kubernetes.io/name: buildkitd
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: buildkitd
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: gitea-actions
|
||||
app.kubernetes.io/name: act-runner
|
||||
toPorts:
|
||||
- ports:
|
||||
- { port: "1234", protocol: TCP }
|
||||
---
|
||||
# WHAT A BUILD NEEDS TO REACH. Build steps download (dnf mirrors, GitHub releases, dl.k8s.io,
|
||||
# npm, PyPI, Fedora/Docker registries) and the result is pushed to the Gitea registry.
|
||||
# * world 443/80: the downloads. 80 because Fedora's mirror list includes plain-http mirrors
|
||||
# (dnf verifies package GPG signatures, not transport).
|
||||
# * ingress (the Gateway's Envoy, reserved:ingress): the push. gitea.home.keithjackson.dev
|
||||
# resolves in-cluster to the Gateway VIP (split-horizon CoreDNS), and Cilium hands that
|
||||
# traffic to the Gateway, whose identity is `ingress` — the same identity gitea's own
|
||||
# ingress allow admits.
|
||||
# Nothing else in the cluster: not the API server, not other pods. A build step is code from a
|
||||
# pull request.
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: buildkitd-allow-egress
|
||||
namespace: buildkit
|
||||
labels:
|
||||
app.kubernetes.io/name: buildkitd
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: buildkitd
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
toPorts:
|
||||
- ports:
|
||||
- { port: "443", protocol: TCP }
|
||||
- { port: "80", protocol: TCP }
|
||||
- toEntities:
|
||||
- ingress
|
||||
@@ -1,19 +0,0 @@
|
||||
---
|
||||
# The address CI dials: tcp://buildkitd.buildkit.svc.cluster.local:1234 (platform/buildkit/
|
||||
# versions.env), mTLS only. ClusterIP — never exposed on the Gateway.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: buildkitd
|
||||
namespace: buildkit
|
||||
labels:
|
||||
app.kubernetes.io/name: buildkitd
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app.kubernetes.io/name: buildkitd
|
||||
ports:
|
||||
- name: buildkit
|
||||
port: 1234
|
||||
targetPort: buildkit
|
||||
protocol: TCP
|
||||
@@ -1,15 +0,0 @@
|
||||
# platform/buildkit/versions.env — the CI side of the builder. Sourced by
|
||||
# .gitea/workflows/scripts/images.sh. NOT a Kubernetes resource (kustomization.yaml omits it).
|
||||
#
|
||||
# The CLIENT. CI drives the daemon with `buildx` (remote driver) rather than bare `buildctl`
|
||||
# because the image pipeline needs `bake`'s `target:` contexts: on a pull request agent-runner is
|
||||
# built FROM the toolbox built in the same run, which never exists in any registry.
|
||||
# Pinned with the sha256 from the release's own checksums.txt
|
||||
# (https://github.com/docker/buildx/releases/download/v0.38.1/checksums.txt); the pipeline
|
||||
# refuses a download that does not match. Bump BOTH lines together.
|
||||
BUILDX_VERSION=0.38.1
|
||||
BUILDX_SHA256=e25d5cd675092bb5d1d03632fdc46cdf3fede7b514dc5f023a105e00d3ce5ad4
|
||||
|
||||
# The DAEMON's address and the name its certificate carries (platform/buildkit/certificates.yaml).
|
||||
BUILDKIT_HOST=tcp://buildkitd.buildkit.svc.cluster.local:1234
|
||||
BUILDKIT_SERVERNAME=buildkitd.buildkit.svc.cluster.local
|
||||
@@ -1,112 +0,0 @@
|
||||
---
|
||||
# =================================================================================================
|
||||
# coredns HA floor — hold in-cluster DNS at >= 2 replicas, which k3s does not.
|
||||
# =================================================================================================
|
||||
# MEASURED on prod 2026-10-18: `kube-system/coredns` ran with **replicas: 1**. In-cluster DNS is a
|
||||
# single pod on a single node, so **draining or rebooting that one node is a cluster-wide
|
||||
# in-cluster DNS outage** for as long as the pod takes to reschedule and become ready — every pod
|
||||
# that looks up a Service, every CNPG replica finding its primary, every operator talking to the
|
||||
# API by name. It is the single cheapest availability defect in the cluster, and it was invisible
|
||||
# because `1/1 Ready` looks perfectly healthy.
|
||||
#
|
||||
# This matters more now than it did: the reboot automation (kured + unattended-upgrades) drains and
|
||||
# reboots one node at a time on a schedule, which turns a latent single point of failure into a
|
||||
# recurring one, unattended, in the middle of the night.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# WHY AN HPA, AND NOT JUST `replicas: 2` — this is the whole decision
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# **k3s owns the coredns Deployment and we cannot durably edit it.** It is applied by k3s's own
|
||||
# deploy controller from a bundled manifest, not by Flux:
|
||||
#
|
||||
# $ kubectl -n kube-system get addon coredns
|
||||
# NAME SOURCE CHECKSUM
|
||||
# coredns /data/k3s/server/manifests/coredns.yaml 8cd91595...
|
||||
# $ kubectl -n kube-system get deploy coredns -o jsonpath='{.metadata.managedFields[*].manager}'
|
||||
# deploy@k3s-prod-1 k3s
|
||||
#
|
||||
# and the Deployment carries an `objectset.rio.cattle.io/hash` label — a wrangler objectset. k3s
|
||||
# RE-STAGES its bundled manifests on startup, so a `replicas: 2` written into that file (or a
|
||||
# `kubectl scale`) is reverted the next time k3s restarts or is upgraded. MEASURED the same day: a
|
||||
# hand `kubectl scale` to 2 was NOT actively reverted (it held, and the AddOn checksum did not
|
||||
# change) — the deploy controller re-applies on manifest change and on restart, not on a drift
|
||||
# timer. So the hand fix survives until exactly the event we are protecting against: a reboot.
|
||||
#
|
||||
# The two durable alternatives were weighed and rejected:
|
||||
# * **`coredns.yaml.skip` + our own templated copy** (the pattern roles/kube_vip already uses for
|
||||
# a manifest in server/manifests). Fully deterministic and never flaps — but it takes OWNERSHIP
|
||||
# of coredns away from k3s permanently, so k3s upgrades would stop updating it and the version
|
||||
# would have to be refreshed by hand forever. Too high a price for a replica count.
|
||||
# * **A Flux-managed patch of the Deployment itself.** Rejected outright: Flux and k3s's deploy
|
||||
# controller would both own one object and fight continuously. That is the shape of the
|
||||
# `default-deny` NetworkPolicy drift loop this cluster already suffered (Kyverno generate + Flux
|
||||
# both owning one resource, ~2500 events per 10 minutes).
|
||||
#
|
||||
# An HPA avoids both. It is a SEPARATE object that k3s does not own and will never reconcile away,
|
||||
# and it drives the Deployment's `/scale` subresource rather than its spec — so there is no
|
||||
# shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA observes it under `minReplicas`
|
||||
# and scales back to 2 within one reconcile (~15s by default).
|
||||
#
|
||||
# THE COST, STATED PLAINLY: there is a brief window — seconds, after a k3s restart — where coredns
|
||||
# is back at 1 replica until the HPA notices. That is strictly better than the status quo, where it
|
||||
# is at 1 replica permanently, but it is not zero. Accepting that window is the trade being made
|
||||
# here; the `.skip` option above is what to reach for if it ever proves unacceptable.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# THIS IS A FLOOR-HOLDER, NOT A REAL AUTOSCALER
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# `minReplicas: 2` is the entire point. The CPU metric exists because an autoscaling/v2 HPA needs a
|
||||
# metric to be a valid object, not because coredns is expected to scale on load: measured via
|
||||
# metrics-server, the two pods use **5m and 4m** against a **100m** request — ~5% utilisation, so
|
||||
# the 80% target will never be approached and the HPA will sit at `minReplicas` indefinitely.
|
||||
#
|
||||
# The CPU request is what makes this legal: coredns ships `requests: {cpu: 100m, memory: 70Mi}`
|
||||
# (measured), and a Resource/Utilization metric is meaningless without a request. metrics-server is
|
||||
# installed as a k3s addon and verified serving (`kubectl top pod -l k8s-app=kube-dns` returns
|
||||
# figures), so the metric resolves rather than leaving the HPA with `ScalingActive=False`.
|
||||
#
|
||||
# `maxReplicas: 3` = the node count. coredns' pod anti-affinity prefers one pod per node, so a
|
||||
# fourth replica would add no failure-domain coverage; the ceiling exists so a metrics glitch can
|
||||
# never balloon DNS pods across the cluster.
|
||||
apiVersion: autoscaling/v2
|
||||
kind: HorizontalPodAutoscaler
|
||||
metadata:
|
||||
name: coredns
|
||||
# kube-system, because that is where k3s puts coredns. This is the only object this component
|
||||
# places outside a namespace it owns, and it is additive: it creates nothing in kube-system and
|
||||
# mutates only the coredns Deployment's scale subresource.
|
||||
namespace: kube-system
|
||||
labels:
|
||||
app.kubernetes.io/name: coredns
|
||||
spec:
|
||||
scaleTargetRef:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
name: coredns
|
||||
minReplicas: 2
|
||||
maxReplicas: 3
|
||||
metrics:
|
||||
- type: Resource
|
||||
resource:
|
||||
name: cpu
|
||||
target:
|
||||
type: Utilization
|
||||
# Deliberately high. See "floor-holder" above: this must not scale on normal DNS load.
|
||||
averageUtilization: 80
|
||||
behavior:
|
||||
scaleDown:
|
||||
# 10 minutes. DNS is the worst thing in the cluster to flap: a scale-down races in-flight
|
||||
# queries and every client's negative cache. The HPA should be reluctant in this direction.
|
||||
stabilizationWindowSeconds: 600
|
||||
policies:
|
||||
- type: Pods
|
||||
value: 1
|
||||
periodSeconds: 600
|
||||
scaleUp:
|
||||
# Prompt in the direction that restores redundancy — this is the path taken after a k3s
|
||||
# restart has re-staged `replicas: 1`.
|
||||
stabilizationWindowSeconds: 0
|
||||
policies:
|
||||
- type: Pods
|
||||
value: 2
|
||||
periodSeconds: 60
|
||||
@@ -7,7 +7,3 @@ apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- coredns-custom.yaml
|
||||
# HA floor for the coredns Deployment itself — a separate concern from the split-horizon server
|
||||
# block above, and deliberately an HPA rather than a replica edit: k3s owns that Deployment via
|
||||
# its own deploy controller and re-stages `replicas: 1` on restart. hpa.yaml has the reasoning.
|
||||
- hpa.yaml
|
||||
@@ -154,74 +154,6 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
|
||||
(it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
|
||||
a realm edit landed.
|
||||
|
||||
## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)
|
||||
|
||||
`keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
|
||||
policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
|
||||
**ALTERNATIVE** beside the password form. Design decisions and their reasoning are in the manifest
|
||||
comments, not repeated here. The headline choices:
|
||||
|
||||
| Decision | Value | One-line reason |
|
||||
|---|---|---|
|
||||
| Additive or replacing? | **Additive** — passkey *or* password | A misfiring mandatory flow locks the single operator out of every SSO app *and* kubectl |
|
||||
| Enrolment | **Optional**, never a default action | Same reason; `webauthn-register-passwordless` is already enabled-not-forced by default |
|
||||
| Relying-party ID | `${BASE_DOMAIN}` (not `keycloak.${BASE_DOMAIN}`) | An unre-keyable credential must not be bound to a hostname that can move |
|
||||
| User verification | `required` | Without it, mere possession of an unlocked device is a complete login |
|
||||
| Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
|
||||
| OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
|
||||
|
||||
### This change does NOT reach the live realm. At all.
|
||||
|
||||
Same one-shot failure this file already documents twice above, third field over. Measured on prod
|
||||
2026-10-17:
|
||||
|
||||
```
|
||||
$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
|
||||
{"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
|
||||
{"status":"False","type":"HasErrors"}]}
|
||||
```
|
||||
|
||||
`Done=True` / `Started=False` — the import has completed and **will not re-run**, so a policy block
|
||||
and three new sub-flows added to this file change nothing about how anyone logs in. The
|
||||
client-secret/client-URL reconciler does **not** close this gap and was never meant to: its hard
|
||||
endpoint allow-list makes `/authentication` unreachable **by design** (see the table above), and
|
||||
realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
|
||||
mapper or a changed scope still needs the realm import to actually re-run" case named above.
|
||||
|
||||
**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
|
||||
an operator applies it by hand.** Two paths, both operator actions:
|
||||
|
||||
1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
|
||||
for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
|
||||
match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
|
||||
into Username Form + Password Form and add the two sub-flows exactly as declared.
|
||||
2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
|
||||
the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
|
||||
is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
|
||||
|
||||
Either way the realm is then configured by hand and git is only the record of intent, which is a
|
||||
live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.
|
||||
|
||||
### Verifying it actually works (nobody has, as of 2026-10-17)
|
||||
|
||||
No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
|
||||
it, in order:
|
||||
|
||||
```sh
|
||||
# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
|
||||
# Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
|
||||
# 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
|
||||
# -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
|
||||
# is wrong for the origin and NOTHING will say so.
|
||||
# 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
|
||||
# prompt. A platform-admin must NOT be asked for TOTP on this path.
|
||||
# 4. regression (the important one): a user with NO passkey must still get the password screen,
|
||||
# and a platform-admin must still be asked for TOTP there.
|
||||
```
|
||||
|
||||
A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak
|
||||
is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.
|
||||
|
||||
## Exceptions that keep native auth (do not force SSO) — ADR-0014
|
||||
|
||||
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
|
||||
@@ -74,142 +74,6 @@ spec:
|
||||
bruteForceProtected: true
|
||||
permanentLockout: false
|
||||
sslRequired: external
|
||||
# ===========================================================================================
|
||||
# PASSKEYS — the WebAuthn PASSWORDLESS policy (R-IDP-04). 2026-10-17.
|
||||
# ===========================================================================================
|
||||
# Read this together with the browser-mfa flow at the bottom of the file: THIS block decides
|
||||
# what a passkey must BE, the FLOW decides when one is OFFERED. Keycloak keeps TWO independent
|
||||
# WebAuthn policies per realm and they are not interchangeable:
|
||||
#
|
||||
# webAuthnPolicy* -> the TWO-FACTOR policy, consumed by the `webauthn-authenticator`
|
||||
# authenticator (a security key presented AFTER a password).
|
||||
# webAuthnPolicyPasswordless* -> the PASSWORDLESS policy, consumed by
|
||||
# `webauthn-authenticator-passwordless` (a passkey presented
|
||||
# INSTEAD of a password). This is the one passkeys use.
|
||||
#
|
||||
# Setting the wrong family is a silent no-op: the passwordless authenticator reads
|
||||
# getWebAuthnPolicyPasswordless() only (WebAuthnPasswordlessAuthenticator.getWebAuthnPolicy),
|
||||
# so a policy written into the two-factor fields is simply never consulted by a passkey login.
|
||||
# Only the passwordless family is declared here; the two-factor family is deliberately left at
|
||||
# Keycloak's defaults because nothing in this realm uses `webauthn-authenticator` — TOTP
|
||||
# (auth-otp-form) is the second factor for platform-admins, see the flow section.
|
||||
#
|
||||
# ADDITIVE AND OPTIONAL — THE DECISION THIS BLOCK EXISTS TO MAKE
|
||||
# -------------------------------------------------------------
|
||||
# Passkeys are offered ALONGSIDE the password, never instead of it, and enrolment is NOT forced.
|
||||
# This is a single-operator homelab whose IdP gates Grafana, Gitea, Nextcloud, Headlamp,
|
||||
# Semaphore, PiHole, Crafty, the *arr stack AND kubectl itself (the `kubernetes` client below).
|
||||
# A mandatory passkey flow that misfires is therefore not an inconvenience, it is a total
|
||||
# lock-out from the whole platform AND from the cluster API in one step — and the only repair
|
||||
# paths left would be the Keycloak bootstrap admin and the break-glass cert kubeconfig. The
|
||||
# expected gain from forcing enrolment (one human stops typing a password) is small; the
|
||||
# expected loss is the platform. So: additive, optional, and password login keeps working for
|
||||
# every user who never enrols.
|
||||
#
|
||||
# Concretely that means: no `requiredActions` block is declared anywhere in this file (see the
|
||||
# long note in the flow section for why declaring one would be actively DANGEROUS), and the
|
||||
# passkey step sits as an ALTERNATIVE beside the password form rather than ahead of it.
|
||||
#
|
||||
# RELYING-PARTY ID — THE FIELD THAT SILENTLY BREAKS EVERY PASSKEY IF IT IS WRONG
|
||||
# ------------------------------------------------------------------------------
|
||||
# The RP ID is baked into every credential at enrolment and can never be changed afterwards:
|
||||
# a passkey created under one RP ID is unusable under another, with no error the user can act
|
||||
# on — the browser simply reports that no credential is available. It is also browser-enforced,
|
||||
# so it cannot be "fixed" server-side: per WebAuthn, the RP ID must be equal to, or a
|
||||
# registrable-domain suffix of, the origin's effective domain. Keycloak is served at
|
||||
# keycloak.<BASE_DOMAIN> (httproute.yaml, and the Keycloak CR's hostname), so the legal choices
|
||||
# are keycloak.<BASE_DOMAIN>, <BASE_DOMAIN>, or the registrable domain one level up. It is NOT
|
||||
# the full URL and NOT a scheme — "https://..." here would fail every ceremony.
|
||||
#
|
||||
# We choose BASE_DOMAIN, the shared platform suffix, NOT the keycloak host, for one concrete
|
||||
# reason drawn from this repo's own history: a credential that cannot be re-keyed must not be
|
||||
# bound to a name that can move. The oauth2-proxy client below ALREADY declares a redirect URI
|
||||
# on auth.<BASE_DOMAIN>, i.e. this platform already contemplates serving auth on a second
|
||||
# hostname; and the client-URL reconciler exists precisely because moving a hostname in git
|
||||
# silently fails to reach the realm (see README, "Client URLs: the same one-shot failure").
|
||||
# Binding passkeys to the keycloak subdomain would mean that relocating Keycloak invalidates
|
||||
# every enrolled passkey; binding them to BASE_DOMAIN survives it. The cost is scope: any
|
||||
# future WebAuthn relying party under this base domain could assert the same RP ID. In a
|
||||
# homelab where every such party is already behind THIS IdP that is not a trust boundary worth
|
||||
# buying. Note the residual, unavoidable risk: changing BASE_DOMAIN itself invalidates every
|
||||
# passkey, which is a re-enrolment, not a repair.
|
||||
#
|
||||
# Written as the cluster-var so it is substituted by the same Flux postBuild pass, with the same
|
||||
# value, as every hostname in this file — there is no second place the domain is decided.
|
||||
webAuthnPolicyPasswordlessRpId: ${BASE_DOMAIN}
|
||||
# The human-readable name the browser/OS shows in the passkey prompt and stores beside the
|
||||
# credential ("Save a passkey for Homelab SSO?"). Matches the realm displayName above so the
|
||||
# OS keychain entry is recognisable. It is also load-bearing on the UPDATE path: Keycloak's
|
||||
# updateRealm applies the passwordless policy ONLY when this field is non-empty
|
||||
# (DefaultExportImportManager line ~842), so an admin-API realm update that omits it drops the
|
||||
# whole policy silently.
|
||||
webAuthnPolicyPasswordlessRpEntityName: "Homelab SSO"
|
||||
# ES256 then RS256 — exactly Keycloak's own DEFAULT_POLICY, declared explicitly rather than
|
||||
# inherited so the realm's crypto is auditable from git. ES256 is what every platform
|
||||
# authenticator (Touch ID/Face ID, Windows Hello, Android) and every FIDO2 key produces;
|
||||
# RS256 is the fallback for older Windows Hello TPM-backed credentials. EdDSA/Ed25519 is
|
||||
# deliberately NOT offered: authenticator and browser support is uneven, and an authenticator
|
||||
# that picks an algorithm the verifier handles badly fails at enrolment, which is the worst
|
||||
# moment to discover it. Order is a preference list, not an allow-list ordering requirement.
|
||||
webAuthnPolicyPasswordlessSignatureAlgorithms:
|
||||
- ES256
|
||||
- RS256
|
||||
# DISCOVERABLE CREDENTIAL REQUIRED. "Yes" is a Keycloak sentinel, not a boolean, and the login
|
||||
# theme compares it as the literal string: webauthnRegister.js maps 'Yes' to
|
||||
# authenticatorSelection.requireResidentKey=true and ANY other non-sentinel value to false.
|
||||
# This is not optional for passkeys — a non-discoverable credential cannot be found by the
|
||||
# browser without the server first naming it, so it is a second factor, not a passkey.
|
||||
# (Keycloak 26.0.6 emits only the LEGACY requireResidentKey member, not the newer
|
||||
# residentKey:"required"; per WebAuthn L2 the client maps requireResidentKey=true onto
|
||||
# residentKey="required", so discoverable credentials are genuinely requested — but it also
|
||||
# means this policy cannot express residentKey:"preferred" at all.)
|
||||
webAuthnPolicyPasswordlessRequireResidentKey: "Yes"
|
||||
# USER VERIFICATION REQUIRED. The authenticator must verify the human (biometric or PIN) and
|
||||
# say so in the assertion. This is the field that makes a passkey login at least as strong as
|
||||
# the password it replaces: with verification merely "preferred", possession of an unlocked
|
||||
# phone or an unattended security key would be a COMPLETE login to every SSO-gated app — a
|
||||
# single-factor possession check, strictly weaker than the password+TOTP path beside it.
|
||||
# Required makes the passkey branch inherently two-factor (something you have + something you
|
||||
# are/know), which is the premise the flow section relies on when it does NOT ask a
|
||||
# passkey-authenticated platform-admin for TOTP as well.
|
||||
webAuthnPolicyPasswordlessUserVerificationRequirement: required
|
||||
# "not specified" is Keycloak's sentinel for "either kind of authenticator"
|
||||
# (Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED; the theme JS omits the WebAuthn member
|
||||
# entirely when it sees this exact string). Deliberately NOT narrowed: "platform" would reject
|
||||
# every YubiKey and every phone used as a cross-device passkey, and "cross-platform" would
|
||||
# reject Touch ID and Windows Hello — i.e. either choice silently halves the usable hardware.
|
||||
# The operator should be able to enrol a laptop AND a phone AND a hardware key.
|
||||
webAuthnPolicyPasswordlessAuthenticatorAttachment: "not specified"
|
||||
# ATTESTATION: none. Requesting "direct" would hand Keycloak an attestation statement it has
|
||||
# no trust anchors to validate here (no metadata service is configured) and nothing to compare
|
||||
# it against, since the AAGUID allow-list below is deliberately empty — so the statement would
|
||||
# be collected, unverifiable, and would add an extra "share model information?" consent prompt
|
||||
# on Safari/iOS for no security gain. Attestation is how you ENFORCE an approved-hardware list;
|
||||
# we do not have one, so asking for it is cost without benefit.
|
||||
webAuthnPolicyPasswordlessAttestationConveyancePreference: none
|
||||
# Empty = accept any authenticator model. An AAGUID allow-list is a hardware whitelist: it
|
||||
# would have to be maintained by hand, and every new phone, laptop or key the operator buys
|
||||
# would be rejected at enrolment with a message that does not explain itself. It is also moot
|
||||
# while attestation is "none" (above) — with no attestation statement there is no trustworthy
|
||||
# AAGUID to match against in the first place.
|
||||
webAuthnPolicyPasswordlessAcceptableAaguids: []
|
||||
# Ceremony timeout in SECONDS (the login theme multiplies by 1000; 0 means "send no timeout and
|
||||
# let the browser decide"). 300s is chosen over a tighter value because the slowest LEGITIMATE
|
||||
# path is the cross-device hybrid flow — Keycloak renders a QR code, the operator unlocks a
|
||||
# phone, scans it, and completes a biometric check — which routinely takes over a minute. A
|
||||
# 60-120s timeout looks tidy and converts that flow into an unexplained failure. Browsers clamp
|
||||
# this value (Chrome to at most 10 minutes), so 300 is honoured rather than reinterpreted.
|
||||
webAuthnPolicyPasswordlessCreateTimeout: 300
|
||||
# Send the user's existing passkeys as excludeCredentials at enrolment, so re-enrolling an
|
||||
# authenticator the user already registered is refused by the authenticator itself instead of
|
||||
# producing a second, indistinguishable credential in the account console. Cheap, and it keeps
|
||||
# "which passkey is which" answerable — which matters when the recovery story is "delete the
|
||||
# lost one".
|
||||
webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister: true
|
||||
# webAuthnPolicyPasswordlessExtraOrigins is intentionally NOT set. It exists to admit non-web
|
||||
# origins (native app facet IDs, e.g. android:apk-key-hash:...). Every ceremony here happens in
|
||||
# a browser on Keycloak's own origin, which is already covered by the RP ID above; adding
|
||||
# origins is only ever a widening of who may ask for these credentials.
|
||||
# --- Authorization model (R-IDP-06): groups drive access; platform-admins carry a realm role. ---
|
||||
roles:
|
||||
realm:
|
||||
@@ -664,105 +528,14 @@ spec:
|
||||
"access.token.claim": "true"
|
||||
"userinfo.token.claim": "true"
|
||||
"claim.name": groups
|
||||
# ===========================================================================================
|
||||
# MFA for platform-admins (R-IDP-04) + PASSKEY login (2026-10-17). The realm browserFlow is
|
||||
# bound to browser-mfa at the bottom of this file.
|
||||
# ===========================================================================================
|
||||
# Non-admins log in without OTP; platform-admins must enroll/present TOTP. VERIFY at bring-up
|
||||
# (see runbook): a malformed flow fails the whole import — if so, the fallback is to comment
|
||||
# this block + browserFlow and configure the flow via the admin console.
|
||||
#
|
||||
# THE SHAPE, AND WHY IT IS THIS SHAPE
|
||||
# -----------------------------------
|
||||
# browser-mfa (top level, bound as browserFlow)
|
||||
# auth-cookie ALTERNATIVE already-authenticated SSO cookie
|
||||
# identity-provider-redirector ALTERNATIVE (no IdPs configured; kept for parity)
|
||||
# browser-mfa-forms ALTERNATIVE the interactive path:
|
||||
# auth-username-form REQUIRED WHO are you (no password yet)
|
||||
# browser-mfa-passkey-or-password REQUIRED ...and PROVE it, one of two ways:
|
||||
# webauthn-authenticator-passwordless ALTERNATIVE a passkey, or
|
||||
# browser-mfa-password-and-otp ALTERNATIVE the password path:
|
||||
# auth-password-form REQUIRED password, then
|
||||
# browser-mfa-conditional-otp CONDITIONAL TOTP iff platform-admin
|
||||
#
|
||||
# WHY THE USERNAME/PASSWORD SCREEN HAD TO BE SPLIT IN TWO. The previous flow used the combined
|
||||
# `auth-username-password-form`, which collects both on one page. A passkey branch cannot live
|
||||
# behind that: the server must know WHICH user is logging in before it can decide whether a
|
||||
# passkey is even available, and the combined form has already taken the password by the time
|
||||
# it knows. So it is replaced by the `auth-username-form` / `auth-password-form` pair — the
|
||||
# same decomposition Keycloak's own passwordless recipe uses. THIS IS A VISIBLE UX CHANGE FOR
|
||||
# EVERY USER, passkey or not: login becomes two screens (username, then password) instead of
|
||||
# one. That cost is accepted deliberately — it is the enabling change, it is what Google and
|
||||
# Microsoft logins already do, and the alternative (below) is worse.
|
||||
#
|
||||
# WHY NOT USERNAMELESS ("just tap your passkey", no username screen at all). Keycloak 26.0.6
|
||||
# does support it: WebAuthnPasswordlessAuthenticator.requiresUser() returns FALSE, so the
|
||||
# authenticator can legally run as a first step with no user in context. We deliberately do not
|
||||
# do that, because of how the flow engine then behaves. In DefaultAuthenticationFlow the
|
||||
# "does this user actually have this credential?" guard sits INSIDE the requiresUser() branch;
|
||||
# with requiresUser() false and no user yet set, the engine skips that guard and simply invokes
|
||||
# the authenticator. A passkey execution placed before the username form is therefore offered
|
||||
# to EVERYONE, so every login — including every password-only user and every family member who
|
||||
# will never own a passkey — would open on a passkey prompt they must dismiss. That is the
|
||||
# opposite of additive. Keeping the passkey step AFTER auth-username-form means a user is in
|
||||
# context, which is what makes the graceful degradation below actually work.
|
||||
#
|
||||
# WHY THE PASSKEY BRANCH DEGRADES CLEANLY TO A PASSWORD (the property the whole "additive"
|
||||
# claim rests on). With a user in context, the engine takes the non-requiresUser branch and
|
||||
# raises CREDENTIAL_SETUP_REQUIRED when the user has no credential of this type and the
|
||||
# authenticator's factory does not allow self-setup — and WebAuthnPasswordlessAuthenticatorFactory
|
||||
# .isUserSetupAllowed() is false, and the authenticator is a CredentialValidator, so both
|
||||
# conditions hold. That exception is raised INSIDE the ALTERNATIVE loop, which catches it,
|
||||
# marks the execution ATTEMPTED and proceeds to the next alternative. Net effect: a user with
|
||||
# no passkey never sees a passkey screen and lands on the password form; a user WITH one gets
|
||||
# the passkey screen and keeps the password as a "Try another way" option. Note that for a
|
||||
# user holding BOTH, which screen appears FIRST is decided by Keycloak's authentication
|
||||
# selection list (the user's own credential ordering), not by the priorities below.
|
||||
#
|
||||
# WHY A PASSKEY LOGIN IS NOT ALSO ASKED FOR TOTP. browser-mfa-conditional-otp hangs off the
|
||||
# PASSWORD branch only, so a platform-admin who authenticates with a passkey is not challenged
|
||||
# for TOTP. That is intentional, and it rests on the policy above: userVerification is
|
||||
# `required`, so the authenticator has already proved possession of hardware AND verified the
|
||||
# human (biometric or PIN) in one phishing-resistant, origin-bound ceremony. Stacking TOTP on
|
||||
# top would add a replayable, phishable shared secret to a credential that is neither, for
|
||||
# friction on every login. If a future operator disagrees, the change is small and local: move
|
||||
# the browser-mfa-conditional-otp reference out of browser-mfa-password-and-otp and into
|
||||
# browser-mfa-passkey-or-password's parent as a REQUIRED step, so it applies to both branches.
|
||||
#
|
||||
# WHY THERE IS NO `requiredActions` BLOCK IN THIS FILE — DO NOT ADD ONE
|
||||
# --------------------------------------------------------------------
|
||||
# Enrolment uses Keycloak's `webauthn-register-passwordless` required action, and the obvious
|
||||
# move is to declare it here. DO NOT. In Keycloak 26.0.6 the realm import treats
|
||||
# requiredActions as REPLACE, NOT MERGE: DefaultExportImportManager.importRealm branches on
|
||||
# `if (rep.getRequiredActions() != null)` and, when the representation declares ANY, imports
|
||||
# exactly those plus delete_account — the `else` branch that installs the full default set
|
||||
# (DefaultRequiredActions.addActions) never runs. RealmManager.setupRequiredActions is no
|
||||
# backstop either: it fires only when the realm has ZERO required-action providers.
|
||||
#
|
||||
# So declaring one action would create this realm with that action and delete_account and
|
||||
# NOTHING ELSE — no CONFIGURE_TOTP, no UPDATE_PASSWORD, no VERIFY_EMAIL, no delete_credential,
|
||||
# no UPDATE_PROFILE. The first casualty would be the MFA requirement this very flow enforces:
|
||||
# with CONFIGURE_TOTP unregistered, a platform-admin who does not yet have TOTP could never
|
||||
# enrol one, while auth-otp-form below still demands it — a hard lock-out of every
|
||||
# platform-admin from every SSO-gated app, including Headlamp and kubectl, delivered by a
|
||||
# change whose stated purpose was to ADD a login method.
|
||||
#
|
||||
# And the declaration buys nothing, because the default set ALREADY CONTAINS IT.
|
||||
# DefaultRequiredActions registers `webauthn-register-passwordless` with enabled=true,
|
||||
# defaultAction=false, priority 80, gated on Profile.Feature.WEB_AUTHN — which is Type.DEFAULT
|
||||
# in 26.0.6, i.e. on unless explicitly disabled. enabled=true / defaultAction=false IS the
|
||||
# additive-and-optional posture this change wants: the action exists and can be triggered for
|
||||
# a user on demand, but it is not forced on anyone at next login. Declaring it would restate
|
||||
# the default while deleting ten siblings.
|
||||
#
|
||||
# (The symmetric question for authenticationFlows has the opposite answer, which is why THIS
|
||||
# block is safe: importAuthenticationFlows adds the declared flows and then backfills each
|
||||
# unbound binding — browser, registration, direct grant, reset credentials, client auth, first
|
||||
# broker login, docker auth — from the built-in defaults individually. Declaring flows does not
|
||||
# suppress the built-ins; declaring required actions does suppress them.)
|
||||
# --- MFA for platform-admins (R-IDP-04): a browser flow whose OTP step is CONDITIONAL on the
|
||||
# platform-admin role. Non-admins log in without OTP; platform-admins must enroll/present TOTP.
|
||||
# Set as the realm browserFlow below. VERIFY at bring-up (see runbook): a malformed flow fails
|
||||
# the whole import — if so, the fallback is to comment this block + browserFlow and configure
|
||||
# the conditional-OTP-by-role flow via the admin console. ---
|
||||
authenticationFlows:
|
||||
- alias: browser-mfa
|
||||
description: "Browser flow: passkey or password, with conditional OTP for platform-admins."
|
||||
description: "Browser flow with conditional OTP required for platform-admins."
|
||||
providerId: basic-flow
|
||||
topLevel: true
|
||||
builtIn: false
|
||||
@@ -782,63 +555,13 @@ spec:
|
||||
priority: 30
|
||||
flowAlias: browser-mfa-forms
|
||||
userSetupAllowed: false
|
||||
# Identify the user FIRST (username only), then prove it. Both steps are REQUIRED: the
|
||||
# username is never optional, and neither is proving possession of the account.
|
||||
- alias: browser-mfa-forms
|
||||
description: "Username, then passkey or password (+ conditional OTP)."
|
||||
description: "Username/password then conditional OTP."
|
||||
providerId: basic-flow
|
||||
topLevel: false
|
||||
builtIn: false
|
||||
authenticationExecutions:
|
||||
# auth-username-form = Keycloak's "Username Form" (UsernameFormFactory) — resolves the
|
||||
# user WITHOUT taking a password, which is the prerequisite for offering a passkey.
|
||||
# Replaces the previous auth-username-password-form; see the header for the UX cost.
|
||||
- authenticator: auth-username-form
|
||||
authenticatorFlow: false
|
||||
requirement: REQUIRED
|
||||
priority: 10
|
||||
userSetupAllowed: false
|
||||
- authenticatorFlow: true
|
||||
requirement: REQUIRED
|
||||
priority: 20
|
||||
flowAlias: browser-mfa-passkey-or-password
|
||||
userSetupAllowed: false
|
||||
# THE FORK. Two ALTERNATIVEs, so satisfying either one completes the flow. Priority puts the
|
||||
# passkey first for users who have one; users who do not fall through to the password branch
|
||||
# automatically (CREDENTIAL_SETUP_REQUIRED is caught by the ALTERNATIVE loop — see header).
|
||||
- alias: browser-mfa-passkey-or-password
|
||||
description: "A passkey (WebAuthn passwordless), or password with conditional OTP."
|
||||
providerId: basic-flow
|
||||
topLevel: false
|
||||
builtIn: false
|
||||
authenticationExecutions:
|
||||
# webauthn-authenticator-passwordless = "WebAuthn Passwordless Authenticator"
|
||||
# (WebAuthnPasswordlessAuthenticatorFactory). It reads the realm's PASSWORDLESS policy
|
||||
# declared above — NOT the two-factor webAuthnPolicy* family. Keep it ALTERNATIVE: as
|
||||
# REQUIRED it would force a passkey on every user, which is the lock-out this change is
|
||||
# explicitly designed not to create.
|
||||
- authenticator: webauthn-authenticator-passwordless
|
||||
authenticatorFlow: false
|
||||
requirement: ALTERNATIVE
|
||||
priority: 10
|
||||
userSetupAllowed: false
|
||||
- authenticatorFlow: true
|
||||
requirement: ALTERNATIVE
|
||||
priority: 20
|
||||
flowAlias: browser-mfa-password-and-otp
|
||||
userSetupAllowed: false
|
||||
# The unchanged pre-passkey behaviour, relocated intact into the password branch: password,
|
||||
# then TOTP iff the user holds the platform-admin role. Nothing about the OTP condition
|
||||
# changed; it simply no longer sits on the path a passkey login takes.
|
||||
- alias: browser-mfa-password-and-otp
|
||||
description: "Password, then conditional OTP for platform-admins."
|
||||
providerId: basic-flow
|
||||
topLevel: false
|
||||
builtIn: false
|
||||
authenticationExecutions:
|
||||
# auth-password-form = Keycloak's "Password Form" (PasswordFormFactory) — the password
|
||||
# half of the old combined form, now that the username has already been collected.
|
||||
- authenticator: auth-password-form
|
||||
- authenticator: auth-username-password-form
|
||||
authenticatorFlow: false
|
||||
requirement: REQUIRED
|
||||
priority: 10
|
||||
|
||||
@@ -45,18 +45,11 @@ waive the sensitive-path human review on its own pull request.
|
||||
|
||||
Both are deliberate, and both fail **visibly** rather than silently:
|
||||
|
||||
1. **The image is published by CI, not by a person.** `platform/pm-agent/deployment.yaml` and
|
||||
`PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder until the first
|
||||
publish. `.gitea/workflows/images.yaml` builds `agent-runner` on rootless BuildKit
|
||||
(`platform/buildkit/`) — the same Containerfile, and the same `verify` stage, that
|
||||
`infra/agent/build-image.sh` builds locally — and on a push to `main` publishes it to
|
||||
`gitea.home.keithjackson.dev/home-network/agent-runner` under `sha-<commit>` and `main`.
|
||||
Renovate then follows the `main` tag through the `# renovate-digest:` marker above **both**
|
||||
lines and moves them together in one pull request. Nobody builds-and-pushes this image from
|
||||
a workstation, and nobody pastes a digest. Until the first publish, and until that PR
|
||||
merges: `ImagePullBackOff`. The one operator prerequisite on the publishing side is the
|
||||
`REGISTRY_PUSH_TOKEN` Actions secret (see the header of `.gitea/workflows/images.yaml`);
|
||||
without it the publish job fails loudly rather than skipping.
|
||||
1. **The image does not exist.** `platform/pm-agent/deployment.yaml` and
|
||||
`PM_WORKER_IMAGE` in `configmap.yaml` carry an all-zero digest placeholder. Build it with
|
||||
`infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the
|
||||
operator, then update **both** places with the digest `podman push --digestfile` returned,
|
||||
in one pull request. Until then: `ImagePullBackOff`.
|
||||
|
||||
**Two things have to be true before any digest will pull, and neither is declared in this
|
||||
repository.** Both were measured after PR #57 merged — this service is the first workload
|
||||
@@ -85,8 +78,8 @@ Both are deliberate, and both fail **visibly** rather than silently:
|
||||
account whose token carries `read:package`, referenced from the Deployment and the
|
||||
committed worker Job template — not built yet, because minting that token is yours.
|
||||
|
||||
The same two apply to `conformance-runner`: same registry path, same publish-then-Renovate
|
||||
model, never pulled either.
|
||||
The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
|
||||
never pulled either.
|
||||
2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
|
||||
repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
|
||||
which Gitea account to create, which scopes, and why it must *not* be an administrator —
|
||||
|
||||
@@ -52,13 +52,12 @@ data:
|
||||
# contract. A tag here is refused at startup (and would be DENIED by admission anyway).
|
||||
#
|
||||
# PLACEHOLDER UNTIL THE IMAGE IS PUBLISHED. The all-zero digest below is deliberately
|
||||
# impossible. CI builds, verifies and — on main — publishes agent-runner
|
||||
# (.gitea/workflows/images.yaml), and Renovate replaces this value with the digest the
|
||||
# REGISTRY returned for the image's `main` tag, in a pull request that also moves the
|
||||
# Deployment's image (same marker, same branch). Nobody pushes this image from a workstation.
|
||||
# Until the first publish the dispatcher starts, fails to pull, and says so — which is the
|
||||
# visible failure, not a silent one.
|
||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
|
||||
# impossible: `infra/agent/build-image.sh` builds and verifies the image, the OPERATOR
|
||||
# pushes it, and the authoritative digest from `--digestfile` replaces this value in a
|
||||
# reviewed pull request (the same two-step the conformance-runner image follows, and for
|
||||
# the same reason: a registry credential is not an agent's to hold). Until then the
|
||||
# dispatcher starts, fails to pull, and says so — which is the visible failure, not a
|
||||
# silent one.
|
||||
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000"
|
||||
|
||||
# --- the model -------------------------------------------------------------
|
||||
|
||||
@@ -66,12 +66,9 @@ spec:
|
||||
- name: dispatch
|
||||
# THE SAME IMAGE THE WORKERS RUN, at the same digest. They share the committed Job
|
||||
# template and the env-name contract baked into it, so a mismatched pair would
|
||||
# disagree about the shape the admission policy enforces. Replaced together, in ONE
|
||||
# Renovate pull request: both lines carry the same marker, so Renovate treats them as
|
||||
# one dependency and bumps them in the same branch. The digest comes from CI's publish
|
||||
# of agent-runner's `main` tag (.gitea/workflows/images.yaml) — the all-zero
|
||||
# disagree about the shape the admission policy enforces. Replaced together, in one
|
||||
# pull request, with the digest `infra/agent/build-image.sh` printed — the all-zero
|
||||
# placeholder below is impossible on purpose (see configmap.yaml PM_WORKER_IMAGE).
|
||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
|
||||
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:0000000000000000000000000000000000000000000000000000000000000000
|
||||
imagePullPolicy: IfNotPresent
|
||||
command:
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
{
|
||||
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
|
||||
"_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
|
||||
"extends": [
|
||||
"config:recommended",
|
||||
":dependencyDashboard",
|
||||
":semanticCommits",
|
||||
"helpers:pinGitHubActionDigests"
|
||||
],
|
||||
"timezone": "America/New_York",
|
||||
"schedule": ["after 3am and before 6am"],
|
||||
"prConcurrentLimit": 5,
|
||||
"prHourlyLimit": 2,
|
||||
"commitMessagePrefix": "chore(deps):",
|
||||
"labels": ["dependencies", "automated-update"],
|
||||
"ignoreUnstable": true,
|
||||
"respectLatest": true,
|
||||
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
|
||||
"flux": {
|
||||
"managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
|
||||
},
|
||||
"kustomize": {
|
||||
"managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
|
||||
},
|
||||
"_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
|
||||
"kubernetes": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)platform/.+\\.ya?ml$/",
|
||||
"/(^|/)platform-config/.+\\.ya?ml$/",
|
||||
"/(^|/)apps/.+\\.ya?ml$/",
|
||||
"/(^|/)catalog/.+\\.ya?ml$/",
|
||||
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
|
||||
"/(^|/)infra/agent/.+\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
"customManagers": [
|
||||
{
|
||||
"customType": "regex",
|
||||
"description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)kustomization\\.ya?ml$/",
|
||||
"/(^|/)versions\\.env$/"
|
||||
],
|
||||
"matchStrings": [
|
||||
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
|
||||
],
|
||||
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
|
||||
}
|
||||
],
|
||||
"packageRules": [
|
||||
{
|
||||
"description": "Auto-merge the allowed low-risk classes on green CI — no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
|
||||
"matchUpdateTypes": ["minor", "patch", "digest"],
|
||||
"automerge": true,
|
||||
"automergeType": "pr",
|
||||
"platformAutomerge": true
|
||||
},
|
||||
{
|
||||
"description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
|
||||
"matchUpdateTypes": ["major"],
|
||||
"automerge": false,
|
||||
"addLabels": ["needs-review", "breaking"]
|
||||
},
|
||||
{
|
||||
"description": "Keep the KubeVirt + CDI operator pair on a compatible track — grouped so they bump together (platform/kubevirt/kustomization.yaml).",
|
||||
"matchDepNames": ["kubevirt/kubevirt", "kubevirt/containerized-data-importer"],
|
||||
"groupName": "kubevirt-cdi"
|
||||
},
|
||||
{
|
||||
"description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface — label but do not auto-merge; a version bump there changes CI reproducibility.",
|
||||
"matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
|
||||
"automerge": false,
|
||||
"addLabels": ["ci-tooling"]
|
||||
},
|
||||
{
|
||||
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
|
||||
"matchPackageNames": [
|
||||
"kured",
|
||||
"longhorn",
|
||||
"cilium",
|
||||
"cloudnative-pg",
|
||||
"kube-prometheus-stack"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": ["needs-review", "blast-radius"]
|
||||
}
|
||||
],
|
||||
"_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
|
||||
"ignorePaths": [
|
||||
"**/*.example",
|
||||
"**/*.sops.yaml",
|
||||
"conformance/**",
|
||||
"platform/image-automation/per-service/**",
|
||||
"platform/kyverno/tests/**"
|
||||
],
|
||||
"_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
|
||||
"_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
|
||||
"docker": { "enabled": true }
|
||||
}
|
||||
-272
@@ -1,272 +0,0 @@
|
||||
// =================================================================================================
|
||||
// Renovate configuration — JSON5, so the rationale can live NEXT TO what it explains.
|
||||
// =================================================================================================
|
||||
// THIS FILE WAS `renovate.json` AND IT WAS INVALID. It carried five `_comment` / `_*_comment`
|
||||
// pseudo-comment keys, because JSON has no comments. Renovate REJECTS unknown top-level options, so
|
||||
// the whole config failed validation and the first manual run of the workflow ERRORED OUT. Proven
|
||||
// with the official validator (`ghcr.io/renovatebot/renovate renovate-config-validator`):
|
||||
//
|
||||
// ERROR: Found errors in configuration
|
||||
// Invalid configuration option: _comment
|
||||
// Invalid configuration option: _docker_comment
|
||||
// Invalid configuration option: _ignorePaths_comment
|
||||
// Invalid configuration option: _kubernetes_comment
|
||||
// Invalid configuration option: _unmanageable_comment
|
||||
//
|
||||
// Renovate discovers `renovate.json5` natively (no RENOVATE_CONFIG_FILE needed, and nothing in
|
||||
// .gitea/ or ci/ pins the old filename — only prose in comments), and JSON5 permits real `//`
|
||||
// comments. So the fix keeps every word of the original rationale, positioned, instead of
|
||||
// flattening it into a `description` string.
|
||||
//
|
||||
// VALIDATE BEFORE COMMITTING — it is one command and it is the only thing that catches this class:
|
||||
// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
|
||||
// ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5
|
||||
//
|
||||
// CORRECTION (2026-10-29): THE VALIDATOR ABOVE WAS NOT THE RENOVATE THE WORKFLOW RUNS. The bare
|
||||
// `ghcr.io/renovatebot/renovate` tag is `latest` (44.167.0 when this was measured), while
|
||||
// renovatebot/[email protected] defaults `renovate-version` to `39` (its action.yml), i.e.
|
||||
// 39.264.0. Validated against THAT, this file failed with five errors — every
|
||||
// `managerFilePatterns` key ("Custom Manager contains disallowed fields", "Invalid configuration
|
||||
// option: flux.managerFilePatterns", ...), because 39.x only knows `fileMatch`; the rename landed
|
||||
// in 40.2.0. So even after the workflow found this file, every run would have stopped at
|
||||
// CONFIG_VALIDATION. renovate.yaml now pins `renovate-version` to the exact release this file is
|
||||
// validated against. VALIDATE WITH THAT SAME TAG:
|
||||
// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
|
||||
// ghcr.io/renovatebot/renovate:<renovate-version from renovate.yaml> \
|
||||
// renovate-config-validator /tmp/renovate.json5
|
||||
// (Not `--strict`: that also fails on a pending migration of the pre-existing top-level
|
||||
// `"docker": {"enabled": true}` block at the bottom, which is a warning, not an error.)
|
||||
//
|
||||
// `regex` vs `custom.regex` in enabledManagers, RESOLVED FROM SOURCE (Renovate 39.264.0 and
|
||||
// later): both work. lib/config/migrations/custom/enabled-managers-migration.ts rewrites
|
||||
// `regex` -> `custom.regex`, and that migration runs on every real repository run
|
||||
// (lib/workers/repository/init/merge.ts -> migrateAndValidate), not only in the validator;
|
||||
// lib/modules/manager/index.ts getEnabledManagersList accepts either spelling. `custom.regex` is
|
||||
// the canonical name (introduced in 37.51.1, renovatebot/renovate#24079), so it is used below and
|
||||
// nothing depends on the migration any more.
|
||||
{ "$schema": "https://docs.renovatebot.com/renovate-schema.json",
|
||||
// Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17
|
||||
// RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags.
|
||||
// It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009
|
||||
// split that gave image tags to Flux image-automation is formally closed — see
|
||||
// .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than
|
||||
// opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with
|
||||
// user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write
|
||||
// anyway; and enabling it would need an in-cluster git WRITE credential for the repo that
|
||||
// defines the cluster). image-automation has been commented out at
|
||||
// platform/kustomization.yaml:52 since it was authored and the cluster holds zero
|
||||
// ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down.
|
||||
// CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid
|
||||
// fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag`
|
||||
// line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth
|
||||
// sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER:
|
||||
// .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.
|
||||
"extends": ["config:recommended", ":dependencyDashboard", ":semanticCommits", "helpers:pinGitHubActionDigests"],
|
||||
"timezone": "America/New_York",
|
||||
"schedule": ["after 3am and before 6am"],
|
||||
"prConcurrentLimit": 5,
|
||||
"prHourlyLimit": 2,
|
||||
"commitMessagePrefix": "chore(deps):",
|
||||
"labels": ["dependencies", "automated-update"],
|
||||
"ignoreUnstable": true,
|
||||
"respectLatest": true,
|
||||
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "custom.regex", "github-actions"],
|
||||
"flux": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)platform/.+\\.ya?ml$/",
|
||||
"/(^|/)apps/.+\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
"kustomize": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)kustomization\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
// ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes
|
||||
// manifests, and it is NOT auto-detected — it only looks at files matched here, which is why
|
||||
// this block is as broad as it is. Scope covers every place this repo writes a container image:
|
||||
// platform components, the rendered app waves, the ServiceDefinition catalog that generates
|
||||
// them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as
|
||||
// `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the
|
||||
// rendered output is reverted by the next render and would also fail ci/render-golden.sh.
|
||||
// Renovate will raise BOTH, and both must land together.
|
||||
"kubernetes": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)platform/.+\\.ya?ml$/",
|
||||
"/(^|/)platform-config/.+\\.ya?ml$/",
|
||||
"/(^|/)apps/.+\\.ya?ml$/",
|
||||
"/(^|/)catalog/.+\\.ya?ml$/",
|
||||
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
|
||||
"/(^|/)infra/agent/.+\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
"customManagers": [
|
||||
{
|
||||
"customType": "regex",
|
||||
"description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)kustomization\\.ya?ml$/",
|
||||
"/(^|/)versions\\.env$/"
|
||||
],
|
||||
"matchStrings": [
|
||||
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
|
||||
],
|
||||
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
|
||||
},
|
||||
// ADDED 2026-10-29 — THIS REPO'S OWN IMAGES (toolbox, agent-runner, conformance-runner).
|
||||
// CI publishes them (.gitea/workflows/images.yaml) under an immutable `sha-<commit>` tag and a
|
||||
// moving `main` tag, and never writes to git. Every reference to them is DIGEST-ONLY
|
||||
// (`<repo>@sha256:<hex>`, no tag), because the admission policies
|
||||
// (platform/kyverno/admission/pm-agent-job-shape.yaml, conformance-trigger-job-shape.yaml)
|
||||
// admit exactly that form. A digest alone gives Renovate nothing to follow, so the tag to
|
||||
// follow rides in a marker comment on the line ABOVE the reference:
|
||||
//
|
||||
// # renovate-digest: image=<registry>/<namespace>/<image> tag=main
|
||||
// <anything>sha256:<64 hex>
|
||||
//
|
||||
// `main` is not a version, so Renovate skips the release lookup and proposes a `digest`
|
||||
// update whenever the tag's digest moves (lib/workers/repository/process/lookup/index.ts:
|
||||
// an unversioned currentValue WITH a currentDigest yields updateType `digest`). The
|
||||
// `sha256:` prefix is INSIDE currentDigest on purpose: Renovate replaces that capture
|
||||
// verbatim with the new digest, which carries the prefix.
|
||||
//
|
||||
// DELIBERATELY NOT TRACKED: infra/agent/worker-job-template.yaml. It is baked INTO
|
||||
// agent-runner, so bumping it would rebuild agent-runner and loop forever; pm-dispatch
|
||||
// overwrites that field from PM_WORKER_IMAGE at runtime anyway.
|
||||
// REGISTRY AUTH: the Gitea registry answers 401 to anonymous reads, and the platform token is
|
||||
// scoped to hostType `gitea`, which a docker lookup never matches
|
||||
// (lib/util/host-rules.ts find). renovate.yaml supplies a hostType `docker` rule.
|
||||
{
|
||||
"customType": "regex",
|
||||
"description": "This repo's own CI-published images, pinned by digest, following the tag named in a `# renovate-digest:` marker on the previous line.",
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)versions\\.env$/",
|
||||
"/(^|/)platform/pm-agent/.+\\.ya?ml$/"
|
||||
],
|
||||
"matchStrings": [
|
||||
"#\\s*renovate-digest:\\s*image=(?<depName>\\S+)\\s+tag=(?<currentValue>\\S+)\\s*\\n[^\\n]*?(?<currentDigest>sha256:[a-f0-9]{64})"
|
||||
],
|
||||
"datasourceTemplate": "docker",
|
||||
"versioningTemplate": "docker"
|
||||
}
|
||||
],
|
||||
"packageRules": [
|
||||
{
|
||||
"description": "Auto-merge the allowed low-risk classes on green CI \u2014 no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
|
||||
"matchUpdateTypes": [
|
||||
"minor",
|
||||
"patch",
|
||||
"digest"
|
||||
],
|
||||
"automerge": true,
|
||||
"automergeType": "pr",
|
||||
"platformAutomerge": true
|
||||
},
|
||||
{
|
||||
"description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
|
||||
"matchUpdateTypes": [
|
||||
"major"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": [
|
||||
"needs-review",
|
||||
"breaking"
|
||||
]
|
||||
},
|
||||
{
|
||||
"description": "Keep the KubeVirt + CDI operator pair on a compatible track \u2014 grouped so they bump together (platform/kubevirt/kustomization.yaml).",
|
||||
"matchDepNames": [
|
||||
"kubevirt/kubevirt",
|
||||
"kubevirt/containerized-data-importer"
|
||||
],
|
||||
"groupName": "kubevirt-cdi"
|
||||
},
|
||||
{
|
||||
"description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface \u2014 label but do not auto-merge; a version bump there changes CI reproducibility.",
|
||||
"matchFileNames": [
|
||||
"ci/versions.env",
|
||||
"infra/devcontainer/versions.env"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": [
|
||||
"ci-tooling"
|
||||
]
|
||||
},
|
||||
{
|
||||
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check \u2014 it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
|
||||
"matchPackageNames": [
|
||||
"kured",
|
||||
"longhorn",
|
||||
"cilium",
|
||||
"cloudnative-pg",
|
||||
"kube-prometheus-stack"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": [
|
||||
"needs-review",
|
||||
"blast-radius"
|
||||
]
|
||||
},
|
||||
{
|
||||
"description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.",
|
||||
"matchManagers": ["kubernetes"],
|
||||
"matchPackageNames": [
|
||||
"gitea.home.keithjackson.dev/home-network/toolbox",
|
||||
"gitea.home.keithjackson.dev/home-network/agent-runner",
|
||||
"gitea.home.keithjackson.dev/home-network/conformance-runner"
|
||||
],
|
||||
"enabled": false
|
||||
},
|
||||
{
|
||||
"description": "ADDED 2026-10-29. Digest bumps for this repo's own images: the image was built and verified by CI before it was published, so these auto-merge on a green gate like any digest (rule 1). Labelled so they are easy to find. NOTE: the conformance-runner bump edits conformance/versions.env, a sensitive path (infra/gitea/approval-policy.json), so it will wait for an administrator's /approve-override regardless of this rule.",
|
||||
"matchManagers": ["custom.regex"],
|
||||
"matchPackageNames": [
|
||||
"gitea.home.keithjackson.dev/home-network/toolbox",
|
||||
"gitea.home.keithjackson.dev/home-network/agent-runner",
|
||||
"gitea.home.keithjackson.dev/home-network/conformance-runner"
|
||||
],
|
||||
"addLabels": ["own-image"]
|
||||
}
|
||||
],
|
||||
// WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A
|
||||
// FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it
|
||||
// and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest,
|
||||
// lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak
|
||||
// (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete
|
||||
// version before any automation can maintain them, and until then they drift on every pull with
|
||||
// nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12,
|
||||
// e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it
|
||||
// has been told not to leave; raising those ceilings is deliberate per-component work and is
|
||||
// NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.
|
||||
// platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest`
|
||||
// and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed
|
||||
// shape, not software to keep current. `conformance/**` was already ignored for the same
|
||||
// reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise
|
||||
// have opened.
|
||||
// conformance/** NARROWED 2026-10-29 to its subdirectories: ignorePaths applies to custom
|
||||
// managers too (lib/workers/repository/extract/file-match.ts getMatchingFiles ->
|
||||
// filterIgnoredFiles), so the blanket glob also hid conformance/versions.env, which now carries
|
||||
// the conformance-runner digest Renovate must maintain. Everything else under conformance/
|
||||
// stays ignored, for the original reason (test fixtures with deliberately fixed shapes).
|
||||
"ignorePaths": [
|
||||
"**/*.example",
|
||||
"**/*.sops.yaml",
|
||||
"conformance/config/**",
|
||||
"conformance/incluster/**",
|
||||
"conformance/lib/**",
|
||||
"conformance/requirements/**",
|
||||
"conformance/runs/**",
|
||||
"conformance/tests/**",
|
||||
"platform/image-automation/per-service/**",
|
||||
"platform/kyverno/tests/**"
|
||||
],
|
||||
// The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were
|
||||
// BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation
|
||||
// over a running tag. That controller was never enabled and never will be here (see _comment),
|
||||
// so the only thing those two flags were achieving was leaving every plain `image:` tag in the
|
||||
// repo unmanaged. `docker` is left out of enabledManagers because this repo contains no
|
||||
// Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.
|
||||
"docker": {
|
||||
"enabled": true
|
||||
}
|
||||
}
|
||||
@@ -1,162 +0,0 @@
|
||||
# cluster-maintenance-shutdown — powering the whole cluster off, and back on
|
||||
|
||||
**Operator procedure, not an alert page.** Nothing links here from an alert.
|
||||
**Script:** `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`)
|
||||
**Related:** `/runbooks/update-rollback` is for rolling node work where the cluster stays up. This
|
||||
page is for taking **all three nodes down at once**.
|
||||
|
||||
First written after the 2026-10-17 UPS move, which was improvised live because no procedure
|
||||
existed — `infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev
|
||||
cluster, so it does nothing for these three baremetal nodes.
|
||||
|
||||
## Why you cannot just run `shutdown -h`
|
||||
|
||||
You mostly can now, and could not before.
|
||||
|
||||
Until 2026-10-17 **kubelet graceful node shutdown was unconfigured on every prod node**, so systemd
|
||||
stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood — 9 Postgres
|
||||
instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes.
|
||||
`roles/k3s_server` now sets `shutdown-grace-period` (and, critically, the
|
||||
`InhibitDelayMaxSec` logind cap that actually enforces it — without that systemd stops waiting
|
||||
after 5s and the drain is silently truncated). A plain reboot of one node is therefore safe.
|
||||
|
||||
**A planned whole-cluster shutdown still wants `quiesce`**, for three things the kubelet's drain
|
||||
does not do:
|
||||
|
||||
1. **The UniFi VM needs an ACPI guest shutdown**, not an eviction. KubeVirt does that on
|
||||
`running: false`; an eviction is a power cut to the guest.
|
||||
2. **CloudNativePG wants hibernation**, not pod deletion — and hibernation is **imperative** state
|
||||
that Flux will neither set nor clear.
|
||||
3. **Draining three nodes one at a time makes pods reschedule onto the survivors**, churning
|
||||
Longhorn replicas for no reason when everything is going down anyway. Scaling in place avoids it.
|
||||
|
||||
## The hazard that bit us — read this before scaling anything by hand
|
||||
|
||||
**A StatefulSet can carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and scaling it
|
||||
to 0 then DELETES its PVCs and their Longhorn volumes.**
|
||||
|
||||
On 2026-10-17 the hand-rolled quiesce scaled `monitoring/loki` (which has that policy) to 0, and
|
||||
`storage-loki-0` was deleted outright. Loki stores with `type: filesystem`, so that PVC was the only
|
||||
copy. Its 72h retention bounded the loss to a rolling window — nothing about the *mechanism* is
|
||||
bounded. The only visible symptom was the Longhorn volume count going 25 → 24, which was first
|
||||
misread as a miscount.
|
||||
|
||||
`quiesce.sh` now refuses to scale any such StatefulSet, leaves it running, and tells you which
|
||||
volumes to expect `degraded` on restart. **If you quiesce by hand, check the policy first:**
|
||||
|
||||
```bash
|
||||
kubectl --context k3s-prod get statefulset -A \
|
||||
-o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,WHEN_SCALED:.spec.persistentVolumeClaimRetentionPolicy.whenScaled'
|
||||
```
|
||||
|
||||
A hard-detached volume comes back `degraded` and rebuilds itself automatically. That is a vastly
|
||||
better outcome than a deleted one — take the rebuild.
|
||||
|
||||
## Shutting down
|
||||
|
||||
Run everything through the pinned toolbox; it supplies `kubectl` and the context.
|
||||
|
||||
```bash
|
||||
# 0. Look before you leap. Refuses to proceed if any Longhorn volume is unhealthy —
|
||||
# powering off on top of a degraded volume is how one replica becomes zero.
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
|
||||
|
||||
# 1. Quiesce: suspend Flux, ACPI-stop the VM, hibernate Postgres, scale down every
|
||||
# PVC-holding workload (NFS consumers first, PiHole last), wait for detach.
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh quiesce'
|
||||
```
|
||||
|
||||
**Expect DNS to stop** when PiHole scales down — it is the LAN resolver. `kubectl` is unaffected
|
||||
because the kubeconfig points at the kube-vip IP `192.168.2.9`, not a name.
|
||||
|
||||
```bash
|
||||
# 2. Power the nodes off. Inside the devcontainer, which holds the become password.
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
|
||||
-m shell -a "shutdown -h +1 \"maintenance\""'
|
||||
```
|
||||
|
||||
`+1` gives Ansible time to disconnect cleanly. The group resolves to all three nodes; confirm with
|
||||
`--list-hosts` if you want to see them first.
|
||||
|
||||
```bash
|
||||
# 3. Confirm they are actually down before touching anything else.
|
||||
for h in 192.168.2.10 192.168.2.11 192.168.2.12; do ping -c1 -W2 $h >/dev/null && echo "$h UP" || echo "$h down"; done
|
||||
```
|
||||
|
||||
**Then, in this order and not before:**
|
||||
|
||||
| # | Machine | How |
|
||||
|---|---|---|
|
||||
| 4 | **TrueNAS** `192.168.2.30` | **Web UI only.** Not in the ansible inventory, and SSH is **refused** (service disabled). `https://192.168.2.30` → power icon → Shut Down. Safe once the nodes are down, because every NFS consumer (`plex-media`, `*-media`, `qbittorrent-downloads`, `nextcloud-legacy-archive`) is already scaled to zero and nothing holds a hard mount. |
|
||||
| 5 | **Router / switch** | Last. |
|
||||
|
||||
TrueNAS must go **after** the k3s nodes: a hard NFS mount on a live node wedges kubelet in
|
||||
D-state (`/runbooks/nfs-export-latency`, and the load-average symptom in the 2026-10-?? note).
|
||||
|
||||
## Bringing it back
|
||||
|
||||
**Power on in the reverse order: router/switch → TrueNAS → all three nodes together.** etcd needs
|
||||
2 of 3 for quorum, so start the nodes as a group rather than one at a time.
|
||||
|
||||
```bash
|
||||
# 1. Nodes Ready, etcd quorum, storage reachable.
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get --raw=/healthz/etcd'
|
||||
|
||||
# 2. Resume: un-hibernate Postgres, then resume Flux.
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh resume'
|
||||
```
|
||||
|
||||
**Un-hibernating Postgres is the step everyone forgets.** The annotation is imperative and survives
|
||||
a Flux resume, so resuming Flux alone leaves all three databases down and Keycloak, Nextcloud and
|
||||
Semaphore looking broken for reasons that point nowhere near Postgres. `resume` does it first, for
|
||||
exactly that reason.
|
||||
|
||||
Flux restores every scaled-down replica count **and** the VM's `running: true` from git, so nothing
|
||||
needs undoing by hand.
|
||||
|
||||
### What "still broken" looks like when it is actually fine
|
||||
|
||||
* **`4/19` Kustomizations Ready for several minutes.** The dependency graph serialises behind
|
||||
`platform-config`; 15 of them report `dependency ... is not ready` until it finishes. Check
|
||||
`platform-config` itself is `Ready=True` and its `lastAppliedRevision` matches the GitRepository
|
||||
artifact — if it is, wait.
|
||||
* **One or more Longhorn volumes `degraded`.** Any volume that was still attached at poweroff
|
||||
rebuilds its lagging replica automatically. On 2026-10-17 it was Prometheus's 50Gi TSDB; it went
|
||||
healthy on its own. Worry only if it is still degraded after ~30 minutes.
|
||||
* **`keycloak-operator` crashlooping, then recovering.** It is the CRD reconciler, not in the
|
||||
request path; Keycloak's StatefulSet stays 2/2 and auth never goes down. See
|
||||
`/runbooks/keycloak-*` and the known OOM-at-450Mi behaviour.
|
||||
|
||||
### What is actually broken and needs you
|
||||
|
||||
* **An `oauth2-proxy` replica stuck in CrashLoopBackOff.** It performs OIDC discovery at startup and
|
||||
**exits 1** on the `503 no healthy upstream` it gets while Keycloak is still coming up, then sits
|
||||
in exponential backoff long after Keycloak is fine. It has no startup dependency on Keycloak.
|
||||
Delete the pod once and it comes up clean:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n oauth2-proxy delete pod <name>'
|
||||
```
|
||||
|
||||
* **Orphaned `Unknown`/`Error` Job pods** from CronJobs that were mid-run at poweroff. They never
|
||||
recover; delete them.
|
||||
|
||||
## Verifying you are actually back
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
|
||||
# nodes 3 Ready | flux 19/19 | longhorn 0 not healthy | postgres 9 pods | unifi 1 VMI
|
||||
|
||||
# DNS, through PiHole's VIP
|
||||
./infra/devcontainer/dev.sh run bash -c 'nslookup home.keithjackson.dev 192.168.2.23'
|
||||
|
||||
# The Gateway actually serving (401 on an auth-gated host is CORRECT)
|
||||
for h in home.keithjackson.dev gitea.home.keithjackson.dev plex.home.keithjackson.dev; do
|
||||
curl -s -o /dev/null -w "$h %{http_code}\n" -k --resolve $h:443:192.168.2.20 https://$h/
|
||||
done
|
||||
```
|
||||
|
||||
Expect `200`, `200`, `401`.
|
||||
@@ -52,13 +52,6 @@ configMapGenerator:
|
||||
- backup-failure=backup-failure.md
|
||||
- backup-missed=backup-missed.md
|
||||
- backup-partial=backup-partial.md
|
||||
# Operator procedure, not an alert page — nothing links here from an alert. Served because
|
||||
# the whole-cluster shutdown had NO written procedure until 2026-10-17 (infra/cluster.sh is
|
||||
# virsh-based and only ever applied to the decommissioned dev cluster), so the UPS move was
|
||||
# improvised live, and the one hazard it exposed -- a StatefulSet with
|
||||
# persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 --
|
||||
# destroyed monitoring/loki's volume before anyone noticed.
|
||||
- cluster-maintenance-shutdown=cluster-maintenance-shutdown.md
|
||||
- cdi-import-stuck=cdi-import-stuck.md
|
||||
- cert-expiry=cert-expiry.md
|
||||
- cert-issuance=cert-issuance.md
|
||||
|
||||
@@ -67,10 +67,10 @@ Work through these in order; the first three are by far the most likely.
|
||||
| Symptom in the log | Cause | Fix |
|
||||
|---|---|---|
|
||||
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
|
||||
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been published; the committed digest is the deliberate placeholder | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
|
||||
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
|
||||
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
||||
| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
|
||||
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | restore the digest form; the digest comes from CI's publish (`.gitea/workflows/images.yaml` job summary) and Renovate's bump PR, never from a hand push |
|
||||
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
|
||||
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
|
||||
| `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
|
||||
| `at the worker cap` / `worker limit reached` | the per-cycle cap, draining a backlog one issue at a time | expected; it will continue next cycle |
|
||||
|
||||
Reference in new issue
Block a user