fix(vpn): default client profiles to MTU 1280; correct the VPN runbook

new-peer.sh generated MTU 1420, which stalled traffic on mobile data
despite a healthy handshake (one phone, one carrier, 2026-11-09). Default
to 1280, overridable through WG_MTU.

Correct stale claims in the runbook and the DaemonSet header against
live BPF conntrack: LoadBalancer VIPs see 10.90.0.x as world, not a
SNATed host identity. Add a troubleshooting section.

The DaemonSet render is unchanged, so WireGuard pods do not roll.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

.gitea/workflows/README.md

@@ -15,7 +15,7 @@ it touches is green** (R-TEST-03).
 ## The pipeline
 
 ```
- lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
+ decisions-log ─▶ lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
  └──────────────── run in CI, NO cluster ────────────────┘        └── in-cluster Chainsaw Job ──┘
                     live now                              cluster-gated (SKIPS in Phase 0)
 ```
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
 *collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
 (`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
 
-## Honest SKIP — never fake green (Phase 0)
+## Honest SKIP — never fake green
 
-**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
+**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
+exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
+skip logic below is still correct and still honest, but read it as describing a **live
+coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
+phrased as a phase is part of why the gap went unexamined.
+
+Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
 they do not pass**:
 
 - The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
 ## Triggers & branch model (CONTRIBUTING.md)
 
 - `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
-- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
+- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
+  so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
+  **No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
+  protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
 
-## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
+## Transition: COMPLETE (2026-09-29)
 
-Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
+This section used to describe a migration as pending. It has happened. Gitea is live at
-[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
+`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
+`gate.yml` runs on every PR.
 
-1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
+**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
-   but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
+anyone checking the platform. In reality `main` had **no branch protection** and **zero
-   local feature-branch → `main`, the gate enforced by convention + the local runner.
+runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
-2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
+between rested on an assertion nobody verified. PR #1 — the first pull request this
-3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
+repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
-   enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
+
-   `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
+So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
-   **platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
+Query the API. `infra/gitea/` holds the scripts and the measured state.
+
+Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
+`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
+CI. That is a real coverage gap, not a phase.
 
 ## Configuration (set in Gitea; referenced by name only)
 
@@ -101,14 +115,60 @@ Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
 | variable | `REQUIRE_CLUSTER_TESTS` | `'true'` (Phase 1+) makes unverified cluster-needing touched reqs a hard block |
 | secret | `CONFORMANCE_TRIGGER_KUBECONFIG` | narrow creds to create/await the in-cluster Chainsaw Job (not cluster-admin) |
 
+## How images are built
+
+This repo's three images — `toolbox` (`infra/devcontainer/`), `agent-runner` (`infra/agent/`,
+built `FROM` the toolbox) and `conformance-runner` (`conformance/incluster/`) — are built,
+verified and published by [`images.yaml`](images.yaml), never from a workstation
+(operator rule, 2026-10-29). The logic is in [`scripts/images.sh`](scripts/images.sh).
+
+**The builder is rootless BuildKit** ([`platform/buildkit/`](../../platform/buildkit/README.md)),
+reached over mTLS with a pinned `buildx` — no job here touches a Docker daemon, privileged or
+otherwise (operator decision on PR #81). The client certificate reaches jobs as the Actions
+secrets `BUILDKIT_CA`, `BUILDKIT_CLIENT_CERT`, `BUILDKIT_CLIENT_KEY`.
+
+**Verification is a build target.** Each Containerfile ends in a `verify` stage that runs every
+in-image assertion at build time as the non-root uid with `RUN --network=none` (tool presence and
+pinned versions, `PM_SELFTEST=1 pm-dispatch`, the pm-worker probe, the `conformance-clone`
+refusals, no writable path outside the workspace). The pipeline builds `<image>-verify` before it
+builds, let alone pushes, the image. Build args come from each image's own
+`build-image.sh --print-build-args`, which is also the local podman path — so the recipe and the
+proof exist once.
+
+| Trigger | What happens |
+|---|---|
+| pull request touching an image's inputs | build + verify only. Nothing is pushed and **no registry credential is referenced** |
+| push to `main` touching an image's inputs | build + verify + publish to `gitea.home.keithjackson.dev/home-network/<image>` as `sha-<commit>` (immutable) and `main` (moving); the **registry-returned** digest is in the job summary |
+| `workflow_dispatch` | the same as a push when run on `main` (all images by default); verify-only elsewhere |
+
+Only images whose inputs changed are built. The publish job **never writes to git**: Renovate
+follows each image's `main` tag through a `# renovate-digest:` marker next to every digest
+reference (`platform/pm-agent/`, `infra/agent/versions.env`, `conformance/versions.env`) and
+opens the bump PR, which goes through the gate like anything else. A PR opened with the Actions
+token would not trigger the required gate, which is why it is Renovate's job and not this one's.
+
+Credential: the repository secret **`REGISTRY_PUSH_TOKEN`** (scope `write:package`, owned by an
+account with package write on the org). The automatic Actions token cannot push in Gitea 1.24.3
+— the reasoning, with source citations, is in the header of `images.yaml`. Retention:
+[`infra/gitea/registry-retention.sh`](../../infra/gitea/registry-retention.sh), an operator
+action, keeps every digest git pins plus the newest tags.
+
+Cost: the job shares the single act_runner slot with the gate, but the build itself runs in the
+`buildkitd` pod, not in DinD, so the runner's disk and memory are untouched. Until
+`platform/buildkit` is deployed and its client certificate copied
+(`infra/gitea/buildkit-client-secret.sh`), the job fails loudly at its builder preflight.
+
 ## Files
 
 ```
 .gitea/workflows/
   merge-gate.yaml            the pipeline (lint → l0 → l1 → conformance → decision)
+  images.yaml                build/verify/publish this repo's container images (see above)
   scripts/
-    install-tools.sh         installs the pinned tools per stage (from ci/versions.env)
+    install-tools.sh         installs the pinned tools per stage (from ci/versions.env) — used by the
+                             dormant merge-gate.yaml blueprint only; gate.yml runs in the toolbox image
     decide.sh                the R-TEST-03 merge decision over job results
+    images.sh                the image pipeline's driver (what to build, base, push, digest)
   README.md                  this file
 ```
 

.gitea/workflows/gate.yml

@@ -1,40 +1,103 @@
 # .gitea/workflows/gate.yml — the ENFORCED PR merge gate (Phase 6, WS-GITOSS).
 #
 # This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
-# Docker executor + DinD). It invokes the AGGREGATE merge gate — `make -C ci gate` — so CI
+# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
-# runs BYTE-FOR-BYTE the same command a human runs in the devcontainer (R-TEST-02): the
+# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
-# runner just installs the pinned tools from ci/versions.env first. `make -C ci gate`
+# (R-TEST-02) — and since 2026-11-08 it runs INSIDE the same image: the job `container:` is the
-# (ci/gate.sh) runs every stage in order — spec-immutability, lint, Kyverno L0,
+# CI-built devcontainer toolbox (see "THE JOB IMAGE" below), not a generic runner image that
-# render==golden, L1 offline render — and applies the R-TEST-03 decision, printing
+# downloads the pinned tools on every run. (See further below for why the script, not `make`,
-# "MERGE GATE: PASS" / "MERGE GATE: BLOCKED". It is the single required status check on
+# is invoked here.) It runs every stage in order —
-# `main` (see branch protection). A non-zero exit fails the job, so a BLOCKED gate blocks
+# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
-# the merge.
+# R-TEST-03 decision, printing
+# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
+# single required status check on `main` (see branch protection). A non-zero exit fails the job,
+# so a BLOCKED gate blocks the merge.
+#
+# WHY THIS STEP DECLARES GATE_STRICT_SKIPS AND TOLERATES EXIT 3 (2026-09-13). A SKIPPED stage is
+# never green, and since 2026-09-13 gate.sh refuses to print PASS when one did not run (it used to
+# print "— not green" and then "MERGE GATE: PASS", exit 0 — fail-open in the gate itself). Two
+# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
+# environment it is:
+#   * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
+#     baked into the job image and checked against ci/versions.env two steps up.
+#     GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
+#     "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
+#   * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
+#     not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
+#     is the ONLY skip that can still reach exit 3 under strict skips, which is what makes
+#     `|| [ "$?" -eq 3 ]` a narrow, reviewed tolerance rather than a blanket one. The log still
+#     says UNDETERMINED and names R-TEST-06 as unverified. Wire the spec checkout (or set
+#     REQUIRE_SPEC_VERIFICATION=true) and this tolerance should be deleted.
+# Called DIRECTLY, not through `make`: GNU make reports any recipe failure as its own exit 2, so
+# `make -C ci gate || [ "$?" -eq 3 ]` would see 2 and fail the job. Same reason the
+# spec-immutability pre-step below calls the script directly.
 #
 # The cluster-gated L2/L3 conformance stage SKIPS honestly here (exit 3, surfaced not faked
-# — HARD RULE 4) because this runner declares no cluster: `make -C ci gate` does not set
+# — HARD RULE 4) because this runner declares no cluster: nothing here sets CLUSTER_AVAILABLE,
-# CLUSTER_AVAILABLE, so gate.sh reports the touched requirements UNVERIFIED without blocking
+# so gate.sh reports the touched requirements UNVERIFIED — and NAMES that declared gap in its
-# Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
+# PASS headline — without blocking Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
 # no broad kubeconfig) driven by the separate `conformance` job below.
 #
 # NOTE (R-TEST-06 / gate-semantics): spec-immutability is ALSO surfaced as its own explicit
 # early step (below) so its result is a distinct, tolerant status in the Gitea UI; gate.sh
-# re-runs it as its first stage, which is the authoritative one. `make -C ci gate` here is
+# re-runs it as its first stage, which is the authoritative one. The gate step below is the
-# the byte-identical laptop gate; the extra pre-step is a read-only git-diff check and is
+# same script the laptop runs; the extra pre-step is a read-only git-diff check and is
 # purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
 # is retained dormant in merge-gate.yaml.
+#
+# THE JOB IMAGE (2026-11-08, feat/gate-toolbox-container). The gate job runs in
+# gitea.home.keithjackson.dev/home-network/toolbox, pinned BY DIGEST — the image
+# .gitea/workflows/images.yaml builds from infra/devcontainer/Containerfile, verifies, and
+# publishes. It is the SAME image `infra/devcontainer/dev.sh` runs, so a local gate and the CI
+# gate now execute the identical toolchain (R-TEST-02), not two installs that agree by intent.
+# Before this, the job ran in ghcr.io/catthehacker/ubuntu:act-22.04 and spent ~11s of every run
+# re-downloading yamllint, check-jsonschema, kyverno, promtool and chainsaw
+# (.gitea/workflows/scripts/install-tools.sh l0 + l1; measured on runs 494/495).
+#   * PINNED BY DIGEST, MOVED BY RENOVATE. The `# renovate-digest:` marker above the image line
+#     is the repo-wide convention (renovate.json5, "This repo's own CI-published images"):
+#     Renovate follows the `main` tag and opens a digest-bump PR, whose OWN gate run then
+#     executes on the new image — the bump tests itself. infra/gitea/registry-retention.sh reads
+#     the same marker, so the pinned digest is never pruned. Never hand-edit it to a tag.
+#   * PULL AUTH: none needed. The Home-Network org is public and the registry issues an
+#     anonymous pull token (measured 2026-11-08: anonymous /v2/token -> manifest 200), so the
+#     runner's DinD pulls it like any public image. If the org is ever made private, add
+#     `credentials:` here with a read:package token secret — the automatic Actions token has no
+#     package scope at /v2/token (see images.yaml's header).
+#   * COLD PULL: ~467 MB compressed (1.54 GB unpacked). The DinD store is an emptyDir, so the
+#     first job after a runner-pod restart or a digest bump pays one pull; every other run finds
+#     it cached (forcePull=false), exactly as it already did for the catthehacker image.
+#   * NO NODE in the toolbox, so `actions/checkout` (a Node action) cannot run here. The checkout
+#     step is plain git instead — see its comment for how it keeps fetch-depth-0 semantics.
+#   * install-tools.sh is NOT deleted: merge-gate.yaml (the dormant blueprint) still calls it,
+#     and conformance/tests/l0-static/alert-rules-unit-tested asserts its promtool wiring.
 ---
 name: gate
 
+# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
+#
+# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
+# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
+#
+#   * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
+#     SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
+#     clock on the one result that actually gates the merge. That cost real time during the
+#     2026-09-29 plex claim, where the loop is racing a short-lived token.
+#   * The list silently did not match this repo's own convention. Branches here are `feat/**`,
+#     which was absent, so the "fast feedback" never fired for the most common prefix anyway —
+#     a mismatch nobody noticed for a month because its only symptom was a MISSING run.
+#     Any hand-maintained prefix list has that failure mode; deleting it removes the class.
+#   * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
+#     you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
+#     same checks buys nothing.
+#
+# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
+# which no PR context exists.
 on:
   pull_request:
     branches: [main]
   push:
     branches:
       - main
-      - 'feature/**'
-      - 'fix/**'
-      - 'docs/**'
-      - 'chore/**'
 
 concurrency:
   group: gate-${{ github.ref }}
@@ -44,16 +107,76 @@ jobs:
   # Single required status check: the static merge gate.
   gate:
     runs-on: ubuntu-latest
+    # The CI-built toolbox — the same image dev.sh runs. See "THE JOB IMAGE" in the header.
+    container:
+      # renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
+      image: gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501efe34fa0b039e3f0d360738686a0f59109866c49779768f5d5ed0ef81
     steps:
-      - name: Checkout
+      # Plain git, because actions/checkout is a Node action and the toolbox has no Node. It
-        uses: actions/checkout@v4
+      # reproduces what actions/checkout@v7 with `fetch-depth: 0` did here (read from its logged
-        with:
+      # commands on runs 490/495): full history of every branch + tags, so
-          fetch-depth: 0        # full history so ci/touched-requirements.sh can diff vs main
+      # ci/touched-requirements.sh can diff against origin/<base>; the exact event commit
-
+      # checked out (detached for a PR, as `-B <branch>` for a push). One difference, on
-      - name: Install pinned CI toolchain (ci/versions.env)
+      # purpose: the job token is passed to the single fetch through GIT_CONFIG_* environment
+      # variables and is NOT persisted into .git/config — nothing after this step needs
+      # authenticated git (ci/ and conformance/ make no network git calls), so there is no
+      # reason to leave a credential in the workspace for the gate's scripts to read.
+      - name: Checkout (git, full history)
+        env:
+          GIT_TOKEN: ${{ github.token }}
+          SERVER_URL: ${{ github.server_url }}
+          REPOSITORY: ${{ github.repository }}
+          SHA: ${{ github.sha }}
+          REF: ${{ github.ref }}
         run: |
-          ./.gitea/workflows/scripts/install-tools.sh l0
+          set -euo pipefail
-          ./.gitea/workflows/scripts/install-tools.sh l1
+          # act runs steps with umask 0000, so without this every checked-out directory is
+          # world-writable — and Ansible then IGNORES infra/ansible/ansible.cfg as a source
+          # ("Ansible is being run in a world writable directory"), which made the L0 check
+          # prod-baremetal-not-dev-surface fail on this PR's first CI run (run 500). The
+          # toolbox has Ansible, so that check now drives it for real. 0022 = what a
+          # workstation clone gets (reproduced locally: 0000 fails the check, 0022 passes).
+          umask 0022
+          git config --global --add safe.directory "$PWD"
+          git init -q .
+          git remote add origin "${SERVER_URL}/${REPOSITORY}"
+          auth="$(printf 'x-access-token:%s' "$GIT_TOKEN" | base64 -w0)"
+          GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraHeader \
+          GIT_CONFIG_VALUE_0="AUTHORIZATION: basic ${auth}" \
+            git -c protocol.version=2 fetch --quiet --no-tags --prune --no-recurse-submodules origin \
+              '+refs/heads/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*' \
+              "+${SHA}:refs/remotes/checkout/head"
+          unset auth GIT_TOKEN
+          case "$REF" in
+            refs/heads/*) git checkout -q --force -B "${REF#refs/heads/}" "$SHA" ;;
+            *)            git checkout -q --force --detach "$SHA" ;;
+          esac
+          [ "$(git rev-parse HEAD)" = "$SHA" ] || { echo "::error::HEAD is not ${SHA}" >&2; exit 1; }
+          echo "checked out $(git log -1 --format='%H %s') (ref ${REF}); $(git rev-list --count HEAD) commits of history"
+
+      # The image is the toolchain, so prove it is the PINNED one before trusting it: every
+      # *_VERSION in ci/versions.env must equal what the image recorded at build time
+      # (/etc/dev-toolchain-versions, written by infra/devcontainer/Containerfile; its `verify`
+      # stage already asserted the version-sensitive binaries report those pins). This replaces
+      # the per-run install: a drifted toolbox fails HERE, naming the tool, instead of gating with
+      # a different engine. ci/versions.env and infra/devcontainer/versions.env must move together.
+      - name: Toolchain is the one ci/versions.env pins
+        run: |
+          set -euo pipefail
+          . ci/versions.env
+          drift=0
+          for var in $(grep -oE '^[A-Z_]+_VERSION' ci/versions.env); do
+            tool="$(printf '%s' "${var%_VERSION}" | tr 'A-Z_' 'a-z-')"
+            want="${!var}"
+            got="$(sed -n "s/^${tool}=//p" /etc/dev-toolchain-versions)"
+            if [ "$got" = "$want" ]; then
+              printf 'ok     %-18s %s\n' "$tool" "$want"
+            else
+              printf '::error::DRIFT  %s: ci/versions.env pins %s, the job image has %s\n' "$tool" "$want" "${got:-nothing}"
+              drift=1
+            fi
+          done
+          [ "$drift" -eq 0 ]
 
       # R-TEST-06: the spec is read-only to agents. An EXPLICIT early surfacing of the
       # spec-immutability check, so its result is a distinct, tolerant status line in the
@@ -69,13 +192,39 @@ jobs:
       - name: Spec immutability (R-TEST-06, skips if spec repo absent)
         run: ./ci/spec-immutability.sh || [ "$?" -eq 3 ]
 
-      # Aggregate merge gate: byte-for-byte the command a human runs in the devcontainer
+      # Aggregate merge gate: the same script a human runs in the devcontainer via
-      # (`make -C ci gate` -> ci/gate.sh). Runs spec-immutability, lint, Kyverno L0,
+      # `make -C ci gate`. Runs spec-immutability, lint, Kyverno L0, render==golden and L1
-      # render==golden and L1 render, then the R-TEST-03 decision. Exits non-zero on a
+      # render, then the R-TEST-03 decision. Exits non-zero on a BLOCKED gate, which fails this
-      # BLOCKED gate, which fails this job. The cluster-gated L2/L3 stage SKIPS here (no
+      # job. The cluster-gated L2/L3 stage SKIPS here (no CLUSTER_AVAILABLE) — surfaced as
-      # CLUSTER_AVAILABLE) — surfaced as UNVERIFIED, never faked green (see header).
+      # UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
-      - name: Merge gate — make -C ci gate
+      # tolerated exit 3 are explained in the file header; the tolerance covers exactly one
-        run: make -C ci gate
+      # thing, the unverifiable spec repo, and nothing else can produce a 3 here.
+      # GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
+      # OWN database (branch protection + runner registration) — the blind spot that let `main`
+      # sit unprotected with zero runners for a month while this repo recorded both as working.
+      # Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
+      #
+      # WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
+      # (spec-immutability). It now also covers gitea-settings skipping when that secret is
+      # absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
+      # every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
+      # exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
+      # tolerance narrows back to spec-immutability alone.
+      - name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
+        env:
+          GATE_STRICT_SKIPS: "true"
+          SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
+          # ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
+          # UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
+          # skipping, this line turns that into a failure and the PR goes red instead of
+          # merging. Green here means the token resolved and the live Gitea matched the
+          # declaration in infra/gitea/expected-branch-protection.json.
+          REQUIRE_GITEA_SETTINGS: "true"
+          # ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
+          # PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
+          GITHUB_EVENT_NAME: ${{ github.event_name }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
       # ("not supported on GHES" — Gitea does not implement the artifact-v2 backend), which

.gitea/workflows/images.yaml

@@ -0,0 +1,185 @@
+# Container images — built, verified and published by CI, never from a workstation.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY THIS FILE EXISTS (operator rule, 2026-10-29)
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# "Container image creation, hosting and lifecycle must be part of CI/CD and the codebase, never a
+# one-off build/push from a workstation." Before this workflow, three images existed only as
+# workstation builds and NOTHING had ever been pushed to the Gitea registry: the Home-Network org's
+# package list was empty, platform/pm-agent/ pinned an all-zero digest and sat in ImagePullBackOff,
+# and conformance/versions.env carried an empty digest. Both build scripts said "publishing is an
+# operator action" — a step with no log, no recorded inputs and no verification anyone could read.
+#
+# WHAT IT DOES — the logic is in scripts/images.sh; this file only wires triggers and secrets.
+#   pull_request -> build + VERIFY only. No push, and no registry credential is referenced.
+#   push to main -> build + verify + PUBLISH to gitea.home.keithjackson.dev/home-network/<image>
+#   dispatch     -> same as a push when run on main; verify-only on any other ref.
+# Images: toolbox (infra/devcontainer), agent-runner (infra/agent, FROM toolbox) and
+# conformance-runner (conformance/incluster). Only images whose inputs changed are built.
+#
+# THE DIGEST LIFECYCLE — this workflow NEVER writes to git.
+#   1. publish pushes `sha-<commit>` (immutable) and `main` (moving), and prints the digest the
+#      REGISTRY returns in the job summary.
+#   2. Renovate (renovate.yaml) follows the `main` tag of each image through the
+#      `# renovate-digest:` markers next to every digest reference, and opens the bump PR.
+#   3. That PR goes through the gate like any other change, and Flux deploys it.
+# The bump PR is Renovate's and not this job's because a PR opened with the Actions token does not
+# trigger workflows, so it could never pass the REQUIRED gate check; Renovate's token is a real
+# user's.
+#
+# THE PUSH CREDENTIAL — `REGISTRY_PUSH_TOKEN`, a repository Actions secret, referenced BY NAME.
+# The automatic Actions token CANNOT be used. Gitea 1.24.3 source, verified:
+#   * services/auth/basic.go (Basic.Verify) turns a task token into the pseudo-user
+#     `gitea-actions` (ActionsUserID -2) and sets NO token scope;
+#   * routers/api/packages/container/container.go (Authenticate, the /v2/token exchange every
+#     docker client uses) then requires read:package / write:package / all scope and answers 401;
+#   * services/context/package.go (determineAccessMode) has `// TODO: ActionUser permission check`
+#     and no branch for that user, so even past login it would get at most READ on a public org.
+# So: a personal access token with scope `write:package`, minted by an account that has package
+# WRITE on the Home-Network org (org owner, or a team with the Packages unit at write). A dedicated
+# bot account in a packages-only team is the least-privilege choice. The username sent with it is
+# not checked by Gitea (basic.go resolves the owner from the token), so `vars.REGISTRY_PUSH_USER`
+# is optional and only labels the login.
+#
+# THE BUILDER — rootless BuildKit (operator decision on PR #81, 2026-10-29).
+# The act runner's job containers get NO Docker daemon (act_runner 0.2.11 forces docker_host "-"
+# for a tcp:// DinD, and gitea/act strips every bind when valid_volumes is empty — measured on
+# this PR's first run), and handing them the privileged DinD daemon would be root on the node for
+# anyone who can push a branch. So images are built by platform/buildkit/: a ROOTLESS buildkitd
+# (not privileged; uid 1000 in a user namespace), reached over mTLS at
+# buildkitd.buildkit.svc.cluster.local:1234 with a pinned `buildx` (remote driver). Its client
+# certificate reaches jobs as three Actions secrets — BUILDKIT_CA, BUILDKIT_CLIENT_CERT,
+# BUILDKIT_CLIENT_KEY — copied from cert-manager by infra/gitea/buildkit-client-secret.sh.
+# Pull requests get the client certificate (they must build) but never a registry credential.
+#
+# UNTIL platform/buildkit IS DEPLOYED (Flux applies it only after merge) THIS JOB FAILS, LOUDLY,
+# at its builder preflight. A skipped Gitea job reports `success`, so it is red on purpose.
+#
+# VERIFICATION is each Containerfile's `verify` stage, built before anything is pushed.
+#
+name: images
+
+on:
+  pull_request:
+    branches: [main]
+    paths:
+      - infra/devcontainer/Containerfile
+      - infra/devcontainer/versions.env
+      - infra/devcontainer/build-image.sh
+      - infra/agent/**
+      - conformance/incluster/Containerfile
+      - conformance/incluster/conformance-clone
+      - conformance/incluster/build-image.sh
+      - .gitea/workflows/images.yaml
+      - .gitea/workflows/scripts/images.sh
+      - platform/buildkit/versions.env
+  push:
+    branches: [main]
+    paths:
+      - infra/devcontainer/Containerfile
+      - infra/devcontainer/versions.env
+      - infra/devcontainer/build-image.sh
+      - infra/agent/**
+      - conformance/incluster/Containerfile
+      - conformance/incluster/conformance-clone
+      - conformance/incluster/build-image.sh
+      - .gitea/workflows/images.yaml
+      - .gitea/workflows/scripts/images.sh
+      - platform/buildkit/versions.env
+  workflow_dispatch:
+    inputs:
+      images:
+        description: "Which images: 'all', or a space-separated subset of: toolbox agent-runner conformance-runner"
+        required: false
+        default: "all"
+
+concurrency:
+  # Never cancel a publish half-way: a pushed `sha-` tag with no matching `main` move is
+  # recoverable (a rerun skips what is already published) but pointless to provoke.
+  group: images-${{ github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  images:
+    runs-on: ubuntu-latest
+    # Measured on the PR that introduced this file — see its decisions entry. Generous because a
+    # cold DinD store downloads the Fedora base and every pinned tool on each run.
+    timeout-minutes: 75
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+        with:
+          fetch-depth: 0   # the per-image change detection diffs against the base commit
+
+      # ── pull requests: build + verify, NO credential ───────────────────────────────────────
+      - name: Build and verify (pull request — nothing is pushed)
+        if: github.event_name == 'pull_request'
+        env:
+          MODE: verify
+          IMAGES: auto
+          PR_BASE_REF: ${{ github.base_ref }}
+          # The BUILDER's client certificate (platform/buildkit). Not a registry credential:
+          # a pull request can build, never push.
+          BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
+          BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
+          BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
+        run: |
+          set -euo pipefail
+          BASE_SHA="$(git merge-base "origin/${PR_BASE_REF}" HEAD)"
+          export BASE_SHA
+          ./.gitea/workflows/scripts/images.sh
+
+      # ── non-main dispatch: verify only ──────────────────────────────────────────────────────
+      - name: Build and verify (dispatch off main — nothing is pushed)
+        if: github.event_name == 'workflow_dispatch' && github.ref != 'refs/heads/main'
+        env:
+          MODE: verify
+          IMAGES: ${{ inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images }}
+          BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
+          BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
+          BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
+        run: ./.gitea/workflows/scripts/images.sh
+
+      # ── main: publish ───────────────────────────────────────────────────────────────────────
+      - name: Preflight - the push token must exist, and its absence must be LOUD
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        env:
+          REGISTRY_PUSH_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${REGISTRY_PUSH_TOKEN:-}" ]; then
+            echo "::error::REGISTRY_PUSH_TOKEN is not set on this repository." >&2
+            cat >&2 <<'MSG'
+
+          Images cannot be published without a registry credential, and this step fails rather
+          than skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would
+          make "nothing was published" look exactly like "published". That is how this repo went
+          a month with images that existed only on one workstation.
+
+          WHY NOT THE BUILT-IN ACTIONS TOKEN: Gitea 1.24.3 gives it no package scope at the
+          registry's /v2/token exchange (routers/api/packages/container/container.go,
+          Authenticate) — see the header of .gitea/workflows/images.yaml.
+
+          TO FIX: as an account with package WRITE on the Home-Network org (ideally a dedicated
+          bot in a packages-only team), create an access token with scope `write:package` and
+          nothing else, then add it to this repository as the Actions secret
+          REGISTRY_PUSH_TOKEN. Optionally set the Actions variable REGISTRY_PUSH_USER to that
+          account's name. Do not commit the token anywhere. Then re-run this workflow
+          (workflow_dispatch on main, images: all).
+          MSG
+            exit 1
+          fi
+          echo "REGISTRY_PUSH_TOKEN is present (value not printed)."
+
+      - name: Build, verify and publish (main)
+        if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main'
+        env:
+          MODE: publish
+          IMAGES: ${{ github.event_name == 'workflow_dispatch' && (inputs.images == 'all' && 'toolbox agent-runner conformance-runner' || inputs.images) || 'auto' }}
+          BASE_SHA: ${{ github.event.before }}
+          REGISTRY_TOKEN: ${{ secrets.REGISTRY_PUSH_TOKEN }}
+          REGISTRY_USER: ${{ vars.REGISTRY_PUSH_USER }}
+          BUILDKIT_CA: ${{ secrets.BUILDKIT_CA }}
+          BUILDKIT_CLIENT_CERT: ${{ secrets.BUILDKIT_CLIENT_CERT }}
+          BUILDKIT_CLIENT_KEY: ${{ secrets.BUILDKIT_CLIENT_KEY }}
+        run: ./.gitea/workflows/scripts/images.sh

.gitea/workflows/merge-gate.yaml

@@ -11,11 +11,14 @@
 #                                          v
 #                                     merge-gate decision  [R-TEST-03: touched subset green]
 #
-# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
+# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
-# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
+# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
-# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
+# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
-# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
+# as a plan and never revisited once the plan completed.
-# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
+#
+# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
+# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
+# statement about the live platform; query the API instead (see infra/gitea/).
 #
 # CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
 # scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a
@@ -57,9 +60,9 @@ jobs:
   spec-immutability:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v4                 # the implementation repo
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1                 # v7                 # the implementation repo
       - name: Check out the read-only spec (sibling path)
-        uses: actions/checkout@v4
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with:
           repository: ${{ vars.SPEC_REPO_SLUG }}  # e.g. Home-Network/home-network-k3s (set in Gitea)
           path: ../home-network-k3s
@@ -71,7 +74,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [spec-immutability]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }          # full history so touched-requirement diff works
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh lint
@@ -82,7 +85,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [lint]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l0
@@ -93,7 +96,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [l0]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -104,7 +107,7 @@ jobs:
     runs-on: ubuntu-latest
     needs: [render-golden]
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh l1
@@ -112,7 +115,7 @@ jobs:
         run: make -C ci l1
       - name: Upload L1 artifacts
         if: always()
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
         with:
           name: l1-junit-coverage
           path: |
@@ -128,7 +131,7 @@ jobs:
     needs: [l1]
     if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
         with: { fetch-depth: 0 }
       - name: Install pinned tools
         run: ./.gitea/workflows/scripts/install-tools.sh conformance
@@ -139,7 +142,7 @@ jobs:
         run: make -C ci conformance
       - name: Upload L2/L3 artifacts
         if: always()
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
         with:
           name: conformance-junit-coverage
           path: |
@@ -157,7 +160,7 @@ jobs:
     needs: [lint, l0, render-golden, l1, conformance]
     if: always()
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
       - name: Decide
         env:
           R_LINT: ${{ needs.lint.result }}

.gitea/workflows/renovate.yaml

@@ -0,0 +1,318 @@
+# Renovate — the dependency/image update automation. SCHEDULED, self-hosted against this Gitea.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY THIS FILE EXISTS: renovate.json HAS BEEN IN THIS REPO AND HAS NEVER ONCE RUN.
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# Measured 2026-10-17: `renovate.json` is present, complete and thoughtfully written — it declares
+# managers, schedules, automerge classes and grouping. Nothing executes it. There is no Renovate
+# workflow (this is the first), no Renovate CronJob or Deployment in the cluster
+# (`kubectl get cronjob,deploy,job -A | grep -i renovate` -> nothing), and no bot account doing it
+# from outside. Meanwhile SEVEN files carry comments asserting "Renovate maintains this pin"
+# (platform/homepage/deployment.yaml:40, runbooks/serving/deployment.yaml:87,
+# platform/crafty/deployment.yaml:63, platform/vpn-ingress/daemonset.yaml:259,
+# platform/keycloak/postgres-cluster.yaml:75, apps/wave-0/nextcloud-db.yaml:302, and
+# platform/headlamp/helmrepository.yaml:2). Those comments have been false for the life of the file.
+#
+# The cost is measurable. With nothing bumping anything: the six per-app oauth2-proxy auth sidecars
+# sit on v7.8.1 while the platform oauth2-proxy in the SAME cluster runs v7.15.5; Keycloak (the IdP)
+# runs 26.0.6, a November-2024 release, 8 minors behind; Velero runs a RELEASE CANDIDATE
+# (velero-plugin-for-aws v1.14.3-rc.2) with a stable available; the CNPG Postgres image is 16.4,
+# missing every 16.5+ security release. A pin nothing maintains is not a pin, it is a fossil.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# WHY RENOVATE AND NOT FLUX IMAGE-AUTOMATION (the decision, and it reverses ADR-0009's split)
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# ADR-0009 divided the work: Renovate owns Helm chart/dependency bumps, Flux image-automation owns
+# per-service container image TAGS, deliberately non-overlapping. The second half of that split
+# cannot be delivered on this repo's terms, for three measured reasons:
+#
+#   1. IT WOULD BYPASS THE MERGE GATE, WHICH CLAUDE.md §1 FORBIDS. ImageUpdateAutomation does not
+#      open pull requests; it COMMITS AND PUSHES. Its default target is the source's own branch —
+#      `main` — so it would write straight past the PR flow. It can be pointed at a side branch
+#      instead, but then nothing opens the PR and bumps silently accumulate on a branch nobody
+#      merges. Either way a human or a second bot has to close the loop.
+#   2. ON THIS CLUSTER IT CANNOT EVEN DO THAT. Measured 2026-10-17: `main` is protected with
+#      `user_can_push: false`, and the `flux-system` GitRepository has NO `spec.secretRef` at all —
+#      the source is anonymous and read-only. Enabling write-back means minting an in-cluster git
+#      WRITE credential for the repository that defines the cluster, which is a materially larger
+#      blast radius than a CI-scoped token.
+#   3. RENOVATE ALREADY OPENS PRs NATIVELY, so every change it proposes goes through the gate with
+#      no extra machinery.
+#
+# So: Renovate owns BOTH halves, and Flux image-automation stays disabled. Note this is a change of
+# plan, not a change of state — image-automation has been commented out at
+# platform/kustomization.yaml:52 since it was authored, and the cluster has ZERO
+# ImageRepository/ImagePolicy/ImageUpdateAutomation objects (measured). Its two controllers ARE
+# installed and have idled for 43 days. Nothing is being torn down here; a path that was never
+# taken is being formally closed, and renovate.json's `docker`/`kubernetes` managers — disabled
+# purely to avoid fighting a controller that never ran — are switched on to fill the gap.
+#
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# THIS WORKFLOW MUST NOT BE ABLE TO SUCCEED SILENTLY
+# ═══════════════════════════════════════════════════════════════════════════════════════════════
+# A SKIPPED Gitea job reports `success` (CLAUDE.md §2), so "green" is not evidence this ran. Two
+# guards:
+#   * The preflight step FAILS HARD (exit 1) if RENOVATE_TOKEN is absent, rather than skipping.
+#     A missing credential on a dependency bot is indistinguishable from "nothing to update", and
+#     that is precisely how renovate.json stayed inert for months.
+#   * The summary step records what ran into the job summary unconditionally.
+# What this still does NOT give you is an alert when the SCHEDULE stops firing. See the handoff
+# entry's "What I did NOT verify" — that gap is stated, not papered over.
+#
+# OPERATOR PREREQUISITE — ONE SECRET, NOT COMMITTED HERE:
+#   `RENOVATE_TOKEN` — a Gitea access token for the bot account (`renovate-bot`) with FIVE scopes.
+#   Each is required by a call Renovate 44.x makes against Gitea 1.24.3, and Gitea refuses a
+#   token missing one with `403 token does not have at least one of required scope(s)`:
+#     read:user          GET /user                        (platform init: initPlatform)
+#     read:organization  GET /orgs/{owner}, /orgs/{o}/labels  (initRepo -> isOrg, then org labels)
+#     write:repository   clone/push branches, /pulls, /statuses
+#     write:issue        /issues, /labels, /issues/{n}/labels  (dependency dashboard, PR labels;
+#                        Gitea files all of these under the ISSUE scope, not repository)
+#     read:package       the Gitea container registry (RENOVATE_HOST_RULES below)
+#   CORRECTED 2026-10-30. This list used to name only write:repository + read:user (+ read:package),
+#   and the operator minted exactly that. Run 350 then failed at
+#   `GET /api/v1/orgs/Home-Network 403 ... required=[read:organization], token scope=read:package,
+#   write:repository,read:user`; write:issue would have been the next wall. The preflight below now
+#   PROBES these scopes, so a short token fails in seconds with the missing scope named.
+#   Set it as an Actions secret on this repository. Nothing in this repo mints,
+#   stores or references its value; it is consumed by name only.
+#   Until it exists, this workflow FAILS LOUDLY on every scheduled run. That is intended: a
+#   visible red is the correct state for "automation configured but not credentialed".
+#
+# THE FIRST RUN WILL WANT TO OPEN A LOT OF PRs. ~30 dependencies are behind. renovate.json caps the
+# blast radius (`prConcurrentLimit: 5`, `prHourlyLimit: 2`), but run it manually with
+# `dry_run: true` FIRST and read the log before letting the schedule loose. That is what
+# workflow_dispatch's default is for.
+name: renovate
+
+on:
+  schedule:
+    # 08:15 UTC daily. renovate.json ALSO carries `"schedule": ["after 3am and before 6am"]` with
+    # `"timezone": "America/New_York"`, and THAT inner schedule is authoritative — Renovate re-checks
+    # it and DECLINES TO OPEN PRs outside it. So this cron must fire INSIDE the local window, not
+    # merely near it.
+    #
+    # WAS `15 3 * * *`, WHICH NEVER WOULD HAVE WORKED (fixed 2026-10-18, before the first scheduled
+    # run). Gitea Actions cron is UTC, so 03:15 UTC is 23:15 the PREVIOUS EVENING in New York --
+    # 3h45m BEFORE the window opens, not "slightly ahead" of it as the original comment claimed.
+    # Every nightly run would have started, logged "not within schedule", opened zero PRs and
+    # exited SUCCESSFULLY. A green tick forever, and no updates ever: the exact "failure mode is a
+    # green tick" shape this repo keeps getting burned by, and indistinguishable from "nothing
+    # needed updating".
+    #
+    # THE WINDOW IN UTC MOVES WITH US DST, and a cron cannot follow it:
+    #   EDT (UTC-4, Mar-Nov): local 03:00-06:00 == 07:00-10:00 UTC
+    #   EST (UTC-5, Nov-Mar): local 03:00-06:00 == 08:00-11:00 UTC
+    # 08:15 UTC is the only quarter-hour comfortably inside BOTH (04:15 EDT / 03:15 EST), so the
+    # schedule keeps working across both switchovers without an edit. Do not "tidy" this back to an
+    # early-morning-UTC value without redoing this arithmetic.
+    - cron: "15 8 * * *"
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: "Dry run (no PRs, no branches). Leave true for the first run."
+        required: false
+        default: "true"
+      log_level:
+        description: "Renovate LOG_LEVEL (info | debug)"
+        required: false
+        default: "info"
+
+concurrency:
+  # Never two Renovate runs at once: concurrent runs race on the same branches and can open
+  # duplicate PRs. Not cancel-in-progress — a half-finished Renovate run can leave a branch pushed
+  # with no PR, so let the first one finish.
+  group: renovate
+  cancel-in-progress: false
+
+jobs:
+  renovate:
+    runs-on: ubuntu-latest
+    # RENOVATE RUNS AS THE JOB CONTAINER, NOT THROUGH renovatebot/github-action. FIXED 2026-10-30.
+    # That action does exactly one thing: `docker run ghcr.io/renovatebot/renovate:<version>`. On
+    # this runner a job has NO Docker daemon, by decision (decisions/2026-10-29-image-pipeline.md:
+    # act_runner sets container.docker_host to `-` when DOCKER_HOST is tcp://, and the DinD daemon
+    # is privileged, so a job that could reach it would be root on the node). So every run of the
+    # action died before Renovate started. Dispatch run 343 (2026-10-30), verbatim:
+    #   [command]/usr/bin/docker run --env LOG_LEVEL ... --env RENOVATE_TOKEN=*** ...
+    #   failed to connect to the docker API at unix:///var/run/docker.sock; ... no such file or directory
+    #   ::error::Error: The process '/usr/bin/docker' failed with exit code 1
+    # No auth, scope, config or network question was ever reached. Making the Renovate image the
+    # job container needs no daemon: act starts it through the runner's own DinD exactly as it
+    # starts the default ubuntu image, then `exec`s each step inside it.
+    #
+    # PINNED here (was the action's `renovate-version: 44.167.0`). renovate.json5 is validated
+    # against this exact tag — keep them in step. Renovate's github-actions manager reads
+    # `.gitea/workflows` and extracts `container.image`, so this pin is still self-maintained.
+    #
+    # The image runs as uid 12021 (`ubuntu`), and that is fine under act: the step command files
+    # ($GITHUB_STEP_SUMMARY etc.) are created mode 0666 (gitea/act v0.261.2 pkg/runner/step.go),
+    # HOME comes from the image env, and no step writes the workspace. There is deliberately NO
+    # actions/checkout: Renovate clones the repo itself from RENOVATE_ENDPOINT, and as uid 12021
+    # checkout could not `git init` the root-owned workspace volume anyway.
+    container:
+      image: ghcr.io/renovatebot/renovate:44.167.0
+    # Renovate scans every manager across the whole repo; 40m is generous for this repo's size and
+    # well under a runner's patience. Without a timeout a wedged run holds the `renovate`
+    # concurrency group and silently blocks every subsequent scheduled run.
+    timeout-minutes: 40
+    steps:
+      - name: Preflight - the token must exist, and its absence must be LOUD
+        shell: bash
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [ -z "${RENOVATE_TOKEN:-}" ]; then
+            echo "::error::RENOVATE_TOKEN is not set on this repository." >&2
+            cat >&2 <<'MSG'
+
+          Renovate cannot run without a Gitea access token, and this step fails rather than
+          skipping ON PURPOSE: a skipped Gitea job reports `success`, so skipping here would make
+          "dependency automation is not running" look exactly like "everything is up to date".
+          That is the failure mode that let renovate.json sit inert in this repo for months while
+          seven files claimed "Renovate maintains this pin".
+
+          TO FIX: create a Gitea access token for the bot account (renovate-bot) with scopes
+          read:user, read:organization, write:repository, write:issue and read:package (the
+          header of .gitea/workflows/renovate.yaml says which call needs which), then add it to
+          this repository as the Actions secret RENOVATE_TOKEN. Do not commit it anywhere.
+          MSG
+            exit 1
+          fi
+          echo "RENOVATE_TOKEN is present (value not printed)."
+          # Proves the step is running INSIDE the pinned Renovate image, not the default job image.
+          echo "renovate $(renovate --version)"
+
+      - name: Preflight - the token has every scope Renovate needs
+        shell: bash
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+        # ADDED 2026-10-30. Renovate reports EVERY platform-init failure as the bare
+        # "Authentication failure" (lib/modules/platform/gitea/index.ts initPlatform swallows the
+        # HTTP error), and a later scope gap surfaces as "Repository has unknown error". Two
+        # dispatches were spent learning one missing scope at a time. These probes are GETs only,
+        # and they name the missing scope. The token never reaches argv: curl reads it from stdin
+        # (`--config -`), so it is not in `ps` output or in the log.
+        run: |
+          set -uo pipefail
+          GITEA=https://gitea.home.keithjackson.dev
+          REPO=Home-Network/home-network-k3s-impl
+          fail=0
+          body=$(mktemp)
+          probe() { # <scope it proves> <path> <auth: bearer|basic>
+            local code
+            if [ "$3" = basic ]; then
+              code=$(printf 'user = "renovate:%s"\n' "$RENOVATE_TOKEN" \
+                | curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
+            else
+              code=$(printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
+                | curl -sS --config - -o "$body" -w '%{http_code}' "$GITEA$2") || code=000
+            fi
+            if [ "$code" = 200 ]; then
+              echo "ok    $1  (GET ${2%%\?*} -> 200)"
+              return
+            fi
+            fail=1
+            echo "::error::scope $1: GET ${2%%\?*} -> $code"
+            case "$code" in
+              401) echo "      401 = Gitea rejected the token VALUE (wrong, expired or revoked)." ;;
+              403) if grep -q 'Access denied' "$body"; then
+                     echo "      403 'Access denied' = a Cilium/Envoy policy denial, not Gitea."
+                   else
+                     echo "      Gitea: $(jq -r '.message // empty' "$body" 2>/dev/null | head -c 300)"
+                   fi ;;
+              *)   echo "      body: $(head -c 200 "$body")" ;;
+            esac
+          }
+          probe read:user          "/api/v1/user"                          bearer
+          probe read:organization  "/api/v1/orgs/${REPO%%/*}"              bearer
+          probe read:repository    "/api/v1/repos/$REPO"                   bearer
+          probe read:issue         "/api/v1/repos/$REPO/labels?limit=1"    bearer
+          probe read:package       "/v2/token?service=container_registry&scope=repository:home-network/toolbox:pull" basic
+          # WRITE scopes cannot be proven by a GET. Gitea's scope-refusal message, however, lists
+          # the token's scopes (names only, never the value): ask for an admin route this bot
+          # cannot have, and read `token scope=` from the 403. Advisory if the format changes.
+          printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
+            | curl -sS --config - -o "$body" "$GITEA/api/v1/admin/orgs?limit=1" || true
+          scopes=$(jq -r '.message // empty' "$body" 2>/dev/null | sed -n 's/.*token scope=//p')
+          if [ -n "$scopes" ]; then
+            echo "token scopes (from Gitea): $scopes"
+            for want in write:repository write:issue; do
+              case ",$scopes," in
+                *",$want,"*|*",all,"*) echo "ok    $want" ;;
+                *) echo "::error::scope $want is missing (token scope=$scopes)"; fail=1 ;;
+              esac
+            done
+          else
+            echo "::warning::could not read the token's scope list; write:repository and write:issue are UNVERIFIED"
+          fi
+          rm -f "$body"
+          if [ "$fail" -ne 0 ]; then
+            echo "::error::RENOVATE_TOKEN failed the checks above (a 401 is the value, a 403 is a scope). Re-mint it for renovate-bot with read:user, read:organization, write:repository, write:issue, read:package and overwrite the secret."
+            exit 1
+          fi
+
+      - name: Run Renovate
+        shell: bash
+        # The repo config is discovered by Renovate from the cloned repo (renovate.json5,
+        # RENOVATE_REQUIRE_CONFIG=required). The action used to ALSO load it as the global config
+        # via RENOVATE_CONFIG_FILE; that double-loading is gone. ci/lint.sh's `configurationFile:`
+        # check therefore has nothing to resolve in this file, and passes.
+        env:
+          RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
+          # Self-hosted Gitea. `platform` + `endpoint` were the two values renovate.json's own
+          # header said were "set at runtime by the operator's renovate runner" — this is that
+          # runner, so they are set here rather than in the repo config, which keeps renovate.json
+          # portable and free of this instance's URL.
+          RENOVATE_PLATFORM: gitea
+          RENOVATE_ENDPOINT: https://gitea.home.keithjackson.dev/api/v1
+          RENOVATE_REPOSITORIES: Home-Network/home-network-k3s-impl
+          # Renovate must never push to a protected branch; it works on its own branches and opens
+          # PRs against this base. Stated explicitly rather than inherited, because this is the
+          # property that keeps the whole mechanism inside CLAUDE.md §1's branch -> PR -> gate path.
+          RENOVATE_BASE_BRANCHES: main
+          RENOVATE_ONBOARDING: "false"
+          RENOVATE_REQUIRE_CONFIG: "required"
+          RENOVATE_DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run == 'true' && 'full' || '' }}
+          LOG_LEVEL: ${{ github.event_name == 'workflow_dispatch' && inputs.log_level || 'info' }}
+          # ADDED 2026-10-29: registry read access for this repo's OWN images (toolbox,
+          # agent-runner, conformance-runner — published by .gitea/workflows/images.yaml and
+          # followed through `# renovate-digest:` markers). The Gitea registry answers 401 to
+          # anonymous reads, and Renovate scopes the platform token to hostType `gitea`, which a
+          # docker lookup never matches — so without this rule every digest lookup fails. Same
+          # token; it therefore needs scope `read:package` in addition to the ones listed in the
+          # preflight above. Gitea ignores the username for token auth (services/auth/basic.go).
+          RENOVATE_HOST_RULES: '[{"hostType":"docker","matchHost":"gitea.home.keithjackson.dev","username":"renovate","password":"${{ secrets.RENOVATE_TOKEN }}"}]'
+          # OPTIONAL, ADDED 2026-10-30. A github.com token (a fine-grained PAT with NO permissions,
+          # public-repo read only) for lookups of GitHub-hosted dependencies. Without it Renovate
+          # SKIPS them with `github-token-required` — measured in a local dry run: 17 deps, i.e.
+          # every `uses: actions/*` here plus kubevirt/kubevirt, kubevirt/containerized-data-importer
+          # and intel/intel-device-plugins-for-kubernetes. Unset is safe: the step below drops the
+          # empty variable, and those deps simply stay unmaintained, with a WARN in the log.
+          GITHUB_COM_TOKEN: ${{ secrets.RENOVATE_GITHUB_COM_TOKEN }}
+        run: |
+          set -euo pipefail
+          # An expression that evaluates to '' still EXPORTS the variable, as an empty string.
+          # Drop those so an unset value means "Renovate's default", never "set to empty".
+          for v in RENOVATE_DRY_RUN GITHUB_COM_TOKEN; do
+            [ -n "${!v:-}" ] || unset "$v"
+          done
+          # Through the image's entrypoint, as `docker run` would have gone: it sources the
+          # containerbase env and initialises the bundled tools before exec'ing Renovate.
+          exec renovate-entrypoint.sh renovate
+
+      - name: Summary
+        if: always()
+        run: |
+          {
+            echo "## Renovate"
+            echo
+            echo "- trigger: \`${{ github.event_name }}\`"
+            echo "- dry run: \`${{ github.event_name == 'workflow_dispatch' && inputs.dry_run || 'false' }}\`"
+            echo "- outcome: \`${{ job.status }}\`"
+            echo
+            echo "Renovate owns BOTH Helm chart bumps and container image tags here; Flux"
+            echo "image-automation is deliberately disabled (platform/kustomization.yaml)."
+            echo "Every change it proposes arrives as a pull request and must pass the merge gate."
+          } >> "$GITHUB_STEP_SUMMARY"

.gitea/workflows/scripts/images.sh

@@ -0,0 +1,373 @@
+#!/usr/bin/env bash
+###############################################################################
+# .gitea/workflows/scripts/images.sh — the image pipeline's driver (images.yaml calls this).
+#
+# Builds, verifies and (on main) PUBLISHES the three images this repo ships:
+#
+#   toolbox             infra/devcontainer/    the pinned dev/gate toolchain
+#   agent-runner        infra/agent/           FROM toolbox; the PM dispatcher + worker
+#   conformance-runner  conformance/incluster/ the in-cluster L2/L3 Chainsaw runner
+#
+# THE BUILDER (operator decision on PR #81): a ROOTLESS BuildKit daemon, platform/buildkit/,
+# reached over mTLS. No CI job may reach a privileged daemon, so there is no `docker build` and no
+# `docker run` here: this script drives buildkitd with a pinned `buildx` (remote driver) and
+# nothing else. The client certificate arrives as Actions secrets (BUILDKIT_CA,
+# BUILDKIT_CLIENT_CERT, BUILDKIT_CLIENT_KEY; infra/gitea/buildkit-client-secret.sh puts them
+# there). If the daemon is unreachable this FAILS LOUDLY — it never skips.
+#
+# VERIFICATION IS A BUILD TARGET. Each Containerfile ends in a `verify` stage that runs every
+# in-image assertion at build time (as the non-root uid, with RUN --network=none); this script
+# builds `<image>-verify` and only then the image. Build args come from each image's own
+# `build-image.sh --print-build-args`, the same list the local podman path uses.
+#
+# MODES (env MODE):
+#   verify   build the verify targets, never push, hold no registry credential. Pull requests.
+#   publish  verify, then build + push to ${REGISTRY}/${NAMESPACE}/<image> with tags
+#            `sha-<12 hex>` (immutable: never re-pushed) and `main` (moving; Renovate follows
+#            it). The digest recorded is the one the REGISTRY returns, read back after the push.
+#
+# WHAT GETS BUILT (env IMAGES): `auto` (default) diffs BASE_SHA..HEAD_SHA against each image's
+# inputs; otherwise a space-separated list. A change to the pipeline itself (images.yaml, this
+# file, platform/buildkit/versions.env) builds everything in verify mode, and nothing extra in
+# publish mode: re-publishing unchanged inputs would only mint new digests and churn Renovate PRs.
+#
+# THE agent-runner BASE:
+#   * verify mode -> FROM the toolbox built in this run, linked with a bake `target:` context —
+#     a pull request holds no registry credential and that toolbox exists in no registry. If the
+#     toolbox's inputs did not change it is built anyway, as the base.
+#   * publish mode, toolbox published in this run -> FROM the digest the registry just returned.
+#   * publish mode otherwise -> AGENT_BASE_IMAGE from infra/agent/versions.env (the published
+#     toolbox digest Renovate keeps current). If that is still the all-zero placeholder the
+#     toolbox has never been published, so it is built and published first (the bootstrap).
+#
+# LOOP-FREEDOM. Renovate turns each publish into a digest-bump PR. Those bumps land in
+# platform/pm-agent/ (no image input), conformance/versions.env (no image input) and
+# infra/agent/versions.env (agent-runner input -> ONE more agent-runner build, whose own bump
+# lands in platform/pm-agent/ and stops). The worker Job template is deliberately NOT tracked:
+# it is baked INTO agent-runner, so bumping it would rebuild agent-runner, mint a new digest,
+# bump it again, forever. pm-dispatch overwrites that field at runtime anyway.
+###############################################################################
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
+cd "${REPO_ROOT}"
+
+# shellcheck disable=SC1091
+BK_HOST_OVERRIDE="${BUILDKIT_HOST:-}"; BK_NAME_OVERRIDE="${BUILDKIT_SERVERNAME:-}"
+. ./platform/buildkit/versions.env
+BUILDKIT_HOST="${BK_HOST_OVERRIDE:-${BUILDKIT_HOST}}"
+BUILDKIT_SERVERNAME="${BK_NAME_OVERRIDE:-${BUILDKIT_SERVERNAME}}"
+
+MODE="${MODE:-verify}"
+IMAGES="${IMAGES:-auto}"
+REGISTRY="${REGISTRY:-gitea.home.keithjackson.dev}"
+NAMESPACE="${NAMESPACE:-home-network}"
+HEAD_SHA="${HEAD_SHA:-$(git rev-parse HEAD)}"
+BASE_SHA="${BASE_SHA:-}"
+SUMMARY="${GITHUB_STEP_SUMMARY:-/dev/null}"
+ZERO_DIGEST="sha256:0000000000000000000000000000000000000000000000000000000000000000"
+
+ALL_IMAGES=(toolbox agent-runner conformance-runner)
+SHA_TAG="sha-${HEAD_SHA:0:12}"
+MOVING_TAG="main"
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; printf -- '- **ERROR**: %s\n' "$*" >>"${SUMMARY}"; exit 1; }
+yn()   { if [ -n "$1" ]; then echo "$2"; else echo "$3"; fi; }
+
+case "${MODE}" in verify|publish) ;; *) die "MODE must be verify or publish, got '${MODE}'" ;; esac
+for t in git jq curl sha256sum; do command -v "$t" >/dev/null || die "the job image lacks '$t'"; done
+
+# --- per-image facts ------------------------------------------------------------------------
+# Inputs: a change under any of these rebuilds that image. conformance-runner takes its pins from
+# infra/devcontainer/versions.env. conformance/versions.env is NOT an input (it only RECORDS the
+# published digest), and must not become one: Renovate writes it after every publish -> a loop.
+inputs() {
+  case "$1" in
+    toolbox)            echo "infra/devcontainer/Containerfile infra/devcontainer/versions.env infra/devcontainer/build-image.sh" ;;
+    agent-runner)       echo "infra/agent/" ;;
+    conformance-runner) echo "conformance/incluster/Containerfile conformance/incluster/conformance-clone conformance/incluster/build-image.sh infra/devcontainer/versions.env" ;;
+  esac
+}
+context_of() { case "$1" in toolbox) echo infra/devcontainer ;; agent-runner) echo infra/agent ;; conformance-runner) echo conformance/incluster ;; esac; }
+PIPELINE_FILES=".gitea/workflows/images.yaml .gitea/workflows/scripts/images.sh platform/buildkit/versions.env"
+
+declare -A WANT=() BUILT=() DIGEST=() SECS=() NOTE=()
+
+changed_any() { # <paths...> -> 0 if BASE_SHA..HEAD_SHA touches any
+  [ -n "$(git diff --name-only "${BASE_SHA}" "${HEAD_SHA}" -- "$@")" ]
+}
+
+select_images() {
+  local i
+  if [ "${IMAGES}" != "auto" ]; then
+    for i in ${IMAGES}; do
+      case " ${ALL_IMAGES[*]} " in *" ${i} "*) WANT[$i]=1; NOTE[$i]="requested" ;; *) die "unknown image '${i}'" ;; esac
+    done
+    return
+  fi
+  if [ -z "${BASE_SHA}" ] || [[ "${BASE_SHA}" =~ ^0+$ ]] || ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
+    info "no usable diff base ('${BASE_SHA}'): building everything"
+    for i in "${ALL_IMAGES[@]}"; do WANT[$i]=1; NOTE[$i]="no diff base"; done
+    return
+  fi
+  info "diff base ${BASE_SHA} .. ${HEAD_SHA}"
+  for i in "${ALL_IMAGES[@]}"; do
+    # shellcheck disable=SC2046
+    if changed_any $(inputs "$i"); then WANT[$i]=1; NOTE[$i]="inputs changed"; fi
+  done
+  # shellcheck disable=SC2086
+  if [ "${MODE}" = verify ] && changed_any ${PIPELINE_FILES}; then
+    for i in "${ALL_IMAGES[@]}"; do [ -n "${WANT[$i]:-}" ] || { WANT[$i]=1; NOTE[$i]="pipeline changed"; }; done
+  fi
+}
+
+# --- the builder: pinned buildx + mTLS to rootless buildkitd ---------------------------------
+WORK="$(mktemp -d)"
+trap 'rm -rf "${WORK}"' EXIT          # client key + registry auth live only here, only this long
+export BUILDX_CONFIG="${WORK}/buildx" DOCKER_CONFIG="${WORK}/docker"
+# Single-manifest images: the admission policies, Renovate and registry-retention.sh all reason
+# about one manifest per tag. (Provenance attestations would turn every push into an index.)
+export BUILDX_NO_DEFAULT_ATTESTATIONS=1
+mkdir -p "${BUILDX_CONFIG}" "${DOCKER_CONFIG}" "${WORK}/certs"
+echo '{}' > "${DOCKER_CONFIG}/config.json"
+
+setup_builder() {
+  log "Builder: rootless buildkitd at ${BUILDKIT_HOST} (mTLS)"
+  BUILDX="${BUILDX_BIN:-}"
+  if [ -z "${BUILDX}" ]; then
+    BUILDX="${WORK}/buildx-bin"
+    curl -fsSL --retry 3 -o "${BUILDX}" \
+      "https://github.com/docker/buildx/releases/download/v${BUILDX_VERSION}/buildx-v${BUILDX_VERSION}.linux-amd64" \
+      || die "could not download buildx v${BUILDX_VERSION}"
+    echo "${BUILDX_SHA256}  ${BUILDX}" | sha256sum -c --quiet - \
+      || die "buildx v${BUILDX_VERSION} does not match the sha256 pinned in platform/buildkit/versions.env — refusing to run it"
+    chmod +x "${BUILDX}"
+  fi
+  info "client: $("${BUILDX}" version)"
+
+  local missing=()
+  [ -n "${BUILDKIT_CA:-}" ]          || missing+=(BUILDKIT_CA)
+  [ -n "${BUILDKIT_CLIENT_CERT:-}" ] || missing+=(BUILDKIT_CLIENT_CERT)
+  [ -n "${BUILDKIT_CLIENT_KEY:-}" ]  || missing+=(BUILDKIT_CLIENT_KEY)
+  [ "${#missing[@]}" -eq 0 ] || die "the builder's client certificate is missing: ${missing[*]} not set.
+       These Actions secrets are copied from the cert-manager Secret buildkit/buildkit-client-ci by
+       infra/gitea/buildkit-client-secret.sh (an operator action, after platform/buildkit is
+       deployed). Without them nothing can be built — this fails rather than skipping ON PURPOSE."
+  umask 077
+  printf '%s\n' "${BUILDKIT_CA}"          > "${WORK}/certs/ca.pem"
+  printf '%s\n' "${BUILDKIT_CLIENT_CERT}" > "${WORK}/certs/cert.pem"
+  printf '%s\n' "${BUILDKIT_CLIENT_KEY}"  > "${WORK}/certs/key.pem"
+  umask 022
+
+  "${BUILDX}" create --name ci-remote --driver remote \
+    --driver-opt "cacert=${WORK}/certs/ca.pem,cert=${WORK}/certs/cert.pem,key=${WORK}/certs/key.pem,servername=${BUILDKIT_SERVERNAME}" \
+    "${BUILDKIT_HOST}" >/dev/null
+  if ! out="$(timeout 90 "${BUILDX}" inspect --bootstrap --builder ci-remote 2>&1)"; then
+    printf '%s\n' "${out}" >&2
+    die "buildkitd is UNREACHABLE at ${BUILDKIT_HOST} — nothing can be built, and this job fails
+       rather than skipping. In order of likelihood:
+         1. platform/buildkit is not deployed yet. Flux applies it only after it merges to main
+            (Kustomization 'buildkit'); a pull request that ADDS it cannot reach it.
+         2. the BUILDKIT_* secrets are stale (the client cert was renewed) or from another CA:
+            re-run infra/gitea/buildkit-client-secret.sh.
+         3. the NetworkPolicy (platform/buildkit/networkpolicy.yaml) does not admit this runner."
+  fi
+  printf '%s\n' "${out}" | grep -E '^(Name|Driver|Status|BuildKit version|Platforms):' | sed 's/^/    /' || true
+}
+
+# KEY=VALUE lines from an image's build-image.sh --print-build-args -> a JSON object.
+build_args_json() {
+  local ctx; ctx="$(context_of "$1")"
+  "./${ctx}/build-image.sh" --print-build-args | jq -Rn '[inputs | select(length>0) | capture("^(?<k>[^=]+)=(?<v>.*)$")] | map({(.k): .v}) | add'
+}
+
+# bake_run <bakefile> <targets...>
+bake_run() {
+  local f="$1"; shift
+  "${BUILDX}" bake --builder ci-remote --progress plain -f "$f" \
+    --metadata-file "${WORK}/meta.json" "$@"
+}
+
+# Write a bake file with a verify target and a ship target for one image.
+#   $1 image, $2 agent base mode: "" | "link" | <image ref>
+write_bake() {
+  local img="$1" base="${2:-}" ctx args f
+  ctx="$(context_of "$img")"
+  args="$(build_args_json "$img")"
+  f="${WORK}/bake-${img}.json"
+  local contexts='{}'
+  if [ "${img}" = agent-runner ]; then
+    if [ "${base}" = link ]; then
+      # FROM ${AGENT_BASE_IMAGE} resolves to the named context below, i.e. the toolbox target
+      # of this same bake file — built here, never pushed anywhere.
+      args="$(jq -c '. + {AGENT_BASE_IMAGE: "toolbox-in-this-run"}' <<<"${args}")"
+      contexts='{"toolbox-in-this-run": "target:toolbox"}'
+    else
+      args="$(jq -c --arg b "${base}" '. + {AGENT_BASE_IMAGE: $b}' <<<"${args}")"
+    fi
+  fi
+  jq -n --arg img "$img" --arg ctx "$ctx" --argjson args "$args" --argjson contexts "$contexts" \
+        --arg tbargs "$( [ "${base}" = link ] && build_args_json toolbox || echo '{}')" \
+        --arg rev "${HEAD_SHA}" --arg repo "${REGISTRY}/${NAMESPACE}/${img}" \
+        --arg sha "${SHA_TAG}" --arg moving "${MOVING_TAG}" --arg mode "${MODE}" '
+    {target: (
+      {
+        ($img + "-verify"): {context: $ctx, dockerfile: "Containerfile", target: "verify",
+                              args: $args, contexts: $contexts, output: ["type=cacheonly"]},
+        ($img): ({context: $ctx, dockerfile: "Containerfile", target: $img, args: $args,
+                  contexts: $contexts,
+                  labels: {"org.opencontainers.image.revision": $rev}}
+                 + (if $mode == "publish"
+                    then {tags: [($repo + ":" + $sha), ($repo + ":" + $moving)], output: ["type=image,push=true"]}
+                    else {output: ["type=cacheonly"]} end))
+      }
+      + (if ($contexts | has("toolbox-in-this-run"))
+         then {toolbox: {context: "infra/devcontainer", dockerfile: "Containerfile", target: "toolbox",
+                         args: ($tbargs | fromjson), output: ["type=cacheonly"]}}
+         else {} end)
+    )}' > "$f"
+  echo "$f"
+}
+
+# --- registry helpers (publish mode only) ----------------------------------------------------
+REG_TOKEN=""
+registry_token() {
+  [ -n "${REG_TOKEN}" ] && return 0
+  REG_TOKEN="$(curl -fsS --config <(printf 'user = "%s:%s"\n' "${REGISTRY_USER}" "${REGISTRY_TOKEN}") \
+    "https://${REGISTRY}/v2/token?service=container_registry&scope=*" | jq -r '.token // empty')" \
+    || die "could not obtain a registry bearer token from https://${REGISTRY}/v2/token"
+  [ -n "${REG_TOKEN}" ] || die "registry returned no bearer token"
+}
+# Prints the manifest digest the REGISTRY serves for <image>:<tag>; nothing if it is absent (404).
+# Any other answer is fatal: "cannot tell" must never be read as "not published", or the
+# immutable-tag guard would be bypassed by an auth hiccup.
+registry_digest() {
+  registry_token
+  local hdrs code digest
+  hdrs="$(curl -sS -I --config <(printf 'header = "Authorization: Bearer %s"\n' "${REG_TOKEN}") \
+    -H 'Accept: application/vnd.oci.image.manifest.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
+    -H 'Accept: application/vnd.oci.image.index.v1+json' \
+    -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
+    "https://${REGISTRY}/v2/${NAMESPACE}/$1/manifests/$2" | tr -d '\r')" \
+    || die "registry unreachable while reading $1:$2"
+  code="$(awk 'NR==1{print $2}' <<<"${hdrs}")"
+  digest="$(awk -F': ' 'tolower($1)=="docker-content-digest"{print $2}' <<<"${hdrs}" | tail -1)"
+  case "${code}" in
+    200) [[ "${digest}" =~ ^sha256:[0-9a-f]{64}$ ]] || die "registry answered 200 for $1:$2 with no usable digest"
+         echo "${digest}" ;;
+    404) ;;
+    *)   die "registry answered HTTP ${code} for $1:$2 — cannot tell whether it exists" ;;
+  esac
+}
+
+# build_one <image> [agent base]  — verify, then (publish) build + push + read back.
+build_one() {
+  local img="$1" base="${2:-}" t0 f pushed got existing
+  t0=$(date +%s)
+  f="$(write_bake "$img" "$base")"
+  log "VERIFY ${img} (Containerfile target: verify, on rootless BuildKit)"
+  bake_run "$f" "${img}-verify"
+  BUILT[$img]=1
+  if [ "${MODE}" = publish ]; then
+    existing="$(registry_digest "${img}" "${SHA_TAG}")"
+    # Immutable tag: never overwrite. A rebuild is not byte-reproducible, so re-pushing would
+    # silently change what `sha-<commit>` means after something may already have pinned it.
+    [ -z "${existing}" ] || die "${REGISTRY}/${NAMESPACE}/${img}:${SHA_TAG} already exists (${existing}); tags are immutable"
+    log "BUILD + PUSH ${img} -> ${REGISTRY}/${NAMESPACE}/${img}:{${SHA_TAG},${MOVING_TAG}}"
+    bake_run "$f" "${img}"
+    pushed="$(jq -r --arg t "$img" '.[$t]["containerimage.digest"] // empty' "${WORK}/meta.json")"
+    # The AUTHORITATIVE digest is what the registry serves, read back — not what the client
+    # reported. They should agree; if they do not, nothing downstream may trust either.
+    got="$(registry_digest "${img}" "${SHA_TAG}")"
+    [ -n "${got}" ] || die "pushed ${img}:${SHA_TAG} but the registry does not serve it"
+    [ -z "${pushed}" ] || [ "${pushed}" = "${got}" ] || die "${img}: client reported ${pushed} but the registry serves ${got}"
+    [ "$(registry_digest "${img}" "${MOVING_TAG}")" = "${got}" ] \
+      || die "${img}: ${MOVING_TAG} does not point at the digest just pushed (${got})"
+    DIGEST[$img]="${got}"
+    info "${img}: registry digest ${got}"
+  fi
+  SECS[$img]=$(( $(date +%s) - t0 ))
+  info "${img}: done in ${SECS[$img]}s"
+}
+
+# --- main ---------------------------------------------------------------------------------
+log "image pipeline: MODE=${MODE} IMAGES=${IMAGES} HEAD=${HEAD_SHA}"
+select_images
+
+if [ "${MODE}" = publish ]; then
+  : "${REGISTRY_TOKEN:?REGISTRY_TOKEN is required in publish mode}"
+  REGISTRY_USER="${REGISTRY_USER:-ci-image-publisher}"
+  # buildx (and through it buildkitd, which does the push) reads registry auth from this file.
+  jq -n --arg r "${REGISTRY}" --arg a "$(printf '%s:%s' "${REGISTRY_USER}" "${REGISTRY_TOKEN}" | base64 -w0)" \
+     '{auths: {($r): {auth: $a}}}' > "${DOCKER_CONFIG}/config.json"
+  chmod 600 "${DOCKER_CONFIG}/config.json"
+  registry_token
+  # Publish idempotence: an image already published at THIS commit is not rebuilt. That is what
+  # makes a re-run after a half-finished publish safe — the immutable tag is never pushed twice.
+  for i in "${ALL_IMAGES[@]}"; do
+    [ -n "${WANT[$i]:-}" ] || continue
+    d="$(registry_digest "${i}" "${SHA_TAG}")"
+    if [ -n "${d}" ]; then
+      info "${i}: already published at ${SHA_TAG} (${d}); not rebuilding"
+      unset "WANT[$i]"; DIGEST[$i]="${d}"; NOTE[$i]="already published at this commit"
+    fi
+  done
+fi
+
+# Decide the agent-runner base (see header).
+AGENT_BASE=""
+if [ -n "${WANT[agent-runner]:-}" ]; then
+  PINNED_BASE="$(. ./infra/agent/versions.env; echo "${AGENT_BASE_IMAGE}")"
+  if [ "${MODE}" = verify ]; then
+    AGENT_BASE=link
+    [ -n "${WANT[toolbox]:-}" ] || { WANT[toolbox]=1; NOTE[toolbox]="built as agent-runner's base (verify only)"; }
+  elif [ -z "${WANT[toolbox]:-}" ] && [ -z "${DIGEST[toolbox]:-}" ]; then
+    case "${PINNED_BASE}" in
+      *@"${ZERO_DIGEST}") WANT[toolbox]=1; NOTE[toolbox]="never published; required as agent-runner's base" ;;
+      *@sha256:*)         AGENT_BASE="${PINNED_BASE}" ;;
+      *) die "infra/agent/versions.env AGENT_BASE_IMAGE='${PINNED_BASE}' is not digest-pinned; CI will not build on a floating base" ;;
+    esac
+  fi
+fi
+
+for i in "${ALL_IMAGES[@]}"; do info "plan: ${i}: $(yn "${WANT[$i]:-}" BUILD skip)  (${NOTE[$i]:-inputs unchanged})"; done
+if [ -z "${WANT[toolbox]:-}${WANT[agent-runner]:-}${WANT[conformance-runner]:-}" ]; then
+  info "nothing to build"
+else
+  setup_builder
+fi
+
+if [ -n "${WANT[toolbox]:-}" ]; then build_one toolbox; fi
+if [ -n "${WANT[agent-runner]:-}" ]; then
+  if [ -z "${AGENT_BASE}" ]; then
+    AGENT_BASE="${REGISTRY}/${NAMESPACE}/toolbox@${DIGEST[toolbox]:?no toolbox digest to build agent-runner from}"
+  fi
+  info "agent-runner base: $( [ "${AGENT_BASE}" = link ] && echo "the toolbox built in this run (bake target: context)" || echo "${AGENT_BASE}")"
+  build_one agent-runner "${AGENT_BASE}"
+fi
+if [ -n "${WANT[conformance-runner]:-}" ]; then build_one conformance-runner; fi
+
+# --- summary ------------------------------------------------------------------------------
+{
+  echo "## Images (${MODE})"
+  echo
+  echo "commit \`${HEAD_SHA}\` — builder: rootless BuildKit \`${BUILDKIT_HOST}\` (mTLS)"
+  echo
+  echo "| image | verify target built | seconds | published digest (registry-returned) | note |"
+  echo "|---|---|---|---|---|"
+  for i in "${ALL_IMAGES[@]}"; do
+    d="${DIGEST[$i]:-}"
+    echo "| \`${i}\` | $(yn "${BUILT[$i]:-}" yes no) | ${SECS[$i]:--} | $(yn "${d}" "\`${REGISTRY}/${NAMESPACE}/${i}@${d}\`" -) | ${NOTE[$i]:-inputs unchanged} |"
+  done
+  echo
+  if [ "${MODE}" = publish ]; then
+    echo "Tags pushed: \`${SHA_TAG}\` (immutable) and \`${MOVING_TAG}\` (moving)."
+    echo "Nothing here writes to git. Renovate opens the digest-bump PR on its next run."
+  else
+    echo "Verify mode: nothing was pushed and no registry credential was present."
+  fi
+} >>"${SUMMARY}"
+log "done"

.gitea/workflows/scripts/install-tools.sh

@@ -39,6 +39,21 @@ install_chainsaw() {
   curl -fsSL "$url" | tar -xz -C "$bindir" chainsaw
 }
 
+# promtool — the alert-rule UNIT-TEST engine the L0 check
+# conformance/tests/l0-static/alert-rules-unit-tested runs. Installed for the l0 stage because
+# without it that check exits 3 (SKIPPED, never green) on the runner and the alert unit tests
+# would be enforced nowhere — the exact "coverage that looks present" failure it exists to end.
+# Pinned from ci/versions.env to the Prometheus version prod runs (promtool IS the rule engine);
+# the check asserts `promtool --version` equals that pin, so a drifted install FAILS, not skips.
+install_promtool() {
+  have promtool && return 0
+  log "promtool ${PROMTOOL_VERSION} (from the prometheus release tarball)"
+  local tarball="prometheus-${PROMTOOL_VERSION}.${os}-${arch}"
+  curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/${tarball}.tar.gz" \
+    | tar -xz -C "$bindir" --strip-components=1 "${tarball}/promtool"
+  chmod +x "${bindir}/promtool"
+}
+
 install_kubectl() {
   have kubectl && return 0
   log "kubectl ${KUBECTL_VERSION}"
@@ -50,10 +65,10 @@ have() { command -v "$1" >/dev/null 2>&1; }
 
 case "$stage" in
   lint)        install_yamllint ;;
-  l0)          install_yamllint; install_check_jsonschema; install_kyverno ;;  # schema (check-jsonschema) + Kyverno L0
+  l0)          install_yamllint; install_check_jsonschema; install_kyverno; install_promtool ;;  # schema (check-jsonschema) + Kyverno L0 + alert unit tests
   l1)          install_chainsaw ;;              # L1 render via conformance runner (Chainsaw)
   conformance) install_chainsaw; install_kubectl ;;   # trigger the in-cluster Job
-  all)         install_yamllint; install_check_jsonschema; install_kyverno; install_chainsaw; install_kubectl ;;
+  all)         install_yamllint; install_check_jsonschema; install_kyverno; install_promtool; install_chainsaw; install_kubectl ;;
   *) echo "unknown stage: $stage" >&2; exit 2 ;;
 esac
 log "stage '${stage}' tools ready"

.gitignore

@@ -37,6 +37,14 @@ coverage/
 infra/tofu/backend.hcl
 infra/tofu/.r2.env
 
+# Operator scratch, writable from inside the dev container (the repo is its only persistent
+# read-write mount). Holds things that must never be committed: a staged copy of the SOPS age or
+# WireGuard key, rendered VPN client profiles (infra/vpn-ingress/new-peer.sh writes here by
+# default). `infra/ansible/.gitignore` already does this for its own .cache/; this is the same
+# convention at the root. Do NOT rely on `*.key` alone to protect a staged key — that only covers
+# one filename shape, which is why this rule exists.
+.cache/
+
 # Claude Code local scratch: agent worktrees + session state. Nothing here is tracked;
 # ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
 # stage (which scans untracked files) from tripping on .claude/worktrees/.

.standup-dns/dns-20260914-pre-externaldns-scope.txt

@@ -0,0 +1,97 @@
+# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev`
+# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied.
+#
+# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers,
+# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not
+# merely left alone. That is a one-way, outward-facing operation against a live authoritative
+# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the
+# reconcile removes can be restored by hand from this file. It is the same discipline as
+# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable
+# change rather than to a migration.
+#
+# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones/<id>/dns_records, all pages) using the
+# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the
+# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the
+# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the
+# devcontainer, which has previously produced false "no records" readings — the API is the only
+# reading that is both complete and authoritative.
+#
+# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the
+# TTL and the proxy flag, not just the value.
+#
+#   <type>|<name>|<content>|<ttl>|<proxied>
+#
+# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here
+# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address.
+#
+# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry
+# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which
+# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy
+# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS
+# cannot touch them regardless of what this branch does:
+#
+#   *.keithjackson.dev            -> 136.51.115.252   legacy stack (the zone wildcard)
+#   brickbreak.keithjackson.dev   -> 20.236.242.14    unrelated, off-prem
+#   dev.brickbreak…               -> 20.236.242.14    unrelated, off-prem
+#
+# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in
+# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of
+# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does
+# NOT cover `<svc>.home.keithjackson.dev`: when a specific `<svc>.home` record is removed, that
+# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended
+# end state for `internal` services and it is also precisely why Pi-hole must already be
+# answering the LAN authoritatively before this branch merges.
+#
+# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog
+# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its
+# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed
+# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an
+# HTTPRoute that already had a published record produced
+# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile
+# interval (~60s) later, and nothing else in the zone moved.
+#
+# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its
+# own): recreate the A record with its value/ttl/proxied from this file, then recreate the
+# matching `a-<name>` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod,
+# external-dns/resource=…` string is what re-establishes ownership. An A record restored without
+# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value
+# points the registry at a source object that does not exist.
+A|*.keithjackson.dev|136.51.115.252|1|false
+A|auth.home.keithjackson.dev|192.168.2.20|1|false
+A|brickbreak.keithjackson.dev|20.236.242.14|1|false
+A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false
+A|crafty.home.keithjackson.dev|192.168.2.20|1|false
+A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false
+A|gitea.home.keithjackson.dev|192.168.2.20|1|false
+A|grafana.home.keithjackson.dev|192.168.2.20|1|false
+A|headlamp.home.keithjackson.dev|192.168.2.20|1|false
+A|home.keithjackson.dev|192.168.2.20|1|false
+A|jackett.home.keithjackson.dev|192.168.2.20|1|false
+A|keycloak.home.keithjackson.dev|192.168.2.20|1|false
+A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false
+A|pihole.home.keithjackson.dev|192.168.2.20|1|false
+A|plex.home.keithjackson.dev|192.168.2.20|1|false
+A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false
+A|radarr.home.keithjackson.dev|192.168.2.20|1|false
+A|seerr.home.keithjackson.dev|192.168.2.20|1|false
+A|semaphore.home.keithjackson.dev|192.168.2.20|1|false
+A|sonarr.home.keithjackson.dev|192.168.2.20|1|false
+A|unifi.home.keithjackson.dev|192.168.2.21|1|false
+TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false
+TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false
+TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false
+TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false
+TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false
+TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false
+TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false
+TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false
+TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false
+TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false
+TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false
+TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false
+TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false
+TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false
+TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false
+TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false
+TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false
+TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false

.standup-dns/dns-before.txt

@@ -0,0 +1,57 @@
+A|*.keithjackson.dev|136.51.115.252
+A|auth.home.keithjackson.dev|192.168.1.53
+A|brickbreak.keithjackson.dev|20.236.242.14
+A|crafty-mc.home.keithjackson.dev|192.168.1.55
+A|crafty.home.keithjackson.dev|192.168.1.53
+A|dev.brickbreak.keithjackson.dev|20.236.242.14
+A|gitea.home.keithjackson.dev|192.168.1.53
+A|grafana.home.keithjackson.dev|192.168.1.53
+A|headlamp.home.keithjackson.dev|192.168.1.53
+A|home.keithjackson.dev|192.168.1.53
+A|jackett.home.keithjackson.dev|192.168.1.53
+A|keycloak.home.keithjackson.dev|192.168.1.53
+A|nextcloud.home.keithjackson.dev|192.168.1.53
+A|pihole.home.keithjackson.dev|192.168.1.53
+A|plex.home.keithjackson.dev|192.168.1.53
+A|qbittorrent.home.keithjackson.dev|192.168.1.53
+A|radarr.home.keithjackson.dev|192.168.1.53
+A|seerr.home.keithjackson.dev|192.168.1.53
+A|semaphore.home.keithjackson.dev|192.168.1.53
+A|sonarr.home.keithjackson.dev|192.168.1.53
+A|unifi.home.keithjackson.dev|192.168.1.54
+TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
+TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
+TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
+TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
+TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
+TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
+TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
+TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
+TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
+TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
+TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
+TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
+TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
+TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
+TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
+TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
+TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
+TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
+TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
+TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
+TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
+TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
+TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
+TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp

.standup-dns/dns-post-l2.txt

@@ -0,0 +1,95 @@
+A|*.keithjackson.dev|136.51.115.252
+A|auth.home.keithjackson.dev|192.168.1.53
+A|auth.prod.home.keithjackson.dev|192.168.2.20
+A|brickbreak.keithjackson.dev|20.236.242.14
+A|conformance-dns.home.keithjackson.dev|192.168.1.56
+A|crafty-mc.home.keithjackson.dev|192.168.1.55
+A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
+A|crafty.home.keithjackson.dev|192.168.1.53
+A|crafty.prod.home.keithjackson.dev|192.168.2.20
+A|dev.brickbreak.keithjackson.dev|20.236.242.14
+A|gitea.home.keithjackson.dev|192.168.1.53
+A|gitea.prod.home.keithjackson.dev|192.168.2.20
+A|grafana.home.keithjackson.dev|192.168.1.53
+A|grafana.prod.home.keithjackson.dev|192.168.2.20
+A|headlamp.home.keithjackson.dev|192.168.1.53
+A|headlamp.prod.home.keithjackson.dev|192.168.2.20
+A|home.keithjackson.dev|192.168.1.53
+A|jackett.home.keithjackson.dev|192.168.1.53
+A|jackett.prod.home.keithjackson.dev|192.168.2.20
+A|keycloak.home.keithjackson.dev|192.168.1.53
+A|keycloak.prod.home.keithjackson.dev|192.168.2.20
+A|nextcloud.home.keithjackson.dev|192.168.1.53
+A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
+A|pihole.home.keithjackson.dev|192.168.1.53
+A|pihole.prod.home.keithjackson.dev|192.168.2.20
+A|plex.home.keithjackson.dev|192.168.1.53
+A|plex.prod.home.keithjackson.dev|192.168.2.20
+A|prod.home.keithjackson.dev|192.168.2.20
+A|qbittorrent.home.keithjackson.dev|192.168.1.53
+A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
+A|radarr.home.keithjackson.dev|192.168.1.53
+A|radarr.prod.home.keithjackson.dev|192.168.2.20
+A|seerr.home.keithjackson.dev|192.168.1.53
+A|seerr.prod.home.keithjackson.dev|192.168.2.20
+A|semaphore.home.keithjackson.dev|192.168.1.53
+A|semaphore.prod.home.keithjackson.dev|192.168.2.20
+A|sonarr.home.keithjackson.dev|192.168.1.53
+A|sonarr.prod.home.keithjackson.dev|192.168.2.20
+A|unifi.home.keithjackson.dev|192.168.1.54
+A|unifi.prod.home.keithjackson.dev|192.168.2.21
+TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|a-conformance-dns.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/chainsaw-growing-grouse/dns-probe-svc
+TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
+TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
+TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
+TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
+TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
+TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
+TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
+TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
+TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
+TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
+TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
+TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
+TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
+TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
+TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
+TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
+TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
+TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
+TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
+TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
+TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
+TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
+TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
+TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
+TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
+TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
+TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
+TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
+TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
+TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
+TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
+TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
+TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
+TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
+TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
+TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp

.standup-dns/dns-verify.txt

@@ -0,0 +1,93 @@
+A|*.keithjackson.dev|136.51.115.252
+A|auth.home.keithjackson.dev|192.168.1.53
+A|auth.prod.home.keithjackson.dev|192.168.2.20
+A|brickbreak.keithjackson.dev|20.236.242.14
+A|crafty-mc.home.keithjackson.dev|192.168.1.55
+A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
+A|crafty.home.keithjackson.dev|192.168.1.53
+A|crafty.prod.home.keithjackson.dev|192.168.2.20
+A|dev.brickbreak.keithjackson.dev|20.236.242.14
+A|gitea.home.keithjackson.dev|192.168.1.53
+A|gitea.prod.home.keithjackson.dev|192.168.2.20
+A|grafana.home.keithjackson.dev|192.168.1.53
+A|grafana.prod.home.keithjackson.dev|192.168.2.20
+A|headlamp.home.keithjackson.dev|192.168.1.53
+A|headlamp.prod.home.keithjackson.dev|192.168.2.20
+A|home.keithjackson.dev|192.168.1.53
+A|jackett.home.keithjackson.dev|192.168.1.53
+A|jackett.prod.home.keithjackson.dev|192.168.2.20
+A|keycloak.home.keithjackson.dev|192.168.1.53
+A|keycloak.prod.home.keithjackson.dev|192.168.2.20
+A|nextcloud.home.keithjackson.dev|192.168.1.53
+A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
+A|pihole.home.keithjackson.dev|192.168.1.53
+A|pihole.prod.home.keithjackson.dev|192.168.2.20
+A|plex.home.keithjackson.dev|192.168.1.53
+A|plex.prod.home.keithjackson.dev|192.168.2.20
+A|prod.home.keithjackson.dev|192.168.2.20
+A|qbittorrent.home.keithjackson.dev|192.168.1.53
+A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
+A|radarr.home.keithjackson.dev|192.168.1.53
+A|radarr.prod.home.keithjackson.dev|192.168.2.20
+A|seerr.home.keithjackson.dev|192.168.1.53
+A|seerr.prod.home.keithjackson.dev|192.168.2.20
+A|semaphore.home.keithjackson.dev|192.168.1.53
+A|semaphore.prod.home.keithjackson.dev|192.168.2.20
+A|sonarr.home.keithjackson.dev|192.168.1.53
+A|sonarr.prod.home.keithjackson.dev|192.168.2.20
+A|unifi.home.keithjackson.dev|192.168.1.54
+A|unifi.prod.home.keithjackson.dev|192.168.2.21
+TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
+TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
+TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
+TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
+TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
+TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
+TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
+TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
+TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
+TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
+TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
+TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
+TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
+TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
+TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
+TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
+TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
+TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
+TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
+TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
+TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
+TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
+TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
+TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
+TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
+TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
+TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
+TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
+TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
+TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
+TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
+TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
+TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
+TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
+TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
+TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp

CLAUDE.md

@@ -0,0 +1,361 @@
+# CLAUDE.md — how work lands in this repo
+
+Read this before doing anything durable. It is the delivery contract for AI coding
+sessions and agents. The newest entry in `decisions/` is the current PM handoff and
+takes precedence on *what* to work on; this file governs *how* it lands.
+
+**Start with §0.** It is first because it is the rule you can break before you have read
+anything else: the very first `git checkout -b` in the wrong directory has already done the
+damage.
+
+## 0. Work in your OWN workspace. Never in the shared checkout.
+
+Before you create a branch, edit a file, or run `git` anything: **make yourself a git
+worktree outside the repository root, and do all of your work there.** One workspace per
+agent run, no exceptions.
+
+    # from the shared checkout, ONCE, at the start of your run
+    git fetch origin main
+    git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+    cd /path/outside/the/repo/wt-<slug>        # everything after this happens HERE
+
+`/path/outside/the/repo` means outside this repository's tree — your session scratchpad,
+or a sibling like `/var/home/keith/source/wt-<slug>`. Branch from **`origin/main`**, never
+from `main` and never from `HEAD`; see "stale base" below.
+
+**The shared checkout is `/var/home/keith/source/home-network-k3s-impl`** — the operator's
+working copy, the one you are probably started in. In it, do **not**:
+
+- **`git checkout` / `git switch` / `git checkout -b`.** Its HEAD is shared state; moving
+  it moves the tree out from under whoever else is using it.
+- **edit, create or delete a file.** Including "just one doc", including a file you intend
+  to move later.
+- **`git add` / `git commit` / `git stash` / `git restore`.** `git add -A` there stages
+  whatever another agent left lying around; `git stash` *removes* their uncommitted work
+  into a place they will not look.
+- **`git worktree remove` anything you did not create.**
+
+Read-only commands there are fine, and the one write you do need is creating your own
+worktree.
+
+### Why — measured in the shared checkout, 2026-10-05
+
+| What was measured | Value |
+|---|---|
+| `checkout: moving` entries in its reflog | **107** — the tree has been yanked between branches 107 times |
+| the branch it was sitting on | `feat/pm-agent-issue-to-pr` — **not `main`** |
+| uncommitted trees left in it by other runs | **2** (`infra/agent/`, `platform/pm-agent/`), mtimes five minutes apart the same evening |
+| its local `main` vs `origin/main` | **two commits behind** (`4386d14` vs `22c94ef`) |
+| registered worktrees | **42**, of which **31** are on branches already merged to `main` |
+| gate scripts carrying an explicit `.claude/` exclusion | **9** |
+
+Each row is a distinct way the shared checkout damages work:
+
+- **Dirty tree, foreign files.** Two other runs' uncommitted directories were sitting in
+  it. An agent that edits there and runs `git add -A` ships someone else's half-finished
+  work inside its own PR, and the PR's author has no idea what those files are. This is not
+  hypothetical and it is not historical: **eleven minutes after those two directories were
+  measured, while this very section was being written, another run committed them** —
+  `5334abe feat(pm-agent): ...`, 1,000+ lines across `infra/agent/`, `clusters/`, `catalog/`
+  and `apps/` — onto the shared checkout's checked-out branch, moving its HEAD under anyone
+  reading it. Had this rule's author been working in that tree rather than in a worktree,
+  that commit and this one would have collided.
+- **Stale base.** Its `main` was two commits behind `origin/main`, so
+  `git checkout -b x main` there branches off a tree that is already not what CI will merge
+  into — the gate you run locally proves a base nobody else has. Worse, it was not even *on*
+  `main`: a bare `git checkout -b x` would have branched off `feat/pm-agent-issue-to-pr` and
+  silently inherited that work into your PR. `git fetch` first, and name `origin/main`
+  explicitly as the start point.
+- **Shared HEAD.** 107 branch switches in one tree is 107 chances to switch it while
+  another agent is mid-edit. The loser's changes are either carried onto the wrong branch
+  or lost to a checkout conflict — and the symptom surfaces in *their* session, not yours.
+
+### Put the workspace OUTSIDE the repo root
+
+A worktree *inside* the tree (`.claude/worktrees/agent-*/` — which is where Claude Code's
+own default puts it) is a full second copy of the repo that every tree-walking check then
+finds. Measured 2026-09-12: `find`-based discovery reported **the same manifest under test
+once per worktree, up to 18 times**, and the L0 check comparing that listing against the
+tree failed on paths that are not authored surface at all
+(`platform/monitoring/tests/run.sh:115-123`). Nine gate scripts now carry an explicit
+`.claude/` prune for this reason — which means the hazard is *worked around*, not fixed,
+and the next check that walks the tree will hit it again. Keep the copy out of the tree and
+none of that applies.
+
+### Clean up — and expect not to be able to
+
+31 of the 42 registered worktrees are on branches already merged to `main`. A registered
+worktree **pins its branch**: while it exists, that branch cannot be deleted or checked out
+anywhere else. So a left-behind workspace is a lock on a branch name, and the cost lands on
+whoever next needs it.
+
+Remove yours when you are done:
+
+    git worktree remove /path/outside/the/repo/wt-<slug>
+
+Measured 2026-10-05: on a worktree **you created yourself, in the session scratchpad**, that
+command **succeeds** — rc 0, no prompt, with or without a branch checked out. The "worktree
+removal is sandbox-blocked" note in circulation is about the `.claude/worktrees/agent-*` ones
+belonging to *other* sessions, some of which are additionally `locked` on git's own terms. So:
+remove yours, and only yours. If removal is refused, say so in your handoff and leave the
+worktree registered — do **not** `rm -rf` the directory, which leaves a registered worktree
+pointing at nothing and still holding the branch.
+
+If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the
+branch name. Push a `rework/<pr-number>` branch to the PR's refspec instead.
+
+### If you are already standing in the shared checkout with edits
+
+Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as
+above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly
+as you found them.
+
+### Gating from a worktree
+
+Three known breakages, all with workarounds, all required every run: see §3, "Running the
+gate from a worktree".
+
+### If you are writing a brief for another agent
+
+The brief must name **its own explicit branch** and tell it to create **its own worktree**
+from `origin/main`. "Work on the current branch" and "continue in the checkout" are how
+branch sprawl and cross-contamination got here: an agent given no workspace uses the one it
+is started in, which is the shared one. Also carry §3's three gate workarounds into the
+brief — an agent that cannot gate will report rc 3 as green.
+
+## 1. Never push to `main`. Ever.
+
+Every durable change reaches `main` exactly one way:
+
+    branch  ->  push the branch  ->  open a PR  ->  CI gate green  ->  human approval  ->  merge
+
+There is no other path. Specifically:
+
+- **Do not `git push origin main`.**
+- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
+- **Do not merge your own PR.** See the approval model below — and either way, the merge is
+  someone else's action, never yours.
+
+### Who approves
+
+**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
+branch requires no human approval for one. That is a deliberate transfer of authority, on the
+operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
+good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
+add a human.
+
+**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
+admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
+approving review from someone who is not its author. The reason is narrow and structural — **a
+change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
+the weakened check. Everything else merges on green.
+
+The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
+sensitive path, so the policy cannot be widened or narrowed without a human.
+
+That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
+missing credential is a lock-out, here a skip would be a bypass.
+
+**The administrator override (2026-10-01).** The rule above was unsatisfiable for the one human
+who runs this repo: Gitea refuses to let an author approve their own PR, and `jakman85` is the
+only collaborator — so a sensitive-path PR the operator opened could never be merged, *including
+the PR that would fix that.* An administrator may therefore waive the review by posting this on
+the PR:
+
+    /approve-override <reason>
+
+**Post it as a PR *review* of type "Comment", not in the plain comment box.** CI's
+`SETTINGS_API_TOKEN` is minted `[read:repository, read:admin]`, and `/issues/{n}/comments` sits
+behind Gitea's *issue* scope — so an override typed into the comment box is **invisible to CI**,
+and the stage reports it as a missing approval rather than as a permission problem (measured on
+PR #38, 2026-10-02). `/pulls/{n}/reviews` is readable, so a review body works today. Both sources
+are accepted and the gate names which it used; the comment box starts working once
+`infra/gitea/set-ci-token.sh` is re-run to pick up `read:issue`.
+
+The directive must **start a line**, the **reason is mandatory**, the poster's `admin`/`owner`
+permission is re-checked **live** when the gate runs, and it must be **newer than the
+head commit** — push again and you must override again. The overrider, the timestamp and the
+reason are printed in the gate log and the job summary. For an administrator, "a change to the
+gate cannot be gated by the gate" is therefore **advisory, not structural**; what is kept is an
+attributed record of each waiver. Note that the agent account `gitea-admin` also holds `owner`
+here, so an agent *can* clear this on its own PR — **do not**: posting that directive to unblock
+your own agent work is exactly the self-approval the stage exists to prevent. Get the operator
+to post it, or move the change out of the sensitive paths.
+
+**`required_approvals` on `main` is `0`**, not 1 — the gate is the approver. That drops the
+approval *count* only: `block_admin_merge_override` and `enable_status_check` both remain `true`,
+so **a red gate is unmergeable by everyone, administrators included.**
+- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
+  anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
+
+> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
+> That wording is **stale** — it predates the PR flow. It describes what Flux does
+> (main is what the cluster follows), not what you are permitted to do. Honour this
+> file, not that line. Fixing those runbooks is tracked work.
+
+## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
+
+Verified against the live Gitea on 2026-09-29:
+
+| Expected | Actual |
+|---|---|
+| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
+| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
+| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
+| Gate cannot be overridden, by anyone | ✅ `block_admin_merge_override: true` |
+| Human approval required before merge | ⬜ **No — `required_approvals: 0` since 2026-10-01.** The gate is the approver; sensitive paths are gated by `ci/approval-policy.sh` instead. See §1 |
+
+The first three were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
+true. They are true now because each was measured, and the push rejection was provoked rather
+than assumed. **Re-measure before relying on any of them** — see §2b.
+
+The fourth row deliberately changed on 2026-10-01. `required_approvals: 1` was a **lock-out**, not
+a protection: with a single collaborator and Gitea refusing author self-approval, nothing could
+satisfy it (measured on PR #35 — green gate, zero reviews, unmergeable). It also contradicted the
+model already written into `ci/approval-policy.sh` and `ci/gate.sh`. Rationale and the live-change
+script: `infra/gitea/allow-admin-self-merge.sh`.
+
+Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
+CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
+
+**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
+the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
+strength of a check you have not seen report — and note that a **skipped** Gitea job
+reports as `success`, so a green tick is not by itself evidence that anything ran.
+
+## 2b. Never infer platform state from this repository
+
+**This repo is not evidence about the live platform.** Neither is the handoff log
+(`decisions/`, and the frozen `DECISIONS-LOG.md` archive). Both describe what someone
+believed at the time of writing, and both have been confidently wrong for a month at a
+stretch.
+
+On 2026-09-29 the log stated the merge gate was a required status check with a runner
+online. The API said `branch_protections: []` and **zero** runners. Everything downstream
+of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
+was wrong, and nothing in the tree could have revealed it, because the tree was the source
+of the error.
+
+**So, before asserting that any gate, protection, runner, secret, or workload is in a
+given state: query the thing itself.**
+
+| Claim about... | Check |
+|---|---|
+| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
+| a runner exists | `GET /api/v1/admin/actions/runners` |
+| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
+| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
+| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
+
+Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
+"unverified" is a legitimate report, and a far better one than a confident guess that
+matches the docs.
+
+**A green tick is not proof anything ran.** Read what actually executed.
+
+## 3. Gate before you push
+
+    ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+
+Exit codes are a contract — do not flatten them:
+
+| rc | Meaning |
+|---|---|
+| 0 | PASS |
+| 1 | BLOCKED — something failed |
+| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
+
+`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
+beside this one; that single skip is expected and tolerated — and from a worktree it is
+avoidable outright (§3's worktree list, item 3). `gitea-settings` is the other expected
+skip: it needs `SETTINGS_API_TOKEN`, which exists only in CI, so **a local run cannot
+return 0** — the honest best local result is rc 3 with `gitea-settings` as the only stage
+that did not run. Any *further* skip is a real gap in your run. Report the rc and name the
+skipped stages; never round rc 3 up.
+
+**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
+so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
+
+### Running the gate from a worktree
+
+This is the **normal** path, not an edge case — §0 requires you to be in a worktree, so
+every gate run you do hits all of these. Three known breakages, all with
+workarounds, all of them yours to apply every run — put them in any agent brief:
+
+1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
+   and exists only in the main checkout. Do **not** copy it (it is a credential).
+   Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
+   normally.
+2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
+   gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
+   the `ENGINE` hook — a shim that intercepts `run` and re-execs
+   `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
+   `tracked_files()` is this bug, not a broken tree.
+3. **`spec-immutability` skips** — the skip called tolerated above is also *avoidable*. `dev.sh`
+   derives the sibling spec path from its own `REPO_ROOT`, which in a worktree is your
+   workspace, so `../home-network-k3s` does not resolve and it adds no mount. Mount the spec
+   repo read-only in the same `ENGINE` shim and pass `-e SPEC_REPO=/home-network-k3s`: with
+   both mounts the stage reports `pass`, not SKIPPED, and R-TEST-06 is actually covered
+   (measured — `decisions/2026-10-01-render-selftest-runs-in-gate.md`).
+
+## 4. Branch naming
+
+Use `feat/`, `fix/`, `docs/`, `chore/`. **One branch, one workspace, one PR** — if you
+need a second branch, make a second worktree (§0); do not re-point an existing one. Note that `.gitea/workflows/gate.yml`'s **push**
+trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
+The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
+
+## 5. Cluster access
+
+`kubectl` does not work on the operator workstation; the toolchain lives only in the
+pinned container. Always name the context explicitly:
+
+    ./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+`dev.sh` merges both clusters and picks a current-context **per invocation**; that
+default has already changed once. Never rely on it. Do not run two `dev.sh`
+invocations concurrently.
+
+**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
+
+## 6. Secrets
+
+- **Never `sops -d` a file into a terminal, a log, or a transcript.**
+- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
+  list key names or object names only. A password has been leaked into a transcript
+  this way before.
+- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
+  the single creation-rules file; do not add a second one in a subtree.
+- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
+  flow. Do not design around committing one — read the expiry from the issuing page and
+  treat a burnt token as free to retry.
+
+## 7. Before you finish — write a handoff entry, in its OWN file
+
+Add **one new file**:
+
+    decisions/YYYY-MM-DD-<slug>.md          <slug> = your branch's last segment
+
+with frontmatter (`date`, `branch`, `title`) and exactly these three sections:
+
+    ## What changed
+    ## What I verified
+    ## What I did NOT verify
+
+State gaps plainly — an honest "UNVERIFIED" is worth more than an optimistic "green",
+and this repo has already been burned by the difference. All three headings are
+**enforced** by `ci/decisions-log.sh`, because the third is the one that gets dropped.
+Template and full rationale: `decisions/README.md`.
+
+**Do NOT append to `DECISIONS-LOG.md`.** It is a frozen archive (pre-2026-09-29), its
+SHA-256 is pinned in `decisions/ARCHIVE.sha256`, and the merge gate rejects any change
+to it. Appending to its tail is what every agent used to do, and because every branch
+appended to the *same last hunk*, no two concurrent branches could auto-merge. Measured
+on 2026-09-29: **four of the five open PRs were `mergeable: false`, with
+`DECISIONS-LOG.md` the only conflicting path in all four** — none of those changes had
+anything to do with each other. Separate files have nothing to merge, which is the
+entire fix. Note that the damage from getting this wrong lands on *other people's*
+branches, later, which is why it is a gate stage and not a request.
+
+Many older handoff entries and runbooks say "append to `DECISIONS-LOG.md`". That
+wording is **stale**, in the same way §1's `git push origin main` wording is stale.

README.md

@@ -11,6 +11,9 @@ specification** at [`../home-network-k3s`](../home-network-k3s).
 - **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a
   manual demo (spec `docs/14`, `agents/definition-of-done.md`).
 - **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`).
+- **No work in a shared checkout.** Every agent run gets its own git worktree, outside the repo
+  root, branched from `origin/main`. This checkout is shared mutable state — see
+  [`CLAUDE.md` §0](CLAUDE.md).
 - **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`).
 - **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`).
 
@@ -21,7 +24,8 @@ infra/          node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible,
 bootstrap/      one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
 platform/       platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
                 longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
-                vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
+                vpn-egress, vpn-ingress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
+                (vpn-egress = pods OUT via PIA; vpn-ingress = remote clients IN via WireGuard)
 platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
                 (ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
                 Split out because CRs and the CRDs they use cannot apply in one pass.
@@ -38,28 +42,76 @@ ci/             the gate as scripts (lint · schema · Kyverno L0 · L1 render
 
 ## Status
 
-Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
+Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in
-open** — `clusters/prod` is a design-only overlay.
+progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the
+apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration:
+dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
+hardware rather than on Proxmox VMs.
+
+**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring +
+alerts live*:
+
+| criterion | state |
+|---|---|
+| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
+| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
+| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only |
+
+**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 —
+the first requirement in this repo to reach that state. Both of its verifiers are recorded:
+`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`)
+and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn`
+class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy**
+(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
+*different* PersistentVolume — so an in-cluster survival cannot pass as a restore.
+
+**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned
+dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
+exercise rather than something to run in passing.
+
+Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a
+bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
 
 | Phase | Delivered |
 |---|---|
 | 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
-| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
+| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
-| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
+| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the LAN resolver**, answering an explicit list of `${BASE_DOMAIN}` names (LAN-only apps, gitea/nextcloud/keycloak/auth, the L4 VIPs — everything else resolves publicly to Cloudflare, since 2026-11-09) on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
 | 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
-| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
+| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
 | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
 | 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
+| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
 
-**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
+**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
-(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
+2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with
-the full report is `conformance/_artifacts/coverage/coverage.md`.
+`./conformance/run.sh --coverage-only`; the full report is
+`conformance/_artifacts/coverage/coverage.md`.
+
+Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and
+*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current
+SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the
+second.
 
 The running narrative — decisions, root causes, incidents, follow-ups — is
 [`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.
 
 ## Working on this repo
 
+**First, get out of this checkout.** It is the operator's working copy and several agents
+share it; concurrent runs here have overwritten each other's branches and left each other's
+uncommitted files behind. One workspace per change:
+
+```bash
+git fetch origin main
+git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+cd /path/outside/the/repo/wt-<slug>
+```
+
+Outside the repo root specifically — a worktree under `.claude/worktrees/` is a second full
+copy that tree-walking gate checks then count twice. The rule, the measurements behind it, and
+the three gate workarounds a worktree needs are in [`CLAUDE.md` §0](CLAUDE.md) and §3.
+
 Everything the gate needs is baked into the pinned dev container — **do not install the
 toolchain on the host**:
 

apps/vms/unifi/30-virtualmachine.yaml

@@ -41,10 +41,41 @@ spec:
       # disk KubeVirt cannot migrate and eviction would hard-stop the guest.
       evictionStrategy: LiveMigrate
       domain:
+        # =====================================================================================
+        # FIRMWARE UUID == THE CLOUD-INIT INSTANCE-ID. THIS IS THE RE-RUN LEVER. READ BEFORE EDIT.
+        # =====================================================================================
+        # MEASURED on prod 2026-09-14, not assumed:
+        #   * the NoCloud seed KubeVirt builds for this VM carries
+        #       {"instance-id":"de5c3338-6d32-4956-89d1-12c8272b2c07","local-hostname":"unifi"}
+        #     (read out of noCloud.iso inside virt-launcher-unifi-gp594), and that value was
+        #     EXACTLY `spec.template.spec.domain.firmware.uuid`, which KubeVirt's mutating webhook
+        #     had stamped onto the VM OBJECT — so it is PERSISTENT, not per-VMI (the VMI's own uid
+        #     was b532fd7c-…, a different value). It is not a hash of the VM name either: the four
+        #     standard uuid5 namespaces over "unifi" / "unifi.unifi" all produce something else.
+        #   * cloud-init's NoCloud datasource decides "is this the same instance?" by comparing the
+        #     cached instance-id against the guest's DMI product_uuid — which IS this firmware UUID.
+        #
+        # CONSEQUENCE, and it is the whole reason the previous unattended attempt could never have
+        # worked: EDITING `userData` BELOW AND RESTARTING THE VM CHANGES NOTHING. cloud-init sees an
+        # unchanged instance-id, restores its CACHED datasource, never re-reads the seed, and skips
+        # every per-instance module (write_files, runcmd). The new payload is silently ignored.
+        #
+        # So the UUID is pinned HERE, in git, and BUMPING IT IS THE SUPPORTED WAY to make the guest
+        # process a new cloud-init payload: change the trailing hex, commit, restart the VM. Pinning
+        # it also makes a rebuild reproducible — a fresh apply would otherwise take a random
+        # webhook-generated UUID and give the guest a different identity every time.
+        #
+        # The value below is DELIBERATELY NOT the de5c3338-…c07 the live guest booted with: that
+        # difference is what makes the next boot a NEW cloud-init instance and installs the
+        # `install-uos` helper + its systemd unit. `serial` is pinned to the value the webhook
+        # already stamped, so this change does not also churn the guest's DMI serial.
+        firmware:
+          uuid: "de5c3338-6d32-4956-89d1-12c8272b2c11"
+          serial: "24bb12ac-d311-45f6-a75b-90a9e1b63e18"
         # FLAG (verify-at-bringup): Ubuntu 24.04 amd64 cloud images are hybrid BIOS+UEFI bootable,
         # so KubeVirt's default SeaBIOS SHOULD boot cleanly (unlike the Fedora containerdisk, which
-        # is UEFI-only and needed an explicit efi block). If the guest drops to emergency/no-boot,
+        # is UEFI-only and needed an explicit efi block). RESOLVED at bring-up: boots as shipped;
-        # add:  firmware: { bootloader: { efi: { secureBoot: false } } }
+        # the guest reports Ubuntu 24.04.4 LTS.
         cpu:
           cores: 2
         memory:
@@ -58,9 +89,28 @@ spec:
             - name: cloudinit
               disk:
                 bus: virtio
+            # The operator's installer URL (+ optional SSH key), delivered as a tiny read-only
+            # iso9660 disk — see the `uosdata` volume near the bottom of this file.
+            - name: uosdata
+              disk:
+                bus: virtio
           interfaces:
+            # MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
+            # KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
+            # /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
+            # that file while the NoCloud instance-id is unchanged — and the instance-id here is
+            # domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
+            # handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
+            # administratively DOWN with no address and no DHCP attempt.
+            # That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
+            # device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
+            # (because the per-instance modules were skipped), yet UOS is installed and
+            # qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
+            # Pinning the MAC to the value the current VMI already holds makes the guest's netplan
+            # match permanently, so this cannot recur on the next recreation.
             - name: default
               masquerade: {}
+              macAddress: a6:d7:c0:b3:72:fa
               # Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
               # these and (b) surfaces them as containerPorts on the virt-launcher pod so the
               # LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
@@ -83,41 +133,112 @@ spec:
         - name: rootdisk
           dataVolume:
             name: unifi-rootdisk
+        # =====================================================================================
+        # THE OPERATOR'S ONE FIELD — SOPS Secret `unifi-uos-installer`, delivered as a DISK.
+        # =====================================================================================
+        # KubeVirt exposes a Secret as a small read-only iso9660 disk whose volume label we choose;
+        # each Secret key becomes a file on it. The guest mounts it by label (LABEL=UOSDATA) and
+        # reads `installer-url` plus an optional `console-password`. The Secret ships ENCRYPTED with
+        # REPLACE-ME placeholders, and the guest-side script treats anything that is not an https
+        # URL as "not provided yet" and exits 0 — so the VM boots and stays healthy until the
+        # operator fills it in. See unifi-uos-installer.sops.yaml.example next to this file.
+        #
+        # `console-password` exists because THERE IS CURRENTLY NO WAY INTO THIS GUEST. The stock
+        # Ubuntu cloud image ships a locked `ubuntu` account, this cloud-init never installed a key,
+        # and SSH could not help anyway: the masquerade interface above forwards ONLY the declared
+        # ports, and 22 is not one of them — probed from inside virt-launcher on 2026-09-14, the
+        # guest answers RST on 22 exactly as it does on the UniFi ports. The serial console is the
+        # only door, and a console needs a password. Set it and `virtctl -n unifi console unifi`
+        # works; `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` already gives
+        # read-only console output with no virtctl at all.
+        #
+        # WHY A SEPARATE DISK AND NOT `cloudInitNoCloud.secretRef` (which would also lift the
+        # 2048-byte inline cap): secretRef moves the ENTIRE cloud-config into ciphertext. Two
+        # concrete costs. (a) The shell below stops being reviewable in a diff and falls outside the
+        # L0 dollar-collapse guard, which can only read plaintext. (b) Flux runs its postBuild
+        # envsubst over the DECRYPTED resources, so the escape contract in
+        # clusters/base/apps-vms.yaml would still govern text nobody can see in git — the exact
+        # shape of the 2026-09-13 nextcloud-db incident. Keeping the script in plaintext and putting
+        # ONLY the operator's two values behind SOPS keeps both properties.
+        #
+        # A CHANGED SECRET REACHES THE GUEST ON THE NEXT VMI START with NO firmware-UUID bump: the
+        # disk is rebuilt from the Secret every time the VMI starts, and the systemd unit below
+        # re-reads it on EVERY boot instead of relying on cloud-init (which will not re-run — see
+        # the firmware block above). The UUID bump is a ONE-TIME step, to get that unit installed.
+        - name: uosdata
+          secret:
+            secretName: unifi-uos-installer
+            volumeLabel: UOSDATA
         - name: cloudinit
           cloudInitNoCloud:
-            # INLINE (kept < KubeVirt's 2048-byte userData limit — so NO SSH key here; add one back via
+            # INLINE, and it MUST stay <= 2048 bytes. That ceiling was MEASURED against the
-            # userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
+            # DEPLOYED KubeVirt v1.9.0 rather than quoted: a server-side dry-run apply of an
-            # signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
+            # otherwise-identical VM accepted 2048 bytes of userData and rejected 2049 with
-            # Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
+            # "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for
-            # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
+            # larger data." The payload below is ~1.5 KB; keep additions small.
-            # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
+            #
-            # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
+            # WHAT IT DOES: installs qemu-guest-agent (the KubeVirt AgentConnected signal, which is
-            # as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
+            # also the proof cloud-init ran) and preps Podman for UOS, then writes `install-uos`
-            # by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
+            # plus a systemd unit that runs it on EVERY boot. The unit — not cloud-init — is what
-            # an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
+            # makes a later URL change take effect, because cloud-init will not re-run per-instance
-            # and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
+            # modules while the instance-id is unchanged.
-            # flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
+            #
-            # text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
+            # `install-uos` IS IDEMPOTENT AND WILL NOT REINSTALL OVER A WORKING CONTROLLER. It bails
-            # wording in clusters/base/apps-vms.yaml overstated it.
+            # out the moment systemd knows a `uosserver.service` — which the live guest ALREADY has:
+            # the serial console log for the current boot shows both `uosserver.service` and
+            # `uosserver-updater.service` starting, and the root filesystem is 6.3 GB of 40 GB used.
+            # UOS is installed here; see the README for what is actually wrong (the guest lost its
+            # IP). A stamp file at /var/lib/uos-installed is the second, weaker guard for the case
+            # where the unit name changes in some future UOS release.
+            #
+            # ENVSUBST: apps-vms declares postBuild.substituteFrom (for ${UNIFI_L4_VIP} in
+            # 50-service-l4.yaml), so Flux's envsubst DOES run over this text. ${BASE_DOMAIN} below
+            # is a deliberate substitution. Everything else is bare `$name` or `$(cmd)`, both of
+            # which envsubst leaves untouched — verified against the live VMI, whose rendered
+            # userData still carries its shell variables verbatim. There is deliberately NO
+            # brace-form shell expansion left in this payload, so nothing here needs escaping; if
+            # you add one, escape it per the contract in clusters/base/apps-vms.yaml, and NEVER
+            # write an indexed expansion.
             userData: |-
               #cloud-config
               hostname: unifi
-              fqdn: unifi.home.keithjackson.dev
+              fqdn: unifi.${BASE_DOMAIN}
               package_update: true
               package_upgrade: false
-              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
+              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl]
               write_files:
                 - path: /usr/local/sbin/install-uos
                   permissions: '0755'
                   content: |
-                    #!/usr/bin/env bash
+                    #!/bin/bash
                     set -euo pipefail
-                    if [ -z "$${UOS_URL:-}" ]; then
+                    d=/run/uos; mkdir -p $d
-                      UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
+                    mountpoint -q $d || mount -o ro LABEL=UOSDATA $d
-                    fi
+                    p=$(tr -d '\r\n' < $d/console-password || true)
-                    [ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
+                    case "$p" in ""|REPLACE-ME) ;; *) echo "ubuntu:$p" | chpasswd ;; esac
-                    t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
+                    systemctl cat uosserver.service >/dev/null 2>&1 && exit 0
+                    s=/var/lib/uos-installed
+                    [ -e $s ] && exit 0
+                    u=$(tr -d '\r\n' < $d/installer-url || true)
+                    case "$u" in https://*) ;; *) echo "UOS installer-url not set yet"; exit 0;; esac
+                    t=$(mktemp)
+                    curl -fL --retry 5 --retry-delay 10 -o $t "$u"
+                    chmod +x $t
+                    yes y | $t install || yes y | $t
+                    rm -f $t; date -Is > $s
+                - path: /etc/systemd/system/uos-install.service
+                  content: |
+                    [Unit]
+                    Description=Install UniFi OS Server
+                    Wants=network-online.target
+                    After=network-online.target
+                    [Service]
+                    Type=oneshot
+                    ExecStart=/usr/local/sbin/install-uos
+                    StandardOutput=journal+console
+                    [Install]
+                    WantedBy=multi-user.target
               runcmd:
-                - [ systemctl, enable, --now, qemu-guest-agent ]
+                - [systemctl, enable, --now, qemu-guest-agent]
-                - [ systemctl, enable, --now, podman.socket ]
+                - [systemctl, enable, --now, podman.socket]
-                - [ bash, -lc, "/usr/local/sbin/install-uos 2>&1 | tee /var/log/uos-install.log > /dev/console" ]
+                - [systemctl, enable, --now, uos-install.service]

apps/vms/unifi/50-service-l4.yaml

@@ -21,8 +21,61 @@
 # k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
 #
 # selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
-# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
+# into the guest.
-# VIP pinned to the node running the VM (also what L2 announcement does).
+#
+# -------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
+# -------------------------------------------------------------------------------------------
+# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
+# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
+# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
+#
+# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
+#    per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
+#    matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
+#    MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
+#    `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
+#    announcing node that holds no backend has nowhere to send the packet and answers ICMP port
+#    unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
+#    (the leases were 8 days old); the operator's controller UI and device inform path were down.
+#    It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
+#    deleting the Lease forces a re-election — and that re-election is random, so it can pick a
+#    backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
+#    reference implementation this file now follows.
+#
+# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
+#    interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
+#    SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
+#    controller has always seen the pod-side masquerade gateway as the source, never the LAN
+#    client. There is a SECOND translation below the Service, and it already destroyed the very
+#    thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
+#    it is giving up an address the guest could not see in the first place.
+#
+#    Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
+#    MAC (and the device reports its own IP inside the inform payload), not the packet source,
+#    and guest-portal authorization runs on the gateway appliance, not on this controller.
+#
+# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
+# is what makes this survive a reschedule and a rebuild.
+#
+# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
+# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
+# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
+# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
+# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
+# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
+# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
+# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
+# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
+#
+# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
+# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
+# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
+# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
+# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
+# fix would be a NEW restriction that could silently break a remote site's device informing over
+# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
+# change, made deliberately and tested against the device fleet.
 #
 # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
 # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
@@ -46,10 +99,15 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
     # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
     # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
-    external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
+    external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
+    # DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
+    # gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
+    # orange-clouded A for these L4 ports would be broken. Required by the L0 check
+    # internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
+    external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:
@@ -72,7 +130,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:

apps/vms/unifi/README.md

@@ -1,18 +1,48 @@
 # UniFi Network controller (UniFi OS Server) — VM onboarding
 
-> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
+> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
-> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
+> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
-> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
+> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
-> or install cannot stall the app tier.
+> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
+> cannot stall the app tier.
 >
-> **Status (verified 2026-08-26):**
+> **Status (re-verified against live prod 2026-09-14):**
-> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
+> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
-> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
+> - CDI import — **Succeeded** (100%).
-> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
+> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
-> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
+>   `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
-> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
+> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
->   (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
+> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
->   The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
+> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
+>   `:8080` refuse connections.
+
+## What is actually wrong (2026-09-14)
+
+The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
+That was wrong, and acting on it would have meant reinstalling over a working controller.
+
+Evidence, all read-only, from the live cluster:
+
+| Claim | How it was checked | Result |
+|---|---|---|
+| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
+| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
+| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
+| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
+| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
+
+So the controller is installed and its units start; nothing downstream of the guest's own network
+stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
+~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
+UOS taking over host networking, but **that has not been proven and cannot be, from outside the
+guest.** Diagnosing it needs a shell, which is the second problem below.
+
+**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
+no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
+nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
+ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
+guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
+new Secret is for.
 
 ## What this is
 
@@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
 | `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
 | `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
 | `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
-| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
+| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
 | `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
-| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
+| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
+| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
+| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
 | `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
 
-The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
+The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
-serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
+`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
-this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
+version. Browser access is via the L4 VIP. Re-add it if/when that lands.
 
-### Cloud-init delivery — two constraints learned the hard way
+### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)
 
-1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
+1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
-   prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
+   of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
-   shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
+   userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
-   Secret deliberately.
+   payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
-2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
+   is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
-   `$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
+   validate as an unknown field.
-   therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
+2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
+   nothing.** The NoCloud seed's `instance-id` is exactly
+   `spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
+   (so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
+   product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
+   `runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
+   effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
+   to force a re-run. See the long comment in `30-virtualmachine.yaml`.
+3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
+   been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
+   envsubst therefore runs over the cloud-init payload, and the escape contract in
+   `clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
+   against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
+   indexed expansion is a hard error that fails the whole Kustomization.
 
 ### Node prerequisites this VM forced onto the cluster
 
@@ -56,15 +101,31 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 - **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
   3-replica class made the import fail with insufficient storage.
 
-## UOS install method (researched)
+## UOS install method (researched 2026-09-14)
 
-- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
+- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
-  2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
+  Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
-- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
+  (the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
-  `ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
+  Podman must be at `/usr/bin/podman`.
-  **`https://<host>:11443`**.
+- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
-- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
+  in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
-  independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
+- **Is the URL a short-lived signed link?** *Probably not.* The links look like
+  `https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
+  **no signature query string**, and public community installers hard-code them. It is
+  account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
+  account-derived material out of plaintext config — but do **not** build logic around expiry.
+- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
+  Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
+  ecosystem actually does is pipe confirmations in: the widely used community installer
+  (`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
+  either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
+  `yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
+  fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
+  (no URL has been supplied, and UOS is already installed here), so treat it as "established by
+  convergent third-party practice", not as verified-by-us.
+- The previously shipped helper had two defects worth recording: it resolved the URL from
+  `fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
+  `unifi-os-server`, and it invoked the binary with **no `install` subcommand**.
 
 ### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
 
@@ -78,76 +139,143 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 | 3478 | UDP | **STUN** (device ⇄ controller) |
 | 10001 | UDP | **L2 device discovery** |
 
+**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
+the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.
+
 ## L4 exposure design (the hard part)
 
 The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
 or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
 
 - **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
-  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
+  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
-  The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
+  (`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
-  flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
+  `default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
+  the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
+  existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
+  / `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
+  decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
 - `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
-  declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
+  declared ports into the guest.
-- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
+- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
-  — via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
+  used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
-  Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
+  Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
-  reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
+  leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
-  masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
+  and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
-  explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
+  it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
-- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
+  why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
-  LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
+- **Device-adoption story:** point devices' **inform host** at the VIP
-  `minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
+  (`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
+  `set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
+  subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
+  broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
+  zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
+  on other VLANs must route to the VIP and be inform-pointed anyway.
+- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
+  per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.
 
-**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
+**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
-`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
+the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
-(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
+`apps/wave-0/pihole.yaml` (LAN split-horizon, an explicit name list since 2026-11-09) point
+`unifi.${BASE_DOMAIN}` at `${UNIFI_L4_VIP}` rather than `${GATEWAY_VIP}`. `${BASE_DOMAIN}` on prod is the
+apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
+flip, so this hostname is current, not a dev leftover.
+
+## The operator's one field — installing UOS without a shell
+
+`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
+the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
+`uos-install.service` reads it **on every boot**:
+
+```bash
+sops apps/vms/unifi/unifi-uos-installer.sops.yaml    # fill installer-url and/or console-password
+git commit && git push                               # merge to the branch prod's Flux tracks
+kubectl -n unifi delete vmi unifi                    # `running: true` recreates it; no virtctl needed
+kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
+  -c guest-console-log                               # watch the boot + the install, read-only
+```
+
+Design notes that matter if you change this:
+
+- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
+  cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
+  L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
+  resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
+  operator values in SOPS.
+- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
+  the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
+  firmware-UUID bump.
+- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
+  a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
+  guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
+  controller on next boot".
+- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
+  `apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
+  `clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
+  guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.
 
 ## Open questions / risks
 
-1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
+1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
-   account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
+   requires a shell, which requires `console-password` above plus one restart. First things to run
-   **interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
+   on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
-   the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
+   `ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
-   **Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
+   own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
-   invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
+   capture the state before rebooting a second time if you can.
-2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
+2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
+   install-method section: no official silent flag exists, `yes y | <installer> install` is the
+   convergent community practice, and it has not been executed on our guest. Treat the first real
+   install as something to watch on the console log, not as a fire-and-forget.
+3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
    import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
    Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
    fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
-3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
+4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
-4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
+5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
-   `192.168.1.53–.60` pool (`.53` is the shared Gateway).
+   prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
-5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
+   (`.23`) or the node addresses (`.10/.11/.12`).
+6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
    migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
-6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
+7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
    installer and for optional remote access.
-7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
+8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
    masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
+9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
+   `kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
+   `virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
+   the cluster's **v1.9.0**.
 
 ## Remaining bring-up
 
-Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
+Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
+is written for a from-scratch cluster; the prod addresses are the live ones.
 
-1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
+1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
 2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
-   Kustomization (not an app wave).
+   Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
 3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
 4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
-   `192.168.1.54`.
+   `192.168.2.21`.
-5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
+5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
-   and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
+   `uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
-   ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
+   fill `installer-url`, commit, and the guest installs itself on first boot.
-6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
+6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
+   console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
+7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
    complete the first-run wizard.
-7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
+8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
-8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
+   split-horizon) and from in-cluster (coredns-custom).
+9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
    connects (green). Test guest portal / throughput if used.
-9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
+10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
-10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
+11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
 
 **Quick health check today:**
 ```bash
 kubectl -n unifi get dv,vmi,svc
 kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
+# the two that would have caught the current failure:
+kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}'   # expect an ipAddress
+kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
+  -c compute -- ip neigh                                                 # expect 10.0.2.2 REACHABLE
 ```

apps/vms/unifi/kustomization.yaml

@@ -11,3 +11,11 @@ resources:
   - 30-virtualmachine.yaml
   - 40-networkpolicy.yaml
   - 50-service-l4.yaml
+  # SOPS-encrypted. Carries the operator's UOS installer URL (+ optional SSH key) and is mounted
+  # into the guest as the UOSDATA disk by 30-virtualmachine.yaml. Decrypted by the `apps-vms` Flux
+  # Kustomization, which clusters/dev and clusters/prod patch with the `decryption` stanza — the
+  # FIRST encrypted content under apps/vms, so that patch was added with this file. The plaintext
+  # template + the operator hand-off steps live in the `.example` sibling of the file below, which
+  # is deliberately NOT named here: the L0 no-plaintext-secrets check fails any kustomization whose
+  # text contains an example template's filename, on the assumption it is being applied.
+  - unifi-uos-installer.sops.yaml

apps/vms/unifi/unifi-uos-installer.sops.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: unifi-uos-installer
+    namespace: unifi
+type: Opaque
+stringData:
+    installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
+    console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
+            WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
+            TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
+            cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
+            kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-19T02:47:13Z"
+    mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/vms/unifi/unifi-uos-installer.sops.yaml.example

@@ -0,0 +1,71 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the two values that (a) let the UniFi VM install UniFi OS Server by itself, with
+# nobody typing commands inside the guest, and (b) give the operator a way INTO that guest at all.
+# KubeVirt mounts this Secret into the VM as a tiny read-only iso9660 disk labelled `UOSDATA` (see
+# the `uosdata` volume in 30-virtualmachine.yaml); each key below becomes a file of the same name on
+# that disk, and the in-guest `uos-install.service` reads them on EVERY boot.
+#
+# READ THIS FIRST — UOS IS ALREADY INSTALLED ON THE LIVE PROD GUEST (verified 2026-09-14, see the
+# README "What is actually wrong"). `installer-url` is therefore NOT the fix for today's outage; it
+# is the rebuild path and the "the install never happened" path. `install-uos` refuses to run the
+# installer at all while systemd knows a `uosserver.service`, so filling this in cannot clobber a
+# working controller. `console-password` IS useful today: it is the only way to get a shell.
+#
+# WHY THE URL IS BEHIND SOPS: it is obtained through the operator's own (free) Ubiquiti account
+# session at ui.com/download/software/unifi-os-server, it identifies a specific build, and it is not
+# ours to redistribute. Treat it as account-derived material, not as public configuration.
+# HONEST CAVEAT, checked 2026-09-14: the resulting `https://fw-download.ubnt.com/data/
+# unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` URL carries NO signature query string,
+# and public community install scripts hard-code these URLs — so it does not behave like a
+# short-lived signed link. It does change every release. Encrypting it costs nothing and keeps the
+# repo's "account-derived material is not config" line clean, but do NOT design around "it expires".
+#
+# HAND-OFF STEPS (operator) — this is the whole job:
+#  (a) Sign in at https://ui.com/download/software/unifi-os-server, pick the Linux (x64) build, and
+#      copy the download link (right-click -> Copy link address). Do NOT download it here; the guest
+#      fetches it itself.
+#  (b) Open the encrypted sibling in your editor:
+#         sops apps/vms/unifi/unifi-uos-installer.sops.yaml
+#  (c) Replace the two REPLACE-ME values. Either may be left as the placeholder — the guest skips a
+#      placeholder rather than failing.
+#  (d) Save/quit (sops re-encrypts in place), commit, push, merge to the branch prod tracks.
+#  (e) Restart the VM once so the new cloud-init instance and the new disk are picked up:
+#         kubectl -n unifi delete vmi unifi        # `running: true` recreates it; no virtctl needed
+#      Watch it come up WITHOUT virtctl:
+#         kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher \
+#           -o name | head -1) -c guest-console-log
+#  (f) UI: https://192.168.2.21:11443 (prod) or https://unifi.home.keithjackson.dev:11443.
+#
+# IF YOU LATER NEED A NEW URL (new release, dead link): repeat (b)-(e). No firmware-UUID bump is
+# needed for that — the disk is rebuilt from this Secret on every VMI start and the systemd unit
+# re-reads it every boot. The UUID bump in 30-virtualmachine.yaml is only needed when the cloud-init
+# payload ITSELF changes.
+#
+# The file is already listed under `resources:` in apps/vms/unifi/kustomization.yaml, and `apps-vms`
+# is patched with the SOPS `decryption` stanza in clusters/dev + clusters/prod.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: unifi-uos-installer
+  namespace: unifi
+type: Opaque
+stringData:
+  # KEY NAMES ARE LOAD-BEARING: each key becomes a FILE of that exact name on the UOSDATA disk, and
+  # /usr/local/sbin/install-uos reads them by path. Renaming a key breaks the install silently.
+  #
+  # The signed-in download link for UniFi OS Server, Linux x64. Anything that is not an https URL
+  # (this placeholder included) is treated by the guest as "not provided yet": the unit logs
+  # "UOS installer-url not set yet" and exits 0, so the VM stays healthy and simply does not install.
+  installer-url: REPLACE-ME
+  # Password set on the guest's `ubuntu` account on every boot, for `virtctl -n unifi console unifi`.
+  # THIS IS THE ONLY DOOR INTO THE GUEST. The stock Ubuntu 24.04 cloud image ships `ubuntu` locked,
+  # no key was ever installed, and an SSH key would not help: the VM's masquerade interface forwards
+  # only the ports declared in 30-virtualmachine.yaml and 22 is not among them (probed from inside
+  # virt-launcher, 2026-09-14 — the guest RSTs on 22 just as it does on 11443). Generate one with
+  #     openssl rand -base64 24
+  # Leave it as REPLACE-ME to keep the account locked; the script skips the placeholder by name.
+  console-password: REPLACE-ME

apps/wave-0/gitea.yaml

@@ -78,7 +78,7 @@ spec:
     spec:
       containers:
       - name: gitea
-        image: gitea/gitea:1.24.3
+        image: gitea/gitea:28.3.0
         ports:
         - containerPort: 3000
         env:
@@ -89,14 +89,18 @@ spec:
             secretKeyRef:
               name: gitea-bootstrap
               key: secret-key
-        - name: GITEA__server__DOMAIN
-          value: gitea.home.keithjackson.dev
         - name: GITEA__server__ROOT_URL
-          value: https://gitea.home.keithjackson.dev/
+          value: https://gitea.${BASE_DOMAIN}/
         - name: GITEA__actions__ENABLED
           value: 'true'
         - name: GITEA__actions__DEFAULT_ACTIONS_URL
           value: github
+        - name: GITEA__actions__RUN_RETENTION_DAYS
+          value: '400'
+        - name: GITEA__security__EGRESS_MODE
+          value: lax
+        - name: GITEA__service__DISABLE_REGISTRATION
+          value: 'true'
         - name: GITEA_ADMIN_USERNAME
           valueFrom:
             secretKeyRef:
@@ -127,7 +131,7 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
+              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
                 \ break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
       volumes:
       - name: data
@@ -170,6 +174,48 @@ spec:
       - port: '3000'
         protocol: TCP
 ---
+# CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): gitea-allow-flux-ingress
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: gitea-allow-flux-ingress
+  namespace: gitea
+  labels:
+    app.kubernetes.io/name: gitea
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: gitea
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: flux-system
+    toPorts:
+    - ports:
+      - port: '3000'
+        protocol: TCP
+---
+# CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): gitea-allow-pm-agent-ingress
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: gitea-allow-pm-agent-ingress
+  namespace: gitea
+  labels:
+    app.kubernetes.io/name: gitea
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: gitea
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: pm-agent
+    toPorts:
+    - ports:
+      - port: '3000'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): gitea-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -207,7 +253,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - gitea.home.keithjackson.dev
+  - gitea.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -217,26 +263,6 @@ spec:
     - name: gitea
       port: 3000
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: gitea
-  namespace: gitea
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: gitea
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: gitea
-  namespaceSelector:
-    matchNames:
-    - gitea
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -251,11 +277,11 @@ spec:
   - name: gitea.slis
     rules:
     - alert: gitea-down
-      expr: up{job="gitea"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"} == 0 or absent(kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"})
       for: 5m
       labels:
         severity: critical
         service: gitea
       annotations:
         summary: gitea is down
-        description: 'gitea has been unreachable for 5m (SLI: up).'
+        description: 'gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/jackett.yaml

@@ -74,36 +74,41 @@ spec:
         homelab.keithjackson.dev/auth-mode: forward-auth
       annotations:
         homelab.keithjackson.dev/auth-mode: forward-auth
-        homelab.keithjackson.dev/auth-groups: media-users
+        homelab.keithjackson.dev/auth-groups: media-users,platform-admins
     spec:
       containers:
       - name: jackett
-        image: lscr.io/linuxserver/jackett:latest
+        image: lscr.io/linuxserver/jackett:v0.24.3025-ls29
         ports:
         - containerPort: 9117
         volumeMounts:
         - name: config
-          mountPath: /data/config
+          mountPath: /config
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:9117
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
+        - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://jackett.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --api-route=^/api/
-        - --whitelist-domain=.home.keithjackson.dev
+        - --cookie-name=_oauth2_proxy_jackett
+        - --cookie-domain=jackett.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +129,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -236,6 +244,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): jackett-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: jackett-allow-monitoring-scrape
+  namespace: jackett
+  labels:
+    app.kubernetes.io/name: jackett
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: jackett
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -266,14 +295,15 @@ metadata:
     app.kubernetes.io/name: jackett
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
-    homelab.keithjackson.dev/auth-groups: media-users
+    homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - jackett.home.keithjackson.dev
+  - jackett.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -283,26 +313,6 @@ spec:
     - name: jackett
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: jackett
-  namespace: jackett
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: jackett
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: jackett
-  namespaceSelector:
-    matchNames:
-    - jackett
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -317,11 +327,11 @@ spec:
   - name: jackett.slis
     rules:
     - alert: jackett-down
-      expr: up{job="jackett"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"} == 0 or absent(kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"})
       for: 5m
       labels:
         severity: critical
         service: jackett
       annotations:
         summary: jackett is down
-        description: 'jackett has been unreachable for 5m (SLI: up).'
+        description: 'jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/kustomization.yaml

@@ -11,13 +11,20 @@ resources:
 - jackett-oauth2-proxy-credentials.sops.yaml
 - jackett.yaml
 - nextcloud-db-app.sops.yaml
+- nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
+- pihole-bootstrap-secret.sops.yaml
+- pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
+- plex-claim-secret.sops.yaml
+- plex-remote.yaml
 - plex.yaml
+- qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
 - qbittorrent.yaml
 - semaphore-db-app.sops.yaml
+- semaphore-db-backup-credentials-prod.sops.yaml
 - semaphore-db-backup-credentials.sops.yaml
 - semaphore-db.yaml

apps/wave-0/nextcloud-db-backup-credentials-prod.sops.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: nextcloud-db-backup-credentials-prod
+    namespace: nextcloud-db
+type: Opaque
+stringData:
+    ACCESS_KEY_ID: ENC[AES256_GCM,data:4lx6et4HcXDegulyIoa7jZR/lVgapGPEkukajfCKjvk=,iv:p6V7tizbikk55tOvpPOsPHTe/p52QLX5A5xrwuB14rA=,tag:oNJuipT/Lb6zynJ2K2Q71A==,type:str]
+    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:V5bHPoPeBOVyLp1n1Jilyso4i5P2aPSH0hrryfweg2q/SctWZxRbLBQ/FLzOigbJsAhlwOGu069mQPma6xjfdw==,iv:8u4UOvI8dC/cm6mEWDlyN5E3SfRZmwKY9uZY/DjPEsQ=,tag:tw7tGs29TmUvMPBCtCszAA==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiSUt5N2k2N2lDU2ZsMy9m
+            M1NFS0M2QjFsNFhnR0hCMzFyaTdiQWFZRFJZCmhBTWRXNmxLMGtsc3Rxd2FSMDZp
+            aktjcTE2c1hmU1EzM3RWNksyb3Q1YVUKLS0tIDBFS1I5aDBER2cvcWtLZ3BXTEp5
+            ZGI5ZmZIcStheUNQamxIVVVpc0ZMOHMKlE5aHewtHJpeU24F5sodc9CEm9KTjHQQ
+            8ThEQ/8vyazL9V4ojlICxSHU/SHkHpD1eBaTKcAQd/LZzujIcJ+QfA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-07T06:36:33Z"
+    mac: ENC[AES256_GCM,data:8awz1Jh8t4zjDMyd93/v+6on9iiIaSi01293A7uYhxj9L1+g63zeO+DPmSn6+KbZvHdqU2A7up09H1NzfXAEdGuvVbB36cdGq0zKank3PUi8c7IT6qNuWydxRAcqLnbbD+c30Z6GB+gAuZHstNOe4Rgc4aJvMN6FuP6x6Z0870c=,iv:KGGKJq2QA5xztvoHs/SViXW72DYwT16i+W+dJBUeIJQ=,tag:3UpYqx2+RIK0QdMH/BSIpA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/nextcloud-db.yaml

@@ -37,16 +37,53 @@ metadata:
   name: nextcloud-db
   namespace: nextcloud-db
 spec:
-  # Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
+  # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db; see
-  # keycloak-db). The storage is already 3-way replicated by Longhorn; raise to 3 for Postgres-level
+  # platform/keycloak/postgres-cluster.yaml for the full rationale. Short version: Longhorn's 3-way
-  # HA as a future hardening.
+  # replication saves the DATA on node loss, not the SERVICE, and with `instances: 1` all three CNPG
-  instances: 1
+  # primaries had drifted onto k3s-prod-2, so one node took Nextcloud down alongside SSO and
-  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
+  # Semaphore. A streaming standby lets CNPG promote in seconds and repoint the -rw Service.
+  # Gated on the Barman archive below actually working (ContinuousArchiving=True with a real
+  # firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
+  #
+  # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
+  # pg_basebackup from the running primary, which is neither restarted nor disconnected.
+  instances: 3
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.15
   # Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
   # nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
   inheritedMetadata:
     labels:
       app.kubernetes.io/name: nextcloud-db
+  # HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
+  # primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
+  # recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
+  # version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
+  # after a node loss it may stack two instances on one node — which is how `instances: 3` decays
+  # back into the single-node failure domain this change removes. `required` makes it a constraint;
+  # the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
+  # serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
+  #
+  # APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
+  # switchover. Expect a single brief connection reset for Nextcloud.
+  affinity:
+    enablePodAntiAffinity: true
+    podAntiAffinityType: required
+  primaryUpdateMethod: switchover
+  # SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
+  # rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
+  # last few transactions when a primary dies before its standbys catch up, and `method: any` +
+  # `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
+  # never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
+  # are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
+  # standby into a total write outage. `preferred` relaxes to async instead.
+  #
+  # NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
+  # no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
+  postgresql:
+    synchronous:
+      method: any
+      number: 1
+      dataDurability: preferred
   storage:
     storageClass: longhorn
     size: 5Gi
@@ -68,25 +105,38 @@ spec:
       memory: 256Mi
     limits:
       memory: 512Mi
-  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
+  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
   # Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
-  # backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
+  # backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
-  # (defaults to the Cluster name), so objects land under s3://cnpg/nextcloud-db/. Path-style HTTP.
+  # destinationPath, so objects land under ${CNPG_BACKUP_PATH}/nextcloud-db/ — dev
-  # Credentials: the scoped key from the in-namespace SOPS Secret nextcloud-db-backup-credentials
+  # s3://cnpg/nextcloud-db/ on the on-Proxmox MinIO, prod
-  # (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
+  # s3://k3s-prod-backup/barman/nextcloud-db/ on Cloudflare R2. Both endpointURL and
-  # endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
+  # destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
-  # substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
+  # Credentials: PER-ENVIRONMENT (2026-09-07). dev reads nextcloud-db-backup-credentials (the
+  # scoped MinIO key), prod reads nextcloud-db-backup-credentials-prod (the Cloudflare R2
+  # token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
+  # generated golden — render.py places the -prod sidecar in this DB's wave); the
+  # cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
+  # one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
+  # ONE shared Secret until the split, so giving prod its R2 token would have left dev
+  # signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
+  # disk-full clock below rather than a merely missing backup.
+  #
+  # archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
+  # Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
+  # the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
+  # endpoint. Treat ContinuousArchiving=False here as an availability incident.
   backup:
     retentionPolicy: "30d"
     barmanObjectStore:
-      destinationPath: s3://cnpg
+      destinationPath: ${CNPG_BACKUP_PATH}
       endpointURL: ${S3_BACKUP_ENDPOINT}
       s3Credentials:
         accessKeyId:
-          name: nextcloud-db-backup-credentials
+          name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
           key: ACCESS_KEY_ID
         secretAccessKey:
-          name: nextcloud-db-backup-credentials
+          name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
           key: ACCESS_SECRET_KEY
       wal:
         compression: gzip
@@ -108,3 +158,243 @@ spec:
   cluster:
     name: nextcloud-db
   method: barmanObjectStore
+---
+# =============================================================================
+# CNPG METRICS-EXPORTER CONNECT RECONCILER (R-OBS-01)
+#
+# THE BUG THIS CLOSES (true positive, verified live on prod 2026-09-12, and the
+# condition platform/cnpg/alerts.yaml already recorded on BOTH clusters 2026-09-06):
+# the `nextcloud` database's ACL had been narrowed out-of-band so that PUBLIC held
+# nothing, and CNPG's metrics exporter role therefore could not even CONNECT:
+#
+#   nextcloud-db-1/-2/-3, every scrape:
+#     FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
+#     DETAIL: User does not have CONNECT privilege.
+#     msg="Error collecting user query" targetDatabase=nextcloud
+#
+# CONSEQUENCE: `cnpg_last_error` = 1 on all three instances and the ENTIRE
+# cnpg_pg_stat_* / pg_settings / pg_postmaster family MISSING for this database,
+# while the scrape target still reported up=1 — a blind spot that looks like health.
+# Measured before the fix: cnpg_pg_settings_setting had 765 series for keycloak and
+# 765 for semaphore-db and ZERO for nextcloud-db. Nextcloud's database had never
+# been monitored.
+#
+# ROOT CAUSE, AND WHY ONLY THIS DATABASE. All three Cluster specs are identical in
+# their bootstrap; the difference is entirely in the live ACL:
+#
+#   nextcloud : datacl = {nextcloud=CTc/nextcloud}          <-- explicit, no PUBLIC entry
+#   keycloak  : datacl = NULL
+#   semaphore : datacl = NULL
+#
+# A NULL datacl means "PostgreSQL defaults", which include CONNECT and TEMP for
+# PUBLIC — so keycloak and semaphore let the exporter in only by inheriting PUBLIC's
+# implicit CONNECT. On `nextcloud` something ran the equivalent of
+# `REVOKE ALL ON DATABASE nextcloud FROM PUBLIC`, which MATERIALISES datacl and
+# drops the implicit PUBLIC entry with it. That REVOKE is not in this repo and not
+# in DECISIONS-LOG.md — it was applied out of band, which is exactly why a
+# git-declared fix is needed rather than another hand-run psql.
+#
+# WHY THE GRANT IS EXPLICIT RATHER THAN "RE-GRANT CONNECT TO PUBLIC": restoring
+# PUBLIC's CONNECT would undo whatever hardening that REVOKE intended and would
+# leave the exporter depending on an implicit default again — the precise fragility
+# that broke here and that keycloak/semaphore are still relying on. Naming the
+# exporter role makes the dependency explicit and survives a future REVOKE FROM
+# PUBLIC on any of the three databases.
+#
+# WHY A CronJob AND NOT A CNPG FIELD — CNPG 1.30.0's API GENUINELY CANNOT EXPRESS
+# THIS. Checked against the live CRDs on prod:
+#   * `Database` (databases.postgresql.cnpg.io) has no database-ACL field at all.
+#     `allowConnections` is the pg_database.datallowconn FLAG (open/closed to
+#     everyone), not a per-role GRANT; its only grant/revoke verbs are the `usage`
+#     lists on `fdws` and `servers`.
+#   * `spec.managed.roles` and `DatabaseRole` manage ROLE ATTRIBUTES (login,
+#     superuser, connectionLimit, ...) plus `inRoles` MEMBERSHIP. Membership is the
+#     wrong tool: the only local role holding CONNECT here is the OWNER, so
+#     `inRoles: [nextcloud]` would hand the exporter the owner's CREATE and TEMP on
+#     the database and read/write on every application table — the opposite of
+#     least privilege, to buy one CONNECT.
+#   * `bootstrap.initdb.postInitApplicationSQL` runs ONCE, at bootstrap. It cannot
+#     repair these three already-bootstrapped instances (the actual bug), and on a
+#     FRESH cluster the grant is unnecessary anyway (datacl starts NULL). Adding it
+#     would be a second mechanism that fixes nothing this one does not.
+# So the SQL has to be applied by something, and this repo already has the pattern
+# for state Flux cannot reach: the level-triggered CronJob reconciler in
+# platform/keycloak/client-secret-reconciler.yaml, whose reasoning transfers
+# verbatim — the drifted state lives inside PostgreSQL, not in a Kubernetes object,
+# so no admission or mutation policy can see it. A one-shot Job is deliberately NOT
+# used: a completed Job is never re-run, so a DB rebuild or a PITR restore would
+# silently reopen the blind spot. A loop converges instead.
+#
+# PRIVILEGE GRANTED — EXACTLY ONE, AND NOTHING ELSE:
+#   GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
+# Resulting ACL entry is `cnpg_metrics_exporter=c/nextcloud` — the `c` is CONNECT;
+# there is no `C` (CREATE) and no `T` (TEMPORARY). No ownership, no superuser, no
+# write, no table or schema grants. Nothing further is needed because CNPG already
+# creates this role as a member of the built-in `pg_monitor` role (verified on all
+# three prod clusters), and pg_monitor already carries the read access every
+# collector query wants — they read pg_catalog views (pg_stat_database,
+# pg_stat_archiver, pg_settings, pg_postmaster) and no application tables. Proof
+# that CONNECT alone is sufficient: after this grant, `Error collecting user query`
+# went to ZERO on all three pods, i.e. every collector query now succeeds.
+#
+# IT RUNS AS THE DATABASE OWNER, NOT AS A SUPERUSER. `enableSuperuserAccess` is off
+# on this Cluster (there is no nextcloud-db-superuser Secret), and it stays off: the
+# owner of a database holds the grant option on that database's privileges, so
+# `nextcloud` can issue this GRANT itself. Verified live before committing, by
+# running the statement under `SET ROLE nextcloud`. Credentials come from the
+# in-namespace SOPS Secret nextcloud-db-app that already exists for the app.
+#
+# NETWORKING: nextcloud-db is intentionally NOT an app-namespace (see this file's
+# header), so Kyverno generates no default-deny baseline here and this same-namespace
+# pod -> nextcloud-db-rw:5432 connection needs no rendered allow.
+#
+# THE POD LABEL IS DELIBERATELY *NOT* app.kubernetes.io/name=nextcloud-db. That
+# label is what `inheritedMetadata` stamps on the CNPG instance pods and what the
+# rendered nextcloud egress CiliumNetworkPolicy selects; reusing it here would quietly
+# make this reconciler a permitted destination for the Nextcloud app. It gets its own
+# name so that selector keeps meaning "the database".
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: nextcloud-db-exporter-grant
+  namespace: nextcloud-db
+  labels:
+    app.kubernetes.io/name: nextcloud-db-exporter-grant
+    app.kubernetes.io/part-of: nextcloud-db
+spec:
+  # Every 15 minutes, matching the keycloak reconciler's cadence. A converged
+  # database makes this one idempotent GRANT and exits; the cost is negligible, and
+  # after a rebuild or PITR restore monitoring is blind for at most one interval
+  # (well inside the alert's `for: 6h`).
+  schedule: "*/15 * * * *"
+  # A run still waiting on the primary must not be lapped by the next tick.
+  concurrencyPolicy: Forbid
+  startingDeadlineSeconds: 300
+  successfulJobsHistoryLimit: 1
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 300
+      ttlSecondsAfterFinished: 3600
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/name: nextcloud-db-exporter-grant
+            app.kubernetes.io/part-of: nextcloud-db
+        spec:
+          restartPolicy: Never
+          # Talks only to PostgreSQL; its one input is a mounted Secret, so it needs
+          # no Kubernetes API access at all (R-SEC-02, least privilege).
+          automountServiceAccountToken: false
+          securityContext:
+            runAsNonRoot: true
+            # uid/gid 26 is `postgres` in the CNPG PostgreSQL image.
+            runAsUser: 26
+            runAsGroup: 26
+            seccompProfile:
+              type: RuntimeDefault
+          containers:
+            - name: grant
+              # The SAME image the instances above run, pinned to the same tag. psql
+              # is already in it and it is already present on every node, so this
+              # adds no new image and no new supply-chain surface. Keep the tag in
+              # step with spec.imageName above when Renovate bumps it.
+              image: ghcr.io/cloudnative-pg/postgresql:16.15
+              imagePullPolicy: IfNotPresent
+              command:
+                - psql
+                # -w never prompts; -X skips ~/.psqlrc; ON_ERROR_STOP makes the Job
+                # FAIL LOUDLY rather than reporting success on a refused grant.
+                - "-w"
+                - "-X"
+                - "-v"
+                - "ON_ERROR_STOP=1"
+                - "-f"
+                - /etc/grant/grant.sql
+              env:
+                # libpq connection, assembled from the owner Secret. Target the -rw
+                # Service: GRANT is a write and must reach the primary.
+                - name: PGHOST
+                  value: nextcloud-db-rw.nextcloud-db.svc.cluster.local
+                - name: PGPORT
+                  value: "5432"
+                # Connecting to the application database itself is deliberate: the
+                # owner has CONNECT on it, and a successful connection is the same
+                # check the exporter was failing.
+                - name: PGDATABASE
+                  value: nextcloud
+                - name: PGUSER
+                  valueFrom:
+                    secretKeyRef:
+                      name: nextcloud-db-app
+                      key: username
+                # Read by libpq from the environment and never logged; psql echoes
+                # no connection parameters.
+                - name: PGPASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: nextcloud-db-app
+                      key: password
+                - name: PGCONNECT_TIMEOUT
+                  value: "10"
+              volumeMounts:
+                - name: sql
+                  mountPath: /etc/grant
+                  readOnly: true
+              securityContext:
+                allowPrivilegeEscalation: false
+                readOnlyRootFilesystem: true
+                capabilities:
+                  drop:
+                    - ALL
+              resources:
+                requests:
+                  cpu: 10m
+                  memory: 32Mi
+                limits:
+                  memory: 128Mi
+          volumes:
+            - name: sql
+              configMap:
+                name: nextcloud-db-exporter-grant
+                defaultMode: 0444
+---
+# The desired state, as SQL. Idempotent: re-granting an existing privilege is a
+# no-op, so a converged run changes nothing. The DO block is the ASSERTION — it
+# turns "the grant did not take" into a failed Job instead of a silent success.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nextcloud-db-exporter-grant
+  namespace: nextcloud-db
+  labels:
+    app.kubernetes.io/name: nextcloud-db-exporter-grant
+    app.kubernetes.io/part-of: nextcloud-db
+data:
+  grant.sql: |
+    -- The ONE privilege the CNPG metrics exporter is missing on this database.
+    -- CONNECT only: no CREATE, no TEMPORARY, no ownership, no table/schema grants.
+    -- Read access for the collector's pg_catalog queries already comes from the
+    -- built-in pg_monitor role, which CNPG grants this role when it creates it.
+    GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
+
+    -- Fail the Job if the privilege is somehow still absent, so a broken
+    -- reconciler surfaces as a failed Job rather than as a quiet blind spot.
+    -- NO PL/pgSQL, AND NOT ONE DOLLAR SIGN IN THIS FILE — DELIBERATE, including in these
+    -- comments. This assertion began life as a dollar-quoted DO block, which cannot survive
+    -- being shipped this way: the ConfigMap is rendered through Flux's postBuild
+    -- substitution, and envsubst collapses a doubled dollar into a single one. The block
+    -- reached Postgres with its opening quote halved and every Flux-reconciled run died
+    -- with a syntax error pointing at that character.
+    -- It PASSED when hand-applied with kubectl, because that path runs no envsubst — so
+    -- the defect was invisible until Flux owned the object, and three runs of Error pods
+    -- were the first sign. Escaping by doubling again would work and would also be the
+    -- next person's trap; plain SQL leaves substitution nothing to touch.
+    --
+    -- has_database_privilege() returns boolean; ::int is 1 when granted and 0 when not, so
+    -- the division raises division_by_zero and ON_ERROR_STOP=1 fails the Job. The divisor
+    -- is not a constant, so it cannot be folded away at plan time.
+    \echo 'asserting cnpg_metrics_exporter has CONNECT on database nextcloud'
+    SELECT 1 / has_database_privilege('cnpg_metrics_exporter', 'nextcloud', 'CONNECT')::int
+      AS exporter_can_connect;

apps/wave-0/pihole-bootstrap-secret.sops.yaml

@@ -0,0 +1,72 @@
+# =================================================================================================
+# PiHole WEB/API ADMIN PASSWORD — the one piece of PiHole state that was NOT declared anywhere.
+# =================================================================================================
+# WHY THIS FILE EXISTS. Measured on live prod 2026-10-04, `pihole.toml` marks every non-default
+# setting with `### CHANGED`, and names the ones that came from the environment `### CHANGED (env)`.
+# The complete list was:
+#
+#     dns.upstreams          ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.listeningMode      ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.rateLimit.count    ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     misc.dnsmasq_lines     ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     files.macvendor        ### CHANGED         <- set by the image itself, container path
+#     webserver.api.pwhash   ### CHANGED         <- SET OUT OF BAND. Declared NOWHERE. This file.
+#
+# So exactly ONE operator-set value lived only on the Longhorn PVC: the admin password, as a
+# one-way BALLOON-SHA256 hash. Nobody could recover it (a hash does not reverse), nothing in git
+# produced it, and a rebuild from git would have come up with a DIFFERENT password — or none.
+#
+# THE REASON THIS BLOCKS HA, which is why it is fixed first. The next step is running more than one
+# PiHole replica, and a replica with a fresh PVC would come up with its own unrelated password. The
+# admin UI would then accept or reject the same credential depending on which pod answered. Declaring
+# it here makes every replica, and every rebuild, identical by construction.
+#
+# MECHANISM: `webserver.api.password` is a real, settable FTL config key — verified in the running
+# image, where `pihole setpassword` is itself just `setFTLConfigValue "webserver.api.password"` with
+# the comment "pihole-FTL will automatically hash the password". So FTL takes CLEARTEXT on this key
+# and writes the hash into `webserver.api.pwhash` itself. catalog/pihole.yaml therefore passes this
+# value as `FTLCONF_webserver_api_password`, and the hash on the PVC becomes a derived artefact
+# rather than the source of truth.
+#
+# DEFENCE IN DEPTH, NOT THE ONLY GATE. The admin UI is already behind the oauth2-proxy sidecar
+# (`--allowed-group=platform-admins`), so this password is a SECOND gate. It is kept rather than set
+# empty because the `pihole` Service still publishes port 80 (PiHole direct, bypassing the proxy on
+# :4180) and only the CiliumNetworkPolicy closes it — and the default-deny policy in this repo has a
+# documented drift history. An unauthenticated admin UI that can rewrite LAN DNS is not a good thing
+# to leave resting on one policy staying correct. Narrowing that Service port is tracked follow-up.
+#
+# ROTATION: `sops` this file in your own terminal, change the value, commit. Flux re-applies the
+# Secret, the pod restarts, FTL re-hashes. Do NOT `sops -d` it into a shared terminal or a log
+# (CLAUDE.md section 6); read it in an editor. The value below was generated randomly at authoring
+# time and has never been printed to a transcript.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: pihole-bootstrap
+    namespace: pihole
+    labels:
+        app.kubernetes.io/name: pihole
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:TMOuRyJeb5IVn3lZPXY7lgh1KGxu7GY9+WIOLHpMB50JqzmM2tHIaWUWKTt1JtZoX/E5/Y7WCUTnfkJLhRBN2zV2MwWRBSLYVy5CMg7xV8PTLRZJYAVUYS5Trg==,iv:q+DpfT6PQfY5ZuNsmP7Ddl+PDLjSm89fJICAhYoW64k=,tag:axijaO3DaxIZgZ3LeK2hXA==,type:comment]
+    webui-password: ENC[AES256_GCM,data:7FcK/bUW79M46f/MlouAmATKbMnoUTNuMUjtqJ5vpU4=,iv:I8DBFB39uXZV42Ie2IPwSefmMgdKEtiDknfc9Tu7h5E=,tag:L423784UrDU3TvylpHc6KQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0MkRSNjZLZlpXYkcvaVJL
+            cjlqb2wrcWREZnZLdGpsTHNPWEpseHlRZnpjCnJVM1FDQ2M2dXlaS1JBZ2k2V1BP
+            Yi9QUDZSMzZsL09pTi93RTdjdGxtY2MKLS0tIDIrQTZIVWdhMGIxSCtlUFN2Mlho
+            SWVybld4VjJyaHRSKzZZU0xsUUV2SEEKbDYHV/qBZVspK7DpBW3vYjRMJ/HmAN+9
+            G9bOlNEp0jAtcNZEn6lpQ9ORgvmIrhJHoBQRLabdMWzNWSZsNn56tA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-04T06:01:27Z"
+    mac: ENC[AES256_GCM,data:rhTMTOwLOtHXHgqI5M2ZW/6w+UpLpxBlTk7NoN0wmGSqOcE+OGrXA1DhkZrJweyQT0RSNSSBm/6QMztLP81eepOzdp2rEkwvbQPK/l6K9g16Ww9c1IatS3NQsKEB8AoMDx7jBQ6exXYgggPx3MJsNYbE+WyloQ96U9KPdrecAM8=,iv:n34v2zX9D5xsaR1PFCH94mNMiuRfhUyI2lseUCoqiTA=,tag:9YmN9nIVLXfsGYIb0bUl2w==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/pihole-dns.yaml

@@ -0,0 +1,210 @@
+---
+# =================================================================================================
+# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole`
+# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS
+# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition
+# schema genuinely cannot express it: schemas/servicedefinition.schema.json is
+# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned
+# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at
+# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and
+# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this
+# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and
+# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog
+# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the
+# schema has no word for.
+#
+# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is
+# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service
+# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09
+# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on
+# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that
+# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0).
+#
+# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the
+# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config
+# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is
+# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it
+# can never be allowed to move.
+#
+# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53
+# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same
+# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the
+# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a
+# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently
+# and confusingly.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC
+# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself
+# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS,
+# and publishing a resolver's address in a public zone invites exactly the traffic the source-range
+# filter below exists to refuse. Internal name, internal answer, no public record — which is also
+# why this change touches nothing under platform/externaldns/.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to
+# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured,
+# not assumed (Cilium 1.20.1, prod, 2026-09-14):
+#
+#   * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the
+#     CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does
+#     NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services
+#     landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3,
+#     `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1.
+#   * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the
+#     packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three
+#     nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol.
+#   * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod
+#     rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three
+#     minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a
+#     re-election — and that re-election is random, so it may pick a backend-less node again.
+#
+# A resolver that works one time in three and stays broken after any reschedule is not a resolver.
+# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is
+# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement
+# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all
+# answering.
+#
+# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in
+# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the
+# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client
+# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two
+# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole
+# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts
+# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity —
+# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode:
+# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract
+# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
+#
+# -------------------------------------------------------------------------------------------------
+# loadBalancerSourceRanges — THE LAN + VPN RESTRICTION, ENFORCED IN THE DATAPATH.
+# -------------------------------------------------------------------------------------------------
+# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
+# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} and ${WG_CLIENT_CIDR} is
+# dropped by the datapath — it never reaches a backend and never reaches a network policy. This is what makes
+# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
+# range to a /24 that excludes the test client, after which the same query that had been answering
+# stopped being answered at all, and restoring it, after which it answered again.
+#
+# ${WG_CLIENT_CIDR} IS LISTED BECAUSE VPN CLIENTS REACH THIS CHECK UNTRANSLATED. wg0 is one of
+# Cilium's kube-proxy-replacement devices (measured 2026-11-08: `cilium-dbg status` lists eno1, wg0,
+# tun0), so a tunnel client's query is load-balanced as it enters from wg0, ahead of the iptables
+# MASQUERADE in platform/vpn-ingress/daemonset.yaml, and the source tested here is still 10.90.0.x
+# rather than the node's address. Without this entry a full-tunnel client's DNS is predicted to die
+# with DROP_NOT_IN_SRC_RANGE (derived from Cilium's source; NOT observed live, because no peer has
+# ever completed a handshake). The range is private and reachable only by a key in peers.conf, so
+# PiHole stays off the internet; conformance/tests/l0-static/pihole-dns-source-ranges fails the gate
+# if either entry goes missing or any range here stops being RFC1918.
+#
+# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
+# restriction once the source has been rewritten: measured on this cluster, the forwarded query
+# arrives from the ingress node's internal router address and Cilium resolves that source to the
+# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the
+# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction
+# lives here, where it is actually enforceable, and the policy below restricts the PORTS.
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-udp
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+  annotations:
+    lbipam.cilium.io/sharing-key: "pihole-dns"
+    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  loadBalancerSourceRanges:
+    - ${LAN_CIDR}
+    - ${WG_CLIENT_CIDR}
+  selector:
+    app.kubernetes.io/name: pihole
+  ports:
+    - { name: dns-udp, port: 53, targetPort: 53, protocol: UDP }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-tcp
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+  annotations:
+    lbipam.cilium.io/sharing-key: "pihole-dns"   # same key + same IP => shares the VIP with -udp
+    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  loadBalancerSourceRanges:
+    - ${LAN_CIDR}
+    - ${WG_CLIENT_CIDR}
+  selector:
+    app.kubernetes.io/name: pihole
+  ports:
+    - { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP }
+---
+# =================================================================================================
+# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports.
+# =================================================================================================
+# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered
+# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the
+# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything
+# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the
+# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule:
+# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that
+# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for
+# it and no policy for it.
+#
+# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is
+# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's
+# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium
+# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was
+# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced
+# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
+# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
+# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
+# policy is consulted. Together: ${LAN_CIDR} + ${WG_CLIENT_CIDR} in the datapath, udp/53 + tcp/53
+# here. A VPN client delivered to a backend on its own ingress node keeps its 10.90.0.x source,
+# which Cilium also classifies as `world`, so this rule needs no new peer for it.
+#
+# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
+# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
+# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none
+# needs PiHole on :53) on fewer ports, behind a filter those two do not have.
+#
+# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the
+# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to
+# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same
+# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not
+# an internet-facing open resolver. Say so out loud rather than implying the filter is total.
+#
+# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved
+# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the
+# node-to-node forwarding that IS the Cluster-policy datapath both need them.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-lan-dns-ingress
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+    - fromEntities:
+        - world         # LAN + VPN clients, post-LoadBalancer: see the block above for why not fromCIDR
+        - host          # this node's kubelet probes
+        - remote-node   # the node-to-node hop that forwards a query to wherever the pod runs
+      toPorts:
+        - ports:
+            - { port: "53", protocol: UDP }
+            - { port: "53", protocol: TCP }

apps/wave-0/pihole.yaml

@@ -35,34 +35,18 @@ spec:
     - protocol: TCP
       port: 53
 ---
-# PVC pihole-config (R-STATE)
+# StatefulSet (R-PLAT-02)
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: pihole-config
-  namespace: pihole
-  labels:
-    app.kubernetes.io/name: pihole
-spec:
-  accessModes:
-  - ReadWriteOnce
-  storageClassName: longhorn
-  resources:
-    requests:
-      storage: 2Gi
----
-# Deployment (R-PLAT-02)
 apiVersion: apps/v1
-kind: Deployment
+kind: StatefulSet
 metadata:
   name: pihole
   namespace: pihole
   labels:
     app.kubernetes.io/name: pihole
 spec:
-  replicas: 1
+  replicas: 2
-  strategy:
+  serviceName: pihole
-    type: Recreate
+  podManagementPolicy: Parallel
   selector:
     matchLabels:
       app.kubernetes.io/name: pihole
@@ -78,32 +62,56 @@ spec:
     spec:
       containers:
       - name: pihole
-        image: pihole/pihole
+        image: pihole/pihole:2026.10.1
         ports:
         - containerPort: 80
+        env:
+        - name: FTLCONF_dns_listeningMode
+          value: ALL
+        - name: FTLCONF_dns_upstreams
+          value: 8.8.8.8;8.8.4.4
+        - name: FTLCONF_dns_rateLimit_count
+          value: '0'
+        - name: FTLCONF_misc_dnsmasq_lines
+          value: local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
+        - name: FTLCONF_webserver_api_password
+          valueFrom:
+            secretKeyRef:
+              name: pihole-bootstrap
+              key: webui-password
         volumeMounts:
         - name: config
-          mountPath: /data/config
+          mountPath: /etc/pihole
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n  echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n  exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n  have=$(sql \"select count(*) from adlist where address='$url';\")\n  if [ \"$have\" = \"0\" ]; then\n    echo \"pihole-adlists: adding $url\"\n    sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n    CHANGED=1\n  else\n    en=$(sql \"select enabled from adlist where address='$url';\")\n    if [ \"$en\" != \"1\" ]; then\n      echo \"pihole-adlists: re-enabling $url\"\n      sql \"update adlist set enabled=1 where address='$url';\"\n      CHANGED=1\n    fi\n  fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n  [ -n \"$addr\" ] || continue\n  if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n    echo \"pihole-adlists: disabling undeclared $addr\"\n    sql \"update adlist set enabled=0 where address='$addr';\"\n    echo changed >> /tmp/adlist-changed\n  fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n  echo \"pihole-adlists: declared set changed - running gravity update\"\n  pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n  echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:80
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://pihole.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-name=_oauth2_proxy_pihole
-        - --whitelist-domain=.home.keithjackson.dev
+        - --cookie-domain=pihole.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +132,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -148,10 +159,39 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
-      volumes:
+      affinity:
-      - name: config
+        podAntiAffinity:
-        persistentVolumeClaim:
+          requiredDuringSchedulingIgnoredDuringExecution:
-          claimName: pihole-config
+          - labelSelector:
+              matchLabels:
+                app.kubernetes.io/name: pihole
+            topologyKey: kubernetes.io/hostname
+  volumeClaimTemplates:
+  - metadata:
+      name: config
+      labels:
+        app.kubernetes.io/name: pihole
+    spec:
+      accessModes:
+      - ReadWriteOnce
+      storageClassName: longhorn
+      resources:
+        requests:
+          storage: 2Gi
+---
+# PodDisruptionBudget — keep one replica through a drain (availability.replicas)
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: pihole
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  minAvailable: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
 ---
 # Service
 apiVersion: v1
@@ -192,6 +232,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-monitoring-scrape
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -223,13 +284,14 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - pihole.home.keithjackson.dev
+  - pihole.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -239,26 +301,6 @@ spec:
     - name: pihole
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: pihole
-  namespace: pihole
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: pihole
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: pihole
-  namespaceSelector:
-    matchNames:
-    - pihole
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -273,11 +315,11 @@ spec:
   - name: pihole.slis
     rules:
     - alert: pihole-down
-      expr: up{job="pihole"} == 0
+      expr: kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"})
       for: 5m
       labels:
         severity: critical
         service: pihole
       annotations:
         summary: pihole is down
-        description: 'pihole has been unreachable for 5m (SLI: up).'
+        description: 'pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'

apps/wave-0/plex-claim-secret.sops.yaml

@@ -0,0 +1,77 @@
+# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
+# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
+#
+# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
+# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
+# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
+# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
+# not prose, so the upstream conditions below are described in words.
+#
+# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
+# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
+# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
+# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
+# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
+#
+# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
+# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
+# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
+# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
+#
+# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
+# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
+# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
+#
+# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
+# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
+# the same file extracted from the running `plexinc/pms-docker:latest` image):
+#   - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
+#     PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
+#     skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
+#     README: "If server is already logged in, this parameter is ignored."
+#   - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
+#     empty body. curl's exit status is never checked, the sed for the authentication-token element
+#     yields the empty string, and the success branch at line 120 simply does not fire. There is no
+#     else branch and no non-zero exit; the script has no `set -e` and ends by touching
+#     /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
+#     It does NOT crashloop.
+#   - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
+#     if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
+# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
+# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
+# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
+# by hand at claim time.
+#
+# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
+#   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+# Never decrypt this file into a shell, a log, or a shared terminal.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: plex-claim
+    namespace: plex
+    labels:
+        app.kubernetes.io/name: plex
+type: Opaque
+stringData:
+    PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
+            cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
+            THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
+            QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
+            RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-29T06:31:16Z"
+    mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/plex-remote.yaml

@@ -0,0 +1,126 @@
+---
+# =================================================================================================
+# PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
+# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
+# mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
+# schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
+# type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
+# `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
+#
+# WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
+# Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
+# returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
+# (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
+#
+# The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
+# `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
+# list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
+# publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
+# 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
+# `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
+# publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
+# file) over advertising the Gateway URL: Plex then terminates its own TLS with its
+# `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
+#
+# VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
+# CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
+# this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
+# port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
+# 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
+# IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
+# .22 (crafty-l4-*) and .23 (pihole-dns-*).
+#
+# ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
+# lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
+# DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
+#
+# NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
+# its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
+# resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
+# clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
+# to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
+# without a token), which is what makes an unfiltered port defensible here.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
+# Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
+# address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
+# from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
+# HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
+# through the Gateway exactly as before.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the tempting choice because it preserves the client source IP, which this service
+# actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
+# reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
+# holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
+# by a node that has no Plex pod, which black-holes the port while every object still reports
+# healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
+# scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
+#
+# KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
+# to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
+# catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
+# Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
+# `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
+# authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
+# on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
+# without also de-classifying genuine LAN clients, so it is accepted.
+apiVersion: v1
+kind: Service
+metadata:
+  name: plex-l4-tcp
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+  annotations:
+    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  selector:
+    app.kubernetes.io/name: plex
+  ports:
+    - name: plex-remote
+      port: 32400
+      targetPort: 32400
+      protocol: TCP
+---
+# INGRESS allow for the raw port (R-NET-01/02).
+#
+# The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
+# 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
+# this file adds the off-cluster case without touching those.
+#
+# `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
+# `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
+# this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
+# NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
+# `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
+# four entities are named.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-remote-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+    - fromEntities:
+        - world
+        - cluster
+        - host
+        - remote-node
+      toPorts:
+        - ports:
+            - { port: "32400", protocol: TCP }

apps/wave-0/plex.yaml

@@ -35,6 +35,34 @@ spec:
     - protocol: TCP
       port: 53
 ---
+# PersistentVolume plex-media - static external NFS (bulk tier)
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: plex-media
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  capacity:
+    storage: 9Ti
+  accessModes:
+  - ReadOnlyMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ''
+  mountOptions:
+  - nfsvers=4.1
+  - hard
+  - noatime
+  - timeo=600
+  - retrans=2
+  nfs:
+    server: ${BULK_NFS_SERVER}
+    path: /mnt/Storage/plex
+    readOnly: true
+  claimRef:
+    namespace: plex
+    name: plex-media
+---
 # PVC plex-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -49,7 +77,24 @@ spec:
   storageClassName: longhorn
   resources:
     requests:
-      storage: 8Gi
+      storage: 16Gi
+---
+# PVC plex-media (R-STATE)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: plex-media
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  accessModes:
+  - ReadOnlyMany
+  storageClassName: ''
+  volumeName: plex-media
+  resources:
+    requests:
+      storage: 9Ti
 ---
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
@@ -75,7 +120,7 @@ spec:
     spec:
       containers:
       - name: plex
-        image: plexinc/pms-docker:latest
+        image: plexinc/pms-docker:1.43.6.11085-d1e52e169
         ports:
         - containerPort: 32400
         env:
@@ -87,13 +132,49 @@ spec:
           value: Etc/UTC
         - name: ALLOWED_NETWORKS
           value: 10.0.0.0/8,192.168.0.0/16
+        - name: PLEX_CLAIM
+          valueFrom:
+            secretKeyRef:
+              name: plex-claim
+              key: PLEX_CLAIM
+        - name: ADVERTISE_IP
+          value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
+        - name: PLEX_TLS_CRT
+          valueFrom:
+            secretKeyRef:
+              name: plex-vip-tls
+              key: tls.crt
+              optional: true
+        - name: PLEX_TLS_KEY
+          valueFrom:
+            secretKeyRef:
+              name: plex-vip-tls
+              key: tls.key
+              optional: true
+        resources:
+          limits:
+            gpu.intel.com/i915: 1
         volumeMounts:
         - name: config
           mountPath: /config
+        - name: media
+          mountPath: /media
+          readOnly: true
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# NOTE \u2014 LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.\n# This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a\n# brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD\n# Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be\n# replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` \u2014 bare. Only\n# BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be\n# substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell\n# variable written with braces would be blanked or would fail the whole Kustomization \u2014\n# this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and\n# catalog/render.py now refuses to emit it (which is how this very comment got caught).\n# DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the\n# certificate has not issued yet they are genuinely UNSET, not empty \u2014 and under `set -u`\n# the first `[ -n \"$PLEX_TLS_CRT\" ]` would abort the script, which for a postStart hook\n# means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists\n# to prevent, so turning it back on here would defeat the whole design. (The alternative,\n# a brace-wrapped `:-` default, would need envsubst escaping \u2014 the trap the note above\n# describes.) Every value below is explicitly guarded instead.\nPX=\"/config/Library/Application Support/Plex Media Server/Preferences.xml\"\nCERTDIR=/config/certs\nP12=$CERTDIR/plex-vip.p12\n# The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it\n# in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a\n# container-format requirement, not an access control.\nP12PASS=plexvip\nLANNETS='10.0.0.0/8,192.168.0.0/16'\nCERTDOMAIN=plex-vip.${BASE_DOMAIN}\nAPI=http://127.0.0.1:32400\n\n# (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so\n# a cert-manager RENEWAL is picked up with no extra moving part.\nif [ -n \"$PLEX_TLS_CRT\" ] && [ -n \"$PLEX_TLS_KEY\" ]; then\n  mkdir -p $CERTDIR && chmod 700 $CERTDIR\n  printf '%s' \"$PLEX_TLS_CRT\" > $CERTDIR/tls.crt\n  printf '%s' \"$PLEX_TLS_KEY\" > $CERTDIR/tls.key\n  chmod 600 $CERTDIR/tls.key\n  if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \\\n       -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then\n    mv -f $P12.tmp $P12\n    chown 1000:1000 $P12 2>/dev/null || true\n    chmod 600 $P12\n    echo \"[plex-settings] PKCS#12 bundle written\"\n  else\n    rm -f $P12.tmp\n    echo \"[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle\" >&2\n  fi\n  rm -f $CERTDIR/tls.key\nelse\n  echo \"[plex-settings] WARN: TLS env empty; certificate not installed this start\" >&2\nfi\n\n# (2) Wait, bounded (~90s), for the local API. /identity needs no auth.\ni=0\nwhile [ \"$i\" -lt 45 ]; do\n  if [ \"$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)\" = \"200\" ]; then\n    break\n  fi\n  i=$(expr $i + 1); sleep 2\ndone\n\n# (3) The prefs API needs the server token. It already lives in Preferences.xml on this\n# PVC; it is never echoed here, and Plex redacts tokens in its own request log. An\n# UNCLAIMED server has none \u2014 a normal state on a fresh /config \u2014 so skip, do not fail.\nTOKEN=$(grep -o 'PlexOnlineToken=\"[^\"]*\"' \"$PX\" 2>/dev/null | head -1 | cut -d'\"' -f2)\nif [ -z \"$TOKEN\" ]; then\n  echo \"[plex-settings] not claimed yet \u2014 skipping prefs, will retry next start\" >&2\n  exit 0\nfi\n\n# Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on\n# a PUT, so a raw '/' or ',' would be read as structure rather than data.\nenc() {\n  printf '%s' \"$1\" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \\\n                         -e 's|:|%3A|g' -e 's| |%20|g'\n}\nsetpref() {\n  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT\
+                \ \\\n    -H \"X-Plex-Token: $TOKEN\" \"$API/:/prefs?$1=$(enc \"$2\")\" 2>/dev/null)\n  if [ \"$code\" = \"200\" ]; then\n    echo \"[plex-settings] set $1\"\n  else\n    echo \"[plex-settings] WARN: $1 -> HTTP $code\" >&2\n  fi\n}\n\n# (4) THE LAN DEFINITION \u2014 the reason this hook was written. Without it Plex has NO notion\n# of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests\n# logged \"Allowed Network (WAN)\" and ZERO \"(LAN)\", including Seerr from inside the\n# cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks\n# (the ALLOWED_NETWORKS env) only waives AUTH \u2014 it does NOT make a client local. Two\n# different settings; only one of them has an env hook.\nsetpref LanNetworksBandwidth \"$LANNETS\"\n\n# (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only\n# if the bundle exists \u2014 pointing Plex at a missing file would break its TLS listener.\nif [ -s \"$P12\" ]; then\n  setpref customCertificatePath \"$P12\"\n  setpref customCertificateKey \"$P12PASS\"\n  setpref customCertificateDomain \"$CERTDOMAIN\"\nelse\n  echo \"[plex-settings] WARN: no bundle; leaving certificate prefs untouched\" >&2\nfi\nexit 0\n"
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: plex-config
+      - name: media
+        persistentVolumeClaim:
+          claimName: plex-media
 ---
 # Service
 apiVersion: v1
@@ -111,6 +192,28 @@ spec:
     port: 32400
     targetPort: 32400
 ---
+# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-radarr-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-radarr-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: radarr
+        k8s:io.kubernetes.pod.namespace: radarr
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-seerr-32400
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -133,6 +236,28 @@ spec:
       - port: '32400'
         protocol: TCP
 ---
+# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-sonarr-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-sonarr-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: sonarr
+        k8s:io.kubernetes.pod.namespace: sonarr
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): plex-allow-gateway-ingress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -177,13 +302,15 @@ metadata:
   namespace: plex
   labels:
     app.kubernetes.io/name: plex
+  annotations:
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - plex.home.keithjackson.dev
+  - plex.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -193,26 +320,6 @@ spec:
     - name: plex
       port: 32400
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: plex
-  namespace: plex
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: plex
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: plex
-  namespaceSelector:
-    matchNames:
-    - plex
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -227,11 +334,11 @@ spec:
   - name: plex.slis
     rules:
     - alert: plex-down
-      expr: up{job="plex"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})
       for: 5m
       labels:
         severity: critical
         service: plex
       annotations:
         summary: plex is down
-        description: 'plex has been unreachable for 5m (SLI: up).'
+        description: 'plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml

@@ -0,0 +1,60 @@
+# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux.
+#
+# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13):
+#   qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR
+#   `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW
+#   random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
+#   password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's
+#   saved download-client credentials stop working at the next qbittorrent pod restart, silently
+#   breaking Completed Download Handling; (b) a human operator has no stable login at all and must
+#   read the pod log after every restart.
+#
+# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env
+#   (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as
+#   gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores
+#   and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then
+#   verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore
+#   from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME
+#   credential, because the hook re-asserts it on every start rather than once.
+#
+# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is
+#   the name its own temporary-password path uses, so keeping it avoids inventing a second identity.
+# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client
+#   entries store in their own databases (app state on their config PVCs, NOT in git — see
+#   DECISIONS-LOG), so rotating it here means re-pointing both arrs.
+#   Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON
+#   payloads, and a shell/URL-special character in any of those is a bug waiting to happen.
+#
+# NEVER put a value in a comment: SOPS does NOT encrypt comments.
+# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml
+apiVersion: v1
+kind: Secret
+metadata:
+    name: qbittorrent-bootstrap
+    namespace: qbittorrent
+    labels:
+        app.kubernetes.io/name: qbittorrent
+type: Opaque
+stringData:
+    webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str]
+    webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2
+            TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3
+            UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv
+            VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv
+            FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-13T05:23:23Z"
+    mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/qbittorrent.yaml

@@ -35,6 +35,33 @@ spec:
     - protocol: TCP
       port: 53
 ---
+# PersistentVolume qbittorrent-downloads - static external NFS (bulk tier)
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: qbittorrent-downloads
+  labels:
+    app.kubernetes.io/name: qbittorrent
+spec:
+  capacity:
+    storage: 9Ti
+  accessModes:
+  - ReadWriteMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ''
+  mountOptions:
+  - nfsvers=4.1
+  - hard
+  - noatime
+  - timeo=600
+  - retrans=2
+  nfs:
+    server: ${BULK_NFS_SERVER}
+    path: /mnt/Storage/plex
+  claimRef:
+    namespace: qbittorrent
+    name: qbittorrent-downloads
+---
 # PVC qbittorrent-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -51,6 +78,23 @@ spec:
     requests:
       storage: 5Gi
 ---
+# PVC qbittorrent-downloads (R-STATE)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: qbittorrent-downloads
+  namespace: qbittorrent
+  labels:
+    app.kubernetes.io/name: qbittorrent
+spec:
+  accessModes:
+  - ReadWriteMany
+  storageClassName: ''
+  volumeName: qbittorrent-downloads
+  resources:
+    requests:
+      storage: 9Ti
+---
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -74,36 +118,67 @@ spec:
         homelab.keithjackson.dev/auth-mode: forward-auth
       annotations:
         homelab.keithjackson.dev/auth-mode: forward-auth
-        homelab.keithjackson.dev/auth-groups: media-users
+        homelab.keithjackson.dev/auth-groups: media-users,platform-admins
     spec:
       containers:
       - name: qbittorrent
-        image: lscr.io/linuxserver/qbittorrent:5.1.2
+        image: lscr.io/linuxserver/qbittorrent:5.3.0
         ports:
         - containerPort: 8080
+        env:
+        - name: PUID
+          value: '1000'
+        - name: PGID
+          value: '3000'
+        - name: QBT_WEBUI_USERNAME
+          valueFrom:
+            secretKeyRef:
+              name: qbittorrent-bootstrap
+              key: webui-username
+        - name: QBT_WEBUI_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: qbittorrent-bootstrap
+              key: webui-password
         volumeMounts:
         - name: config
-          mountPath: /data/config
+          mountPath: /config
+        - name: downloads
+          mountPath: /downloads
+          subPath: downloads
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n  echo \"[qbt-webui-cred] $*\"\n  printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n    >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n#     before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n#     racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n  attempt=$((attempt + 1))\n\n  # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n  #     10 = rewritten, anything else = could not write.\n  python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n    sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n    raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n#   WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
+                , \"[Preferences]\"\n\ndef matches(raw):\n    \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n    v = raw.strip().strip('\"')\n    if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n        v = v[len(\"@ByteArray(\"):-1]\n    try:\n        salt_b64, digest_b64 = v.split(\":\", 1)\n        salt = base64.b64decode(salt_b64, validate=True)\n        digest = base64.b64decode(digest_b64, validate=True)\n    except Exception:\n        return False\n    if not salt or not digest:\n        return False\n    return hmac.compare_digest(\n        hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n    lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n    lines = []\n\ndef value_of(key):\n    for i, line in enumerate(lines):\n        if line.startswith(key + \"=\"):\n            return i, line.split(\"=\", 1)[1]\n    return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n    raise SystemExit(0)          # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n    at = out.index(SECTION) + 1\nelse:\n    if out and out[-1].strip():\n        out.append(\"\")\n    out.append(SECTION)\n    at = len(out)\nout[at:at] = desired\n\ntry:\n    os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n    tmp = conf + \".webui-cred.new\"\n    with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n        fh.write(\"\\n\".join(out) + \"\\n\")\n        fh.flush()\n        os.fsync(fh.fileno())\n    # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n    # cost the app the ability to save its own settings, so carry the ownership over.\n    try:\n        st = os.stat(conf)\n        os.chown(tmp, st.st_uid, st.st_gid)\n        os.chmod(tmp, stat.S_IMODE(st.st_mode))\n    except FileNotFoundError:\n        os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n                 int(os.environ.get(\"PGID\", \"1000\")))\n        os.chmod(tmp, 0o664)\n    os.replace(tmp, conf)        # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n    sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n    raise SystemExit(1)\nraise SystemExit(10)\nPY\n  rc=$?\n\n  if [ \"$rc\" = 0 ]; then\n    log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n  elif [ \"$rc\" = 10 ]; then\n    log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n    # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n    # have read the old, credential-less conf and be serving a random temporary password.\n    # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n    # Nothing to do when the service has not started yet - it will read the new file.\n    # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n    # 2026-09-13 failure spent the entire verification budget before the first login and\n    # then fired attempts 2 and 3 with no wait left at all.\n    sv=0\n    while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n      sv=$((sv + 1)); sleep 1\n    done\n    if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n      log \"qbittorrent-nox was already running - restarting it to load the credential\"\n      s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n      sleep 3\n    fi\n  else\n    log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n    log \"minting a random password\
+                \ every start and BOTH arrs will break. Fix and restart.\" >&2\n    exit 0\n  fi\n\n  # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n  #     (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n  #     at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n  #\n  # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n  # conflating them is what made this hook lie once already (see the PORT comment). An\n  # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n  # its remaining login attempts to produce two more identical wrong diagnoses.\n  api_ok=0\n  while [ \"$waited\" -lt 120 ]; do\n    code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n    if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n    waited=$((waited + 1)); sleep 1\n  done\n  if [ \"$api_ok\" != 1 ]; then\n    log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n    log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n    log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n    exit 0\n  fi\n  body=$(curl -sS -m 10 \\\n           --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n           --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n           \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n  if [ \"$body\" = \"Ok.\" ]; then\n    log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n    exit 0\n  fi\n  log \"WebUI rejected the durable credential (login said '$body')\" >&2\n  # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n  # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n  # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n  # make the process RE-READ the file, so restart the supervised service and let the next\n  # attempt re-verify. Without this the retry loop could never change its own outcome.\n  if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n    log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n    s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n    sleep 3\n  fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8080
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
+        - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://qbittorrent.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-name=_oauth2_proxy_qbittorrent
-        - --whitelist-domain=.home.keithjackson.dev
+        - --cookie-domain=qbittorrent.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +199,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -148,10 +226,15 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
+      securityContext:
+        fsGroup: 3000
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: qbittorrent-config
+      - name: downloads
+        persistentVolumeClaim:
+          claimName: qbittorrent-downloads
 ---
 # Service
 apiVersion: v1
@@ -236,6 +319,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): qbittorrent-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: qbittorrent-allow-monitoring-scrape
+  namespace: qbittorrent
+  labels:
+    app.kubernetes.io/name: qbittorrent
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: qbittorrent
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -266,14 +370,15 @@ metadata:
     app.kubernetes.io/name: qbittorrent
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
-    homelab.keithjackson.dev/auth-groups: media-users
+    homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - qbittorrent.home.keithjackson.dev
+  - qbittorrent.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -283,26 +388,6 @@ spec:
     - name: qbittorrent
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: qbittorrent
-  namespace: qbittorrent
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: qbittorrent
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: qbittorrent
-  namespaceSelector:
-    matchNames:
-    - qbittorrent
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -317,11 +402,11 @@ spec:
   - name: qbittorrent.slis
     rules:
     - alert: qbittorrent-down
-      expr: up{job="qbittorrent"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"} == 0 or absent(kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"})
       for: 5m
       labels:
         severity: critical
         service: qbittorrent
       annotations:
         summary: qbittorrent is down
-        description: 'qbittorrent has been unreachable for 5m (SLI: up).'
+        description: 'qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/semaphore-db-backup-credentials-prod.sops.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: semaphore-db-backup-credentials-prod
+    namespace: semaphore-db
+type: Opaque
+stringData:
+    ACCESS_KEY_ID: ENC[AES256_GCM,data:OhWEEYqrvmDB40NmiETgeUisBvaO9ocevCZAsurNijw=,iv:kEjOIg9NHbRYm+Ul1WrBAzb3gLbIv+ipiYCxFK0QoVc=,tag:KAB1JAIw0ATwbjDEKZVT9A==,type:str]
+    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:C0PNKv07LCYpFONLqd55sz+ioyU/i2YJUkbZ8OX/dCErM/DilqchyUAT5EqnT/r3YJmdaA1pX85S3fxlVVHkbg==,iv:4ha28b+xIebvX2JPITmtTovIe4vdmj2XfI3cgkUHH4c=,tag:xbMyflcHojytPzgaoOYcXQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZVTJ3cXR6cks3Y0RNS2hx
+            K3pjZEduYUxmL0V3SmFUcU9XbGVHSVFITWh3Cjl1eHlTLytoOW5PaVNFVkJoeTBo
+            T1FnRHNDVzJ6dWpuR09WaElvbWF4YU0KLS0tIEJoQ1VsM1cxRlg0R1dtNGdOenNn
+            aVNjMjVBTytLdHlFdmZCS2tPelo1Y2MKYt1G5t+sDiHgcg+Q9ECBzJn2gS/paSEC
+            7aoX60SBrPgHvnGls5lzc3Tin9nH4jWYl6Nu8DAvIE0YPXDFr5a0UA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-07T06:36:33Z"
+    mac: ENC[AES256_GCM,data:mhVP7tjdZK2FR2imvpJbTyT7BigvcIKEpLSnBS1oSITUirB8xZC7aHBfyFuIoKOrpOJgEe40nHUdDYCHRJtb4dLeVja8+7meVaQWzxC8BWVHsksW+73hsqhjAzlHeobx6Irbk4PAMqIY0/jnf4MHryjnRn3dUuwu9U1CqL30JbE=,iv:57oCcp6FW9EAIVKnYGtkaTrnbijyv54T+vmZxMbMqnw=,tag:agrRP/JLypU60KOeN0s/5w==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/semaphore-db.yaml

@@ -38,16 +38,53 @@ metadata:
   name: semaphore-db
   namespace: semaphore-db
 spec:
-  # Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
+  # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db /
-  # keycloak-db / nextcloud-db). The storage is already 3-way replicated by Longhorn; raise to 3 for
+  # nextcloud-db; see platform/keycloak/postgres-cluster.yaml for the full rationale. Short version:
-  # Postgres-level HA as a future hardening.
+  # Longhorn's 3-way replication saves the DATA on node loss, not the SERVICE, and with
-  instances: 1
+  # `instances: 1` all three CNPG primaries had drifted onto k3s-prod-2, so one node took Semaphore
-  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
+  # down alongside SSO and Nextcloud. A streaming standby lets CNPG promote in seconds and repoint
+  # the -rw Service. Gated on the Barman archive below actually working (ContinuousArchiving=True
+  # with a real firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
+  #
+  # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
+  # pg_basebackup from the running primary, which is neither restarted nor disconnected.
+  instances: 3
+  imageName: ghcr.io/cloudnative-pg/postgresql:16.15
   # Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
   # semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
   inheritedMetadata:
     labels:
       app.kubernetes.io/name: semaphore-db
+  # HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
+  # primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
+  # recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
+  # version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
+  # after a node loss it may stack two instances on one node — which is how `instances: 3` decays
+  # back into the single-node failure domain this change removes. `required` makes it a constraint;
+  # the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
+  # serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
+  #
+  # APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
+  # switchover. Expect a single brief connection reset for Semaphore.
+  affinity:
+    enablePodAntiAffinity: true
+    podAntiAffinityType: required
+  primaryUpdateMethod: switchover
+  # SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
+  # rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
+  # last few transactions when a primary dies before its standbys catch up, and `method: any` +
+  # `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
+  # never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
+  # are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
+  # standby into a total write outage. `preferred` relaxes to async instead.
+  #
+  # NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
+  # no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
+  postgresql:
+    synchronous:
+      method: any
+      number: 1
+      dataDurability: preferred
   storage:
     storageClass: longhorn
     size: 5Gi
@@ -69,25 +106,38 @@ spec:
       memory: 256Mi
     limits:
       memory: 512Mi
-  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
+  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
   # Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
-  # backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
+  # backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
-  # (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP.
+  # destinationPath, so objects land under ${CNPG_BACKUP_PATH}/semaphore-db/ — dev
-  # Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
+  # s3://cnpg/semaphore-db/ on the on-Proxmox MinIO, prod
-  # (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
+  # s3://k3s-prod-backup/barman/semaphore-db/ on Cloudflare R2. Both endpointURL and
-  # endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
+  # destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
-  # substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
+  # Credentials: PER-ENVIRONMENT (2026-09-07). dev reads semaphore-db-backup-credentials (the
+  # scoped MinIO key), prod reads semaphore-db-backup-credentials-prod (the Cloudflare R2
+  # token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
+  # generated golden — render.py places the -prod sidecar in this DB's wave); the
+  # cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
+  # one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
+  # ONE shared Secret until the split, so giving prod its R2 token would have left dev
+  # signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
+  # disk-full clock below rather than a merely missing backup.
+  #
+  # archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
+  # Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
+  # the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
+  # endpoint. Treat ContinuousArchiving=False here as an availability incident.
   backup:
     retentionPolicy: "30d"
     barmanObjectStore:
-      destinationPath: s3://cnpg
+      destinationPath: ${CNPG_BACKUP_PATH}
       endpointURL: ${S3_BACKUP_ENDPOINT}
       s3Credentials:
         accessKeyId:
-          name: semaphore-db-backup-credentials
+          name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
           key: ACCESS_KEY_ID
         secretAccessKey:
-          name: semaphore-db-backup-credentials
+          name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
           key: ACCESS_SECRET_KEY
       wal:
         compression: gzip

apps/wave-0/tests/app-slis.test.yaml

@@ -0,0 +1,269 @@
+# promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
+# Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
+# qbittorrent (container/Deployment), pihole (container/StatefulSet — `availability.replicas: 2`, so
+# its SLI reads kube_statefulset_status_replicas_ready; see schemas/renderer-contract.md
+# `### availability`) and vm-demo (KubeVirt VMI). The harness discovers this file by
+# the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
+# tests run against the exact expressions Flux applies, never a copy.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHY THESE EXIST. Until 2026-09-13 every one of these rules read `up{job="<app>"} == 0` while the
+# rendered ServiceMonitor named a Service port (`metrics`) that no rendered Service defined. The
+# monitor discovered ZERO targets, so no `up{job="<app>"}` series existed and the rule could not
+# fire — on prod, `count by (job) (up)` listed 24 jobs and not one of the twelve appeared.
+# `PrometheusTargetsMissing` could not see it either: a target that was never discovered has no `up`
+# series to be 0. The rules now read kube-state-metrics / kubevirt series that DO exist for these
+# workloads (verified on prod), and every rule below is pinned in BOTH directions:
+#   * QUIET on the healthy state -> it is not permanently firing;
+#   * FIRING when the workload has no ready replica -> it is not inert;
+#   * FIRING when the signal is ABSENT -> the failure mode that produced this whole defect class
+#     (a rule that goes quiet because its series vanished) is itself alerted on.
+#
+# CLOCK/`for` CONVENTION: sample interval 1m, series 21 samples long, evaluated at 10m — comfortably
+# past every rule's `for: 5m` and well inside Prometheus' 5m lookback.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # ===========================================================================================
+  # 1. HEALTHY WAVE-0 (what prod looks like right now) -> every rule SILENT.
+  #    This is the "not permanently firing" half. It is also the test that catches a mis-spelled
+  #    label: get `deployment=` wrong and the `== 0` half matches nothing while the `absent()` half
+  #    matches everything, so this test goes red instead of the rule going quietly wrong.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 healthy - every app has a ready replica, the demo VM runs - all quiet
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running", node="k3s-prod-1"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 2. EACH APP DOWN IN TURN (0 available replicas) -> ONLY that app's rule fires.
+  #    The per-app isolation matters: a rule that forgot its `namespace=`/`deployment=` matchers
+  #    would light up every app at once and be useless for routing.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 plex has no available replica - only plex-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: plex
+              namespace: plex
+              deployment: plex
+            exp_annotations:
+              summary: "plex is down"
+              description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+    # Before `for: 5m` has elapsed the rule must be PENDING, not firing — proof the 5m is real and
+    # a single scrape blip does not page anyone.
+    promql_expr_test:
+      - expr: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})'
+        eval_time: 2m
+        exp_samples:
+          - labels: 'kube_deployment_status_replicas_available{deployment="plex", namespace="plex"}'
+            value: 0
+
+  - interval: 1m
+    name: wave-0 gitea/jackett/pihole/qbittorrent each down - each own rule fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '0+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '0+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: gitea
+              namespace: gitea
+              deployment: gitea
+            exp_annotations:
+              summary: "gitea is down"
+              description: "gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: jackett
+              namespace: jackett
+              deployment: jackett
+            exp_annotations:
+              summary: "jackett is down"
+              description: "jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: pihole
+              namespace: pihole
+              statefulset: pihole
+            exp_annotations:
+              summary: "pihole is down"
+              description: "pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: qbittorrent
+              namespace: qbittorrent
+              deployment: qbittorrent
+            exp_annotations:
+              summary: "qbittorrent is down"
+              description: "qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 3. THE SIGNAL VANISHES -> the rule FIRES. This is the defect class itself, asserted on.
+  #    The old `up{job=...} == 0` rule had exactly this shape of hole and was silent for it. Here
+  #    the Deployment series for plex is simply not in the input at all, and plex-down must still
+  #    fire (via the `absent()` half), with the labels the selector's equality matchers carry.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 plex Deployment series ABSENT (never scraped / workload gone) - plex-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: plex
+              namespace: plex
+              deployment: plex
+            exp_annotations:
+              summary: "plex is down"
+              description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 4. THE VM: a VMI carries no replica count, so liveness is the EXISTENCE of a running-phase
+  #    series. Two ways to be down, both asserted: the VMI is gone entirely, and the VMI exists in
+  #    some OTHER phase. The second is the one a naive `kubevirt_vmi_info == 1` would miss, and it
+  #    is why the phase value is matched — LOWERCASE `running`, the value KubeVirt actually writes
+  #    (a capital-R matcher selects nothing and would be just as inert).
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 demo VM has no running-phase VMI - vm-demo-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: vm-demo
+              namespace: vm-demo
+              name: vm-demo
+              phase: running
+            exp_annotations:
+              summary: "vm-demo is down"
+              description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."
+
+  - interval: 1m
+    name: wave-0 demo VM exists but is Failed, not running - vm-demo-down still fires
+    input_series:
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="failed"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: vm-demo
+              namespace: vm-demo
+              name: vm-demo
+              phase: running
+            exp_annotations:
+              summary: "vm-demo is down"
+              description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."

apps/wave-0/vm-demo.yaml

@@ -0,0 +1,139 @@
+---
+# Namespace (trust zone; default-deny + policies apply)
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: vm-demo
+  labels:
+    homelab.keithjackson.dev/app-namespace: 'true'
+  annotations:
+    homelab.keithjackson.dev/backup-policy: daily-offcluster
+---
+# Default-deny baseline (R-NET-01): deny all, allow only DNS
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: default-deny
+  namespace: vm-demo
+  labels:
+    homelab.keithjackson.dev/baseline: default-deny
+spec:
+  podSelector: {}
+  policyTypes:
+  - Ingress
+  - Egress
+  ingress: []
+  egress:
+  - to:
+    - namespaceSelector:
+        matchLabels:
+          kubernetes.io/metadata.name: kube-system
+    ports:
+    - protocol: UDP
+      port: 53
+    - protocol: TCP
+      port: 53
+---
+# CDI DataVolume (standalone; survives rebuild, R-STATE-01)
+apiVersion: cdi.kubevirt.io/v1beta1
+kind: DataVolume
+metadata:
+  name: vm-demo-rootdisk
+  namespace: vm-demo
+spec:
+  source:
+    registry:
+      url: docker://quay.io/containerdisks/fedora:41
+  storage:
+    accessModes:
+    - ReadWriteOnce
+    storageClassName: longhorn
+    resources:
+      requests:
+        storage: 6Gi
+---
+# KubeVirt VirtualMachine (R-PLAT-03)
+apiVersion: kubevirt.io/v1
+kind: VirtualMachine
+metadata:
+  name: vm-demo
+  namespace: vm-demo
+  labels:
+    app.kubernetes.io/name: vm-demo
+spec:
+  running: false
+  template:
+    metadata:
+      labels:
+        kubevirt.io/vm: vm-demo
+        homelab.keithjackson.dev/egress-class: internet
+        homelab.keithjackson.dev/auth-mode: none
+    spec:
+      domain:
+        cpu:
+          cores: 1
+        memory:
+          guest: 1536Mi
+        devices:
+          disks:
+          - name: rootdisk
+            disk:
+              bus: virtio
+          - name: cloudinit
+            disk:
+              bus: virtio
+          interfaces:
+          - name: default
+            masquerade: {}
+        resources:
+          requests:
+            memory: 1536Mi
+      networks:
+      - name: default
+        pod: {}
+      volumes:
+      - name: rootdisk
+        dataVolume:
+          name: vm-demo-rootdisk
+      - name: cloudinit
+        cloudInitNoCloud:
+          userData: "#cloud-config\npackage_update: true\npackages:\n  - dnf-automatic\nwrite_files:\n  - path: /etc/dnf/automatic.conf\n    content: |\n      [commands]\n      upgrade_type = default\n      random_sleep = 0\n      download_updates = yes\n      apply_updates = yes\nruncmd:\n  - [ systemctl, enable, --now, dnf-automatic.timer ]\n"
+---
+# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): vm-demo-allow-internet-egress
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: vm-demo-allow-internet-egress
+  namespace: vm-demo
+  labels:
+    app.kubernetes.io/name: vm-demo
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: vm-demo
+  egress:
+  - toEntities:
+    - world
+---
+# PrometheusRule SLI alerts (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: vm-demo-slis
+  namespace: vm-demo
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: vm-demo
+spec:
+  groups:
+  - name: vm-demo.slis
+    rules:
+    - alert: vm-demo-down
+      expr: absent(kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"})
+      for: 5m
+      labels:
+        severity: critical
+        service: vm-demo
+      annotations:
+        summary: vm-demo is down
+        description: 'vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info).'

apps/wave-1/gitea-actions-runner-secret.sops.yaml

@@ -4,7 +4,13 @@
 # against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
 # decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
 # in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
-# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
+# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
+# Gitea API and writes it here via `sops set`, without the value ever being printed.
+# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
+# it still decrypts, redirect: `sops -d <file> > /dev/null`.
+# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
+# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
+# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
 apiVersion: v1
 kind: Secret
 metadata:
@@ -14,7 +20,7 @@ metadata:
         app.kubernetes.io/name: act-runner
 type: Opaque
 stringData:
-    token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
+    token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -30,8 +36,8 @@ sops:
             MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
             UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-08-26T04:48:14Z"
+    lastmodified: "2026-09-29T05:48:35Z"
-    mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
+    mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

apps/wave-1/gitea-actions.yaml

@@ -77,6 +77,42 @@ spec:
     requests:
       storage: 1Gi
 ---
+# act_runner config file — DELTAS ONLY. act_runner v0.6.1 LoadDefault()
+# (internal/pkg/config/config.go) unmarshals this file onto a ZERO Config and then fills every
+# unset field with the same default it uses when there is no file at all, so a file holding
+# only `runner.capacity` changes exactly that and nothing else. Do NOT paste in the output of
+# `act_runner generate-config`: that example is NOT the defaults. Its `runner.labels` would
+# OVERRIDE the registered labels in /data/.runner (daemon.go: config labels win when non-empty)
+# and repoint ubuntu-latest at a different job image; its `force_pull: true` differs from the
+# no-file default (false); and its sample `envs` would be injected into every job.
+#
+# Using a file also makes act_runner IGNORE the legacy GITEA_RUNNER_CAPACITY / _FILE /
+# _ENVIRON / _ENV_FILE / GITEA_DEBUG / GITEA_TRACE env vars (deprecated.go). None of them is
+# set on this Deployment; the GITEA_INSTANCE_URL / _REGISTRATION_TOKEN / _NAME / _LABELS env
+# below is consumed by the image's run.sh for registration, not by that legacy path.
+#
+# A ConfigMap edit alone does NOT restart the runner (it reads the file once at start). If a
+# future change touches ONLY this ConfigMap, also bump the Deployment's pod template (e.g. the
+# config-revision annotation below) or the new value stays inert until the next restart.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: act-runner-config
+  namespace: gitea-actions
+  labels:
+    app.kubernetes.io/name: act-runner
+data:
+  config.yaml: |
+    runner:
+      # Two concurrent jobs (default 1). Measured 2026-11-08: queue wait ~1s normally, but
+      # 29-759s whenever gate / images / renovate runs collide (run 472 waited 759s behind a
+      # renovate burst). Both jobs share the ONE dind daemon below; act_runner names each job's
+      # containers GITEA-ACTIONS-TASK-<task id> and creates a network per job, so they do not
+      # collide. Workflow `concurrency:` groups are enforced by Gitea server-side and are
+      # unaffected: gate.yml's gate-<ref> group still cancels a superseded run on the SAME ref,
+      # and renovate's group still serialises renovate. The dind resources are sized for TWO jobs.
+      capacity: 2
+---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -95,77 +131,101 @@ spec:
     metadata:
       labels:
         app.kubernetes.io/name: act-runner
+      annotations:
+        # Bump when act-runner-config changes without any other pod-template change, so the
+        # Deployment rolls and act_runner re-reads its config (see the ConfigMap header).
+        homelab.keithjackson.dev/act-runner-config-revision: "1"   # capacity 2
     spec:
       # CI holds no cluster credentials — do not mount a SA token into the runner or
       # (via the shared netns) the job containers (R-SEC-02, docs/14).
       automountServiceAccountToken: false
+      # ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
+      #
+      # NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
+      # `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
+      # running alongside them, and is terminated after them. kubelet holds `runner` until
+      # this container's startupProbe passes, so the ordering is enforced by the kubelet
+      # instead of hand-rolled shell in the runner.
+      #
+      # This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
+      # Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
+      # postBuild substitution over this ENTIRE manifest, comments included, and those loops
+      # depended on bare dollar-names surviving it untouched. There is now no shell here at
+      # all, so there is nothing to escape or mis-escape.
+      #
+      # WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
+      # re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
+      # and only THEN begins listening, so the port can accept a connection while the certs
+      # are absent or incomplete -- and act_runner hard-exits on a missing
+      # /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
+      # are usable and that the daemon answers.
+      #
+      # CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
+      # introduced this claimed the conversion would REMOVE the one-per-start dockerd line
+      # `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
+      # appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
+      # attempt, landing microseconds after dockerd begins listening and cut off by
+      # timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
+      # was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
+      # startup line would slow every subsequent poll. Left documented rather than silenced.
+      #
+      # periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
+      initContainers:
+        - name: dind
+          restartPolicy: Always   # <- this field is what makes it a SIDECAR
+          image: docker:29-dind
+          securityContext:
+            privileged: true
+          env:
+            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
+            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
+            - name: DOCKER_TLS_CERTDIR
+              value: /certs
+          volumeMounts:
+            - name: certs
+              mountPath: /certs
+            - name: dind-storage
+              mountPath: /var/lib/docker
+          # Sized for runner.capacity: 2. Every job's containers run UNDER this daemon, so this
+          # container's cgroup carries the whole CI workload. Measured 2026-11-08 (Prometheus,
+          # capacity 1): peak CPU 0.69 core (1m rate, 30d), peak working set 447Mi (30d), 0 OOMs.
+          #   cpu request 2: two jobs at today's peak is ~1.4 core, and the gate is being
+          #     parallelised, so per-job demand rises. The request is what the CFS share is
+          #     weighted by under contention (100m was a 1/20 share); NO cpu limit, deliberately,
+          #     so an idle node's spare cores are still usable. k3s-prod-2 (8 CPU) had 2945m
+          #     requested; +1.9 core = ~4.85 (61%), and prod-1/prod-3 fit too if it reschedules.
+          #   memory request 1Gi / limit 4Gi: two jobs at today's peak is ~0.9Gi. An OOM kill here
+          #     takes down dockerd and with it BOTH running jobs, so the limit carries ~4x headroom
+          #     over that for the parallelised gate. Nodes have ~31Gi allocatable.
+          resources:
+            requests:
+              cpu: "2"
+              memory: 1Gi
+            limits:
+              memory: 4Gi
+          startupProbe:
+            exec:
+              command:
+                - docker
+                - --host=tcp://127.0.0.1:2376
+                - --tlsverify
+                - --tlscacert=/certs/client/ca.pem
+                - --tlscert=/certs/client/cert.pem
+                - --tlskey=/certs/client/key.pem
+                - info
+            periodSeconds: 1
+            failureThreshold: 60
       containers:
         # ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
         - name: runner
-          image: gitea/act_runner:0.2.11
+          image: gitea/act_runner:0.6.1
-          # TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
+          # No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
-          # entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
+          # registers from env on first boot and runs `act_runner daemon`. The two wait-loops
-          # boot and runs `act_runner daemon`.
+          # that used to live here are gone: kubelet now holds this container until the dind
-          #
+          # sidecar's startupProbe passes (see initContainers above).
-          # STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
-          #   and hard-exits if it is missing (a startup race against the sidecar).
-          # STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
-          #   necessary but NOT sufficient. dockerd writes the certs and only THEN begins
-          #   listening on :2376, so a runner that proceeds on the file alone still hard-exits
-          #   with "cannot ping the docker daemon at tcp://localhost:2376" — observed
-          #   crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
-          # Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
-          # surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
-          #
-          # ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
-          #   braced form or the double-dollar form; both are wrong here, for different reasons.
-          #   clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
-          #   Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
-          #   Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
-          #   and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
-          #     - A bare  $name  is NOT a reference to envsubst: it passes through untouched, and
-          #       that holds even when the name IS set in cluster-vars. So $i and $j reach the
-          #       cluster verbatim and both 60s bounds below are real, enforced bounds.
-          #     - $((i + 1))  is likewise untouched: "(" cannot begin a variable name.
-          #     - A BRACED dollar-name IS a reference and gets replaced — with the empty string
-          #       when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
-          #       which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
-          #     - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
-          #       form — still wrong for us, because on any path where envsubst does NOT run
-          #       (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
-          #       reads that leading double-dollar as its own PID, so a bound would compare a
-          #       string like 1234{i} against 60. The bare form is the only spelling that is
-          #       correct on BOTH paths.
-          #     - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
-          #       name") that fails this whole Kustomization — and this Kustomization gates every
-          #       merge in the repo. So never write one anywhere in this file, INCLUDING in a
-          #       comment. That is why this note names the three forms in words rather than
-          #       showing them: an earlier draft of it spelled them out literally and broke
-          #       `flux envsubst` on the file. (Same trap the indexed-expansion warning in
-          #       clusters/base/apps-vms.yaml describes.)
-          command:
-            - /bin/sh
-            - -c
-            - |
-              # Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
-              i=0
-              until [ -f /certs/client/ca.pem ]; do
-                i=$((i + 1))
-                [ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
-                echo "waiting for dind TLS certs ($i)"; sleep 1
-              done
-              # Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
-              # Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
-              j=0
-              until nc -z -w1 127.0.0.1 2376; do
-                j=$((j + 1))
-                [ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
-                echo "waiting for dind daemon on :2376 ($j)"; sleep 1
-              done
-              exec /sbin/tini -- /opt/act/run.sh
           env:
             - name: GITEA_INSTANCE_URL
-              value: https://gitea.home.keithjackson.dev
+              value: https://gitea.${BASE_DOMAIN}
             - name: GITEA_RUNNER_REGISTRATION_TOKEN
               valueFrom:
                 secretKeyRef:
@@ -186,69 +246,36 @@ spec:
               value: /certs/client
             - name: DOCKER_TLS_VERIFY
               value: "1"
+            # The image's run.sh passes `--config <this>` to BOTH `act_runner register` and
+            # `act_runner daemon` when CONFIG_FILE is set (act_runner v0.6.1 scripts/run.sh).
+            - name: CONFIG_FILE
+              value: /etc/act_runner/config.yaml
           volumeMounts:
             - name: data
               mountPath: /data
             - name: certs
               mountPath: /certs
               readOnly: true
-          resources:
+            - name: config
-            requests:
+              mountPath: /etc/act_runner
-              cpu: 100m
+              readOnly: true
-              memory: 128Mi
+          # Measured 2026-11-08 at capacity 1: peak CPU 0.24 core, peak working set 314Mi (30d)
-            limits:
+          # against the old 512Mi limit. It holds per-task state, log buffers and the embedded
-              memory: 512Mi
+          # actions/cache server for every concurrent task, so the limit is doubled for capacity 2.
-        # ---- Docker-in-Docker: the job executor daemon (privileged) ----
-        #
-        # FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
-        #   Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
-        #   k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
-        #   ordering that the loops above hand-roll — which also removes the envsubst hazard
-        #   from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
-        #   Shape to build:
-        #     1. Move this whole container into `initContainers:` as the FIRST entry.
-        #     2. Add `restartPolicy: Always` to it. That single field is what makes an
-        #        initContainer a sidecar: it starts before the regular containers, keeps
-        #        running alongside them, and is terminated after them.
-        #     3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
-        #        connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
-        #        --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
-        #        --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
-        #        it would re-open the same "port accepting but TLS/certs not ready" gap that
-        #        Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
-        #        preserves today's 60s cap.
-        #     4. kubelet then holds the `runner` container until that startupProbe passes, so
-        #        `runner` can drop its `command:` override completely and run the image's
-        #        default entrypoint.
-        #   Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
-        #   entire repository. If the conversion is wrong, nothing can merge — so it must land
-        #   alone, where the failure is unambiguous and not entangled with anything else, and
-        #   only after a real workflow run has been observed green on the converted pod.
-        - name: dind
-          image: docker:27-dind
-          securityContext:
-            privileged: true
-          env:
-            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
-            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
-            - name: DOCKER_TLS_CERTDIR
-              value: /certs
-          volumeMounts:
-            - name: certs
-              mountPath: /certs
-            - name: dind-storage
-              mountPath: /var/lib/docker
           resources:
             requests:
               cpu: 100m
               memory: 256Mi
             limits:
-              memory: 2Gi
+              memory: 1Gi
       volumes:
         - name: data
           persistentVolumeClaim:
             claimName: act-runner-data
         - name: certs
           emptyDir: {}
+        - name: config
+          configMap:
+            name: act-runner-config
         - name: dind-storage
           emptyDir: {}

apps/wave-1/keycloak.yaml

@@ -0,0 +1,208 @@
+---
+# Namespace (trust zone; default-deny + policies apply)
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: keycloak
+  labels:
+    homelab.keithjackson.dev/app-namespace: 'true'
+    gateway-access: shared
+  annotations:
+    homelab.keithjackson.dev/backup-policy: daily-offcluster
+---
+# Default-deny baseline (R-NET-01): deny all, allow only DNS
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: default-deny
+  namespace: keycloak
+  labels:
+    homelab.keithjackson.dev/baseline: default-deny
+spec:
+  podSelector: {}
+  policyTypes:
+  - Ingress
+  - Egress
+  ingress: []
+  egress:
+  - to:
+    - namespaceSelector:
+        matchLabels:
+          kubernetes.io/metadata.name: kube-system
+    ports:
+    - protocol: UDP
+      port: 53
+    - protocol: TCP
+      port: 53
+---
+# PVC keycloak-db (R-STATE)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: keycloak-db
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  accessModes:
+  - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 10Gi
+---
+# Deployment (R-PLAT-02)
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: keycloak
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: keycloak
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: keycloak
+        homelab.keithjackson.dev/egress-class: internet
+        homelab.keithjackson.dev/auth-mode: native
+    spec:
+      containers:
+      - name: keycloak
+        image: quay.io/keycloak/keycloak:26.0.6
+        ports:
+        - containerPort: 8080
+        volumeMounts:
+        - name: db
+          mountPath: /data/db
+      volumes:
+      - name: db
+        persistentVolumeClaim:
+          claimName: keycloak-db
+---
+# Service
+apiVersion: v1
+kind: Service
+metadata:
+  name: keycloak
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  selector:
+    app.kubernetes.io/name: keycloak
+  ports:
+  - name: port-8080
+    port: 8080
+    targetPort: 8080
+---
+# CiliumNetworkPolicy egress allow (R-NET-02): keycloak-allow-keycloak-db-5432
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: keycloak-allow-keycloak-db-5432
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: keycloak
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: keycloak-db
+        k8s:io.kubernetes.pod.namespace: keycloak-db
+    toPorts:
+    - ports:
+      - port: '5432'
+        protocol: TCP
+---
+# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): keycloak-allow-gateway-ingress
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: keycloak-allow-gateway-ingress
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: keycloak
+  ingress:
+  - fromEntities:
+    - ingress
+    toPorts:
+    - ports:
+      - port: '8080'
+        protocol: TCP
+---
+# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): keycloak-allow-internet-egress
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: keycloak-allow-internet-egress
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: keycloak
+  egress:
+  - toEntities:
+    - world
+---
+# HTTPRoute keycloak-8080 (R-TLS/R-DNS)
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: keycloak-8080
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  parentRefs:
+  - name: shared
+    namespace: gateway
+    sectionName: https
+  hostnames:
+  - keycloak.${BASE_DOMAIN}
+  rules:
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /
+    backendRefs:
+    - name: keycloak
+      port: 8080
+---
+# PrometheusRule SLI alerts (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: keycloak-slis
+  namespace: keycloak
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: keycloak
+spec:
+  groups:
+  - name: keycloak.slis
+    rules:
+    - alert: keycloak-down
+      expr: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"})
+      for: 5m
+      labels:
+        severity: critical
+        service: keycloak
+      annotations:
+        summary: keycloak is down
+        description: 'keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'

apps/wave-1/nextcloud.yaml

@@ -35,6 +35,34 @@ spec:
     - protocol: TCP
       port: 53
 ---
+# PersistentVolume nextcloud-legacy-archive - static external NFS (bulk tier)
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: nextcloud-legacy-archive
+  labels:
+    app.kubernetes.io/name: nextcloud
+spec:
+  capacity:
+    storage: 270Gi
+  accessModes:
+  - ReadOnlyMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ''
+  mountOptions:
+  - nfsvers=4.1
+  - hard
+  - noatime
+  - timeo=600
+  - retrans=2
+  nfs:
+    server: ${BULK_NFS_SERVER}
+    path: /mnt/Storage/nextcloud
+    readOnly: true
+  claimRef:
+    namespace: nextcloud
+    name: nextcloud-legacy-archive
+---
 # PVC nextcloud-data (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -51,6 +79,23 @@ spec:
     requests:
       storage: 20Gi
 ---
+# PVC nextcloud-legacy-archive (R-STATE)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: nextcloud-legacy-archive
+  namespace: nextcloud
+  labels:
+    app.kubernetes.io/name: nextcloud
+spec:
+  accessModes:
+  - ReadOnlyMany
+  storageClassName: ''
+  volumeName: nextcloud-legacy-archive
+  resources:
+    requests:
+      storage: 270Gi
+---
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -104,13 +149,13 @@ spec:
               name: nextcloud-bootstrap
               key: admin-password
         - name: NEXTCLOUD_TRUSTED_DOMAINS
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
         - name: OVERWRITEHOST
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
         - name: OVERWRITEPROTOCOL
           value: https
         - name: OVERWRITECLIURL
-          value: https://nextcloud.home.keithjackson.dev
+          value: https://nextcloud.${BASE_DOMAIN}
         - name: NEXTCLOUD_OIDC_CLIENT_SECRET
           valueFrom:
             secretKeyRef:
@@ -119,19 +164,26 @@ spec:
         volumeMounts:
         - name: data
           mountPath: /var/www/html
+        - name: legacy-archive
+          mountPath: /mnt/legacy-nextcloud
+          subPath: nextcloud
+          readOnly: true
         lifecycle:
           postStart:
             exec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"\
+              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc]\
-                [nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"\
+                \ client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
-                $ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
+                ); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:
           claimName: nextcloud-data
+      - name: legacy-archive
+        persistentVolumeClaim:
+          claimName: nextcloud-legacy-archive
 ---
 # Service
 apiVersion: v1
@@ -228,7 +280,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - nextcloud.home.keithjackson.dev
+  - nextcloud.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -238,26 +290,6 @@ spec:
     - name: nextcloud
       port: 80
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: nextcloud
-  namespace: nextcloud
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: nextcloud
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: nextcloud
-  namespaceSelector:
-    matchNames:
-    - nextcloud
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -272,11 +304,11 @@ spec:
   - name: nextcloud.slis
     rules:
     - alert: nextcloud-down
-      expr: up{job="nextcloud"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"} == 0 or absent(kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"})
       for: 5m
       labels:
         severity: critical
         service: nextcloud
       annotations:
         summary: nextcloud is down
-        description: 'nextcloud has been unreachable for 5m (SLI: up).'
+        description: 'nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/radarr.yaml

@@ -35,6 +35,33 @@ spec:
     - protocol: TCP
       port: 53
 ---
+# PersistentVolume radarr-media - static external NFS (bulk tier)
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: radarr-media
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  capacity:
+    storage: 9Ti
+  accessModes:
+  - ReadWriteMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ''
+  mountOptions:
+  - nfsvers=4.1
+  - hard
+  - noatime
+  - timeo=600
+  - retrans=2
+  nfs:
+    server: ${BULK_NFS_SERVER}
+    path: /mnt/Storage/plex
+  claimRef:
+    namespace: radarr
+    name: radarr-media
+---
 # PVC radarr-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -51,6 +78,23 @@ spec:
     requests:
       storage: 5Gi
 ---
+# PVC radarr-media (R-STATE)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: radarr-media
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  accessModes:
+  - ReadWriteMany
+  storageClassName: ''
+  volumeName: radarr-media
+  resources:
+    requests:
+      storage: 9Ti
+---
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -74,36 +118,51 @@ spec:
         homelab.keithjackson.dev/auth-mode: forward-auth
       annotations:
         homelab.keithjackson.dev/auth-mode: forward-auth
-        homelab.keithjackson.dev/auth-groups: media-users
+        homelab.keithjackson.dev/auth-groups: media-users,platform-admins
     spec:
       containers:
       - name: radarr
-        image: lscr.io/linuxserver/radarr:5.21.1
+        image: lscr.io/linuxserver/radarr:5.28.0
         ports:
         - containerPort: 7878
+        env:
+        - name: PUID
+          value: '1000'
+        - name: PGID
+          value: '3000'
+        - name: TZ
+          value: Etc/UTC
         volumeMounts:
         - name: config
-          mountPath: /data/config
+          mountPath: /config
+        - name: media
+          mountPath: /media
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:7878
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
+        - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://radarr.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --api-route=^/api/
-        - --whitelist-domain=.home.keithjackson.dev
+        - --api-route=^/signalr/
+        - --cookie-name=_oauth2_proxy_radarr
+        - --cookie-domain=radarr.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +183,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -148,10 +210,15 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
+      securityContext:
+        fsGroup: 3000
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: radarr-config
+      - name: media
+        persistentVolumeClaim:
+          claimName: radarr-media
 ---
 # Service
 apiVersion: v1
@@ -216,6 +283,28 @@ spec:
       - port: '9117'
         protocol: TCP
 ---
+# CiliumNetworkPolicy egress allow (R-NET-02): radarr-allow-plex-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-plex-32400
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: plex
+        k8s:io.kubernetes.pod.namespace: plex
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): radarr-allow-seerr-7878
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -258,6 +347,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): radarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-monitoring-scrape
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -288,14 +398,15 @@ metadata:
     app.kubernetes.io/name: radarr
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
-    homelab.keithjackson.dev/auth-groups: media-users
+    homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - radarr.home.keithjackson.dev
+  - radarr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -305,26 +416,6 @@ spec:
     - name: radarr
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: radarr
-  namespace: radarr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: radarr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: radarr
-  namespaceSelector:
-    matchNames:
-    - radarr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -339,11 +430,11 @@ spec:
   - name: radarr.slis
     rules:
     - alert: radarr-down
-      expr: up{job="radarr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"})
       for: 5m
       labels:
         severity: critical
         service: radarr
       annotations:
         summary: radarr is down
-        description: 'radarr has been unreachable for 5m (SLI: up).'
+        description: 'radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/semaphore-bootstrap-secret.sops.yaml

@@ -5,6 +5,15 @@
 #   credential from its own namespace (a Pod cannot read a Secret in another namespace).
 # access-key-encryption: FIXED base64 32-byte AES key Semaphore uses to encrypt stored SSH keys /
 #   credentials; MUST stay stable across restarts or stored secrets become undecryptable.
+# oidc-client-secret: the Keycloak `semaphore` CONFIDENTIAL client secret, and ONLY that. It MIRRORS
+#   `semaphore-client-secret` in platform/keycloak/keycloak-realm-clients.sops.yaml (the realm import
+#   injects that copy into the client), same cross-namespace mirror pattern as db-password above.
+#   It replaced the former `oidc-providers-json` key (2026-09-07): that key held the WHOLE
+#   SEMAPHORE_OIDC_PROVIDERS blob, which buried the environment-specific realm-issuer and redirect
+#   URLs inside ciphertext where Flux's ${BASE_DOMAIN} postBuild substitution could not reach them,
+#   so prod ran with dev's URLs. Those URLs are PUBLIC and now live in plaintext in
+#   catalog/semaphore.yaml (rendered to apps/wave-1/semaphore.yaml); the kubelet splices THIS value
+#   back into the blob via $(SEMAPHORE_OIDC_CLIENT_SECRET). Only the secret half is encrypted.
 # Recover: infra/devcontainer/dev.sh run sops -d apps/semaphore-bootstrap-secret.sops.yaml
 apiVersion: v1
 kind: Secret
@@ -15,11 +24,11 @@ metadata:
         app.kubernetes.io/name: semaphore
 type: Opaque
 stringData:
-    admin-username: ENC[AES256_GCM,data:uD/vO9A9MjerFg==,iv:i/NAqVD6rWg9oWWSGBrRkxzwcPp2IW1HLIKY0pdcPPg=,tag:r27tKLAJ+ydE7Ty3ccMNUw==,type:str]
+    admin-username: ENC[AES256_GCM,data:ju3wVdjNzRUP/w==,iv:d/5ZxgQZ5bympu3gT2ck3DJy9iUPPBAtZYsJvD3To7I=,tag:0UtBaDVqNSK4RCJrQicchw==,type:str]
-    admin-password: ENC[AES256_GCM,data:544nWY1UmBnlsdPJAQZJMXRumu4W9X7UXZvZ6vHN/ArjRgLXKTArOe5JkT/nNrNw,iv:AC1DDbZj2HfpU19EjzA7T+WpIaR4nZc3PNN61y8bAzU=,tag:r28LFi5w6ZeCCaRaX2jRdg==,type:str]
+    admin-password: ENC[AES256_GCM,data:pxIwpgh3BoND2TI0mPTixOTJoictzT77dqtuRdMKgY/2WaW4wM1Ra6H6n/dxnyR2,iv:eYbFQGFYVJsnYGwn6cOI3BgAdzMcin9YggAhiuL+xqM=,tag:/sPWGv4gue0vjc2nMk7wWA==,type:str]
-    db-password: ENC[AES256_GCM,data:ySkdvaGRCW9fMQqChs5l8dY4zCG19i4HOEinRC48mgLuh9WOrjd16T6jwFgiwB55,iv:+1gztJ22HbwgQJiolZMMWOcKIsxrBJpnruJdtxdooOU=,tag:8CfJGwyPTjDGyr0CvalXMQ==,type:str]
+    db-password: ENC[AES256_GCM,data:hBFpWXdpk1R5a23Pntup03t56NBRPS+SZU8CrHVb/ypIaIpgf7kjYtVy6nhTF6u0,iv:eBzoQgH5ZhWWpfKZ6AQmwaSal5QaiuN9gfexQ7GsVFs=,tag:j2k8EwhKDlj/Ud6O39ioVQ==,type:str]
-    access-key-encryption: ENC[AES256_GCM,data:1YAMLo1nVkpJWZc+KBZ7B4ia+Y8P+xMyOhHMcEGGJOXd3o+WC5FqirLBVh4=,iv:j28ozeaDhGWDjoaxFOhn2t9YbYE7L3ah/Mfi+i2DWwY=,tag:czG9Wxc/7Nsuh1W5iATOww==,type:str]
+    access-key-encryption: ENC[AES256_GCM,data:SKtAV4HxJ9pNYDqUK/1uOdI1rtMji3pTfprI2rhQxFQyhc6dFlhqI+OUEcY=,iv:rKCP+FLidcOltOwmMMcO5M8tz4qiifIMI+W5soGO2M8=,tag:HaHzkX/JI+9wmcg2SD9GAQ==,type:str]
-    oidc-providers-json: ENC[AES256_GCM,data:90yt5mIVWLVI2KrlJWpxhL/BMDpEgRhv7+BL+mBRt79diRvU7pN9vtzhZprAKzcU1sEywlT4EwFlsvJpYyo5Qbkp+ZVknbyhAlGvHhJOBHukOPWgf4LignLerO2GzhXjOOzFqWfZ4EyUxNMWTbSe/xc0zIVHDDNQiYplGVRK+E/ukiI7sD/Tn2RrbrdvpC5GyK0NXWRsXn9ffhE3fCoiRHXagj9i/q+uUrQI5EKpPobrDUAGqvKoBXTl/wJ4phnWg9IqtCTE/c3yA7fLLz4j5dCtPZNSVPT4TKDx8Mdv8CfCgziVicjG2s+yHuWpsQn8lf6nvdJmpYPvIqTtLWdxFWuD8Vfo6Qbq2mqPbKR+KmDOGUnjl4gbZdL1SkE9JiqBUobz7UXeLXwYzxMIsinOfYZLcCk7alrD5Yndj9JDxDJZQ54sMMBfVbxDl94W1VHy03RxAR46KKAbPaxOP5wH4Q4z3SDceNwmuGCDRGChRXiHhN5X2X0GqYCbc8E6GdkIMcpmD80e26ydEmAjw7F89mCF4F/yMzJNiG82qaZIn8HeqgGA8urous9r9bYJNlitYTYQ,iv:jbfVdrrZ9fGiOxhQeoTJfFg1QwiOMkoQyaadiVfxX/w=,tag:JkepcRs34EuEgNb4buTTDQ==,type:str]
+    oidc-client-secret: ENC[AES256_GCM,data:uCo7TNzbM2QKw9ie+xzyUG9MFKco1OAlwr5qPn/AGpNvo1YyXSfbZyyM0CLJ1NS821hGg5Zr/m3R/xpy84s3FQ==,iv:O8a4haskufpmorPaxOkJ1Q99Y5GTlzmDFJAsHS+irLk=,tag:VojJpA9jNMbNQmIe1Qr44w==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -29,14 +38,14 @@ sops:
         - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuRDlqRFJ0dk14UHluSGRP
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyTzF6NTBtcFVWK1Vtc0I2
-            S1pESy9PWlpMakU0aTkzNVhqUGdtbWNEdTJjCm5RNG14azJHc28rdEJ4L0RzR0xO
+            VGdySHBsRGgvc1h0VzgvQ01xYVBJV3g2Q1NRCjJyZHVYZGtuamo1MFJHWlBwWkxt
-            MGRocGZ1V1JZTENWQkN3VGZzamp2eTgKLS0tIGt6QkNNaW5hdTBJVHY0SVo5d3JX
+            cHA4VkY1TkRpN3M4UTdMOWZTbFVGemMKLS0tIGVNdDBrWVJveU56VjVBU2JWdHF4
-            elp0SWxHcjg0RWF3aW9mWTJHZ0lhcUEKc0rU5VilAVDleCwqq4SOF+Y9sniu1Pka
+            RGtaYStXYUpONTZjZCtmM0NyZ3Rka1UKIQpmQai8dz+uRAsXxyeuGaRlRwsFqIPQ
-            w/MGDA9THSB0b+YgWRWiX/KiimFqp0ZFODOJ9cHVwFGY71+VadPwOg==
+            9AfrZJcslYwlaK3ptm1oqaCdrlxcMar2nI7thSqzbLe+e1nzA2z7vQ==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-08-27T22:12:32Z"
+    lastmodified: "2026-09-07T07:01:50Z"
-    mac: ENC[AES256_GCM,data:nHftJTGnTr0O5h8yGXwgz3eflggXJfIXM3m8n3bzDJE/P4nvSpZzFeMb+sc2on+AL+t+SeKrIl8blzm33XsQ1lkaY85Zm+bc5cnED1g04BMDVmmK3brh/8w50Bp2wPEVzb4I01MDzhhyIbOYg7pFvZAH3/VuqvM1t6XmfF1NrOA=,iv:4KBa9PH/dfdGwLC16RenZIVB6BtSoGo6x3qt3h1R43I=,tag:KULiA1klP/RZMNigkloGDA==,type:str]
+    mac: ENC[AES256_GCM,data:Xutm1rQt57J4RILElD8FkNOsQT1HK5jZlPtqqLyZ/VqOoDwo+IS1wp7hROprlUvQC0bUXy79tRRIl5Q0CCOD+fvAGIdmade+vB7qXlYN6Q1441dGhuSTPvckso6FaB2ddRjKhKGLc/Z4bJoz9VaRNNxJNpWXIrENTg+I+QhNhlo=,iv:klJOrEQ6QN5dTGkQfGnbMbgCWsgWwZPb5YXDMeWeE34=,tag:fuJr4t3Tt3dT9u3/0SFo8Q==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

apps/wave-1/semaphore.yaml

@@ -78,7 +78,7 @@ spec:
     spec:
       containers:
       - name: semaphore
-        image: semaphoreui/semaphore:v2.19.10
+        image: semaphoreui/semaphore:v2.19.17
         ports:
         - containerPort: 3000
         env:
@@ -110,7 +110,7 @@ spec:
         - name: SEMAPHORE_ADMIN_NAME
           value: Semaphore Admin
         - name: SEMAPHORE_ADMIN_EMAIL
-          value: admin@home.keithjackson.dev
+          value: admin@${BASE_DOMAIN}
         - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
           valueFrom:
             secretKeyRef:
@@ -118,11 +118,13 @@ spec:
               key: access-key-encryption
         - name: SEMAPHORE_PORT
           value: :3000
-        - name: SEMAPHORE_OIDC_PROVIDERS
+        - name: SEMAPHORE_OIDC_CLIENT_SECRET
           valueFrom:
             secretKeyRef:
               name: semaphore-bootstrap
-              key: oidc-providers-json
+              key: oidc-client-secret
+        - name: SEMAPHORE_OIDC_PROVIDERS
+          value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
         volumeMounts:
         - name: config
           mountPath: /etc/semaphore
@@ -225,13 +227,14 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: oidc
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - semaphore.home.keithjackson.dev
+  - semaphore.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -241,26 +244,6 @@ spec:
     - name: semaphore
       port: 3000
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: semaphore
-  namespace: semaphore
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: semaphore
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: semaphore
-  namespaceSelector:
-    matchNames:
-    - semaphore
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -275,11 +258,11 @@ spec:
   - name: semaphore.slis
     rules:
     - alert: semaphore-down
-      expr: up{job="semaphore"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"} == 0 or absent(kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"})
       for: 5m
       labels:
         severity: critical
         service: semaphore
       annotations:
         summary: semaphore is down
-        description: 'semaphore has been unreachable for 5m (SLI: up).'
+        description: 'semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/sonarr.yaml

@@ -35,6 +35,33 @@ spec:
     - protocol: TCP
       port: 53
 ---
+# PersistentVolume sonarr-media - static external NFS (bulk tier)
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: sonarr-media
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  capacity:
+    storage: 9Ti
+  accessModes:
+  - ReadWriteMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ''
+  mountOptions:
+  - nfsvers=4.1
+  - hard
+  - noatime
+  - timeo=600
+  - retrans=2
+  nfs:
+    server: ${BULK_NFS_SERVER}
+    path: /mnt/Storage/plex
+  claimRef:
+    namespace: sonarr
+    name: sonarr-media
+---
 # PVC sonarr-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -51,6 +78,23 @@ spec:
     requests:
       storage: 5Gi
 ---
+# PVC sonarr-media (R-STATE)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: sonarr-media
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  accessModes:
+  - ReadWriteMany
+  storageClassName: ''
+  volumeName: sonarr-media
+  resources:
+    requests:
+      storage: 9Ti
+---
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -74,36 +118,49 @@ spec:
         homelab.keithjackson.dev/auth-mode: forward-auth
       annotations:
         homelab.keithjackson.dev/auth-mode: forward-auth
-        homelab.keithjackson.dev/auth-groups: media-users
+        homelab.keithjackson.dev/auth-groups: media-users,platform-admins
     spec:
       containers:
       - name: sonarr
-        image: lscr.io/linuxserver/sonarr:4.0.15
+        image: lscr.io/linuxserver/sonarr:4.0.20
         ports:
         - containerPort: 8989
+        env:
+        - name: PUID
+          value: '1000'
+        - name: PGID
+          value: '3000'
+        - name: TZ
+          value: Etc/UTC
         volumeMounts:
         - name: config
-          mountPath: /data/config
+          mountPath: /config
+        - name: media
+          mountPath: /media
       - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
         imagePullPolicy: IfNotPresent
         args:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
+        - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-name=_oauth2_proxy_sonarr
-        - --whitelist-domain=.home.keithjackson.dev
+        - --cookie-domain=sonarr.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +181,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -148,10 +208,15 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
+      securityContext:
+        fsGroup: 3000
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: sonarr-config
+      - name: media
+        persistentVolumeClaim:
+          claimName: sonarr-media
 ---
 # Service
 apiVersion: v1
@@ -216,6 +281,28 @@ spec:
       - port: '9117'
         protocol: TCP
 ---
+# CiliumNetworkPolicy egress allow (R-NET-02): sonarr-allow-plex-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-plex-32400
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: plex
+        k8s:io.kubernetes.pod.namespace: plex
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): sonarr-allow-seerr-8989
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -258,6 +345,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -288,14 +396,15 @@ metadata:
     app.kubernetes.io/name: sonarr
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
-    homelab.keithjackson.dev/auth-groups: media-users
+    homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - sonarr.home.keithjackson.dev
+  - sonarr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -305,26 +414,6 @@ spec:
     - name: sonarr
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: sonarr
-  namespace: sonarr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: sonarr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: sonarr
-  namespaceSelector:
-    matchNames:
-    - sonarr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -339,11 +428,11 @@ spec:
   - name: sonarr.slis
     rules:
     - alert: sonarr-down
-      expr: up{job="sonarr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
       for: 5m
       labels:
         severity: critical
         service: sonarr
       annotations:
         summary: sonarr is down
-        description: 'sonarr has been unreachable for 5m (SLI: up).'
+        description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/tests/app-slis.test.yaml

@@ -0,0 +1,245 @@
+# promtool rule-unit tests for the wave-1 per-app `<app>-down` SLI rules.
+# Manifests under test: every PrometheusRule in apps/wave-1/ — nextcloud, radarr, semaphore, sonarr
+# (container/Deployment) and keycloak (StatefulSet, see below). Discovered by the SUITE convention;
+# the flattened manifests are the RENDERED ones Flux applies.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-1/tests/app-slis.test.yaml
+#
+# See apps/wave-0/tests/app-slis.test.yaml for the full history: these rules all read
+# `up{job="<app>"} == 0` until 2026-09-13 against a job no scrape target produced, and were
+# structurally inert on prod. Each is now pinned QUIET on the healthy state, FIRING on no-ready-
+# replica, and FIRING when the signal is ABSENT.
+#
+# KEYCLOAK IS THE INTERESTING ONE. Its server is a StatefulSet run by the Keycloak operator, NOT the
+# Deployment this ServiceDefinition would otherwise imply — so its rule reads
+# kube_statefulset_status_replicas_ready (catalog/keycloak.yaml declares
+# `observability.livenessWorkload: {kind: StatefulSet, name: keycloak}`). Had it been left to the
+# default, the `absent()` half would have matched forever and the alert would have been PERMANENTLY
+# FIRING, which is the other failure mode this repo has shipped to prod. Verified on prod
+# 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}
+# = 2, and the only Deployment in that namespace is keycloak-operator.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # ===========================================================================================
+  # 1. HEALTHY WAVE-1 -> every rule SILENT ("not permanently firing").
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 healthy - every app ready, keycloak StatefulSet at 2 replicas - all quiet
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '2+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '1+0x20'
+      # The namespace's OTHER workloads are present on purpose: keycloak-operator is a Deployment in
+      # the keycloak namespace, and keycloak-db is a StatefulSet in it. A rule that matched the
+      # namespace only (not the object) would be satisfied — or triggered — by these instead of by
+      # the IdP itself. That is the exact masking KeycloakDown had to be fixed for.
+      - series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
+        values: '3+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 2. THE IdP ITSELF IS DOWN while its operator and DB are fine -> ONLY keycloak-down fires.
+  #    This is the anti-masking assertion: the neighbours in that namespace stay ready.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 keycloak StatefulSet has no ready replica (operator + DB fine) - keycloak-down fires
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
+        values: '3+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: keycloak
+              namespace: keycloak
+              statefulset: keycloak
+            exp_annotations:
+              summary: "keycloak is down"
+              description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 3. EACH DEPLOYMENT-BACKED APP DOWN -> its own rule fires, keycloak stays quiet.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 nextcloud/radarr/semaphore/sonarr have no available replica - each own rule fires
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '2+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: nextcloud
+              namespace: nextcloud
+              deployment: nextcloud
+            exp_annotations:
+              summary: "nextcloud is down"
+              description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: radarr
+              namespace: radarr
+              deployment: radarr
+            exp_annotations:
+              summary: "radarr is down"
+              description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: semaphore
+              namespace: semaphore
+              deployment: semaphore
+            exp_annotations:
+              summary: "semaphore is down"
+              description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: sonarr
+              namespace: sonarr
+              deployment: sonarr
+            exp_annotations:
+              summary: "sonarr is down"
+              description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 4. THE SIGNAL VANISHES -> the rules FIRE. The defect class itself, asserted on: a rule that can
+  #    only fire on `== 0` goes SILENT when kube-state-metrics or the workload object disappears,
+  #    which is indistinguishable from healthy. Here NOTHING in wave-1 is scraped.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 all liveness series ABSENT (kube-state-metrics gone) - every rule fires
+    input_series:
+      # One unrelated series so the test has an input at all; no wave-1 liveness series exists.
+      - series: 'up{job="kube-state-metrics"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: keycloak
+              namespace: keycloak
+              statefulset: keycloak
+            exp_annotations:
+              summary: "keycloak is down"
+              description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: nextcloud
+              namespace: nextcloud
+              deployment: nextcloud
+            exp_annotations:
+              summary: "nextcloud is down"
+              description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: radarr
+              namespace: radarr
+              deployment: radarr
+            exp_annotations:
+              summary: "radarr is down"
+              description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: semaphore
+              namespace: semaphore
+              deployment: semaphore
+            exp_annotations:
+              summary: "semaphore is down"
+              description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: sonarr
+              namespace: sonarr
+              deployment: sonarr
+            exp_annotations:
+              summary: "sonarr is down"
+              description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."

apps/wave-2/seerr.yaml

@@ -75,7 +75,7 @@ spec:
     spec:
       containers:
       - name: seerr
-        image: ghcr.io/seerr-team/seerr:v3.4.1
+        image: ghcr.io/seerr-team/seerr:v3.5.1
         ports:
         - containerPort: 5055
         volumeMounts:
@@ -223,7 +223,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - seerr.home.keithjackson.dev
+  - seerr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -233,26 +233,6 @@ spec:
     - name: seerr
       port: 5055
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: seerr
-  namespace: seerr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: seerr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: seerr
-  namespaceSelector:
-    matchNames:
-    - seerr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -267,11 +247,11 @@ spec:
   - name: seerr.slis
     rules:
     - alert: seerr-down
-      expr: up{job="seerr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})
       for: 5m
       labels:
         severity: critical
         service: seerr
       annotations:
         summary: seerr is down
-        description: 'seerr has been unreachable for 5m (SLI: up).'
+        description: 'seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-2/tests/app-slis.test.yaml

@@ -0,0 +1,69 @@
+# promtool rule-unit tests for the wave-2 per-app `<app>-down` SLI rule (seerr).
+# Manifest under test: apps/wave-2/seerr.yaml. Discovered by the SUITE convention; the flattened
+# manifest is the RENDERED one Flux applies.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-2/tests/app-slis.test.yaml
+#
+# seerr (Jellyseerr/Overseerr) exports NO Prometheus metrics of its own, so its `up` SLI could never
+# have been met by scraping the app — see apps/wave-0/tests/app-slis.test.yaml for the full history
+# of the inert `up{job="seerr"} == 0` rule this replaces. Pinned in both directions plus the
+# signal-vanishes case.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  - interval: 1m
+    name: wave-2 healthy - seerr has an available replica - quiet
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts: []
+
+  - interval: 1m
+    name: wave-2 seerr has no available replica - seerr-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: seerr
+              namespace: seerr
+              deployment: seerr
+            exp_annotations:
+              summary: "seerr is down"
+              description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+    # Under `for: 5m` a 2m-old condition must NOT yet be a firing alert, but the expression must
+    # already be selecting — the distinction an inert rule cannot make.
+    promql_expr_test:
+      - expr: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})'
+        eval_time: 2m
+        exp_samples:
+          - labels: 'kube_deployment_status_replicas_available{deployment="seerr", namespace="seerr"}'
+            value: 0
+
+  - interval: 1m
+    name: wave-2 seerr liveness series ABSENT - seerr-down still fires
+    input_series:
+      - series: 'up{job="kube-state-metrics"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: seerr
+              namespace: seerr
+              deployment: seerr
+            exp_annotations:
+              summary: "seerr is down"
+              description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."

bootstrap/flux/bootstrap.sh

@@ -1,6 +1,9 @@
 #!/usr/bin/env bash
 #
-# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
+# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
+# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
+# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
+# same script bootstraps dev and prod with no fork (R-ENV-02).
 #
 # Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
 # =============================================================================
@@ -33,7 +36,7 @@
 #   2. sops-age Secret  — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
 #                         decryption works on first pass (../sops/age-key-secret.md). This script
 #                         only VERIFIES it exists; it never handles key material.
-#   3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
+#   3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
 #                         GitRepository `flux-system` (source) -> Kustomization `platform` ->
 #                         `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
 #                         (chained dependsOn) + `apps-vms`, with cluster-vars overlay
@@ -52,7 +55,10 @@
 set -euo pipefail
 
 # --- config (env-overridable) ------------------------------------------------
-FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"     # dev is the ONLY Phase-1 build target
+# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
+# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
+# every existing caller — override it, never edit it.
+FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
 FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
 SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
 # The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
@@ -62,7 +68,7 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
 
-echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
+echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
 
 # --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
 for bin in flux kubectl; do
@@ -134,7 +140,7 @@ if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null
   exit 1
 fi
 
-# --- 3. apply the authored dev entrypoint ------------------------------------
+# --- 3. apply the authored environment entrypoint ----------------------------
 echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
 kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
 

bootstrap/gitea/bootstrap-gitea.sh

@@ -1,11 +1,30 @@
 #!/usr/bin/env bash
 # bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
-# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
+# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
-# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
+# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
-# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
+# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
-# CONTRIBUTING.md).
+# using it now.
 #
-# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
+# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
+# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
+#
+#   PROD (bare metal, current use):  HOST_NAT_IP=192.168.2.132
+#       The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
+#       NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
+#       three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
+#       apply to prod, so do not "fix" prod by opening zones.
+#
+#   DEV (libvirt VMs, HISTORICAL):   HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
+#       *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
+#       bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
+#       recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
+#
+# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
+# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
+# author-only work and must never be fired off casually or "to see if it works": it starts a
+# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
+# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
+# it will then reconcile. Know which HEAD you are publishing before you run it.
 set -euo pipefail
 ENGINE="${ENGINE:-podman}"
 GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}"   # pinned; bump if needed
@@ -16,11 +35,24 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
 # subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
 DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
 DATA_DIR="podman volume ${DATA_VOLUME}"   # display label only (data lives in the named volume)
-# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
+# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
 # clone/redirect URLs are correct from inside the cluster.
-# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
+#
+# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
+# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
+# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
+# Pass it explicitly, e.g. for prod:   HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
+#
+# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
+# VM->host:3000 by default and had to be opened once with
 #   sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
-HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
+# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
+if [ -z "${HOST_NAT_IP:-}" ]; then
+  echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
+  echo "       prod: HOST_NAT_IP=192.168.2.132   (same LAN as the nodes, no NAT, no firewall change)" >&2
+  echo "       There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
+  exit 1
+fi
 ORG="${GITEA_ORG:-Home-Network}"
 REPO="${GITEA_REPO:-home-network-k3s-impl}"
 
@@ -75,8 +107,17 @@ cat <<EOF
 
 Gitea bootstrap source ready (${DATA_DIR}).
   UI (host):                     ${BASE}/${ORG}/${REPO}
-  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git   (clusters/dev)
+  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
+                                 (patch this url into the overlay being bootstrapped:
+                                  clusters/prod today; clusters/dev has already cut over)
   Admin creds (host, 0600):      ${ADMIN_CRED_FILE}
-Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
+Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
+firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
+must allow VM->host:${HTTP_PORT} —
   ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
+REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
+at this source will reconcile THAT revision.
+TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
+any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
+procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
 EOF

catalog/gitea.yaml

@@ -10,7 +10,7 @@
 name: gitea
 kind: container
 source:
-  image: gitea/gitea:1.24.3     # pinned recent stable tag (R-UPD-01); update.range governs advancement
+  image: gitea/gitea:28.3.0     # pinned recent stable tag (R-UPD-01); update.range governs advancement
 # Headless-install bootstrap (formerly apps/gitea-bootstrap-patch.yaml; now first-class via the
 # ServiceDefinition env/envFrom/lifecycle fields, schema >=1.2.0). Gitea must come up INSTALLED
 # without the interactive web wizard and with a FIXED, persistent SECRET_KEY, so all of this is
@@ -23,14 +23,43 @@ env:
       secretKeyRef:
         name: gitea-bootstrap
         key: secret-key
-  - name: GITEA__server__DOMAIN              # correct public clone/redirect URLs for the TLS-terminated host
+  # ROOT_URL is the ONLY source of the instance domain since Gitea 28 (go-gitea/gitea#39400): [server]
-    value: gitea.home.keithjackson.dev
+  # DOMAIN is no longer read, and the SSH domain defaults from ROOT_URL too. GITEA__server__DOMAIN was
+  # removed in the 28 upgrade; the stale DOMAIN line environment-to-ini already wrote into app.ini is
+  # inert. Correct public clone/redirect URLs for the TLS-terminated host come from this one value.
   - name: GITEA__server__ROOT_URL
-    value: https://gitea.home.keithjackson.dev/
+    value: https://gitea.${BASE_DOMAIN}/
   - name: GITEA__actions__ENABLED            # Phase 6 CI/CD: declared (not implicit upstream default) so the
     value: "true"                            # in-cluster act_runner (ns gitea-actions) registers against it
   - name: GITEA__actions__DEFAULT_ACTIONS_URL  # resolve `uses: <org>/<action>@ref` from github.com (cluster
     value: github                              # has internet egress; not air-gapped)
+  # Gitea 28 added run retention (go-gitea/gitea#38855): completed runs, with their jobs, logs and
+  # artifacts, are deleted by the cleanup_action_runs cron after RUN_RETENTION_DAYS. Upstream's default
+  # (400) is kept but DECLARED, so the deletion horizon is visible here rather than implicit. The spec
+  # repo states no CI-history retention requirement; 0 would mean keep forever.
+  - name: GITEA__actions__RUN_RETENTION_DAYS
+    value: "400"
+  # Gitea 28 (go-gitea/gitea#39426) puts webhooks, the OAuth2 avatar fetch and OpenID 2.0 behind an
+  # egress policy: in the default lax mode, PUBLIC targets are allowed and private/loopback/CGNAT
+  # targets need an IP or CIDR entry in [security] ALLOWED_HOST_LIST (a hostname entry is NOT enough
+  # for a private address). DECLARED lax with an EMPTY allow list, deliberately:
+  #   * the Keycloak OIDC login source is NOT gated by it. In 28.3.0 the openidConnect provider is
+  #     goth's, built with its default HTTP client (services/auth/source/oauth2/providers_openid.go);
+  #     egress.NewSecurityPolicy is used only by routers/web/auth/oauth.go (avatar fetch, and
+  #     [oauth2_client] UPDATE_AVATAR is off here) and modules/auth/openid (OpenID 2.0, unused).
+  #     So keycloak.<domain> resolving to the private Gateway address in-cluster does not matter.
+  #   * this instance has no webhooks and no mirrors (measured 2026-11-07), and 1.27 already refused
+  #     private webhook targets by default, so nothing that worked before is newly denied.
+  # Adding an entry here would only WIDEN the webhook SSRF surface. Add the narrowest CIDR (e.g. the
+  # Gateway address /32) only if a real private webhook target appears.
+  - name: GITEA__security__EGRESS_MODE
+    value: lax
+  # Gitea 28 (go-gitea/gitea#39400) disables LOCAL self-registration unless DISABLE_REGISTRATION is
+  # explicitly false; 1.27 defaulted it open. Declared true: accounts arrive via Keycloak only.
+  # OAuth2 auto-registration below is unaffected — routers/web/auth/oauth.go gates it on
+  # ALLOW_ONLY_INTERNAL_REGISTRATION and ENABLE_AUTO_REGISTRATION, not on DISABLE_REGISTRATION.
+  - name: GITEA__service__DISABLE_REGISTRATION
+    value: "true"
   - name: GITEA_ADMIN_USERNAME               # break-glass local admin creds (consumed by the postStart hook)
     valueFrom:
       secretKeyRef:
@@ -104,7 +133,7 @@ lifecycle:
           # persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
           # file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
           # usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
-          GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
           gitea_oidc_fail() {
             echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
@@ -168,7 +197,7 @@ lifecycle:
           # be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
           # the moment its OIDC guard was fixed and the block actually ran for the first time.
           exit 0
-exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
+exposure: public                # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: data
@@ -178,8 +207,49 @@ storage:
                                 # the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
 dependencies: []                # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
 ingressRoutes:
-  - host: gitea.home.keithjackson.dev
+  - host: gitea.${BASE_DOMAIN}
     port: 3000                  # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
+# THIS IS WHAT KEEPS PROD'S GITOPS OUT OF A DEADLOCK — it is not an incidental allowance. Remove it
+# and prod cannot reconcile AT ALL, in a way that is very hard to diagnose from cold.
+#
+# Prod's Flux source is THIS SERVICE, not a public hostname:
+#   clusters/prod/kustomization.yaml -> GitRepository.spec.url =
+#   http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
+# chosen deliberately so prod's GitOps depends on NEITHER public DNS, Cloudflare, prod's own
+# Gateway, NOR ${BASE_DOMAIN} — a future domain change cannot sever prod from its own source. That
+# file documents the other half of this pair at length; the two must be read together.
+#
+# The only other ingress this entry renders is gitea-allow-gateway-ingress, which admits
+# `fromEntities: [ingress]` — the shared Gateway's Envoy proxy — and nothing else. Flux's
+# source-controller dials the ClusterIP DIRECTLY, so under the namespace default-deny (R-NET-01) its
+# clone was DROPPED (`wget` from source-controller timed out; srcReady=False, "unable to list
+# remote"). The deadlock that made this urgent: the HTTPRoute that would have served the new public
+# name after the 2026-09-13 apex flip lives in a DOWNSTREAM tier, which cannot apply while the
+# source is down — source down -> routes never rename -> source stays down. It was broken on
+# 2026-09-13 with a live `kubectl apply` of a hand-written gitea-allow-flux-ingress, i.e. with a
+# cluster object that existed OUTSIDE git. This declaration renders that object, so one
+# `kubectl delete` or one cluster rebuild can no longer put prod back in the deadlock.
+inClusterClients:
+  # `name` is pinned to `flux` (rather than defaulting to the namespace) so the rendered policy name
+  # equals the live object's, `gitea-allow-flux-ingress`, and Flux ADOPTS it in place: no
+  # near-duplicate policy beside it, and no window in which prod's own source has no allow.
+  - name: flux
+    namespace: flux-system      # matched by NAMESPACE alone — the flux controllers' pod labels belong
+                                # to upstream's manifests, not to this platform's contract
+    port: 3000                  # ONLY the Gitea HTTP port — the same single port the Gateway allow opens
+  # The PM agent service (platform/pm-agent/). Its dispatcher reads issues and posts comments
+  # over this port, and every worker Job clones, pushes and opens pull requests over it. Like
+  # Flux's source-controller above, it dials the ClusterIP DIRECTLY — deliberately, so the
+  # agent's path to the repository depends on neither public DNS, nor Cloudflare, nor the
+  # shared Gateway, nor ${BASE_DOMAIN}. And exactly as for Flux, that means this namespace's
+  # own default-deny (R-NET-01) DROPS the connection unless it is declared here: without this
+  # entry the dispatcher's egress allow is satisfied and the packet is still discarded at the
+  # Gitea end, which presents as an authentication timeout rather than as a policy denial.
+  # That is the 2026-09-13 lesson documented at length above, applied before it is relearned.
+  - name: pm-agent
+    namespace: pm-agent         # matched by NAMESPACE alone, so the dispatcher Deployment and every
+                                # ephemeral worker Job are covered by the one declaration
+    port: 3000                  # ONLY the Gitea HTTP port
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
   groups:
@@ -192,9 +262,14 @@ auth:
   # recovery tools (docs/15 "Availability caveat"). No secret is embedded here.
 update:
   channel: stable               # R-UPD-01
-  range: ">=1.22.0 <2.0.0"      # stay on the 1.x stable line
+  range: ">=28.0.0 <29.0.0"     # Gitea dropped the "1." prefix at 28.0.0 (successor of 1.27); stay on 28.x
   cadence: "0 4 * * *"
-  autoRollbackOnFailedHealth: true   # R-UPD-03
+  # R-UPD-03. INERT FOR THIS SERVICE, and it must stay that way: this flag renders only an annotation
+  # on a Flux ImagePolicy, and apps/ is rendered --no-image-automation (ci/render-golden.sh), so no
+  # ImagePolicy exists and the Deployment (strategy Recreate) is never auto-reverted. That matters
+  # because a Gitea upgrade (1.27 -> 28 included) MIGRATES THE SQLITE DB ONE-WAY: an image revert onto
+  # a migrated DB is not a rollback. Roll Gitea back by RESTORING THE DATA PVC from its pre-upgrade backup.
+  autoRollbackOnFailedHealth: true
 observability:
   metrics: true
   slis:

catalog/jackett.yaml

@@ -16,24 +16,62 @@
 name: jackett
 kind: container
 source:
-  image: lscr.io/linuxserver/jackett:latest
+  image: lscr.io/linuxserver/jackett:v0.24.3025-ls29
-exposure: internal              # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
     size: 2Gi
     class: longhorn             # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
+    mountPath: /config          # THE APP WRITES /config, NOT the renderer default /data/config.
+                                # LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
+                                # /config), so without this the Longhorn PVC mounted at /data/config sat
+                                # holding nothing but lost+found while the app wrote its entire
+                                # configuration to the container's EPHEMERAL writable layer — destroyed on
+                                # every pod recreation: restart, reschedule, image bump. Measured in-cluster
+                                # 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
+                                # dev=2053 on the node's own disk with [Sentry asp config.xml logs].
+                                # This is the SAME failure catalog/plex.yaml:54 already records fixing.
+                                # The live /config contents were copied onto each PVC before this landed,
+                                # so API keys and credentials other apps reference were preserved.
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: jackett.home.keithjackson.dev
+  - host: jackett.${BASE_DOMAIN}
     port: 9117                  # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
   groups:
     - media-users               # authenticated AND in an authorized group (R-IDP-06)
+    # platform-admins is a SUPERSET, not a widening. --allowed-group is repeatable and OR-ed, so
+    # listing both means "media consumers OR platform operators". Without it the media UIs are
+    # reachable by NOBODY the moment media-users is empty: on prod the realm's only user is the
+    # operator, who is in platform-admins only, so jackett/qbittorrent/radarr/sonarr answered an
+    # immediate 403 at the oauth2-proxy callback (no login prompt, because the Keycloak SSO session
+    # already existed) while pihole/crafty — which already gate on platform-admins — worked.
+    # This grants no real privilege: platform-admins already maps to cluster-admin via
+    # k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods.
+    - platform-admins           # operators of these services must not be locked out of them
+  # Answer Jackett's OWN API with 401 instead of a 302 to Keycloak (schema 1.9.0; renders
+  # --api-route, singular and repeatable, onto the oauth2-proxy sidecar).
+  #
+  # MEASURED on this pod's access log, 2026-09-30: on /api/ paths, 33 x 302 against 2 x 401 —
+  # and 4 of 6 POSTs were answered BY THE PROXY and never reached Jackett (access-log upstream
+  # field `-`, not `/`), including both `/api/v2.0/server/update` calls. Jackett's jQuery does
+  # not send the exact `Accept: application/json` oauth2-proxy's AJAX guard requires, so the
+  # 302 was followed invisibly, satisfied silently by the live Keycloak SSO session, and
+  # resolved 200 carrying the dashboard's HTML where the JavaScript expected JSON. A browser
+  # following a 302 also re-issues a POST as a GET and DROPS THE BODY, which is why "adding an
+  # indexer silently does nothing" was the user-visible symptom and nothing logged a failure.
+  #
+  # SCOPE: `/api/` only. Jackett serves its UI at /UI/ and its static assets elsewhere, and a
+  # browser never NAVIGATES to /api/ — so a plain page load is still 302'd into a silent SSO
+  # sign-in, which is the behaviour that must not change.
+  apiRoutes:
+    - ^/api/
 update:
   channel: stable               # R-UPD-01
-  range: ">=0.22.0"             # stay on the Jackett 0.22.x stable line
+  range: ">=0.22.0"             # floor only, no ceiling (0.24.x runs as of 2026-10-30; Jackett has no
+                                # major line to hold back from, so renovate.json5 mirrors no cap)
   cadence: "0 4 * * *"
   autoRollbackOnFailedHealth: true   # R-UPD-03
 observability:

catalog/keycloak.yaml

@@ -5,8 +5,15 @@
 # Validates against schemas/servicedefinition.schema.json.
 name: keycloak
 kind: container
+# DOCUMENTATION-ONLY IMAGE. The running server is NOT chosen here: the Keycloak operator
+# (platform/keycloak/kustomization.yaml, three pinned release URLs) picks it via its
+# RELATED_IMAGE_KEYCLOAK, and the Keycloak CR has no spec.image. This SD's render,
+# apps/wave-1/keycloak.yaml, is not listed in apps/wave-1/kustomization.yaml and deploys nothing
+# (ci/render-golden.sh RENDER_GOLDEN_NOT_ONBOARDED). The tag records the operator's version for
+# readers; renovate.json5 disables Renovate for this image so it cannot raise PRs against a file
+# nothing deploys. The Keycloak version moves with the operator — update this tag by hand then.
 source:
-  image: quay.io/keycloak/keycloak
+  image: quay.io/keycloak/keycloak:26.0.6
 exposure: public              # login/redirect flows for internet-exposed services; hardened (docs/15; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
@@ -18,7 +25,7 @@ dependencies:
     port: 5432
     protocol: tcp
 ingressRoutes:
-  - host: keycloak.home.keithjackson.dev
+  - host: keycloak.${BASE_DOMAIN}
     port: 8080                # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
 auth:
   mode: native                # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.
@@ -31,5 +38,15 @@ observability:
   metrics: true
   slis:
     - up                      # IdP down/unhealthy is a critical alert (docs/10; R-OBS-01)
+  # WHAT "up" IS for Keycloak. The Keycloak OPERATOR runs the server as a StatefulSet named
+  # `keycloak` (pods keycloak-0/1), not as the Deployment this ServiceDefinition would otherwise
+  # imply — the operator owns the workload (platform/keycloak/keycloak-instance.yaml) and this SD
+  # is the catalog's declaration of intent for it. Without this override the rendered SLI would
+  # read a Deployment series that does not exist and its `absent()` half would fire FOREVER.
+  # Verified on prod 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak",
+  # statefulset="keycloak"} = 2, and the only Deployment in that namespace is keycloak-operator.
+  livenessWorkload:
+    kind: StatefulSet
+    name: keycloak
 backup:
   policy: daily-offcluster    # DR-critical: rebuilds from DB + git realm config (docs/15, docs/04; R-STATE-03)

catalog/nextcloud.yaml

@@ -50,13 +50,13 @@ env:
   #     the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
   #     though it speaks plain HTTP behind the proxy. ---
   - name: NEXTCLOUD_TRUSTED_DOMAINS
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
   - name: OVERWRITEHOST
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
   - name: OVERWRITEPROTOCOL
     value: https
   - name: OVERWRITECLIURL
-    value: https://nextcloud.home.keithjackson.dev
+    value: https://nextcloud.${BASE_DOMAIN}
   # --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
   #     `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
   #     (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
@@ -88,7 +88,7 @@ lifecycle:
           # provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
           # ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
           OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
-          NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
           nextcloud_oidc_fail() {
             echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
@@ -223,12 +223,50 @@ storage:
     mountPath: /var/www/html    # 3-replica tier; irreplaceable, DR-critical (docs/04; R-STATE-02).
                                 # The whole html dir is persisted so config.php + user data survive a
                                 # rebuild (the default /data/<name> would strand them on ephemeral rootfs).
+  # THE LEGACY NEXTCLOUD DATASET (Storage/nextcloud, 270 G) — MOUNTED READ-ONLY, DELIBERATELY, AND
+  # NOT AS THIS INSTANCE'S DATA DIRECTORY. Read this before changing either decision.
+  #
+  # WHAT IS ACTUALLY IN THERE (measured 2026-09-07, not assumed):
+  #     /mnt/Storage/nextcloud/nextcloud/config          103 M
+  #     /mnt/Storage/nextcloud/nextcloud/config_old      3.4 G
+  #     /mnt/Storage/nextcloud/nextcloud/config_backup   267 G   <- 99% of the dataset
+  # There is NO `data/` directory. This is the OLD, non-k3s Nextcloud's configuration plus a backup
+  # blob — an ARCHIVE, not a live user-data tree.
+  #
+  # WHY READ-ONLY. Two independent reasons, either sufficient:
+  #   1. The old instance is STILL LIVE on it. The export is currently shared rw to 192.168.1.6 and
+  #      192.168.1.5, and the operator has said the existing non-k3s prod must remain active. Two
+  #      Nextcloud instances writing one instance's config directory, each with its own database and
+  #      therefore its own oc_filecache, is a corruption scenario with no clean recovery.
+  #   2. Nothing here needs writing. An archive is read.
+  # "Do not grant more than needed" therefore lands on `ro`. If the operator's intent is in fact a
+  # MIGRATION (read this data in, then decommission the old instance), that is a different and
+  # larger change than a mount, and it should be a deliberate one — flip `readOnly` and update the
+  # export at that point, not before. Enforced by the PV's nfs.readOnly, so it is a real `ro` mount.
+  #
+  # WHY NOT /var/www/html/data. That path is the LONGHORN PVC above and is this instance's own
+  # state; mounting a foreign tree into it would have Nextcloud scan and index an archive it does
+  # not own. A neutral path keeps the two completely separate. To expose it to users, add it as a
+  # Nextcloud External Storage (Local) mount pointing at this path — an in-app, per-user decision.
+  #
+  # subPath skips the redundant single-child directory: the dataset root holds exactly one entry,
+  # `nextcloud/`, so mounting that subtree puts config/ config_old/ config_backup/ at the top.
+  #
+  # UID: nothing to do. The tree is 0755 owned 1000:3000 and the mount is read-only, so www-data
+  # (uid 33) can read and traverse it. (fsGroup would be a no-op on an NFS volume in any case.)
+  - name: legacy-archive
+    class: bulk
+    export: /mnt/Storage/nextcloud   # the ZFS dataset Storage/nextcloud, on ${BULK_NFS_SERVER}
+    subPath: nextcloud
+    mountPath: /mnt/legacy-nextcloud
+    readOnly: true
+    size: 270Gi                      # ADVISORY ONLY on a static NFS PV — the dataset's used size.
 dependencies:
   - to: nextcloud-db            # explicit allow: Nextcloud -> its PostgreSQL (docs/13 graph; R-NET-02).
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
     protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: nextcloud.home.keithjackson.dev
+  - host: nextcloud.${BASE_DOMAIN}
     port: 80                    # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)

catalog/pihole.yaml

@@ -5,21 +5,369 @@
 name: pihole
 kind: container
 source:
-  image: pihole/pihole
+  image: pihole/pihole:2026.10.1
 exposure: internal            # LAN-only admin UI (docs/13 row 6; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: config
     size: 2Gi
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
+    # THE PVC WAS MOUNTED SOMEWHERE PIHOLE NEVER WRITES, AND SO HELD NOTHING AT ALL. Without an
+    # explicit mountPath the renderer defaults to /data/<name>, i.e. /data/config — but PiHole keeps
+    # every piece of its state in /etc/pihole. MEASURED on live prod 2026-10-04: `/etc/pihole` (7.1 M
+    # — gravity.db, pihole.toml, the query DB) was on the **overlay** filesystem, i.e. the
+    # container's EPHEMERAL layer, while `/dev/longhorn/pvc-849edb86...` sat mounted at
+    # /data/config holding `lost+found` and 24 K of nothing. Three consequences, all live until now:
+    #   * every restart silently discarded PiHole's whole state, re-downloading 72 k gravity domains
+    #     and regenerating the admin password hash;
+    #   * `backup.policy: daily-offcluster` below has been faithfully backing up an EMPTY volume,
+    #     which is worse than no backup because it reports success;
+    #   * it is also why phase 1's password/adlist declaration mattered more than it first appeared —
+    #     that state was not merely undeclared, it was ephemeral.
+    # This is the same class of bug catalog/plex.yaml's `config` entry documents ("Without this the
+    # PVC sat unused and Plex wrote to the container's EPHEMERAL layer").
+    #
+    # /etc/pihole IS THE IMAGE'S INTENDED MOUNT POINT, and an EMPTY volume there self-initialises.
+    # Verified in the running image, not assumed:
+    #   * `/usr/bin/start.sh` (PID 1) calls `migrate_gravity`, whose `migrate_to_database()` creates
+    #     the gravity database when `[ ! -e "$gravityDBfile" ]` — the same branch the postStart hook
+    #     below relies on, and the branch that produced the current ephemeral copy.
+    #   * FTL writes `pihole.toml` itself from the `FTLCONF_*` env above (that is what the
+    #     `### CHANGED (env)` markers in it are).
+    #   * NO `podSecurityContext.fsGroup` is needed and adding one would be inert: `start.sh` line 5
+    #     is `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/` and PID 1 runs as **uid 0**, so
+    #     the image fixes ownership of a fresh root-owned volume before anything else starts. Same
+    #     arrangement as plex's image managing its own /config ownership.
+    #   * the image's own default for `files.macvendor` was moved OUT of this directory (to
+    #     `/macvendor.db`), which is what a maintainer does to a path they expect to be a volume.
+    #
+    # WHY IT IS FIXED IN THIS BRANCH rather than left for its own: `availability.replicas: 2` renders
+    # per-replica volumeClaimTemplates, and with the wrong mountPath those would be two 2 Gi Longhorn
+    # volumes (times three Longhorn replicas each) holding nothing but `lost+found` while all state
+    # stayed ephemeral — paying the cost of persistence and getting none of it. It also REMOVES this
+    # change's main operational risk: with gravity.db persisted, a pod restart is cheap, instead of
+    # every restart re-downloading 72 k domains inside a wave that has a 5 m timeout.
+    mountPath: /etc/pihole
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
+# ------------------------------------------------------------------------------------------------
+# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
+# ------------------------------------------------------------------------------------------------
+# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
+# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
+# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
+# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
+# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
+# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
+# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
+# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
+#
+# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
+# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
+# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
+# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
+# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
+# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
+# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
+# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
+# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
+# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
+# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
+# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
+#
+# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
+# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
+# configured through the UI is lost on the next reschedule — which is exactly what happened to the
+# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
+# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
+# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
+# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
+env:
+  # Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
+  # pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
+  # required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
+  # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}, ${WG_CLIENT_CIDR}]` on the DNS Services,
+  # enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
+  # nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
+  - name: FTLCONF_dns_listeningMode
+    value: "ALL"
+  # Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
+  # router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
+  # answers, and forwarding to a device that may forward back here invites a loop.
+  - name: FTLCONF_dns_upstreams
+    value: "8.8.8.8;8.8.4.4"
+  # Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
+  # because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
+  # externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
+  # announcement lease and the pihole pod land on different nodes, which is most of the time — and
+  # Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
+  # INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
+  # apparent client's budget, and a normal evening of browsing from a handful of machines would trip
+  # a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
+  # off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
+  # limit becomes meaningful again — and should be restored — if Cilium is ever switched to
+  # `loadBalancer.mode: dsr`, which preserves the client address.
+  - name: FTLCONF_dns_rateLimit_count
+    value: "0"
+  # THE LAN ANSWERS — AN EXPLICIT LIST, NOT A ZONE WILDCARD (2026-11-09). A dnsmasq
+  # `address=/<domain>/<ip>` entry answers that domain AND every subdomain of it (on LABEL
+  # boundaries: `pihole.` does not match `pihole-dns.`), and dnsmasq resolves overlapping entries by
+  # LONGEST MATCHING DOMAIN, so the order below is irrelevant. Every name NOT listed here is
+  # FORWARDED to the upstreams above and gets the PUBLIC answer.
+  #
+  # WHY NO WILDCARD ANY MORE. Until 2026-11-09 the first entry was `address=/${BASE_DOMAIN}/
+  # ${GATEWAY_VIP}`, answering the whole zone. That made sense while every public record was a
+  # DNS-only CNAME to `wan.${BASE_DOMAIN}` (no NAT hairpin on this router, so the LAN NEEDED a
+  # private answer). Since PR #156 every public name is a PROXIED CNAME to the Cloudflare Tunnel
+  # (platform/externaldns/README.md), which works identically from the LAN — so a name that is
+  # public now SHOULD resolve publicly from the LAN too, and the LAN no longer depends on PiHole for
+  # it: a client that bypasses PiHole (DoH browser, phone private DNS, the router's fallback) gets
+  # the same working answer. The wildcard had also become actively misleading: it answered names
+  # that exist nowhere (`wan.` itself resolved to the Gateway VIP on the LAN).
+  #
+  # WHAT IS LISTED, and why each — enforced by conformance/tests/l0-static/pihole-lan-names:
+  #   -> ${GATEWAY_VIP}, LAN-ONLY: every HTTPRoute disowned from public DNS
+  #      (`external-dns.alpha.kubernetes.io/controller: none`, i.e. catalog `exposure: internal`,
+  #      plus plex and headlamp) — jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr,
+  #      headlamp. These have NO public record and the tunnel 404s them, so THIS LINE IS THEIR ONLY
+  #      PATH: drop one and the app is unreachable everywhere. A NEWLY ONBOARDED `internal` SERVICE
+  #      MUST BE ADDED HERE (the L0 check fails until it is). A new `public` service needs nothing.
+  #   -> ${GATEWAY_VIP}, PUBLIC BUT KEPT ON THE LAN PATH: gitea (git push, LFS, container registry)
+  #      and nextcloud (uploads) — Cloudflare's proxy caps a request body at 100 MB and a response
+  #      at 100 s, which a large push or upload exceeds; keycloak and auth (oauth2-proxy) — so the
+  #      OIDC login of those two LAN-path apps stays on the same path as the apps themselves.
+  #   -> their own L4 VIPs: unifi (${UNIFI_L4_VIP}) and crafty-mc (${CRAFTY_L4_VIP}) are not behind
+  #      the Gateway (grep `external-dns.alpha.kubernetes.io/hostname` on a Service:
+  #      apps/vms/unifi/50-service-l4.yaml, platform/crafty/service-l4.yaml); their public records
+  #      are DNS-only A records to the same RFC1918 VIPs, which this router's rebind protection
+  #      strips, so the LAN answer is still required. pihole-dns (${PIHOLE_DNS_VIP}) is PiHole's own
+  #      resolver address; plex-vip (${PLEX_L4_VIP}) is below.
+  # NOT listed, deliberately, so they resolve via Cloudflare from the LAN as from anywhere: the apex,
+  # grafana, crafty, seerr. In-cluster (platform/coredns/coredns-custom.yaml) and node
+  # (/etc/hosts pins, node_baseline_hosts_pins) resolution are separate and unchanged.
+  #
+  # AAAA: dnsmasq answers a AAAA query for an `address=`d name with NODATA (NOERROR, no answer),
+  # so IPv6-capable clients fall back to the A record instead of getting NXDOMAIN — the same
+  # behaviour coredns-custom hand-writes a `template IN AAAA` block for. Unlisted (proxied) names
+  # now get Cloudflare's real A and AAAA answers.
+  #
+  # Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
+  # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
+  # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
+  # while resolution for the zone does not work. Confirmed both ways against the live image.
+  #
+  # plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the name of the tcp/32400 LoadBalancer
+  # (apps/wave-0/plex-remote.yaml) and the domain of the certificate Plex serves there
+  # (catalog/plex.yaml CERTDOMAIN / customCertificateDomain, Secret `plex-vip-tls`). It is NOT what
+  # Plex advertises any more (ADVERTISE_IP is a resolver-independent plex.direct name since
+  # 2026-10-05) and it has NO public record, so this entry is its only resolution — kept for
+  # reaching the VIP by name with a matching certificate from a LAN shell. Measured 2026-10-04,
+  # before it existed the zone wildcard answered it with ${GATEWAY_VIP}, where nothing listens on
+  # 32400; without it now, the name is NXDOMAIN.
+  #
+  # ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
+  # MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
+  #
+  #   dig home.keithjackson.dev @192.168.2.23 (pihole)  -> 192.168.2.20                TTL=0
+  #   dig home.keithjackson.dev @1.1.1.1      (public)  -> wan.home.keithjackson.dev   TTL=300
+  #   dig home.keithjackson.dev @192.168.2.1  (router)  -> wan....  TTL=153 -> 148 -> 143
+  #
+  # dnsmasq answers `address=` entries with **TTL 0** unless told otherwise, and that is the whole
+  # bug. The SOHO router forwards LAN queries to PiHole and CACHES the result, so:
+  #   * the CORRECT answer is UNCACHEABLE (TTL 0) -- the router must re-query PiHole on every
+  #     single lookup, which is maximum exposure to any momentary gap; while
+  #   * the WRONG answer is CACHEABLE FOR 5 MINUTES -- the instant the router falls back to its
+  #     upstream (standard forwarder behaviour when the forward target does not answer) it latches
+  #     the public WAN address and serves it to EVERY LAN CLIENT for the public record's 300s TTL.
+  # The decrementing TTL on the router's reply above is the proof it caches rather than forwarding
+  # live. So ONE sub-second PiHole gap -- a pod restart, an L2 announcement lease moving, an ARP
+  # flap, a kured node reboot -- becomes FIVE MINUTES of the whole house resolving to the WAN IP.
+  # That asymmetry, not the gap, is why this is visible as "flapping" rather than a blip.
+  #
+  # 60s is a deliberate floor, not a tuning knob: long enough to ride out a pod restart and a lease
+  # re-election (both seconds), short enough that a genuine VIP change still propagates inside a
+  # minute. These VIPs are pinned by name (`lbipam.cilium.io/ips`) and effectively static, so the
+  # propagation cost is theoretical while the flap it prevents is measured. If flaps persist across
+  # LONGER outages, raising this toward the public record's 300 would cover them -- at the cost of
+  # the same delay on a real change. It does NOT help when PiHole is down longer than the TTL: the
+  # cache expires, the router falls back, and the public answer latches again. That is a real
+  # outage, and the fix for it is PiHole availability, not DNS caching.
+  #
+  # SINCE 2026-11-09 THE LATCH IS MOSTLY HARMLESS. The measurement above predates the tunnel: the
+  # public answer was then `wan.`, unreachable from the LAN. Now the public answer for every
+  # proxied name is Cloudflare, which works from the LAN, and those names are not listed below at
+  # all. A latch still bites only the listed names: gitea/nextcloud/keycloak/auth fall back to
+  # Cloudflare (working, but under its 100 MB / 100 s limits); the LAN-only names and the L4 VIPs
+  # get NXDOMAIN or a stripped answer until the router re-queries PiHole.
+  #
+  # Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
+  # configuration, /etc/hosts and DHCP leases), so one directive covers every listed name.
+  - name: FTLCONF_misc_dnsmasq_lines
+    value: "local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
+  # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
+  # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
+  # carries the full rationale and the measurement that found it.
+  #
+  # `webserver.api.password` takes CLEARTEXT and FTL hashes it itself into `webserver.api.pwhash`.
+  # Verified in the running image, not assumed: `pihole setpassword` is itself
+  # `setFTLConfigValue "webserver.api.password" "$PASSWORD"`, commented upstream as "pihole-FTL will
+  # automatically hash the password". So the BALLOON-SHA256 hash on the PVC is now a DERIVED artefact
+  # and this is its source of truth. Rotating the Secret rotates the password.
+  - name: FTLCONF_webserver_api_password
+    valueFrom:
+      secretKeyRef:
+        name: pihole-bootstrap
+        key: webui-password
+availability:
+  # TWO REPLICAS, ON DIFFERENT NODES. LAN DNS is a critical network service and it was ONE pod on
+  # ONE node. Measured 2026-10-04, that cost more than "until it reschedules":
+  #   * the node running the pod dies -> ~40s to mark it NotReady, then `tolerationSeconds: 300`
+  #     before the pod is even evicted, then reschedule, then Longhorn must force-detach the RWO
+  #     volume from a dead node and reattach it. Roughly 6-10 MINUTES with no LAN resolution.
+  #   * the node holding the `pihole-dns-*` L2 lease dies -> Cilium re-elects an announcer, tens of
+  #     seconds. (These were DIFFERENT nodes: lease on k3s-prod-2, pod on k3s-prod-1. Two of three
+  #     nodes were single points of failure for LAN DNS, with wildly different blast radii.)
+  # The Services already tolerate this: `externalTrafficPolicy: Cluster` on the DNS LoadBalancers
+  # (apps/wave-0/pihole-dns.yaml) means the L2 lease holder forwards to ANY ready backend, so a
+  # second replica is picked up with no change there.
+  #
+  # WHAT THIS DOES NOT FIX, and it is the bigger risk: LAN DNS still depends on the CLUSTER. etcd
+  # quorum loss, a Cilium regression, a Longhorn problem or a bad reconcile takes DNS down
+  # everywhere regardless of replica count. The only thing that removes that dependency is a
+  # resolver outside the cluster carrying the same `address=` lines. Considered and not chosen
+  # (operator decision, 2026-10-04); see decisions/2026-10-04-pihole-multi-replica.md.
+  #
+  # THE PREREQUISITE WAS DONE FIRST, and this field would have been actively harmful without it. A
+  # replica gets a FRESH volume, and until 2026-10-04 PiHole's admin password and adlists lived only
+  # on the PVC — so replica 1 would have come up SERVING DNS AND BLOCKING NOTHING, with a different
+  # password, and answers would have varied by which pod replied. Both are now declared (the
+  # password as a SOPS Secret -> FTLCONF_webserver_api_password; the adlists reconciled into
+  # gravity.db by the postStart hook below), which is what makes a fresh replica identical to its
+  # sibling. See decisions/2026-10-04-pihole-declare-state.md.
+  #
+  # WHY 2 AND NOT 3. Two tolerates the loss of any single node, which is the stated goal, and leaves
+  # a spare node for the evicted replica to land on under the HARD anti-affinity the renderer emits.
+  # Three would pin one replica per node with no spare, so a single node failure would leave a
+  # replica permanently Pending and — `apps-wave-0` being `wait: true` — stall the app tier for as
+  # long as the node was down. Two is the count that degrades gracefully here.
+  replicas: 2
 ingressRoutes:
-  - host: pihole.home.keithjackson.dev
+  - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
 auth:
   mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
   groups:
     - platform-admins         # sensitive admin UI: authenticated AND in group (R-IDP-06)
+  # NO `apiRoutes` HERE, DELIBERATELY. PiHole's admin UI serves no /api/ path, so the `^/api/`
+  # that jackett and radarr need would match nothing on a good day — and a BROADER guess would
+  # answer PiHole's own page loads with 401, leaving a logged-out operator on a bare error page
+  # with no sign-in offered. This app is the reason auth.apiRoutes is per-service (schema 1.9.0)
+  # rather than a renderer constant: the measurement that justifies the flag on two apps is
+  # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
+  # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
+lifecycle:
+  # LEVEL-TRIGGERED ADLIST ASSERTION, run on EVERY container start. Same pattern, and the same
+  # reasoning, as catalog/qbittorrent.yaml's postStart: desired state lives in git and is
+  # RE-ASSERTED unconditionally rather than set once by hand.
+  #
+  # WHY THIS IS NEEDED AT ALL — the blocklists were NOT reproducible from git. Measured on live prod
+  # 2026-10-04: `gravity.db` held exactly ONE adlist (StevenBlack's hosts, enabled) and 72,234
+  # gravity domains, and NOTHING in this repo named it. Worse, a FRESH database does not seed it:
+  # the image's own schema template `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the
+  # `adlist` table and inserts NO rows into it (grepped in the running image — there is no
+  # `INSERT INTO adlist` at all). So a rebuild from git, or a replica with a new PVC, would come up
+  # SERVING DNS AND BLOCKING NOTHING, silently and with every object reporting healthy.
+  #
+  # WHY A postStart HOOK AND NOT A FILE. PiHole v6 keeps adlists as DB rows, not config. The
+  # `/etc/pihole/adlists.list` file still exists in `gravity.sh` but is MIGRATION-ONLY: its import
+  # sits inside `migrate_to_database()`, guarded by `if [ ! -e "$gravityDBfile" ]`, so it fires only
+  # while CREATING a database and never again on an existing one. A ConfigMap mounted there would
+  # therefore be silently ignored on every pod this cluster currently runs. There is also no
+  # `pihole adlist` CLI subcommand (only allow/deny for domains), so SQL against the DB is the
+  # actual interface. FTL ships an embedded shell (`pihole-FTL sqlite3`, SQLite 3.53.1) so no extra
+  # tooling is needed in the image.
+  #
+  # IT RECONCILES, IT DOES NOT JUST ADD: adlists present in the DB but absent from DECLARED below are
+  # DISABLED (not deleted — deleting cascades to `gravity` rows and would force a full re-download of
+  # everything on a typo). So this list is the source of truth in both directions, and `pihole -g`
+  # runs ONLY when something actually changed, which keeps the steady-state start free of a 70k-domain
+  # rebuild.
+  #
+  # NO BRACED DOLLAR EXPANSION ANYWHERE IN THIS SCRIPT, DELIBERATELY — the same contract
+  # catalog/qbittorrent.yaml documents at length. apps-wave-0 declares postBuild.substituteFrom
+  # cluster-vars, so Flux runs envsubst over this whole manifest at reconcile time; a `${VAR}` form
+  # would behave differently on the cluster than under a plain `kubectl apply` of the golden. Bare
+  # `$VAR` and `$(cmd)` are untouched by envsubst and so behave identically however this is run.
+  # `render.py`'s assert_substitution_safe refuses to emit a manifest that breaks this.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and
+          # does NOT run a gravity rebuild.
+          set -u
+          GRAVITY=/etc/pihole/gravity.db
+
+          # THE DECLARED SET — one URL per line, the source of truth for what PiHole blocks.
+          # To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.
+          DECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
+
+          # On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,
+          # and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.
+          i=0
+          while [ ! -s "$GRAVITY" ] && [ "$i" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done
+          if [ ! -s "$GRAVITY" ]; then
+            echo "pihole-adlists: gravity.db absent after 180s - leaving adlists untouched" >&2
+            exit 0
+          fi
+
+          sql() { pihole-FTL sqlite3 "$GRAVITY" "$1"; }
+          CHANGED=0
+
+          # 1) Every declared URL must be present AND enabled.
+          echo "$DECLARED" | while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            printf '%s\n' "$url"
+          done > /tmp/declared.txt
+          while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.
+            have=$(sql "select count(*) from adlist where address='$url';")
+            if [ "$have" = "0" ]; then
+              echo "pihole-adlists: adding $url"
+              sql "insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');"
+              CHANGED=1
+            else
+              en=$(sql "select enabled from adlist where address='$url';")
+              if [ "$en" != "1" ]; then
+                echo "pihole-adlists: re-enabling $url"
+                sql "update adlist set enabled=1 where address='$url';"
+                CHANGED=1
+              fi
+            fi
+          done < /tmp/declared.txt
+
+          # 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).
+          sql "select address from adlist where enabled=1;" | while IFS= read -r addr; do
+            [ -n "$addr" ] || continue
+            if ! grep -qxF "$addr" /tmp/declared.txt; then
+              echo "pihole-adlists: disabling undeclared $addr"
+              sql "update adlist set enabled=0 where address='$addr';"
+              echo changed >> /tmp/adlist-changed
+            fi
+          done
+          [ -s /tmp/adlist-changed ] && CHANGED=1
+          rm -f /tmp/adlist-changed /tmp/declared.txt
+
+          # 3) Rebuild gravity ONLY if the declared set actually moved.
+          if [ "$CHANGED" = "1" ]; then
+            echo "pihole-adlists: declared set changed - running gravity update"
+            pihole -g || echo "pihole-adlists: gravity update failed; existing gravity.db retained" >&2
+          else
+            echo "pihole-adlists: already matches the declared set - nothing to do"
+          fi
+          exit 0
 update:
   channel: stable             # R-UPD-01
   range: ">=2024.0.0"

catalog/plex.yaml

@@ -4,25 +4,15 @@
 # auth: native (Plex accounts, NOT Keycloak — docs/13 + docs/15 documented exception).
 # Validates against schemas/servicedefinition.schema.json.
 #
-# BULK MEDIA (spec bulk-tier requirement, docs/04) — OPEN PREREQUISITE, intentionally NOT declared
+# BULK MEDIA — NOW ATTACHED (docs/04 bulk tier; the long-standing OQ-2 prerequisite, resolved for
-# here yet (the requirement ID is cited in the WS-APPS report, not as a token here: this change does
+# the STATIC case). The library is 8.33 T of ZFS on the Proxmox host at /mnt/Storage/plex, and it
-# NOT exercise the bulk tier, so it must not be selected as a "touched" requirement whose in-cluster
+# already exists: nothing is provisioned, an existing dataset is mounted. See
-# conformance test is still a Phase-1 gap).
+# docs/BULK-NFS-STORAGE.md for the layout, the export change and the uid analysis. (The bulk-tier
-#   The Plex library belongs on the `bulk` tier (docs/04 / docs/13 cross-cutting note: metadata DB
+# requirement ID is deliberately still not typed here — see that document's "gate" note.)
-#   on replicated Longhorn, large library on bulk). The `bulk` StorageClass exists
-#   (platform/longhorn/storageclasses.yaml) and the schema+renderer both support `class: bulk`, but
-#   that class is a documented STUB: OQ-2 (external NFS/SMB backing) is UNRESOLVED and the NFS CSI
-#   driver is not installed, so a `bulk` PVC will not bind and would leave the Plex pod Pending.
-#   To onboard the running Plex shell now (compute + config + public exposure), the media volume is
-#   omitted. Once OQ-2 is resolved and `bulk` provisions, attach the library with a one-line add:
-#     storage:
-#       - name: media
-#         class: bulk        # external mount; size omitted (sized outside the cluster)
-#   then re-render. See the WS-APPS report for the full bulk-media finding.
 name: plex
 kind: container
 source:
-  image: plexinc/pms-docker:latest
+  image: plexinc/pms-docker:1.43.6.11085-d1e52e169
 # plexinc/pms-docker env. Rendered VERBATIM onto the app container by render.py (schema env list).
 env:
   # PLEX_UID/PLEX_GID: the image boots as root (s6 PID1), chowns /config to this uid/gid, then drops
@@ -42,37 +32,373 @@ env:
   # (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
   - name: ALLOWED_NETWORKS
     value: 10.0.0.0/8,192.168.0.0/16
-  # PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
+  # PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
-  # shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
+  # (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
-  # is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
+  # workload). Plex cannot create libraries over the mounted bulk share while the server is
-  # single API POST) against the running server; the claim then persists on the /config PVC and
+  # UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
-  # survives pod rolls (including the roll this env change causes), so it never belongs in git.
+  # half of that reasoning survived contact with the upstream source.
-  # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
+  #
-  # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
+  # A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
-  # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
+  # expires in MINUTES — read the countdown the claim page renders; do not budget against a number
-  # Gateway VIP (conformance/tests/l2-behavioral/dns-internal-resolution), and both its public
+  # written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
-  # reachability and whether the TLS-terminating Cilium Gateway proxies Plex's 32400 API correctly for
+  # UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
-  # remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
+  # for the whole procedure design — including a since-disproven claim that the delivery pipeline was
-  # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
+  # too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
-  # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
+  #
-exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
+  # The live token is injected by the operator inside that window, with an explicit `flux reconcile`
+  # to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
+  # the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
+  # committed value is the spent token from that claim rather than the original placeholder.
+  #
+  # ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
+  # (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+  # the running `:latest` image), NOT assumed:
+  #   - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
+  #     PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
+  #     no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
+  #     logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
+  #   - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
+  #     status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
+  #     there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
+  #     SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
+  #     default 0, which continues startup even when a cont-init script fails.
+  # So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
+  # placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
+  # so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
+  # listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
+  # is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
+  - name: PLEX_CLAIM
+    valueFrom:
+      secretKeyRef:
+        name: plex-claim
+        key: PLEX_CLAIM
+  # ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv.
+  #
+  # THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single
+  # property is what this field keeps getting wrong, so it is the first thing to check before
+  # changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it
+  # happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by
+  # default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some
+  # clients hard-code a public resolver.
+  #
+  # THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly
+  # advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers
+  # (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve
+  # it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the
+  # router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the
+  # router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443`
+  # and the WAN address, while PiHole itself was answering perfectly — both replicas returned
+  # `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on
+  # it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left
+  # the Plex path entirely and were talking to the router.
+  #
+  # SO: `<dashed-ip>.<serverhash>.plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own
+  # mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private
+  # address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH
+  # browser — answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no
+  # split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert
+  # Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL
+  # validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**.
+  #
+  # WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant:
+  # it cannot be made to work for clients that bypass PiHole. A public record for it would resolve
+  # to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too —
+  # so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no
+  # configuration of that name that is right for both. `plex.direct` is right for both.
+  # (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful
+  # for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.)
+  #
+  # THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track
+  # ${PLEX_L4_VIP} and `<serverhash>` is this server's certificate identity — read it from the live
+  # listener with
+  #     openssl s_client -connect <vip>:32400 </dev/null | openssl x509 -noout -subject
+  # Nothing in this tree can derive either, and nothing checks that they still agree: envsubst
+  # cannot dash an IP, so this is a literal. That fragility is real and was the reason the hostname
+  # was tried in the first place — it is accepted here because an unresolvable name is a WORSE
+  # failure (silent, LAN-wide, and it points users at the router) than a stale literal, which at
+  # least fails loudly and only after someone changes the VIP.
+  #
+  # The image's /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections"
+  # "${ADVERTISE_IP}"` (line 127, verified in the running image), OUTSIDE the
+  # PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY
+  # fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web
+  # form, owns the value.
+  - name: ADVERTISE_IP
+    value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
+  # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
+  # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
+  # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
+  # workload). The postStart hook converts it to the PKCS#12 bundle Plex requires.
+  #
+  # WHY ENV AND NOT A VOLUME, which would be the conventional choice. MEASURED 2026-10-04:
+  # `schemas/servicedefinition.schema.json` `storage` accepts only
+  # [class, export, mountPath, name, readOnly, size, subPath] with `additionalProperties: false`,
+  # `catalog/render.py` emits NO secret volumes, and no rendered app mounts a Secret as a volume.
+  # Mounting one therefore needs a schema + renderer change; `env.valueFrom.secretKeyRef` is
+  # already in the schema and already used by PLEX_CLAIM above. PEM is text, so it survives an env
+  # var intact (a raw PKCS#12 would not — it is binary and carries NUL bytes).
+  #
+  # COST, STATED PLAINLY: the private key is readable from this container's environment
+  # (/proc/1/environ, `kubectl exec env`) by anything that can already exec into the pod. That is
+  # the same audience that can read the converted bundle under /config, so the marginal exposure is
+  # small — but a secret VOLUME would still be the better shape, and adding secret-volume support
+  # to the schema is the clean follow-up that removes this note.
+  #
+  # `optional: true` IS LOAD-BEARING, NOT DEFENSIVE CLUTTER (schema 1.11.0, added for this). The
+  # Certificate is issued INTO THIS NAMESPACE, and the namespace is created by the rendered
+  # apps/wave-0/plex.yaml itself — so the cert CANNOT be issued in an earlier tier, and the
+  # Certificate and this Deployment land in the SAME reconcile. On a cold bootstrap the Secret is
+  # therefore briefly absent while cert-manager completes a DNS01 challenge. With a HARD ref, Plex
+  # would sit in CreateContainerConfigError until a certificate issued — and because `apps-wave-0`
+  # is `wait: true` with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE
+  # APP TIER (the same hazard the `hardware` block below documents for the GPU resource). Optional
+  # instead means: no cert -> Plex still starts and serves, the postStart hook logs that the
+  # certificate was not installed, and the next start after issuance converges. Plex falls back to
+  # its own `*.plex.direct` certificate in the meantime, so the degraded state is "advertised URL
+  # not yet servable over TLS", not "media server down".
+  - name: PLEX_TLS_CRT
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.crt
+        optional: true
+  - name: PLEX_TLS_KEY
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.key
+        optional: true
+lifecycle:
+  # LEVEL-TRIGGERED PLEX SETTINGS, re-asserted on EVERY container start — the same pattern and the
+  # same reasoning as catalog/pihole.yaml's and catalog/qbittorrent.yaml's postStart: desired state
+  # lives in git and is re-asserted unconditionally rather than typed into a web form once. Typing
+  # settings into the web form is precisely what took this service down on 2026-10-04.
+  #
+  # IT EXISTS BECAUSE THE IMAGE HAS NO ENV FOR THESE SETTINGS. Measured 2026-10-04 across the whole
+  # init surface (/etc/cont-init.d, /etc/services.d, /plex-common.sh, /usr/local/bin): the image
+  # calls setPref for exactly SIX prefs — MachineIdentifier, ProcessedMachineIdentifier,
+  # PlexOnlineToken, customConnections, allowedNetworks, TranscoderTempDirectory — and supports NO
+  # generic PLEX_PREFERENCE* hook of any kind. The two things we need (a LAN definition and a custom
+  # certificate) are reachable ONLY through Plex's own API, so this hook is the only GitOps path.
+  #
+  # IT MUST NEVER FAIL. A postStart hook that exits non-zero makes kubelet KILL the container, so
+  # every step is bounded and the script ends with an unconditional `exit 0`. A missing token, an
+  # unissued certificate or an unreachable API degrades to "not applied this start" and is retried
+  # on the next one; it never crashloops Plex.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # NOTE — LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.
+          # This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a
+          # brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD
+          # Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be
+          # replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` — bare. Only
+          # BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be
+          # substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell
+          # variable written with braces would be blanked or would fail the whole Kustomization —
+          # this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and
+          # catalog/render.py now refuses to emit it (which is how this very comment got caught).
+          # DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the
+          # certificate has not issued yet they are genuinely UNSET, not empty — and under `set -u`
+          # the first `[ -n "$PLEX_TLS_CRT" ]` would abort the script, which for a postStart hook
+          # means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists
+          # to prevent, so turning it back on here would defeat the whole design. (The alternative,
+          # a brace-wrapped `:-` default, would need envsubst escaping — the trap the note above
+          # describes.) Every value below is explicitly guarded instead.
+          PX="/config/Library/Application Support/Plex Media Server/Preferences.xml"
+          CERTDIR=/config/certs
+          P12=$CERTDIR/plex-vip.p12
+          # The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it
+          # in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a
+          # container-format requirement, not an access control.
+          P12PASS=plexvip
+          LANNETS='10.0.0.0/8,192.168.0.0/16'
+          CERTDOMAIN=plex-vip.${BASE_DOMAIN}
+          API=http://127.0.0.1:32400
+
+          # (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so
+          # a cert-manager RENEWAL is picked up with no extra moving part.
+          if [ -n "$PLEX_TLS_CRT" ] && [ -n "$PLEX_TLS_KEY" ]; then
+            mkdir -p $CERTDIR && chmod 700 $CERTDIR
+            printf '%s' "$PLEX_TLS_CRT" > $CERTDIR/tls.crt
+            printf '%s' "$PLEX_TLS_KEY" > $CERTDIR/tls.key
+            chmod 600 $CERTDIR/tls.key
+            if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \
+                 -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then
+              mv -f $P12.tmp $P12
+              chown 1000:1000 $P12 2>/dev/null || true
+              chmod 600 $P12
+              echo "[plex-settings] PKCS#12 bundle written"
+            else
+              rm -f $P12.tmp
+              echo "[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle" >&2
+            fi
+            rm -f $CERTDIR/tls.key
+          else
+            echo "[plex-settings] WARN: TLS env empty; certificate not installed this start" >&2
+          fi
+
+          # (2) Wait, bounded (~90s), for the local API. /identity needs no auth.
+          i=0
+          while [ "$i" -lt 45 ]; do
+            if [ "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)" = "200" ]; then
+              break
+            fi
+            i=$(expr $i + 1); sleep 2
+          done
+
+          # (3) The prefs API needs the server token. It already lives in Preferences.xml on this
+          # PVC; it is never echoed here, and Plex redacts tokens in its own request log. An
+          # UNCLAIMED server has none — a normal state on a fresh /config — so skip, do not fail.
+          TOKEN=$(grep -o 'PlexOnlineToken="[^"]*"' "$PX" 2>/dev/null | head -1 | cut -d'"' -f2)
+          if [ -z "$TOKEN" ]; then
+            echo "[plex-settings] not claimed yet — skipping prefs, will retry next start" >&2
+            exit 0
+          fi
+
+          # Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on
+          # a PUT, so a raw '/' or ',' would be read as structure rather than data.
+          enc() {
+            printf '%s' "$1" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \
+                                   -e 's|:|%3A|g' -e 's| |%20|g'
+          }
+          setpref() {
+            code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT \
+              -H "X-Plex-Token: $TOKEN" "$API/:/prefs?$1=$(enc "$2")" 2>/dev/null)
+            if [ "$code" = "200" ]; then
+              echo "[plex-settings] set $1"
+            else
+              echo "[plex-settings] WARN: $1 -> HTTP $code" >&2
+            fi
+          }
+
+          # (4) THE LAN DEFINITION — the reason this hook was written. Without it Plex has NO notion
+          # of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests
+          # logged "Allowed Network (WAN)" and ZERO "(LAN)", including Seerr from inside the
+          # cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks
+          # (the ALLOWED_NETWORKS env) only waives AUTH — it does NOT make a client local. Two
+          # different settings; only one of them has an env hook.
+          setpref LanNetworksBandwidth "$LANNETS"
+
+          # (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only
+          # if the bundle exists — pointing Plex at a missing file would break its TLS listener.
+          if [ -s "$P12" ]; then
+            setpref customCertificatePath "$P12"
+            setpref customCertificateKey "$P12PASS"
+            setpref customCertificateDomain "$CERTDOMAIN"
+          else
+            echo "[plex-settings] WARN: no bundle; leaving certificate prefs untouched" >&2
+          fi
+          exit 0
+# HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
+# INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
+# Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
+# How the platform hands the device over is `catalog/render.py`'s business and is documented in
+# `schemas/renderer-contract.md` `### hardware` — read that for the mechanism, never this comment.
+# The payoff is proven, not theoretical: on 2026-10-01 the mechanism was swapped outright (hostPath
+# -> Intel GPU device plugin extended resource) and THIS BLOCK DID NOT MOVE — a `render.py` change
+# plus a re-render, zero catalog edits. Do not "simplify" it into a raw `/dev/...` path; that is the
+# whole point of the enum.
+#
+# THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
+# anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
+# GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
+# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Measured again 2026-10-01, one layer up: all
+# three nodes advertise `gpu.intel.com/i915: 1` in `.status.allocatable`. This field renders no
+# nodeSelector; placement is a separate concern with its own justification.
+#
+# PLEX'S SCHEDULABILITY IS NOW COUPLED TO THE `intel-gpu-plugin` DaemonSet, and that is a real cost,
+# not a footnote. The current mechanism is a hard scheduling constraint: if the resource is
+# unavailable (plugin broken, removed, or not yet reconciled) this pod is UNSCHEDULABLE, and because
+# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that STALLS THE WHOLE APP TIER.
+# The previous hostPath mechanism degraded silently to software transcoding instead — but it never
+# actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep.
+# THE ESCAPE HATCH IS THIS BLOCK: delete these two lines and re-render, and Plex goes back to
+# software transcoding and the tier unblocks. One field, no code change. `runbooks/plex-quicksync.md`
+# carries that procedure for whoever needs it at 2am.
+#
+# NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
+# The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
+# the host `render` GID in as a supplemental group. It cannot work here, and this survived the
+# mechanism swap unchanged. MEASURED IN THE RUNNING CONTAINER, 2026-09-30 and again 2026-10-01: the
+# image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device
+# under `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
+# `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
+# GID renumbering self-healing (no GID is pinned anywhere in this repo) and which works the same way
+# on a PLUGIN-INJECTED device as on a bind-mounted one. Measured live, `Plex Media Server` ran with
+# `Groups: 44 100 992 1000` — 992 IS the host `render` GID, i.e. the hook had already done the job.
+# The service execs via `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from
+# `/etc/group` rather than appending to it, so root's gid `0` — which PID 1 (`s6-svscan`) does hold —
+# is GONE, and anything Kubernetes supplies reaches s6-svscan and is discarded before Plex ever
+# starts. The group is the image's job and it does it correctly. `render.py --selftest` asserts the
+# ABSENCE of a pod securityContext for this field precisely so nobody adds an inert one back later.
+#
+# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Handing over the device does
+# not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
+# is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
+# set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
+# `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support,
+# and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
+# `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
+# unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
+# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
+hardware:
+  gpu: intel-quicksync
+# EXPOSURE: internal (operator decision 2026-11-08, Cloudflare Tunnel cut-over). The plex.${BASE_DOMAIN}
+# web UI is LAN/VPN-only: `internal` renders `external-dns ... controller: none` on the HTTPRoute, so
+# external-dns DELETES the public plex record and its TXT, and platform/cloudflared/config.yaml maps the
+# name to http_status:404 so it can never be served through the tunnel either. REMOTE STREAMING IS NOT
+# AFFECTED: it never used this name. It rides plex.tv + the advertised plex.direct URL (ADVERTISE_IP
+# above) to the raw tcp/32400 L4 VIP (${PLEX_L4_VIP}, apps/wave-0/plex-remote.yaml) via the WAN
+# tcp/32400 forward — none of which this field touches. See runbooks/plex-remote-access.md.
+exposure: internal            # web UI LAN/VPN-only; remote streaming = plex.tv + WAN tcp/32400 (docs/09; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: config
-    size: 8Gi
+    size: 16Gi
     class: longhorn           # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
     mountPath: /config        # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
                               # renderer default /data/config. Without this the PVC sat unused and Plex wrote to the
                               # container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
                               # root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
                               # (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
-                              # makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
+                              # makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
-                              # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
+                              # above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
-                              # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
+                              # exchange is then skipped on every later start. (The older note here claimed a
-                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
+                              # placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
+                              # otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
+                              # failure, and that is why one is committed.)
+  # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
+  # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
+  # downloads/ movies/ photos/ tv/).
+  #
+  # WHY THE ROOT AND NOT THREE subPath MOUNTS. Nothing is gained by narrowing: the mount is `ro`,
+  # so no subPath could remove a privilege that matters, while three mounts would triple the NFS
+  # mounts per pod and mean a manifest change every time a top-level library is added.
+  #
+  # WHY READ-ONLY, when the export is `rw` and the other consumers write. Plex is a SERVER: it
+  # indexes and streams. Every write on that tree belongs to Sonarr and Radarr. Marking it here
+  # costs nothing and removes Plex from the set of things that can damage 8.33 T. Enforced by the
+  # PV's nfs.readOnly (a real `ro` mount), not only by the volumeMount flag.
+  #
+  # UID: no change needed. The tree is 0755 owned 1000:3000 throughout, so it is world-readable and
+  # world-traversable, and PLEX_UID/PLEX_GID above already make the process uid 1000 in any case.
+  # (fsGroup would have been a no-op here regardless — the in-tree NFS plugin does no ownership
+  # management. That is the trap the arrs' PUID/PGID exists to avoid; Plex never hits it.)
+  - name: media
+    class: bulk
+    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
+    mountPath: /media
+    readOnly: true
+    size: 9Ti                   # ADVISORY ONLY (nothing enforces a static NFS PV's capacity): the
+                                # dataset's 8.33 T used, rounded up, so `kubectl get pv` reports
+                                # something true. The real limit is the ZFS pool.
 dependencies: []              # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: plex.home.keithjackson.dev
+  - host: plex.${BASE_DOMAIN}
     port: 32400               # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
 auth:
   mode: native                # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)

catalog/qbittorrent.yaml

@@ -4,19 +4,18 @@
 # internal download client | egress VPN + kill-switch | config on Longhorn | auth forward-auth
 # (bare web UI, no native OIDC).
 #
-# BULK DOWNLOADS TIER — OPEN PREREQUISITE, intentionally NOT declared here (mirrors plex + sonarr/radarr):
+# BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record).
-#   docs/13 row 13 places qBittorrent's downloads/staging volume on the `bulk` tier (large, non-3x-
+#   This entry used to ship config-only, with a comment promising to attach the downloads volume "once
-#   replicated media handoff to the arr stack). The `bulk` StorageClass exists
+#   OQ-2 is resolved". OQ-2 was resolved for STATIC pre-existing datasets (schema 1.5.0 / renderer
-#   (platform/longhorn/storageclasses.yaml) and both the schema and renderer support `class: bulk`, BUT
+#   `class: bulk` + `export` -> a static, pre-bound NFS PV), and plex/sonarr/radarr/nextcloud all took
-#   it is a NON-PROVISIONING STUB pending OQ-2 (its NFS/SMB backing is REPLACE-ME and the NFS CSI driver
+#   their media mounts — but qBittorrent's was never added, so the DOWNLOAD CLIENT was the one member
-#   is not installed) — a `bulk` PVC would never bind and would leave the pod Pending. Declaring it now
+#   of the chain with no media volume at all. Measured on prod 2026-09-12:
-#   would also pull the bulk-tier requirement into the change's "touched" set, whose in-cluster
+#       Session\DefaultSavePath=/downloads/   Session\TempPath=/downloads/incomplete/
-#   conformance test is not yet scaffolded, blocking the merge gate. So the downloads volume is DEFERRED
+#       Downloads\SavePath=/downloads/       `ls /downloads` -> No such file or directory
-#   and this onboarding ships config-only. Once OQ-2 is resolved and `bulk` provisions, attach downloads
+#   i.e. the app was configured to write a path that DID NOT EXIST, in a container with no volume there.
-#   with a one-line add under storage:
+#   Anything it did manage to create would have been invisible to sonarr/radarr, lost on every pod
-#         - name: downloads
+#   recreation, and filling the NODE's ephemeral disk in the meantime. The `downloads` entry under
-#           class: bulk          # external mount; size omitted (sized outside the cluster)
+#   storage: below closes that. See docs/BULK-NFS-STORAGE.md and the bulk-tier L0 conformance test.
-#   then re-render. See the WS-APPS report for the full bulk-downloads finding.
 #
 # VPN-class (docs/13 row 13; docs/03 "VPN-class set"): ALL of this pod's public-internet egress
 # (peer/tracker/DHT traffic) MUST leave via the node PIA tunnel, never the WAN. `egress: vpn` renders
@@ -31,21 +30,430 @@
 name: qbittorrent
 kind: container
 source:
-  image: lscr.io/linuxserver/qbittorrent:5.1.2   # pinned stable tag (R-UPD-01); update.range governs advancement
+  image: lscr.io/linuxserver/qbittorrent:5.3.0   # pinned stable tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
+# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
+# LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
+# 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
+# uid=911 gid=911, and /mnt/Storage/plex/downloads is 0755 owned 1000:1000. 0755 grants write to the
+# OWNER ONLY, so uid 911 would have mounted the export cleanly and then failed every write with
+# EACCES — the "mounts fine, cannot write" failure the sonarr entry warns about at length.
+env:
+  - name: PUID
+    value: "1000"               # owner uid of /mnt/Storage/plex/downloads (and of every dir under the export)
+  - name: PGID
+    value: "3000"               # the export's dominant group, matching sonarr/radarr: files this writes stay
+                                # readable to the EXISTING non-k3s consumers of the same dataset, and land
+                                # group-owned the same way the arrs' own writes do
+  # --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
+  # Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
+  # NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
+  # NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
+  # password is provided for this session: ..."). Two real consequences, not hypotheticals:
+  #   * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
+  #     tested valid against the then-current temporary password; both would have started failing at
+  #     the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
+  #     report a failed client, not a failed import).
+  #   * The human operator had no stable login: every restart meant reading the pod log.
+  # Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
+  # same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
+  - name: QBT_WEBUI_USERNAME
+    valueFrom:
+      secretKeyRef:
+        name: qbittorrent-bootstrap
+        key: webui-username
+  - name: QBT_WEBUI_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: qbittorrent-bootstrap
+        key: webui-password
+lifecycle:
+  # LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
+  #
+  # WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
+  # directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
+  # apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
+  # RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
+  # the container start rather than a clock, because the failure this closes IS a container start:
+  #   * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
+  #     PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
+  #     restore or rebuild in which the app serves a random password and both arrs are broken. The
+  #     hook has no such window: it runs before/with the very start that would otherwise regenerate.
+  #   * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
+  #     PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
+  #     then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
+  #     In-container, before the service starts, none of that exists.
+  #   * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
+  #     privilege than the CronJob alternative (R-SEC/least-privilege).
+  # The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
+  # keeps in its own state), and the schema documents this hook for exactly that use.
+  #
+  # AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
+  # BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
+  # would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
+  #   * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
+  #     credential — because the WebUI is not running yet. That assertion is the most valuable line
+  #     in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
+  #     writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
+  #   * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
+  #     mean a schema version bump plus a renderer change — a blast radius across every service's
+  #     golden — to buy something strictly weaker than what the hook already does.
+  # The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
+  # the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
+  #
+  # DURABILITY, the whole point — all three paths end with the SAME credential:
+  #   pod restart      -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
+  #   config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
+  #                       hook runs and re-asserts over whatever the snapshot contained.
+  #   rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
+  #                       credential into it before the service is serving. Nothing outside git.
+  #
+  # WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
+  # for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
+  # depend on network policy staying correct forever. The WebUI keeps demanding a password; that
+  # password is simply now known, fixed, and reproducible from git.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
+          # running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
+          #
+          # THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
+          # apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
+          # this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
+          # enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
+          # CLUSTER — but only there:
+          # envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
+          # apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
+          # development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;
+          # see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
+          # are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
+          # however it is run.
+          set -u
+          CONF=/config/qBittorrent/qBittorrent.conf
+          SVC=/run/service/svc-qbittorrent
+          # The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
+          # ServiceDefinition — which is the single place the renderer takes the containerPort, the
+          # Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
+          # env var for a lookup to read anyway.
+          # It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
+          # that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
+          # so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
+          # matched. Every curl below then got an unparseable URL, returned nothing, and the hook
+          # logged "the WebUI rejected the durable
+          # credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
+          # service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
+          # against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
+          PORT=8080
+          # A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
+          # that only echoes is effectively silent (gitea's hits the same wall and writes a marker
+          # file for it). Every outcome therefore also lands, timestamped, on the persistent config
+          # volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
+          # credential from git was asserted and accepted on this start.
+          log() {
+            echo "[qbt-webui-cred] $*"
+            printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
+              >> /config/qbittorrent-webui-cred.log 2>/dev/null || true
+          }
+
+          # (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
+          #     before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
+          #     racing to create one it would then not overwrite.
+          n=0
+          while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
+
+          waited=0
+          attempt=0
+          while [ "$attempt" -lt 3 ]; do
+            attempt=$((attempt + 1))
+
+            # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
+            #     10 = rewritten, anything else = could not write.
+            python3 - "$CONF" <<'PY'
+          import base64, hashlib, hmac, os, stat, sys
+
+          conf = sys.argv[1]
+          user = os.environ.get("QBT_WEBUI_USERNAME", "")
+          pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
+          if not user or not pw:
+              sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
+              raise SystemExit(1)
+
+          # qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
+          # iterations, 16-byte random salt, 64-byte digest, stored as
+          #   WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
+          # Get any one of them wrong and the app simply rejects every login.
+          ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
+          UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
+
+          def matches(raw):
+              """True iff the stored digest is one OUR password produces (salt is read from it)."""
+              v = raw.strip().strip('"')
+              if v.startswith("@ByteArray(") and v.endswith(")"):
+                  v = v[len("@ByteArray("):-1]
+              try:
+                  salt_b64, digest_b64 = v.split(":", 1)
+                  salt = base64.b64decode(salt_b64, validate=True)
+                  digest = base64.b64decode(digest_b64, validate=True)
+              except Exception:
+                  return False
+              if not salt or not digest:
+                  return False
+              return hmac.compare_digest(
+                  hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
+
+          try:
+              lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
+          except FileNotFoundError:
+              lines = []
+
+          def value_of(key):
+              for i, line in enumerate(lines):
+                  if line.startswith(key + "="):
+                      return i, line.split("=", 1)[1]
+              return -1, None
+
+          ui, uval = value_of(UKEY)
+          pi, pval = value_of(PKEY)
+          if uval == user and pval is not None and matches(pval):
+              raise SystemExit(0)          # converged - leave the file (and its salt) alone
+
+          salt = os.urandom(SALT_LEN)
+          digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
+          stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
+          desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
+
+          out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
+          if SECTION in out:
+              at = out.index(SECTION) + 1
+          else:
+              if out and out[-1].strip():
+                  out.append("")
+              out.append(SECTION)
+              at = len(out)
+          out[at:at] = desired
+
+          try:
+              os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
+              tmp = conf + ".webui-cred.new"
+              with open(tmp, "w", encoding="utf-8") as fh:
+                  fh.write("\n".join(out) + "\n")
+                  fh.flush()
+                  os.fsync(fh.fileno())
+              # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
+              # cost the app the ability to save its own settings, so carry the ownership over.
+              try:
+                  st = os.stat(conf)
+                  os.chown(tmp, st.st_uid, st.st_gid)
+                  os.chmod(tmp, stat.S_IMODE(st.st_mode))
+              except FileNotFoundError:
+                  os.chown(tmp, int(os.environ.get("PUID", "1000")),
+                           int(os.environ.get("PGID", "1000")))
+                  os.chmod(tmp, 0o664)
+              os.replace(tmp, conf)        # atomic: the app never reads a half-written conf
+          except OSError as exc:
+              sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
+              raise SystemExit(1)
+          raise SystemExit(10)
+          PY
+            rc=$?
+
+            if [ "$rc" = 0 ]; then
+              log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
+            elif [ "$rc" = 10 ]; then
+              log "asserted WebUI Username + Password_PBKDF2 into $CONF"
+              # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
+              # have read the old, credential-less conf and be serving a random temporary password.
+              # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
+              # Nothing to do when the service has not started yet - it will read the new file.
+              # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
+              # 2026-09-13 failure spent the entire verification budget before the first login and
+              # then fired attempts 2 and 3 with no wait left at all.
+              sv=0
+              while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
+                sv=$((sv + 1)); sleep 1
+              done
+              if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
+                log "qbittorrent-nox was already running - restarting it to load the credential"
+                s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
+                sleep 3
+              fi
+            else
+              log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
+              log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
+              exit 0
+            fi
+
+            # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
+            #     (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
+            #     at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
+            #
+            # "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
+            # conflating them is what made this hook lie once already (see the PORT comment). An
+            # unreachable WebUI is reported as unreachable, and the hook stops rather than spending
+            # its remaining login attempts to produce two more identical wrong diagnoses.
+            api_ok=0
+            while [ "$waited" -lt 120 ]; do
+              code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
+              if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
+              waited=$((waited + 1)); sleep 1
+            done
+            if [ "$api_ok" != 1 ]; then
+              log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
+              log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
+              log "never submitted. Do not rotate the credential on the strength of this line." >&2
+              exit 0
+            fi
+            body=$(curl -sS -m 10 \
+                     --data-urlencode "username=$QBT_WEBUI_USERNAME" \
+                     --data-urlencode "password=$QBT_WEBUI_PASSWORD" \
+                     "http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
+            if [ "$body" = "Ok." ]; then
+              log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
+              exit 0
+            fi
+            log "WebUI rejected the durable credential (login said '$body')" >&2
+            # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
+            # the running qbittorrent-nox is serving something it read from an EARLIER conf, so
+            # rewriting the same bytes on the next attempt would change nothing. The remedy is to
+            # make the process RE-READ the file, so restart the supervised service and let the next
+            # attempt re-verify. Without this the retry loop could never change its own outcome.
+            if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
+              log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
+              s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
+              sleep 3
+            fi
+          done
+
+          log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
+          # A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
+          # Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
+          # app, and the arrs' own download-client health check surfaces a broken credential.
+          exit 0
+podSecurityContext:
+  # For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
+  # on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
+  # the kubelet applies it, so /config becomes group-writable by the now-uid-1000 process without
+  # depending on the image chowning it. (This image DOES chown: init-qbittorrent-config runs
+  # `lsiown -R abc:abc /config`. Belt and braces — the live /config was 911:911 before this change,
+  # and losing write access to it would lose the WebUI password hash and the session state.)
+  fsGroup: 3000
+exposure: internal              # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
 egress: vpn                     # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
     size: 5Gi
     class: longhorn             # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
-dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
+    mountPath: /config          # THE APP WRITES /config, NOT the renderer default /data/config.
+                                # LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
+                                # /config), so without this the Longhorn PVC mounted at /data/config sat
+                                # holding nothing but lost+found while the app wrote its entire
+                                # configuration to the container's EPHEMERAL writable layer — destroyed on
+                                # every pod recreation: restart, reschedule, image bump. Measured in-cluster
+                                # 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
+                                # dev=2053 on the node's own disk with [Sentry asp config.xml logs].
+                                # This is the SAME failure catalog/plex.yaml:54 already records fixing.
+                                # The live /config contents were copied onto each PVC before this landed,
+                                # so API keys and credentials other apps reference were preserved.
+  # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
+  # (192.168.2.30:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
+  # /downloads, which is precisely where this app is ALREADY configured to write.
+  #
+  # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
+  #   * LEAST PRIVILEGE, AND IT COSTS NOTHING HERE. A download client has no business writing the
+  #     LIBRARY. The arrs need the root mount because THEY hardlink downloads/ -> tv/ or movies/ and
+  #     link() is a same-MOUNT operation (EXDEV otherwise) — their smallest common ancestor genuinely
+  #     IS the export root. qBittorrent performs no such cross-directory link: it writes into the
+  #     downloads tree and nothing else, so narrowing to that subtree removes movies/, tv/ and photos/
+  #     from this pod's mount namespace without removing any operation it actually performs.
+  #   * HARDLINKS STILL WORK, because they are not done here. The arrs still see ONE mount containing
+  #     both /media/downloads and /media/tv, so their link() stays intra-mount. Proven live 2026-09-12:
+  #     a file created from this pod under /downloads was hardlinked from the sonarr pod between
+  #     /media/downloads/... and /media/tv/... with st_nlink 1 -> 2 and no EXDEV.
+  #   * IT IS ONE MOUNT, so the L0 test `bulk-tier-distinct-from-replicated` assertion (6) — no
+  #     workload mounts one export twice — holds. That assertion is about TWO mounts inside ONE pod;
+  #     a single subPath mount is not the EXDEV trap, two are.
+  #   * The narrowing is a CLIENT/kubelet guarantee, stated honestly: the NFS server still exports the
+  #     whole dataset to this node, so the containment is the mount namespace, not the server's ACLs —
+  #     the same class of caveat docs/BULK-NFS-STORAGE.md §5 records for plex's readOnly.
+  #
+  # NOT readOnly — this is the one workload whose whole job is WRITING here.
+  #
+  # KNOWN, DELIBERATE SIDE EFFECT ON SHARED STATE: the image's init-qbittorrent-config runs
+  # `lsiown abc:abc /downloads` (non-recursive) whenever /downloads is a mount, so starting this pod
+  # chgrps /mnt/Storage/plex/downloads from 1000:1000 to 1000:3000 on the SERVER. Benign and verified:
+  # the directory is mode 0755, whose group bits (r-x) are identical to its other bits, so no consumer
+  # — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with
+  # non-cluster consumers, which should never be a surprise.
+  #
+  # PATH AGREEMENT WITH THE ARRS IS STILL *NOT* CLOSED — see DECISIONS-LOG 2026-09-12 "Remote Path
+  # Mappings". This pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
+  # /media/downloads/<name>. Same inode, different absolute path, so Completed Download Handling needs
+  # ONE Remote Path Mapping per arr (remote host `qbittorrent.qbittorrent.svc.cluster.local`, remote
+  # path /downloads/, local path /media/downloads/).
+  #
+  # THOSE MAPPINGS ARE NOW CONFIGURED AND VERIFIED ON PROD (2026-09-12), together with the qBittorrent
+  # download client itself — which did not exist in EITHER arr until then (both
+  # `GET /api/v3/downloadclient` returned `[]`, so completed-download handling had never been wired up
+  # at all). `downloadclient/testall` reports isValid=true in both, and Sonarr's queue was observed
+  # reporting outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it
+  # present — same torrent, same client.
+  #
+  # BUT THEY LIVE IN THE ARRS' OWN DATABASES (app state on their config PVCs), NOT IN GIT: a rebuild
+  # from git alone does NOT restore them and no gate here notices their absence. The alternative —
+  # repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead —
+  # buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was
+  # not chosen because it changes a running download client's save path, which the deferral above
+  # already shows is easy to get wrong. It is nonetheless the RECOMMENDED end state (it makes the
+  # mappings unnecessary and puts the mountPath half of the fix in git) and is awaiting an operator
+  # decision, not a technical blocker.
+  #
+  # RELATED OPEN BLOCKER: this app has no durable WebUI credential (no WebUI\Username /
+  # WebUI\Password_PBKDF2 in qBittorrent.conf), so v5 issues a NEW temporary `admin` password on every
+  # start. The arrs' saved download-client password therefore goes stale on every pod restart. See the
+  # same DECISIONS-LOG entry for the two fixes, neither of which was taken unilaterally.
+  - name: downloads
+    class: bulk
+    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export
+                                # plex (ro) and sonarr/radarr (rw) mount; one dataset, one set of inodes
+    subPath: downloads          # the handoff subtree ONLY: no movies/, no tv/, no photos/
+    mountPath: /downloads       # matches the app's EXISTING Session\DefaultSavePath / TempPath, so no
+                                # change to config-PVC app state is needed to make this take effect
+    size: 9Ti                   # ADVISORY ONLY on a static NFS PV (the ZFS pool is the real limit): the
+                                # dataset's 8.4 T used rounded up, stated the same way plex/sonarr/radarr
+                                # state it so `kubectl get pv` reports something true
+dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02).
+                                # STILL EMPTY WITH THE NFS MOUNT ABOVE, AND THAT IS CORRECT, NOT AN
+                                # OVERSIGHT. An NFS PersistentVolume is mounted by the NODE KERNEL in the
+                                # HOST netns, not by this pod: there is no pod-side socket to 2049, so no
+                                # CiliumNetworkPolicy can permit or deny it and none is needed. Verified by
+                                # construction — sonarr and radarr have mounted this same export read-write
+                                # for weeks and neither has any CNP naming the NFS server or port 2049
+                                # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
+                                # egress gateway off this traffic (its selector is a podSelector), and
+                                # 192.168.2.30 is inside the 192.168.0.0/16 excludedCIDR besides — see
+                                # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
 ingressRoutes:
-  - host: qbittorrent.home.keithjackson.dev
+  - host: qbittorrent.${BASE_DOMAIN}
     port: 8080                  # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
   groups:
     - media-users               # authenticated AND in an authorized group (R-IDP-06)
+    # See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
+    # platform-admins here is a superset, not a widening (that group already carries cluster-admin
+    # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
+    # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
+    - platform-admins           # operators of these services must not be locked out of them
+  # NO `apiRoutes` HERE: UNMEASURED, not cleared. qBittorrent's WebUI drives everything through
+  # /api/v2/, so it is plausibly affected in the same way jackett is — but plausibility is what
+  # put a renderer constant on the table in the first place. MEASURED 2026-10-01 on the live pod
+  # (qbittorrent-5547bdb9bd-zq557, 0 restarts, 23h): 13,649 log lines, ZERO unauthenticated
+  # requests, nothing but health probes. No traffic means no evidence either way. Add `^/api/`
+  # when someone opens the WebUI and records the 302/401 split, as was done for jackett.
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the qBittorrent 5.x stable line

catalog/radarr.yaml

@@ -8,19 +8,91 @@
 name: radarr
 kind: container
 source:
-  image: lscr.io/linuxserver/radarr:5.21.1   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
+  image: lscr.io/linuxserver/radarr:5.28.0   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
+# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
+# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
+# which cannot write a 1000:3000-owned 0755 tree.
+env:
+  - name: PUID
+    value: "1000"               # owner uid of every directory under /mnt/Storage/plex
+  - name: PGID
+    value: "3000"               # dominant group there; keeps files this writes readable to the
+                                # EXISTING non-k3s consumers of the same export, which stay live
+  - name: TZ
+    value: Etc/UTC              # same safe default plex uses; the arrs schedule by local time
+podSecurityContext:
+  # For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
+  # does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
+  # what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
+  # /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
+  fsGroup: 3000
+exposure: internal              # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
     size: 5Gi
     class: longhorn             # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
-  # NOTE (bulk-media PREREQUISITE): docs/13 row 3 places the media library / downloads volume on the
+    mountPath: /config          # THE APP WRITES /config, NOT the renderer default /data/config.
-  # `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
+                                # LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
-  # but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
+                                # /config), so without this the Longhorn PVC mounted at /data/config sat
-  # volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
+                                # holding nothing but lost+found while the app wrote its entire
-  # therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Radarr deployable today.
+                                # configuration to the container's EPHEMERAL writable layer — destroyed on
+                                # every pod recreation: restart, reschedule, image bump. Measured in-cluster
+                                # 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
+                                # dev=2053 on the node's own disk with [Sentry asp config.xml logs].
+                                # This is the SAME failure catalog/plex.yaml:54 already records fixing.
+                                # The live /config contents were copied onto each PVC before this landed,
+                                # so API keys and credentials other apps reference were preserved.
+  # THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
+  # /media/movies (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
+  #
+  # *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Radarr imports by HARDLINKING or
+  # atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
+  # and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
+  #     /mnt/Storage/plex/movies/                      the library
+  #     /mnt/Storage/plex/downloads/complete/radarr/  where finished releases land
+  # — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
+  # them as two subPath volumes (subPath: movies at /media/movies, subPath: downloads elsewhere) creates
+  # TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Radarr then
+  # silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
+  # seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
+  # is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
+  # requirement, not a shortcut. (The cost is stated plainly: Radarr can also write /media/tv,
+  # which is sonarr's. Narrowing that would cost hardlinks, which is a worse trade.)
+  #
+  # *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
+  # renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
+  # config volume entirely; the pod would come up with its app database shadowed by an 8 T media
+  # tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
+  # the same export, so when qbittorrent is eventually given this mount too, the arrs and the
+  # download client will already agree on absolute paths — which is the other precondition for
+  # hardlinked imports (otherwise every import needs a Remote Path Mapping).
+  #
+  # SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
+  # is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
+  # with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
+  # Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
+  # documents having already been bitten by. Fixing it means starting from an empty config volume, so
+  # it belongs in its own change with the operator present.
+  #
+  # UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
+  # + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
+  # the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
+  # write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
+  # every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
+  #
+  # AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
+  # ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
+  # lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
+  # that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
+  # the image re-chowning /config for itself.
+  - name: media
+    class: bulk
+    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
+    mountPath: /media           # => /media/movies (Root Folder) and /media/downloads (import source)
+    size: 9Ti                   # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
+                                # rounded up, so `kubectl get pv` reports something true.
 dependencies:
   - to: qbittorrent             # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 8080
@@ -28,13 +100,47 @@ dependencies:
   - to: jackett                 # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 9117
     protocol: tcp
+  - to: plex
+    port: 32400                 # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
+    protocol: tcp
 ingressRoutes:
-  - host: radarr.home.keithjackson.dev
+  - host: radarr.${BASE_DOMAIN}
     port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
   groups:
     - media-users               # authenticated AND in the media group (R-IDP-06)
+    # See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
+    # platform-admins here is a superset, not a widening (that group already carries cluster-admin
+    # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
+    # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
+    - platform-admins           # operators of these services must not be locked out of them
+  # Answer Radarr's machine endpoints with 401 instead of a 302 to Keycloak (schema 1.9.0;
+  # renders --api-route, singular and repeatable, onto the oauth2-proxy sidecar).
+  #
+  # `^/signalr/` — MEASURED ON THE LIVE POD, 2026-10-01, 23h after the cookie fix deployed
+  # (radarr-7d55cbffb9-42hmh, 0 restarts): 232 unauthenticated requests in the pod's whole
+  # lifetime and ALL 232 are the same call — `POST /signalr/messages/negotiate` from a browser
+  # UA, answered 302 by the proxy with upstream `-`, i.e. Radarr never saw one of them. That is
+  # a left-open tab whose session the cookie rename orphaned, polling SignalR forever, silently
+  # redirected every time and NEVER re-prompting the operator. It is the reported symptom
+  # ("reloading never fixes it and I am never re-prompted") still running in production on a
+  # service nobody was watching. With 401 the UI's connection breaks LOUDLY, the operator
+  # reloads, and `/` is outside both regexes so the reload signs them back in.
+  #
+  # `^/api/` — measured on radarr, but on the PRE-FIX pod and NOT by me: the 2026-09-30
+  # diagnosis (decisions/2026-09-30-jackett-oauth2-proxy-session.md) recorded radarr's 156
+  # unauthenticated requests as `/api/v3/command` (62), `/api/v3/queue/details` (46) and
+  # `/signalr/messages/negotiate` (43). The current pod has logged ZERO /api/ traffic, so I did
+  # not re-measure it; it is included because 108 of 156 measured rejections on this app were
+  # /api/v3 XHRs, which is this app's own evidence rather than an analogy to another app.
+  #
+  # SCOPE: both are JSON/RPC namespaces a browser never NAVIGATES to. Radarr's UI is served at
+  # `/` with assets under /Content/ and /MediaCover/, none of which match — so a plain page load
+  # is still 302'd into a silent SSO sign-in.
+  apiRoutes:
+    - ^/api/
+    - ^/signalr/
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the Radarr v5 stable line

catalog/seerr.yaml

@@ -24,7 +24,7 @@
 name: seerr
 kind: container
 source:
-  image: ghcr.io/seerr-team/seerr:v3.4.1   # Seerr: maintained Overseerr/Jellyseerr successor, Plex-native request UI (docs/13 row 5); served on 5055
+  image: ghcr.io/seerr-team/seerr:v3.5.1   # Seerr: maintained Overseerr/Jellyseerr successor, Plex-native request UI (docs/13 row 5); served on 5055
 # The ghcr.io/seerr-team/seerr image runs as the non-root user `node:node` (UID 1000 / GID 1000 —
 # the standard node base-image user; confirmed from the image config's User field). On boot the app
 # creates /app/config/logs and writes its settings/DB under the mounted /app/config PVC. A fresh
@@ -57,7 +57,7 @@ dependencies:
     port: 32400
     protocol: tcp
 ingressRoutes:
-  - host: seerr.home.keithjackson.dev
+  - host: seerr.${BASE_DOMAIN}
     port: 5055                   # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: native                   # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)

catalog/semaphore.yaml

@@ -22,7 +22,7 @@
 name: semaphore
 kind: container
 source:
-  image: semaphoreui/semaphore:v2.19.10   # pinned recent stable tag (R-UPD-01); update.range governs advancement
+  image: semaphoreui/semaphore:v2.19.17   # pinned recent stable tag (R-UPD-01); update.range governs advancement
 env:
   # --- PostgreSQL connection (the CNPG-DB pattern). DIALECT/HOST/PORT/DB/USER are literals; PASS is
   #     sourced from the in-namespace SOPS secret (the CNPG -app secret lives in ns semaphore-db,
@@ -58,7 +58,7 @@ env:
   - name: SEMAPHORE_ADMIN_NAME
     value: Semaphore Admin
   - name: SEMAPHORE_ADMIN_EMAIL
-    value: admin@home.keithjackson.dev
+    value: admin@${BASE_DOMAIN}
   # --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
   #     SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
   - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
@@ -78,16 +78,34 @@ env:
   #     (util ConfigType `oidc_providers`, env-tagged SEMAPHORE_OIDC_PROVIDERS). The value is a single
   #     JSON blob for the `keycloak` provider — provider_url is the homelab realm issuer (auto-discovery),
   #     client_id=semaphore, redirect_url=/api/auth/oidc/keycloak/redirect, claims map to the realm's
-  #     token. It CONTAINS the confidential client secret, so the whole blob is held SOPS-encrypted in the
+  #     token. The name is NOT a kubelet service-link var (those are SEMAPHORE_SERVICE_HOST /
-  #     semaphore-bootstrap Secret (key oidc-providers-json; same secret value SOPS-held in
+  #     SEMAPHORE_PORT_*), so no collision. The LOCAL SEMAPHORE_ADMIN above stays break-glass (docs/15).
-  #     keycloak-realm-clients) and injected here by reference — never in cleartext. The name is NOT a
+  #
-  #     kubelet service-link var (those are SEMAPHORE_SERVICE_HOST / SEMAPHORE_PORT_*), so no collision.
+  #     PER-ENVIRONMENT URLS (2026-09-07 fix). The blob USED to live whole inside the SOPS-encrypted
-  #     The LOCAL SEMAPHORE_ADMIN above stays as the break-glass path (docs/15). ---
+  #     semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
-  - name: SEMAPHORE_OIDC_PROVIDERS
+  #     client secret. That hid two environment-specific, NON-SECRET URLs — the realm issuer and the
+  #     redirect URI — inside ciphertext, where Flux's ${BASE_DOMAIN} postBuild substitution CANNOT
+  #     reach them. Both were frozen at the apex `home.keithjackson.dev`, so PROD (BASE_DOMAIN =
+  #     prod.home.keithjackson.dev) pointed its OIDC at DEV's Keycloak — unroutable from prod's
+  #     192.168.2.0/24 (the identical root cause as the Headlamp login timeout; see
+  #     platform/headlamp/README.md). Dev was only ever "right" because dev IS the apex.
+  #
+  #     THE SPLIT: the URLs are public, so the blob is now a PLAINTEXT literal here and gets
+  #     ${BASE_DOMAIN}-substituted per environment like every other hostname. Only the confidential
+  #     client secret stays SOPS-encrypted, in its own key (`oidc-client-secret`), and is spliced back
+  #     in by the KUBELET's dependent-env-var expansion: an env `value` may reference an EARLIER env
+  #     var of the same container as $(NAME), and secretKeyRef-sourced vars are eligible. So
+  #     SEMAPHORE_OIDC_CLIENT_SECRET MUST STAY IMMEDIATELY ABOVE — the kubelet resolves env in list
+  #     order, and a reference to a not-yet-defined name is left as the LITERAL text $(...). It is
+  #     fail-loud in the way that matters: if the Secret key is missing the Pod never starts
+  #     (CreateContainerConfigError), rather than silently authenticating against nothing. ---
+  - name: SEMAPHORE_OIDC_CLIENT_SECRET
     valueFrom:
       secretKeyRef:
         name: semaphore-bootstrap
-        key: oidc-providers-json
+        key: oidc-client-secret
+  - name: SEMAPHORE_OIDC_PROVIDERS
+    value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
 # The semaphoreui/semaphore image runs as the non-root user UID 1001 / GID 0 (tini -> server-wrapper),
 # and on boot the wrapper writes its generated config.json into the config dir. A mounted Longhorn PVC
 # is root:root, so UID 1001 cannot write it -> the pod panics ("permission denied"). fsGroup=1001 makes
@@ -110,17 +128,19 @@ dependencies:
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
     protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: semaphore.home.keithjackson.dev
+  - host: semaphore.${BASE_DOMAIN}
     port: 3000                  # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
   groups:
     - platform-admins           # access scoped to an authorized group (R-IDP-06)
   # WIRED (native OIDC): the Keycloak `semaphore` confidential client (platform/keycloak/keycloak-realm-
-  # import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env (JSON blob held encrypted
+  # import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env — a PLAINTEXT JSON blob
-  # in the semaphore-bootstrap Secret, oidc-providers-json). The `groups` claim is exposed for
+  # (see `env` above) whose per-environment URLs are ${BASE_DOMAIN}-substituted, with the confidential
-  # authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the break-glass path so an IdP outage
+  # client secret spliced in at runtime from the semaphore-bootstrap Secret key `oidc-client-secret`.
-  # never locks the operator out (docs/15 "Availability caveat"). No secret is embedded here.
+  # The `groups` claim is exposed for authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the
+  # break-glass path so an IdP outage never locks the operator out (docs/15 "Availability caveat").
+  # No secret is embedded here.
 update:
   channel: stable               # R-UPD-01
   range: ">=2.19.0 <3.0.0"      # stay on the 2.x stable line

catalog/sonarr.yaml

@@ -8,19 +8,91 @@
 name: sonarr
 kind: container
 source:
-  image: lscr.io/linuxserver/sonarr:4.0.15   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
+  image: lscr.io/linuxserver/sonarr:4.0.20   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
+# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
+# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
+# which cannot write a 1000:3000-owned 0755 tree.
+env:
+  - name: PUID
+    value: "1000"               # owner uid of every directory under /mnt/Storage/plex
+  - name: PGID
+    value: "3000"               # dominant group there; keeps files this writes readable to the
+                                # EXISTING non-k3s consumers of the same export, which stay live
+  - name: TZ
+    value: Etc/UTC              # same safe default plex uses; the arrs schedule by local time
+podSecurityContext:
+  # For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
+  # does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
+  # what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
+  # /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
+  fsGroup: 3000
+exposure: internal              # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
     size: 5Gi
     class: longhorn             # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
-  # NOTE (bulk-media PREREQUISITE): docs/13 row 2 places the media library / downloads volume on the
+    mountPath: /config          # THE APP WRITES /config, NOT the renderer default /data/config.
-  # `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
+                                # LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
-  # but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
+                                # /config), so without this the Longhorn PVC mounted at /data/config sat
-  # volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
+                                # holding nothing but lost+found while the app wrote its entire
-  # therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Sonarr deployable today.
+                                # configuration to the container's EPHEMERAL writable layer — destroyed on
+                                # every pod recreation: restart, reschedule, image bump. Measured in-cluster
+                                # 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
+                                # dev=2053 on the node's own disk with [Sentry asp config.xml logs].
+                                # This is the SAME failure catalog/plex.yaml:54 already records fixing.
+                                # The live /config contents were copied onto each PVC before this landed,
+                                # so API keys and credentials other apps reference were preserved.
+  # THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
+  # /media/tv (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
+  #
+  # *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Sonarr imports by HARDLINKING or
+  # atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
+  # and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
+  #     /mnt/Storage/plex/tv/                      the library
+  #     /mnt/Storage/plex/downloads/complete/tv-sonarr/  where finished releases land
+  # — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
+  # them as two subPath volumes (subPath: tv at /media/tv, subPath: downloads elsewhere) creates
+  # TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Sonarr then
+  # silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
+  # seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
+  # is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
+  # requirement, not a shortcut. (The cost is stated plainly: Sonarr can also write /media/movies,
+  # which is radarr's. Narrowing that would cost hardlinks, which is a worse trade.)
+  #
+  # *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
+  # renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
+  # config volume entirely; the pod would come up with its app database shadowed by an 8 T media
+  # tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
+  # the same export, so when qbittorrent is eventually given this mount too, the arrs and the
+  # download client will already agree on absolute paths — which is the other precondition for
+  # hardlinked imports (otherwise every import needs a Remote Path Mapping).
+  #
+  # SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
+  # is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
+  # with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
+  # Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
+  # documents having already been bitten by. Fixing it means starting from an empty config volume, so
+  # it belongs in its own change with the operator present.
+  #
+  # UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
+  # + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
+  # the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
+  # write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
+  # every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
+  #
+  # AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
+  # ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
+  # lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
+  # that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
+  # the image re-chowning /config for itself.
+  - name: media
+    class: bulk
+    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
+    mountPath: /media           # => /media/tv (Root Folder) and /media/downloads (import source)
+    size: 9Ti                   # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
+                                # rounded up, so `kubectl get pv` reports something true.
 dependencies:
   - to: qbittorrent             # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 8080
@@ -28,13 +100,29 @@ dependencies:
   - to: jackett                 # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 9117
     protocol: tcp
+  - to: plex                    # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
+    port: 32400
+    protocol: tcp
 ingressRoutes:
-  - host: sonarr.home.keithjackson.dev
+  - host: sonarr.${BASE_DOMAIN}
     port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
   groups:
     - media-users               # authenticated AND in the media group (R-IDP-06)
+    # See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
+    # platform-admins here is a superset, not a widening (that group already carries cluster-admin
+    # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
+    # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
+    - platform-admins           # operators of these services must not be locked out of them
+  # NO `apiRoutes` HERE, AND NOT BECAUSE IT NEEDS NONE — because nothing measures that it does.
+  # Sonarr is nearly the same codebase as Radarr, which is exactly what makes `^/api/` +
+  # `^/signalr/` a tempting guess and still a guess. MEASURED 2026-10-01 on the live pod
+  # (sonarr-ddf485775-8ckwt, 0 restarts, 23h): its oauth2-proxy log contains 13,643 lines and
+  # ZERO unauthenticated requests — no /api/, no /signalr/, nothing but health probes. Nobody
+  # has opened this UI since the cookie fix rolled, so the measurement is empty rather than
+  # negative: it is evidence of no TRAFFIC, not evidence of no need. Add the routes when someone
+  # reproduces the 302-on-XHR here, with the numbers.
 update:
   channel: stable               # R-UPD-01
   range: ">=4.0.0 <5.0.0"       # stay on the Sonarr v4 stable line

ci/Makefile

@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
 
 .DEFAULT_GOAL := help
 
-.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
+.PHONY: help spec-immutability decisions-log lint schema l0 render-golden l1 conformance gate all touched versions tools test
 
 help: ## Show this help
 	@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf "  \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
@@ -29,9 +29,18 @@ schema l0: ## ServiceDefinition schema validation + Kyverno L0 policy gate (no c
 render-golden: ## render==golden: canonical render reproduces committed apps/ (no cluster; live now)
 	@"$(CI_DIR)render-golden.sh"
 
+decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-formed (no cluster)
+	@"$(CI_DIR)decisions-log.sh"
+
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"
 
+approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
+	@./approval-policy.sh
+
+gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
+	@./gitea-settings.sh
+
 conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
 	@"$(CI_DIR)conformance.sh"
 

ci/README.md

@@ -1,5 +1,19 @@
 # `ci/` — portable merge gate (runnable today, without Gitea)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 **Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
 
 The same merge gate the Gitea Actions workflow runs
@@ -27,9 +41,31 @@ make -C ci test        # unit tests for these scripts themselves (ci/tests/*.tes
 ```
 
 The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
-(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
+(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo
-per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
+L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
-spec-immutability enforcement). See `conformance/tests/README.md`.
+least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`.
+
+### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped
+
+R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the
+worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
+coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be
+selected *by* the requirements a change touches — because **the change most likely to break it
+attributes no requirement at all**. Requirement-scoped selection then filters the check out of
+exactly that change.
+
+This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
+3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
+`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that
+failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write
+a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"`
+and the runner unions every declaring test into the run. Adding the annotation is the only step.
+
+`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the
+tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the
+gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably,
+by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the
+full rationale and the incidents behind it.
 
 ## Exit-code contract
 
@@ -37,6 +73,48 @@ Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIP
 with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
 the gate stays honest in Phase 0 (no fake green).
 
+### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)
+
+**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It
+used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a
+warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
+`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule
+unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as
+merge-approved. Fail-open in the one component whose whole job is to fail closed.
+
+| verdict | exit | meaning |
+|---|---|---|
+| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. |
+| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. |
+| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. |
+| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. |
+
+Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.
+
+**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so
+`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A
+caller that must distinguish them invokes `./ci/gate.sh` directly.
+
+### Which skips block, and why — a skip is not one thing
+
+| stage | a skip there means | verdict |
+|---|---|---|
+| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is the job image — the CI-built toolbox, `.gitea/workflows/gate.yml` `container:` — so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
+| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
+| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
+| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
+
+That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
+
+`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml`
+sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls
+back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction
+cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI
+strictness.
+
+Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed
+stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate).
+
 ### The touched-requirement subset can also be *undeterminable* (exit 3)
 
 `touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
@@ -87,6 +165,8 @@ ci/
   lint.sh                  Stage: yaml/lint
   l0.sh                    Stage: schema validation + Kyverno L0 gate + L0 static conformance
   l1.sh                    Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
+                           (harness: platform/monitoring/tests/). Catches the two alerting bugs no
+                           other stage can see — permanently-firing and structurally INERT rules.
   conformance.sh           Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
   touched-requirements.sh  compute the R-* subset a change touches (drives the R-TEST-03 filter);
                            exits 3 LOUDLY rather than guessing when the diff base is unreliable
@@ -122,3 +202,15 @@ its owner delivers a runnable entrypoint.
 `CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and
 be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+)
 turns any unverified cluster-needing touched requirement into a hard block.
+
+All four switches are read through one `truthy()` in `lib.sh`, so they spell "on" identically
+(`true|1|yes|y|on`, any case) and — the direction that matters — **anything else is OFF**: a
+misspelled `CLUSTER_AVAILABLE=ture` leaves the capability undeclared instead of silently claiming
+it, which can only make the gate stricter about what it says it verified.
+
+| switch | default | effect |
+|---|---|---|
+| `CLUSTER_AVAILABLE` | `false` | declares a cluster; L2/L3 then run and are required |
+| `REQUIRE_CLUSTER_TESTS` | `false` | Phase 1+: an unverified cluster-needing touched requirement BLOCKS |
+| `REQUIRE_SPEC_VERIFICATION` | `false` | an unverifiable spec repo BLOCKS instead of UNDETERMINED |
+| `GATE_STRICT_SKIPS` | `auto` | `true` = a missing tool / unwired seam BLOCKS (CI); `auto` follows `GITHUB_ACTIONS`/`CI` |

ci/approval-override.py

@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""ci/approval-override.py — select the admin-override on a PR, if there is a valid one.
+
+This script decides NOTHING about authority. It parses a PR's comments and reviews and returns
+the single override *candidate* that ci/approval-policy.sh should then try to authorise against
+the live Gitea permission API. Keeping the two apart is the point:
+
+  * parsing is pure, offline, and therefore TESTABLE (ci/tests/approval-override.test.sh);
+  * authorisation needs the API and must stay in the stage that fails closed.
+
+A candidate is a comment OR a review whose body contains, AT THE START OF A LINE, the directive
+followed by a non-empty reason:
+
+    /approve-override the gate cannot gate itself and I am the only human reviewer
+
+TWO SOURCES, AND WHY THAT IS NOT BELT-AND-BRACES. Measured 2026-10-02 on PR #38: the override was
+posted as an ordinary PR comment, was well-formed, was newer than the head commit — and the stage
+still blocked, because CI could not READ it. `SETTINGS_API_TOKEN` is minted with scopes
+`[read:repository, read:admin]`, and Gitea puts /issues/{n}/comments behind the *issue* scope, so
+the request came back as an error OBJECT rather than an array. `/pulls/{n}/reviews` is under
+read:repository and was readable in the same run. So:
+
+  * a REVIEW body works with the token CI has today (leave a review of type "Comment");
+  * a COMMENT body works once the token carries read:issue (see infra/gitea/set-ci-token.sh).
+
+Accepting both means the hatch is not hostage to one API scope. A source that cannot be read
+contributes no candidates and is reported on stderr; it can only cause a false BLOCK, never a
+false pass, which is why degrading it is safe and blocking on it was not necessary.
+
+WHY EACH RULE IS HERE
+
+  * START OF A LINE, not "anywhere in the body". Otherwise quoting someone else's override
+    ("you could just /approve-override this") re-arms it. Lines beginning with '>' are DROPPED
+    rather than unquoted, for exactly that reason.
+  * A REASON IS MANDATORY. A bare directive records that a review was waived but not why, which
+    is the difference between an audit trail and a rubber stamp.
+  * STALE OVERRIDES ARE DISCARDED. An override earned at commit A must not silently cover commit
+    B. Any candidate created at or before the head commit's timestamp is dropped — the same
+    semantics Gitea's own `dismiss_stale_approvals` applies to approvals. Push new commits and
+    you must re-override.
+  * A PENDING REVIEW IS NOT SUBMITTED. It is a private draft visible only to its author, so it is
+    not an act of anything and is dropped.
+  * THE LATEST CANDIDATE WINS, across both sources, so a later override supersedes an earlier one.
+
+Output: one TSV line (login, timestamp, source, reason) on stdout, or nothing at all when there is
+no valid candidate. Never exits non-zero for "no override" — absence is a normal answer and the
+caller must not be able to confuse it with a parse crash.
+
+Input, all via env so nothing lands in `ps`:
+  COMMENTS_JSON  the Gitea /issues/{n}/comments array (or an error object; or absent)
+  REVIEWS_JSON   the Gitea /pulls/{n}/reviews array   (or an error object; or absent)
+  HEAD_TIME      ISO-8601 timestamp of the PR head commit
+  DIRECTIVE      the override directive, e.g. /approve-override
+"""
+import json
+import os
+import re
+import sys
+from datetime import datetime, timezone
+
+
+def parse_ts(value):
+    """Gitea emits RFC3339. Return an aware datetime, or None if it is unparseable.
+
+    Unparseable is NOT 'treat as fresh': the caller drops such a candidate, because a
+    timestamp we cannot read is a staleness check we cannot perform.
+    """
+    if not value:
+        return None
+    text = str(value).strip()
+    # Python < 3.11 cannot read a trailing 'Z' in fromisoformat.
+    if text.endswith(("Z", "z")):
+        text = text[:-1] + "+00:00"
+    try:
+        parsed = datetime.fromisoformat(text)
+    except ValueError:
+        return None
+    if parsed.tzinfo is None:
+        parsed = parsed.replace(tzinfo=timezone.utc)
+    return parsed
+
+
+def extract_reason(body, directive):
+    """Return the reason from the first line that STARTS with the directive, else None."""
+    pattern = re.compile(
+        r"^[ \t]*" + re.escape(directive) + r"(?=[ \t:]|$)[ \t:]*(.*)$"
+    )
+    for raw in (body or "").splitlines():
+        # A quoted line is somebody repeating an override, not casting one.
+        if raw.lstrip().startswith(">"):
+            continue
+        match = pattern.match(raw)
+        if match:
+            reason = match.group(1).strip()
+            return reason or None
+    return None
+
+
+def load_source(env_name, label):
+    """Return a list of items for one source, or [] with a stderr note if it is unusable."""
+    raw = os.environ.get(env_name)
+    if not raw or not raw.strip():
+        return []
+    try:
+        parsed = json.loads(raw)
+    except (ValueError, TypeError):
+        print(
+            "approval-override: %s is not valid JSON — ignoring that source" % label,
+            file=sys.stderr,
+        )
+        return []
+    if isinstance(parsed, list):
+        return parsed
+    # An error object, typically {"message": "token does not have ... scope"}. Say which,
+    # because "no override found" and "could not read the overrides" look identical from
+    # the outside and the first wasted a real debugging session.
+    detail = ""
+    if isinstance(parsed, dict):
+        detail = str(parsed.get("message") or parsed.get("errors") or "")[:200]
+    print(
+        "approval-override: %s is not a list (Gitea returned %s%s) — ignoring that source"
+        % (label, type(parsed).__name__, ": " + detail if detail else ""),
+        file=sys.stderr,
+    )
+    return []
+
+
+def candidates_from(items, source, directive, head_time):
+    """Yield (when, login, reason, raw_when, source) for every valid override in one source."""
+    found = []
+    for item in items:
+        if not isinstance(item, dict):
+            continue
+        login = ((item.get("user") or {}).get("login") or "").strip()
+        reason = extract_reason(item.get("body"), directive)
+        if not login or not reason:
+            continue
+        # A PENDING review is an unsubmitted private draft — not an act of anything.
+        state = str(item.get("state") or "").upper()
+        if source == "review" and state == "PENDING":
+            print(
+                "approval-override: dropping PENDING (unsubmitted) review by '%s'" % login,
+                file=sys.stderr,
+            )
+            continue
+        raw_when = (
+            item.get("submitted_at") or item.get("created_at") or item.get("updated_at") or ""
+        )
+        when = parse_ts(raw_when)
+        if when is None:
+            print(
+                "approval-override: dropping %s override by '%s' — unreadable timestamp %r"
+                % (source, login, raw_when),
+                file=sys.stderr,
+            )
+            continue
+        # An override must post-date the commit it covers.
+        if head_time is not None and when <= head_time:
+            print(
+                "approval-override: dropping STALE %s override by '%s' (%s) — not newer than "
+                "head commit (%s). Re-post it to override the current commit."
+                % (source, login, when.isoformat(), head_time.isoformat()),
+                file=sys.stderr,
+            )
+            continue
+        found.append((when, login, reason, raw_when, source))
+    return found
+
+
+def main():
+    directive = os.environ.get("DIRECTIVE", "/approve-override").strip()
+    head_time = parse_ts(os.environ.get("HEAD_TIME", ""))
+
+    found = []
+    found += candidates_from(
+        load_source("COMMENTS_JSON", "COMMENTS_JSON"), "comment", directive, head_time
+    )
+    found += candidates_from(
+        load_source("REVIEWS_JSON", "REVIEWS_JSON"), "review", directive, head_time
+    )
+
+    if not found:
+        return 0
+
+    # Latest wins, across both sources: a later override supersedes an earlier one.
+    when, login, reason, raw_when, source = max(found, key=lambda c: c[0])
+    # Keep the reason on one line — this becomes a TSV field.
+    reason = " ".join(reason.split())
+    print("%s\t%s\t%s\t%s" % (login, raw_when, source, reason))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

ci/approval-policy.sh

@@ -0,0 +1,274 @@
+#!/usr/bin/env bash
+# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
+# what the pipeline verifies or the security posture it rests on.
+#
+# THE POLICY, and why it is shaped this way.
+#
+# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
+# pipeline is not good enough to be trusted for this, then it is not good enough to do its
+# intended job and needs to be improved. Manual approvers should only be for architecturally or
+# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
+#
+# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
+# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
+# A PR that weakens a check would be approved by the weakened check. The same applies to the
+# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
+# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
+# requires a human for them.
+#
+# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
+# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
+# policy cannot be widened without a human.
+#
+# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
+# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
+# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
+# BYPASS, not a gap. So: cannot determine -> BLOCK.
+#
+# THE ADMIN OVERRIDE (added 2026-10-01), and the lock-out it clears.
+#
+# The rule above was UNSATISFIABLE for the one human who runs this repo. Gitea refuses to let a
+# pull request's author approve their own PR, and `jakman85` is the only collaborator — so "an
+# APPROVED review from someone who is not its author" is a condition the sole administrator can
+# never meet. Every sensitive-path PR they opened was permanently unmergeable, INCLUDING the PR
+# that would fix it. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, blocked.
+#
+# So an administrator may waive the review by COMMENTING on the PR:
+#
+#     /approve-override <reason>
+#
+# Authority comes from the LIVE Gitea permission API, never from the comment — the commenter must
+# hold a permission listed in `.override.authorized_permissions` (admin/owner) at the moment the
+# gate runs, so an old comment from a since-demoted account is worth nothing. The directive must
+# start a line, a reason is mandatory, and the comment must POST-DATE the head commit: an override
+# earned at commit A does not cover commit B. Choosing the candidate comment is the job of the
+# pure, offline, unit-tested ci/approval-override.py; AUTHORISING it is the job of this stage,
+# which fails closed.
+#
+# WHAT THIS COSTS, STATED PLAINLY. For an administrator, "a change to the gate cannot be gated by
+# the gate" stops being structural and becomes advisory. That is the unavoidable consequence of
+# there being one human; the alternative is a repository its owner cannot change. What is bought
+# back is a RECORD — the waiver is an explicit act, attributed to an account, carrying a stated
+# reason, in the PR history and in this stage's output. A silent bypass keyed on "is the author an
+# admin?" would have been less code and is exactly the fail-open shape this repo keeps finding.
+#
+# AND THE PART THAT IS NOT COMFORTABLE. Authorisation is by permission ALONE, with no allowlist of
+# human accounts (operator decision, 2026-10-01, taken with this consequence on the table). The
+# AGENT account `gitea-admin` also reports permission `owner` here, so an agent can clear the
+# sensitive-path review on its own PR by posting the directive — and agents author most PRs in
+# this repo, so that is the common case, not an edge case. To narrow it without touching this
+# logic, add an `override_admins` list to infra/gitea/approval-policy.json: the check below
+# already requires membership whenever that key is present.
+#
+# EXIT CODES
+#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and
+#      either a non-author APPROVED review or a valid admin override is present
+#   1  sensitive paths touched with neither, OR the policy could not be evaluated
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: approval policy (CI approves; humans approve changes to CI)"
+
+DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
+[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
+require_tool jq
+require_tool python3
+
+# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
+# by branch protection, so this is genuinely not-applicable rather than a skipped check.
+EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
+PR="${PR_NUMBER:-}"
+if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
+  ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
+  exit 0
+fi
+
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+if [ -z "$TOKEN" ]; then
+  err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
+  err "  BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
+  summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
+  exit 1
+fi
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
+PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
+REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
+# PR conversation comments live under the ISSUES endpoint. `/pulls/{n}/comments` returns review
+# (diff) comments ONLY, so an override posted in the normal comment box would never appear in it.
+# Verified against this Gitea (1.24.3) rather than assumed.
+#
+# This one is NOT in the must-parse loop below, and that is deliberate. /issues/{n}/comments sits
+# behind Gitea's *issue* token scope, which `SETTINGS_API_TOKEN` does not carry (it is minted
+# `[read:repository, read:admin]`) — so it returns an error OBJECT, not an array. Blocking on that
+# turned "CI cannot read your override" into a message that said "you did not post one", and cost a
+# real debugging session on PR #38. It is handled explicitly, and loudly, further down: an
+# unreadable source can only cause a false BLOCK, never a false pass.
+COMMENTS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/issues/${PR}/comments?limit=1000" || true)"
+for v in FILES_JSON PR_JSON REVIEWS_JSON; do
+  jq -e . >/dev/null 2>&1 <<<"${!v}" || {
+    err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
+    summary "- approval-policy: **BLOCKED — Gitea unreadable**"
+    exit 1; }
+done
+
+AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
+[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
+
+# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
+# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
+# never read. Caught before shipping, by running it.
+SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
+import json, os, fnmatch
+decl = json.load(open(os.environ["DECL"]))
+files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
+pats = decl["sensitive_paths"]
+hits = sorted({f for f in files for p in pats
+               if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
+print("\n".join(hits))
+')"
+
+if [ -z "$SENSITIVE" ]; then
+  ok "no architecturally/security-relevant paths touched — the GATE is the approver."
+  summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
+  exit 0
+fi
+
+echo "  sensitive paths touched:" >&2
+printf '    %s\n' $SENSITIVE >&2
+
+APPROVER="$(jq -r --arg a "$AUTHOR" \
+  '[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
+  <<<"$REVIEWS_JSON")"
+
+if [ -n "$APPROVER" ]; then
+  ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
+  summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
+  exit 0
+fi
+
+# ─────────────────────────────────────────────────────────────────────────────────────────
+# No non-author approval exists. Before refusing, look for an administrator's explicit
+# override. See the header for why this hatch exists and what it costs.
+# ─────────────────────────────────────────────────────────────────────────────────────────
+OVERRIDE_ENABLED="$(jq -r '.override.enabled // false' "$DECL")"
+DIRECTIVE="$(jq -r '.override.comment_directive // "/approve-override"' "$DECL")"
+HEAD_SHA="$(jq -r '.head.sha // empty' <<<"$PR_JSON")"
+
+if [ "$OVERRIDE_ENABLED" = "true" ]; then
+  # The head commit's timestamp anchors staleness. Read it from GITEA, not from the local
+  # tree: CI checks out a merge commit, whose date is not the head commit's date.
+  HEAD_TIME=""
+  if [ -n "$HEAD_SHA" ]; then
+    HEAD_COMMIT_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/git/commits/${HEAD_SHA}" || true)"
+    HEAD_TIME="$(jq -r '.commit.committer.date // .created // empty' <<<"${HEAD_COMMIT_JSON:-null}" 2>/dev/null || true)"
+  fi
+  if [ -z "$HEAD_TIME" ]; then
+    err "could not read the head commit's timestamp for PR #${PR} — BLOCKING."
+    err "  Without it an override cannot be aged, and an unaged override is one that silently"
+    err "  covers commits nobody waived review on."
+    summary "- approval-policy: **BLOCKED — head commit timestamp unreadable** (cannot age an override)"
+    exit 1
+  fi
+
+  # Is the comments source actually readable? Say so plainly if not — a scope problem and "you
+  # never posted an override" are indistinguishable from the outside, and conflating them is what
+  # made PR #38 look like a code bug.
+  if ! jq -e 'type == "array"' >/dev/null 2>&1 <<<"${COMMENTS_JSON:-}"; then
+    SCOPE_MSG="$(jq -r '.message // empty' <<<"${COMMENTS_JSON:-null}" 2>/dev/null || true)"
+    warn "cannot read PR COMMENTS from Gitea${SCOPE_MSG:+ — ${SCOPE_MSG}}."
+    warn "  /issues/${PR}/comments needs the *issue* token scope; SETTINGS_API_TOKEN is minted"
+    warn "  [read:repository, read:admin] (infra/gitea/set-ci-token.sh). An override posted as an"
+    warn "  ordinary PR COMMENT is therefore invisible to CI — post it as a REVIEW of type"
+    warn "  'Comment' instead, which /pulls/${PR}/reviews exposes under read:repository."
+    summary "- approval-policy: note — PR comments unreadable (token lacks the issue scope); use a **review** to override"
+    COMMENTS_JSON='[]'
+  fi
+
+  # Pure, offline, unit-tested selection: which comment or review, if any, is a live override?
+  OVERRIDE_TSV="$(COMMENTS_JSON="$COMMENTS_JSON" REVIEWS_JSON="$REVIEWS_JSON" \
+    HEAD_TIME="$HEAD_TIME" DIRECTIVE="$DIRECTIVE" \
+    python3 "${CI_DIR}/approval-override.py")" || {
+      err "ci/approval-override.py failed — BLOCKING rather than assuming there is no override."
+      summary "- approval-policy: **BLOCKED — override parser failed**"
+      exit 1; }
+
+  if [ -n "$OVERRIDE_TSV" ]; then
+    OV_USER="$(cut -f1 <<<"$OVERRIDE_TSV")"
+    OV_WHEN="$(cut -f2 <<<"$OVERRIDE_TSV")"
+    OV_SRC="$(cut -f3 <<<"$OVERRIDE_TSV")"
+    OV_WHY="$(cut -f4- <<<"$OVERRIDE_TSV")"
+
+    # AUTHORITY IS MEASURED LIVE, never inferred from the comment or from a name in git.
+    PERM_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/collaborators/${OV_USER}/permission" || true)"
+    OV_PERM="$(jq -r '.permission // empty' <<<"${PERM_JSON:-null}" 2>/dev/null || true)"
+    if [ -z "$OV_PERM" ]; then
+      err "'${OV_USER}' posted ${DIRECTIVE}, but their permission on ${GITEA_REPO} could not be"
+      err "  read — BLOCKING. An override whose authority is unverified is not an override."
+      summary "- approval-policy: **BLOCKED — cannot verify \`${OV_USER}\`'s permission**"
+      exit 1
+    fi
+
+    # Authorised when the LIVE permission is one of the declared ones and — only if an
+    # `override_admins` allowlist exists at all — the account is on it. No allowlist means
+    # permission alone, which is the operator's standing decision; adding the key narrows
+    # this with no change to any code.
+    AUTHORISED="$(DECL="$DECL" OV_PERM="$OV_PERM" OV_USER="$OV_USER" python3 -c '
+import json, os
+d = json.load(open(os.environ["DECL"])).get("override", {})
+perms = d.get("authorized_permissions") or ["admin", "owner"]
+allow = d.get("override_admins")
+ok_perm = os.environ["OV_PERM"] in perms
+ok_list = True if allow is None else os.environ["OV_USER"] in allow
+print("yes" if (ok_perm and ok_list) else "no")
+')"
+
+    if [ "$AUTHORISED" = "yes" ]; then
+      if [ "$OV_USER" = "$AUTHOR" ]; then
+        warn "ADMIN OVERRIDE — SELF-WAIVED. '${OV_USER}' authored this PR and waived the"
+        warn "  non-author review requirement on their own change. Permitted (permission"
+        warn "  '${OV_PERM}'), deliberate, and shouted about here precisely because it is not free."
+      else
+        warn "ADMIN OVERRIDE by '${OV_USER}' (permission '${OV_PERM}'); author is '${AUTHOR}'."
+      fi
+      warn "  via:    PR ${OV_SRC}"
+      warn "  when:   ${OV_WHEN}  (head commit ${HEAD_SHA:0:8}, ${HEAD_TIME})"
+      warn "  reason: ${OV_WHY}"
+      warn "  review waived on: $(printf '%s ' $SENSITIVE)"
+      ok "sensitive paths cleared by administrator override — this stage does not block."
+      SELFNOTE=""; [ "$OV_USER" = "$AUTHOR" ] && SELFNOTE=", **self-waived as author**"
+      summary "- approval-policy: **OVERRIDDEN by \`${OV_USER}\`** (permission \`${OV_PERM}\`, via PR ${OV_SRC}${SELFNOTE})"
+      summary "  - reason: ${OV_WHY}"
+      summary "  - review waived on: $(printf '\`%s\` ' $SENSITIVE)"
+      exit 0
+    fi
+
+    err "'${OV_USER}' posted ${DIRECTIVE} but is NOT authorised to override: live permission is"
+    err "  '${OV_PERM}'. The authorised set is declared in infra/gitea/approval-policy.json."
+    summary "- approval-policy: **BLOCKED — \`${OV_USER}\` may not override** (permission \`${OV_PERM}\`)"
+    exit 1
+  fi
+fi
+
+err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
+err "  review from anyone other than its author ('${AUTHOR}')."
+err "  A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
+err "  honestly approve for itself. Get a human review, or move the change out of:"
+printf '    %s\n' $SENSITIVE >&2
+err "  The list lives in infra/gitea/approval-policy.json, with a reason per entry."
+if [ "$OVERRIDE_ENABLED" = "true" ]; then
+  err "  Or, if you hold admin/owner on this repository, post this on the PR:"
+  err "      ${DIRECTIVE} <reason>"
+  err "  as a REVIEW of type 'Comment' (not the plain comment box — CI's token cannot read"
+  err "  /issues/${PR}/comments without the issue scope; see infra/gitea/set-ci-token.sh)."
+  err "  The reason is mandatory, the directive must START a line, and it must be NEWER than"
+  err "  head commit ${HEAD_SHA:0:8} — push again and you must override again."
+fi
+summary "- approval-policy: **BLOCKED — needs a non-author approval, or an admin \`${DIRECTIVE}\` comment**"
+exit 1

ci/conformance.sh

@@ -47,7 +47,11 @@ fi
 # (conformance/lib/common.sh) now unions a comma/space list of layers, so this is a single
 # invocation with one aggregated report (no per-layer looping needed). Artifacts are then
 # COLLECTED from the fixed location below.
-"$CONFORMANCE_CMD" --layer l2,l3 "${req_args[@]}"
+# --include-scope: union the touched subset with every L2/L3 test DECLARING
+# `conformance.home.k3s/scope: whole-repo` (none today — wired once, here, so a future
+# whole-repo behavioural invariant needs only the annotation; see conformance/lib/common.sh).
+: "${WHOLE_REPO_SCOPE:=whole-repo}"
+"$CONFORMANCE_CMD" --layer l2,l3 --include-scope "$WHOLE_REPO_SCOPE" "${req_args[@]}"
 
 CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
 if [ -f "${CONF_ART}/report.junit.xml" ]; then
@@ -57,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
   cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi
 
+# ---------------------------------------------------------------------------------------
+# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
+#
+# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
+# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
+# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
+# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
+# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
+#
+# MEASURED 2026-09-29, before this guard existed:
+#   CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
+#   "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
+#   "[ ok ] L2/L3 conformance passed for touched subset"
+# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
+# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
+# it because nobody had ever set the variable.
+#
+# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
+# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
+# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
+# makes the stage itself honour that instead of reporting a pass.
+JUNIT="${CONF_ART}/report.junit.xml"
+if [ -f "$JUNIT" ]; then
+  total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
+  skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
+  if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
+    err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
+    err "  (scaffold mode -- run.sh could not reach a cluster and exited 0)."
+    err "  This stage will NOT report that as a pass. Either the cluster is unreachable from"
+    err "  this runner, or CLUSTER_AVAILABLE is set when it should not be."
+    err "  NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
+    err "  docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
+    err "  by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
+    err "  runner has no credential for (automountServiceAccountToken: false, by design)."
+    summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
+    exit 1
+  fi
+  log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
+fi
+
 ok "L2/L3 conformance passed for touched subset"
 summary "- conformance-l2-l3: **passed**"

ci/decisions-log.sh

@@ -0,0 +1,195 @@
+#!/usr/bin/env bash
+# ci/decisions-log.sh — the handoff log is a DIRECTORY, and the old single file is frozen.
+#
+# WHY THIS STAGE EXISTS
+# ---------------------
+# CLAUDE.md §7 tells every agent to record a handoff before finishing. Until 2026-09-29 that
+# meant appending to the tail of one 5,228-line file, DECISIONS-LOG.md. Every branch appended
+# to the SAME last hunk, so git could not auto-merge any two of them. Measured that day against
+# the live Gitea: four of the five open PRs were `mergeable: false`, and `git merge-tree`
+# against `main` named DECISIONS-LOG.md as the ONLY conflicting path in all four:
+#
+#   #22 fix/crafty-oauth2-proxy-signin       CONFLICT (content): DECISIONS-LOG.md
+#   #20 feat/conformance-job-template        CONFLICT (content): DECISIONS-LOG.md
+#   #19 feat/conformance-trigger-kubeconfig  CONFLICT (content): DECISIONS-LOG.md
+#   #18 feat/conformance-runner-image        CONFLICT (content): DECISIONS-LOG.md
+#   #17 feat/ci-is-the-approver              clean — it wrote no handoff entry
+#
+# Nothing was wrong with any of those changes. The log's storage format serialised work that
+# was otherwise parallel, and the cost grew with every concurrent branch (N branches -> N-1
+# rebases, each invalidating the gate run before it).
+#
+# The fix is that handoffs now live one-per-file under decisions/, so concurrent branches touch
+# disjoint paths and there is nothing to merge. See decisions/README.md.
+#
+# WHY IT NEEDS A CHECK AND NOT JUST A DOCUMENT
+# --------------------------------------------
+# The old habit is written into this repo in many places — the runbooks, the muscle memory of
+# every agent that has read the log, and (until this change) CLAUDE.md itself. An agent that
+# appends to DECISIONS-LOG.md anyway does not fail: it produces a perfectly good-looking commit
+# that quietly re-creates the conflict for every other open branch, and the damage is only
+# visible later, to somebody else, in a PR that is not theirs. That is the exact shape this
+# repo keeps getting burned by — a silent regression with a delayed, misattributed symptom — so
+# it gets a guard rather than a paragraph.
+#
+# WHAT IS ASSERTED
+#   (1) DECISIONS-LOG.md is byte-identical to the SHA-256 pinned in decisions/ARCHIVE.sha256.
+#       Re-pinning is a deliberate, reviewable line in a diff; appending is not.
+#   (2) Every decisions/ entry is named YYYY-MM-DD-<slug>.md with a real calendar date, so the
+#       directory sorts chronologically and two agents cannot collide on a filename.
+#   (3) Every entry carries frontmatter (date, branch, title) whose `date` agrees with its
+#       filename — a disagreement means a copied template nobody edited.
+#   (4) Every entry has all three of CLAUDE.md §7's sections with a real body under each:
+#       "What changed", "What I verified", "What I did NOT verify". This is the quality floor
+#       that stops an entry degrading into "done ✅". The third heading is the load-bearing one:
+#       this repo's recurring failure is a confident report of something nobody measured, and an
+#       entry with nowhere to write "UNVERIFIED" does not get one written.
+#
+# EXIT CODES (ci/README.md contract): 0 pass, 1 BLOCKS the merge. This stage has NO skip path —
+# it reads only files that are in the tree it is run against, needs no tool beyond coreutils and
+# no cluster, so there is no environment in which it "could not run". Adding an rc 3 here would
+# add a fail-open surface to a guard whose whole job is to close one.
+set -uo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+set +e   # aggregate every problem in one run; do not stop at the first.
+
+cd "$REPO_ROOT"
+
+ARCHIVE="DECISIONS-LOG.md"
+PIN="decisions/ARCHIVE.sha256"
+rc=0
+
+# ---- (1) the archive is frozen ----------------------------------------------
+log "Stage: decisions log (archive frozen + entries well-formed)"
+
+if [ ! -f "$ARCHIVE" ]; then
+  err "${ARCHIVE} is missing. It is the frozen pre-2026-09-29 history and must stay in the tree."
+  summary "- decisions-log: **FAILED** (\`${ARCHIVE}\` missing)"
+  rc=1
+elif [ ! -f "$PIN" ]; then
+  err "${PIN} is missing, so the archive's integrity cannot be established."
+  err "  If you are re-pinning deliberately: sha256sum ${ARCHIVE} > ${PIN}"
+  summary "- decisions-log: **FAILED** (\`${PIN}\` missing — archive unverifiable)"
+  rc=1
+else
+  want="$(awk 'NR==1{print $1}' "$PIN")"
+  got="$(sha256sum "$ARCHIVE" | awk '{print $1}')"
+  if [ "$want" != "$got" ]; then
+    err "${ARCHIVE} HAS CHANGED. It is a FROZEN ARCHIVE — do not append to it."
+    err "    pinned:  ${want}"
+    err "    on disk: ${got}"
+    err ""
+    err "  Every branch that appends to this file's tail conflicts with every other branch that"
+    err "  does, on the same hunk. That is what made four of five open PRs unmergeable on"
+    err "  2026-09-29, and your commit would re-create it for branches that are not yours."
+    err ""
+    err "  Write your handoff as its OWN FILE instead:"
+    err "    decisions/\$(date +%F)-<your-branch-slug>.md      (see decisions/README.md)"
+    err "  and restore the archive:  git checkout -- ${ARCHIVE}"
+    err ""
+    err "  If the archive genuinely must change (redacting a leaked credential, say), that is a"
+    err "  reviewed act: re-pin it with 'sha256sum ${ARCHIVE} > ${PIN}' and say why in your entry."
+    summary "- decisions-log: **FAILED — the frozen archive \`${ARCHIVE}\` was modified** (append to \`decisions/\` instead)"
+    rc=1
+  else
+    ok "${ARCHIVE}: unchanged (sha256 matches ${PIN})"
+  fi
+fi
+
+# ---- (2)(3)(4) every entry is well-formed ------------------------------------
+shopt -s nullglob
+entries=( decisions/[0-9]*.md )
+shopt -u nullglob
+
+# Zero entries is legitimate only until the first handoff lands; after that the glob always
+# matches. It is never an error here — a branch that changes no behaviour owes no handoff, and
+# CLAUDE.md, not this stage, is what asks for one.
+if [ "${#entries[@]}" -eq 0 ]; then
+  log "no decisions/ entries yet"
+fi
+
+REQUIRED_SECTIONS=("What changed" "What I verified" "What I did NOT verify")
+
+for f in "${entries[@]}"; do
+  base="${f#decisions/}"
+
+  # (2) filename
+  if [[ ! "$base" =~ ^([0-9]{4}-[0-9]{2}-[0-9]{2})-([a-z0-9][a-z0-9-]*)\.md$ ]]; then
+    err "${f}: name must be decisions/YYYY-MM-DD-<slug>.md with a lowercase [a-z0-9-] slug."
+    err "  Use your branch's last segment as the slug — it is already unique per unit of work."
+    summary "- decisions-log: **FAILED** (\`${f}\` misnamed)"
+    rc=1
+    continue
+  fi
+  fname_date="${BASH_REMATCH[1]}"
+
+  if ! date -d "$fname_date" +%F >/dev/null 2>&1 || [ "$(date -d "$fname_date" +%F)" != "$fname_date" ]; then
+    err "${f}: '${fname_date}' is not a real calendar date."
+    summary "- decisions-log: **FAILED** (\`${f}\` — impossible date)"
+    rc=1
+    continue
+  fi
+
+  # (3) frontmatter
+  if [ "$(head -1 "$f")" != "---" ]; then
+    err "${f}: must open with a '---' YAML frontmatter block (date, branch, title)."
+    err "  See decisions/README.md for the template."
+    summary "- decisions-log: **FAILED** (\`${f}\` — no frontmatter)"
+    rc=1
+    continue
+  fi
+  fm="$(awk 'NR==1&&$0=="---"{inb=1;next} inb&&$0=="---"{exit} inb{print}' "$f")"
+  missing_keys=()
+  for k in date branch title; do
+    grep -qE "^${k}:[[:space:]]*\S" <<<"$fm" || missing_keys+=("$k")
+  done
+  if [ "${#missing_keys[@]}" -gt 0 ]; then
+    err "${f}: frontmatter is missing a non-empty ${missing_keys[*]}."
+    summary "- decisions-log: **FAILED** (\`${f}\` — frontmatter missing \`${missing_keys[*]}\`)"
+    rc=1
+    continue
+  fi
+  fm_date="$(grep -m1 -E '^date:' <<<"$fm" | sed -E 's/^date:[[:space:]]*//; s/[[:space:]"'\'']//g')"
+  if [ "$fm_date" != "$fname_date" ]; then
+    err "${f}: frontmatter date '${fm_date}' disagrees with the filename's '${fname_date}'."
+    err "  One of them was copied from a template and not edited; the log's ordering depends on both."
+    summary "- decisions-log: **FAILED** (\`${f}\` — date disagrees with filename)"
+    rc=1
+    continue
+  fi
+
+  # (4) the three required sections, each with a real body
+  body_start="$(awk 'NR>1 && $0=="---" {print NR; exit}' "$f")"
+  [ -n "$body_start" ] || body_start=1
+  for want_sec in "${REQUIRED_SECTIONS[@]}"; do
+    hdr_line="$(awk -v s="$body_start" -v w="## ${want_sec}" 'NR>s && $0==w {print NR; exit}' "$f")"
+    if [ -z "$hdr_line" ]; then
+      err "${f}: missing the required section '## ${want_sec}'."
+      err "  CLAUDE.md §7 asks for what you changed, what you verified, and what you did NOT."
+      err "  All three are structure here, because the third is the one that gets dropped."
+      summary "- decisions-log: **FAILED** (\`${f}\` — missing \`## ${want_sec}\`)"
+      rc=1
+      continue
+    fi
+    # A real body: at least 20 non-blank, non-heading characters before the next '## '.
+    nchars="$(awk -v s="$hdr_line" 'NR>s { if ($0 ~ /^## /) exit; print }' "$f" \
+             | tr -d '[:space:]' | wc -c)"
+    if [ "$nchars" -lt 20 ]; then
+      err "${f}: section '## ${want_sec}' is empty or a stub (${nchars} chars of content)."
+      err "  If nothing was verified, WRITE THAT — 'nothing was verified; the gate was not run'"
+      err "  is a valid and useful entry. A blank section is not."
+      summary "- decisions-log: **FAILED** (\`${f}\` — \`## ${want_sec}\` is a stub)"
+      rc=1
+    fi
+  done
+done
+
+if [ "$rc" -eq 0 ]; then
+  ok "decisions log: archive frozen, ${#entries[@]} entr$([ "${#entries[@]}" -eq 1 ] && echo y || echo ies) well-formed"
+  summary "- decisions-log: **passed** (archive frozen; ${#entries[@]} entries well-formed)"
+else
+  err "decisions log stage FAILED"
+fi
+exit "$rc"

ci/gate.sh

@@ -7,6 +7,75 @@
 #   skip (3)  not run (cluster-gated stage w/ no cluster, or unwired seam) —
 #             NOT counted as green (HARD RULE 4: don't fake green)
 #
+# ...AND THREE OUTCOMES FOR THE GATE ITSELF (2026-09-13). Until today a stage that
+# SKIPPED only printed a warning ("— not green") and the gate then printed
+# `MERGE GATE: PASS` and exited 0 anyway. That is fail-open in the one component whose
+# entire job is to fail closed: a pinned tool missing from the runner, or a seam silently
+# becoming unwired, produced exit 0 and read as merge-approved. Observed live:
+# `prom-rules: skipped (missing tool) — not green` followed by `MERGE GATE: PASS`, and
+# `l0: skipped (kyverno CLI not installed)` likewise. The gate's own vocabulary already
+# had the answer — the 2026-08-31 "local false green" introduced UNDECIDED for exactly
+# this shape of non-answer — the offline stages just did not use it.
+#
+#   MERGE GATE: PASS           exit 0   every stage that was SUPPOSED TO RUN ran and was
+#                                       green, and the R-TEST-03 subset was checked.
+#   MERGE GATE: BLOCKED        exit 1   a stage FAILED, or a stage did not run WHERE IT
+#                                       MUST (see the per-stage rules below), or the
+#                                       decision input is unusable.
+#   MERGE GATE: UNDETERMINED   exit 3   nothing failed, but a stage did not run, so
+#                                       greenness was NOT ESTABLISHED. Never a pass;
+#                                       non-zero so no automation can read it as approval.
+#                                       `3` is this repo's existing "not run / not
+#                                       established" code (ci/lib.sh), now spoken by the
+#                                       gate as a whole and not only by its stages.
+#   MERGE GATE: NOT APPLICABLE exit 0   there is no branch under gate at all (HEAD is on
+#                                       base) — a push-to-main CI run. Stages were green;
+#                                       no subset was checked, and the headline says so.
+#
+# NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so
+# `make -C ci gate` collapses 1 and 3 into 2 — a human sees the headline and that is enough.
+# A caller that must tell BLOCKED from UNDETERMINED invokes ./ci/gate.sh directly; to tolerate
+# an UNDETERMINED run it writes `./ci/gate.sh || [ $? -eq 3 ]`, the same explicit idiom
+# .gitea/workflows/gate.yml already used for its spec-immutability pre-step and now uses for
+# the gate itself (with GATE_STRICT_SKIPS=true, so that tolerance can ONLY ever cover the
+# declared-environmental skip below — a missing tool on the runner is exit 1, untolerated).
+# TOLERATING A NON-ANSWER IS A VISIBLE, REVIEWED CHOICE IN THE CALLER. There is deliberately no
+# knob inside this script that turns UNDETERMINED into exit 0: that would be the fail-open this
+# change exists to remove, just relocated.
+#
+# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
+#
+#   lint / l0 / render-golden / l1   A skip here means a PINNED tool is absent or a wired
+#     seam went unwired. On a CI runner the pinned toolchain is the job image itself (the
+#     CI-built toolbox, .gitea/workflows/gate.yml), so a skip there is a DEFECT, not an
+#     environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
+#     it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
+#     in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
+#
+#   spec-immutability (R-TEST-06)    A skip means the sibling spec repo is not checked out, so
+#     the spec COULD NOT BE VERIFIED. Never a pass — unverified is unverified — so it denies a
+#     PASS and yields UNDETERMINED. It is deliberately NOT upgraded to BLOCKED by CI strictness,
+#     because on the live runner it is not a defect anyone can fix from inside this repo: the
+#     workflow does not check the spec repo out at all (SPEC_REPO_SLUG is unwired — the blueprint
+#     lives in merge-gate.yaml), and blocking would leave the only enforced status check
+#     permanently red for a reason no PR could clear. So THIS is the one skip whose non-answer the
+#     CALLER may knowingly tolerate, and gate.yml does so in one visible line. Locally the
+#     remedy exists and the message names it: check the spec repo out beside this one, or point
+#     SPEC_REPO at it (infra/devcontainer/dev.sh mounts it automatically when it is a sibling).
+#     Set REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a
+#     hard block once that checkout is wired.
+#
+#   conformance (L2/L3)              A skip when CLUSTER_AVAILABLE!=true is NOT a defect and
+#     NOT an accident: it is the environment DECLARING that the capability is absent, which
+#     is the documented Phase-0 behaviour. A declared-inapplicable stage was never "supposed
+#     to run", so it does not deny a PASS — the touched requirements it would have covered
+#     are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that
+#     into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as
+#     it already did. That asymmetry is the whole distinction this file now draws:
+#     DECLARED-not-applicable may pass; COULD-NOT-RUN may not. A declared gap is still NAMED in
+#     the PASS headline (see DECLARED_GAPS below), so a PASS never reads as "everything was
+#     verified" when a layer was declared out of scope.
+#
 # Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
 # MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
 # available, the gate reports those requirements UNVERIFIED and — unless
@@ -20,6 +89,7 @@ source "${CI_DIR}/lib.sh"
 set +e   # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
 
 : "${REQUIRE_CLUSTER_TESTS:=false}"
+: "${REQUIRE_SPEC_VERIFICATION:=false}"
 
 declare -A OUTCOME
 run_stage() {
@@ -38,15 +108,25 @@ summary "## Merge gate (R-TEST-03)"
 summary ""
 
 run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
+run_stage decisions-log "${CI_DIR}/decisions-log.sh"
 run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage approval-policy "${CI_DIR}/approval-policy.sh"
+run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
 echo >&2
 log "==== gate decision ===="
 BLOCK=0
+# Stages that did NOT run and are tolerated here — but never as green. A non-empty list
+# means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3.
+NOT_ESTABLISHED=()
+# Stages the environment DECLARED inapplicable (today: L2/L3 with no cluster declared). These
+# do not deny a PASS — nothing was supposed to run — but they are named IN the PASS headline, so
+# "PASS" is never mistaken for "everything was verified".
+DECLARED_GAPS=()
 
 # --- R-TEST-03 input: the touched-requirement subset --------------------------
 # Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
@@ -64,37 +144,135 @@ rm -f "$_reason_file"
 TOUCHED=()
 [ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
 UNDECIDED=0
+SUBSET_BROKEN=0
 if [ "$TOUCHED_RC" -eq 3 ]; then
   UNDECIDED=1
   echo >&2
   err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
   err "  determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
   summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
+elif [ "$TOUCHED_RC" -ne 0 ]; then
+  # Any OTHER non-zero exit is a hard defect in the subset computation itself — an
+  # incomplete path->requirement map (a changed path matches no row) or an invalid row
+  # would report exactly this. The reasoning is taken from the unlanded
+  # fix/touched-requirements-fail-closed (47068ee): gate.sh special-cased ONLY exit 3, so
+  # any other non-zero exit left TOUCHED empty and UNDECIDED=0, and the decision below
+  # printed "MERGE GATE: PASS" having applied no coverage gate at all. Unreachable on
+  # `main` today (touched-requirements.sh exits only 0 or 3) and wired here so it is
+  # already closed the moment a fail-closed path map makes exit 1 reachable. A subset that
+  # cannot be computed is NOT an empty subset.
+  SUBSET_BROKEN=1
+  echo >&2
+  err "R-TEST-03 DECISION INPUT BROKEN: touched-requirements.sh exited ${TOUCHED_RC}"
+  err "  (${TOUCHED_REASON:-unknown}). Its output above names the paths/rows to fix."
+  summary "- **R-TEST-03 subset UNUSABLE** (\`${TOUCHED_REASON:-unknown}\`, exit ${TOUCHED_RC}) — the coverage gate could not be applied."
 elif [ "${#TOUCHED[@]}" -gt 0 ]; then
   log "touched requirements: ${TOUCHED[*]}"
 fi
 
 # spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
-# A skip means the spec repo wasn't available to verify — surfaced, not green.
+# A skip means the spec repo wasn't available to verify — UNVERIFIED, never green: it
+# denies a PASS (exit 3) and, with REQUIRE_SPEC_VERIFICATION=true, blocks outright. See the
+# file header for why this one is not upgraded to a block by CI detection alone.
 case "${OUTCOME[spec-immutability]}" in
   fail) err  "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
-  skip) warn "spec-immutability: skipped (spec repo unavailable) — not verified" ;;
+  skip)
+    if truthy "$REQUIRE_SPEC_VERIFICATION"; then
+      err "spec-immutability: skipped (spec repo unavailable) and REQUIRE_SPEC_VERIFICATION=true -> blocks merge"
+      summary "- spec-immutability: **SKIPPED** (spec repo unavailable) — blocks merge (REQUIRE_SPEC_VERIFICATION=true)"
+      BLOCK=1
+    else
+      warn "spec-immutability: skipped (spec repo unavailable) — R-TEST-06 NOT VERIFIED, so"
+      warn "  this run cannot be a PASS. Check the spec repo out beside this one (or set"
+      warn "  SPEC_REPO) to verify it."
+      summary "- spec-immutability: **SKIPPED** (spec repo unavailable — R-TEST-06 not verified; not a pass)"
+      NOT_ESTABLISHED+=("spec-immutability")
+    fi ;;
   pass) ok   "spec-immutability: green" ;;
 esac
 
-# L0/L1 stages: any failure blocks. A SKIP here means a seam is unwired — in
+# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
-# Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
+# is the approver, per operator intent. This stage is the single exception: a change to the
+# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
+# cannot honestly be approved BY the thing it changes, so those paths still need a human.
+#
+# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
+# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
+# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
+case "${OUTCOME[approval-policy]}" in
+  fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
+  skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
+  pass) ok  "approval-policy: green" ;;
+esac
+
+# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
+# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
+# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
+# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
+# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
+# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
+# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
+# token is wired, and the skip becomes a block as it should.
+case "${OUTCOME[gitea-settings]}" in
+  fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
+  skip)
+    if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
+      err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
+      summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
+      BLOCK=1
+    else
+      warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
+      warn "  registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
+      warn "  (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
+      summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
+      NOT_ESTABLISHED+=("gitea-settings")
+    fi ;;
+  pass) ok "gitea-settings: green" ;;
+esac
+
+# decisions-log: the handoff log's storage format. A FAIL means either the frozen archive
+# DECISIONS-LOG.md was modified — which re-creates, for every OTHER open branch, the tail-hunk
+# conflict that made four of five PRs unmergeable on 2026-09-29 — or a decisions/ entry is
+# malformed. Always blocks, and has no skip path by construction: the stage reads only files in
+# the tree under test and needs no tool or cluster, so there is no environment in which it
+# "could not run". A guard against a silent, delayed, misattributed regression must not itself
+# have a way to not-run.
+case "${OUTCOME[decisions-log]}" in
+  fail) err "decisions-log: FAILED -> blocks merge (see decisions/README.md)"; BLOCK=1 ;;
+  skip) err "decisions-log: reported SKIPPED, which this stage has no path to produce -> treat"
+        err "  as a defect in the stage itself and block."
+        summary "- decisions-log: **SKIPPED** (impossible by construction) — blocks merge"
+        BLOCK=1 ;;
+  pass) ok  "decisions-log: green" ;;
+esac
+
+# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
+# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
+# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
 # no-cluster reproducibility gate (canonical render == committed apps/ goldens):
 # a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
 for s in lint l0 render-golden l1; do
   case "${OUTCOME[$s]}" in
     fail) err  "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
-    skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;
+    skip)
+      if strict_skips; then
+        err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
+        err "  toolchain is baked into the job image (.gitea/workflows/gate.yml container:) and every"
+        err "  seam is on main — so this is a DEFECT, not an environment -> blocks merge"
+        summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
+        BLOCK=1
+      else
+        warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS"
+        summary "- ${s}: **SKIPPED** (unwired seam / missing tool) — not green"
+        NOT_ESTABLISHED+=("$s")
+      fi ;;
     pass) ok   "${s}: green" ;;
   esac
 done
 
-# L2/L3 conformance: cluster-gated.
+# L2/L3 conformance: cluster-gated. The ONLY stage whose non-execution can still be a PASS,
+# because CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent rather
+# than the stage failing to run where it could have — see the file header.
 case "${OUTCOME[conformance]}" in
   fail)
     err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
@@ -105,18 +283,22 @@ case "${OUTCOME[conformance]}" in
       err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
     elif [ "$UNDECIDED" -eq 1 ]; then
       warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
+      DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
     else
       if [ "${#TOUCHED[@]}" -gt 0 ]; then
         reqs=("${TOUCHED[@]}")
         warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
         warn "  (no cluster — Phase 0). These are NOT certified green."
         summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
-        if [ "$REQUIRE_CLUSTER_TESTS" = "true" ]; then
+        if truthy "$REQUIRE_CLUSTER_TESTS"; then
           err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
           BLOCK=1
+        else
+          DECLARED_GAPS+=("conformance-l2-l3(no cluster declared; ${#TOUCHED[@]} touched reqs unverified)")
         fi
       else
         skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
+        DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
       fi
     fi ;;
 esac
@@ -132,13 +314,34 @@ if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
   err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
   BLOCK=1
 fi
+# An incomplete/invalid path map is not an environment problem to skip past: the change under
+# gate contains paths whose requirements are UNKNOWN, so no subset can be certified.
+if [ "$SUBSET_BROKEN" -eq 1 ]; then
+  err "R-TEST-03 subset unusable (${TOUCHED_REASON:-unknown}) -> blocks merge"
+  BLOCK=1
+fi
 
 echo >&2
+# Precedence: BLOCKED (something is wrong) > UNDETERMINED (a stage did not run, so
+# greenness was not established) > NOT APPLICABLE (nothing to gate) > PASS. UNDETERMINED
+# outranks NOT APPLICABLE because "no branch under gate" AND "a stage did not run" is two
+# reasons this is not a green run, and the stronger, non-zero one must be the headline.
 if [ "$BLOCK" -ne 0 ]; then
   err "MERGE GATE: BLOCKED"
   summary ""
   summary "**Result: BLOCKED** — see failures above."
   exit 1
+elif [ "${#NOT_ESTABLISHED[@]}" -gt 0 ]; then
+  err "MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED."
+  err "  Stages that did not run: ${NOT_ESTABLISHED[*]}"
+  err "  Nothing FAILED, and nothing is certified either: a stage that did not run is not"
+  err "  green (HARD RULE 4). This is exit 3, never 0 — no automation may read it as"
+  err "  merge-approved. Fix the environment (install the pinned tool / wire the seam /"
+  err "  check out the spec repo) and run again, or accept it KNOWINGLY and explicitly:"
+  err "    ./ci/gate.sh || [ \$? -eq 3 ]"
+  summary ""
+  summary "**Result: UNDETERMINED** — these stages did not run, so nothing is certified: \`${NOT_ESTABLISHED[*]}\`. Not a pass (exit 3)."
+  exit 3
 elif [ "$UNDECIDED" -eq 1 ]; then
   warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
   warn "  The stages above ran and did not fail, but NO touched-requirement subset was"
@@ -148,8 +351,16 @@ elif [ "$UNDECIDED" -eq 1 ]; then
   summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
   exit 0
 else
-  ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
+  ok "MERGE GATE: PASS — every stage that was supposed to run RAN and was GREEN, and the"
-  summary ""
+  ok "  R-TEST-03 touched-requirement subset was checked."
-  summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
+  if [ "${#DECLARED_GAPS[@]}" -gt 0 ]; then
+    ok "  NOT verified here, because the environment DECLARED it out of scope (never a silent"
+    ok "  gap — Phase 0 by policy, see the header): ${DECLARED_GAPS[*]}"
+    summary ""
+    summary "**Result: PASS** — every stage that was supposed to run ran green. Declared out of scope (NOT verified): \`${DECLARED_GAPS[*]}\`."
+  else
+    summary ""
+    summary "**Result: PASS** — every stage that was supposed to run ran green; nothing was declared out of scope."
+  fi
   exit 0
 fi

ci/gitea-settings.sh

@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
+# match what git declares.
+#
+# WHY THIS STAGE EXISTS AT ALL
+# ----------------------------
+# Every other check in this repo reads the repository, the rendered manifests, or the
+# cluster. NONE of them can see Gitea's own database — and that is where branch
+# protection and runner registration live. On 2026-09-29 that blind spot was measured:
+#
+#   * `main` had NO branch protection, and
+#   * ZERO Actions runners were registered,
+#
+# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
+# and human in the loop repeated that claim because the repository said so. PR #1 — the
+# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
+# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
+# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
+# merged with zero approvals recorded against `required_approvals: 1`.
+#
+# Three separate failures, one shape: a setting that READ as enforced while not being, with
+# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
+# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
+#
+# WHAT IT ASSERTS
+#   1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
+#      LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
+#   2. at least `.runners_online_min` Actions runners report status "online".
+#
+# (2) is not decoration. A required status check with no runner is the deadlock this repo
+# nearly walked into: every PR pends forever, including the one that would fix it.
+#
+# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
+# `protected`, `enable_status_check` and `required_approvals` but NOT
+# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
+# only account that merges. This stage queries /branch_protections/ deliberately.
+#
+# EXIT CODES (the gate contract)
+#   0  live Gitea matches the declaration
+#   1  DIVERGENCE — something drifted, or was never applied. Blocks the merge.
+#   3  SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
+#      skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
+#      that an unwired credential cannot lock the repository.
+#
+# TO WIRE IT: create a Gitea repo secret SETTINGS_API_TOKEN (a token that can read repo
+# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
+# the drift it exists to catch is still undetected.
+#
+# NOT `GITEA_SETTINGS_TOKEN`. This comment named that until 2026-09-29 and it instructed the
+# reader to do something IMPOSSIBLE: Gitea reserves the `GITEA_` prefix for secret names and
+# rejects the PUT with HTTP 400 "invalid secret name". That is not a style rule — it is why the
+# detector sat inert for hours while a token existed and no secret did. The env var below still
+# accepts GITEA_SETTINGS_TOKEN, but only for a LOCAL run: it can never be the CI wiring, because
+# the secret that would feed it cannot be created at all.
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: Gitea platform settings match the declaration"
+
+DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
+# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
+# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
+# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
+# it looked like it had worked. A token was minted, no secret existed, and this stage would
+# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+
+[ -f "$DECL" ] || die "declaration missing: ${DECL}"
+require_tool jq
+require_tool curl
+
+if [ -z "$TOKEN" ]; then
+  skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
+  skip "  Branch protection and runner registration are therefore UNVERIFIED. They were"
+  skip "  wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
+  summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
+  exit 3
+fi
+
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+BRANCH="$(jq -r '.branch' "$DECL")"
+LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
+if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
+  skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
+  skip "  lacks permission). Not treating this as a pass."
+  summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
+  exit 3
+fi
+
+RC=0
+
+# --- 1. every declared protection key must match the live rule, exactly ---------------
+while IFS= read -r key; do
+  want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
+  got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
+  if [ "$want" = "$got" ]; then
+    ok "  ${key} = ${got}"
+  else
+    err "  ${key}: DECLARED ${want} but LIVE is ${got}"
+    RC=1
+  fi
+done < <(jq -r '.protection | keys[]' "$DECL")
+
+# --- 2. a runner must actually be online ----------------------------------------------
+MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
+RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
+N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
+if [ "${N:-0}" -ge "$MIN" ]; then
+  ok "  runners online: ${N} (>= ${MIN})"
+else
+  err "  runners online: ${N}, DECLARED minimum ${MIN}."
+  err "  A required status check with no runner leaves every PR pending forever —"
+  err "  including the PR that would fix it. This is the 2026-09-29 outage condition."
+  RC=1
+fi
+
+if [ "$RC" -eq 0 ]; then
+  ok "Gitea settings match the declaration."
+  summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
+else
+  err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
+  err "  Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
+  err "  state is what you now want, change the DECLARATION in the same commit — deliberately."
+  summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
+fi
+exit "$RC"

ci/l0.sh

@@ -10,6 +10,18 @@
 #   Stage 3b L0     : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
 #                     a per-resource policy cannot express (secret hygiene across every
 #                     tracked file, least-privilege RBAC, spec-immutability enforcement).
+#                     Selection = the TOUCHED requirement subset (R-TEST-03) UNION every L0
+#                     test DECLARING `conformance.home.k3s/scope: whole-repo`. That union
+#                     replaces what used to be a hand-written "UNCONDITIONAL" stage per
+#                     whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e
+#                     bulk-tier): three near-identical 12-line rationales, three hardcoded
+#                     check.sh paths and three copies of the same rc 0/3/* case block, which
+#                     did not scale and — worse — failed SILENTLY, because whoever added the
+#                     next whole-repo invariant had to REMEMBER to hand-write a stage. Now the
+#                     test declares its own scope and the runner enforces it. The reasoning and
+#                     the incidents those stages recorded were moved INTO the tests they
+#                     describe (see each chainsaw-test.yaml's `scope` annotation) and the
+#                     general mechanism is documented in conformance/lib/common.sh.
 #
 # Integration seams are env-overridable (see ci/lib.sh + README). When an owning
 # subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
@@ -67,12 +79,37 @@ else
   cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
   find "$l0_render" -type f -name '*.yaml' -print0 \
     | xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
-  if kyverno test "$l0_render"; then
+  # DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
-    ok "Kyverno L0 policy tests passed"
+  # error, bad match block, malformed JMESPath) does not fail the run: every result row
-    summary "- kyverno-l0: **passed**"
+  # becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
+  #
+  # MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
+  # `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
+  # rows came back `Skip / Invalid Policy`, and the summary read:
+  #     Test Summary: 9 tests passed and 0 tests failed
+  # `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
+  # own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
+  #
+  # This is not specific to that policy: a typo in restrict-host-namespaces or
+  # secret-provenance would silently disable it and the gate would stay green. Scan the
+  # output for the load failure the exit code hides.
+  l0_kout="$(mktemp)"
+  if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
+    if grep -q 'Invalid Policy' "$l0_kout"; then
+      err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
+      err "  skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
+      err "  must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
+      grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
+      summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
+      rc=1
+    else
+      ok "Kyverno L0 policy tests passed"
+      summary "- kyverno-l0: **passed**"
+    fi
   else
     err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
   fi
+  rm -f "$l0_kout"
   rm -rf "$l0_render"
 fi
 
@@ -83,58 +120,45 @@ fi
 # binding, the spec-immutability gate's own behaviour) live in the conformance
 # harness's tests/l0-static subtree and run with no cluster. This stage executes them
 # through the same single-command runner CI and a laptop both use (R-TEST-02).
-log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
+#
+# THE UNION, AND WHY IT IS NOT OPTIONAL (R-TEST-03's blind spot)
+# --------------------------------------------------------------
+# 3b scopes the conformance run to the TOUCHED requirements. That is right for an invariant
+# whose inputs live under the subtree its requirements map to. It is WRONG for one that
+# quantifies over the WHOLE tracked tree, or over subtrees (apps/, infra/, most of platform/)
+# that ci/touched-requirements.sh's coarse path map keys to NOTHING: for those, the change
+# most likely to break the invariant attributes no requirement at all, so requirement-scoped
+# selection filters the check out of EXACTLY that change. `--include-scope whole-repo` unions
+# in every L0 test that DECLARES that scope, so those run on every gate invocation.
+#
+# rc 3 IS NOT A PASS. The runner returns 3 when a scope-declared check SKIPPED (its subject
+# subtree is absent, or it has no check script): UNWIRED, not green, exactly as the deleted
+# per-check stages had it ("check SKIPPED (clusters/ absent) — not a pass"). It is mapped to
+# the gate's own rc 3 below, never to rc 0.
+: "${WHOLE_REPO_SCOPE:=whole-repo}"
+log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0 --include-scope ${WHOLE_REPO_SCOPE})"
 if ! entrypoint_ready "$CONFORMANCE_CMD"; then
   skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
   summary "- l0-static: **SKIPPED** (conformance runner missing)"
   if [ "$rc" -eq 0 ]; then rc=3; fi
 else
   load_touched_requirements
-  if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
+  l0_static_rc=0
-    ok "L0 static conformance checks passed"
+  "$CONFORMANCE_CMD" --layer l0 --include-scope "$WHOLE_REPO_SCOPE" \
-    summary "- l0-static: **passed**"
+    ${REQ_ARGS[@]+"${REQ_ARGS[@]}"} || l0_static_rc=$?
-  else
+  case "$l0_static_rc" in
-    err "L0 static conformance checks failed"
+    0) ok "L0 static conformance checks passed"
-    summary "- l0-static: **FAILED**"
+       summary "- l0-static: **passed**" ;;
-    rc=1
+    3) skip "an L0 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
-  fi
+       summary "- l0-static: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
-fi
-
-# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ----------
-# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
-# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
-# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
-# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every
-# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED
-# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail
-# a legitimate manifest, and that no expansion silently empties today.
-#
-# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the
-# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys
-# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/
-# attributes no requirement, so the requirement-scoped selection would filter this check
-# out of exactly the change that introduces the bug. The invariant is whole-repo (same
-# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation.
-# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a
-# sub-second offline scan with no side effects, so the duplication is not worth avoiding.
-SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}"
-log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})"
-if [ ! -f "$SUBST_CHECK" ]; then
-  skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green"
-  summary "- subst-safety: **SKIPPED** (check missing)"
-  if [ "$rc" -eq 0 ]; then rc=3; fi
-else
-  subst_rc=0
-  bash "$SUBST_CHECK" || subst_rc=$?
-  case "$subst_rc" in
-    0) ok "Flux postBuild substitution safety passed"
-       summary "- subst-safety: **passed**" ;;
-    3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
-       summary "- subst-safety: **SKIPPED** (nothing to audit)"
        if [ "$rc" -eq 0 ]; then rc=3; fi ;;
-    *) err "Flux postBuild substitution safety FAILED"
+    *) err "L0 static conformance checks failed"
-       summary "- subst-safety: **FAILED**"; rc=1 ;;
+       summary "- l0-static: **FAILED**"
+       rc=1 ;;
   esac
+  # Per-check step-summary lines, so the gate output still NAMES what ran — the generated
+  # replacement for the one hardcoded `summary` line each deleted stage carried.
+  emit_offline_summary
 fi
 
 case "$rc" in

ci/l1.sh

@@ -29,7 +29,18 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
 # [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
 # (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
 # touched subset, then COLLECT its artifacts into the CI artifact dirs.
-"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
+# --include-scope: union the touched subset with every L1 test DECLARING
+# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
+# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
+# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
+# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
+# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
+# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
+# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
+: "${WHOLE_REPO_SCOPE:=whole-repo}"
+l1_rc=0
+"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
+  ${req_args[@]+"${req_args[@]}"} || l1_rc=$?
 
 CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
 if [ -f "${CONF_ART}/report.junit.xml" ]; then
@@ -39,5 +50,16 @@ if [ -d "${CONF_ART}/coverage" ]; then
   cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi
 
-ok "L1 rendering checks passed"
+case "$l1_rc" in
-summary "- l1-render: **passed**"
+  0) ok "L1 rendering checks passed"
+     summary "- l1-render: **passed**"
+     emit_offline_summary ;;
+  3) skip "an L1 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
+     summary "- l1-render: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
+     emit_offline_summary
+     exit 3 ;;
+  *) err "L1 rendering checks FAILED (rc=${l1_rc})"
+     summary "- l1-render: **FAILED**"
+     emit_offline_summary
+     exit "$l1_rc" ;;
+esac

ci/lib.sh

@@ -72,20 +72,83 @@ entrypoint_ready() {
   return 1
 }
 
-# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
+# truthy <value> — the ONE place the repo's "is this switch on" spelling lives, so
-# A cluster is 'available' only if explicitly declared. We NEVER auto-assume a
+# CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS
-# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
+# all agree. Anything not listed is OFF, and that direction matters: a misspelled value
-#   CLUSTER_AVAILABLE=true   -> L2/L3 run and are REQUIRED green
+# (CLUSTER_AVAILABLE=ture, REQUIRE_SPEC_VERIFICATION=yeah) leaves the guard OFF rather than
-#   CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
+# silently declaring a capability "available", so a typo can only make the gate stricter about
-cluster_available() {
+# what it claims to have verified, never looser.
-  case "${CLUSTER_AVAILABLE:-false}" in
+truthy() {
-    true|1|yes) return 0 ;;
+  case "${1:-}" in
+    true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;;
     *) return 1 ;;
   esac
 }
 
+# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
+# A cluster is 'available' only if explicitly DECLARED. We NEVER auto-assume a
+# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
+#   CLUSTER_AVAILABLE=true   -> L2/L3 run and are REQUIRED green
+#   CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
+# That DECLARATION is what makes an L2/L3 skip different IN KIND from every other skip —
+# see "WHICH SKIPS BLOCK, AND WHY" in ci/gate.sh.
+cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
+
+# --- Skip strictness: is this a CI runner, or a workstation? -------------------
+# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
+#
+#   CI       the pinned toolchain IS the job image (the CI-built toolbox; gate.yml `container:`),
+#            and every seam is on `main`. A stage that skips here is a
+#            DEFECT (a tool dropped from the image, a seam that went unwired) -> BLOCK.
+#   local    a missing pinned tool is a real, ordinary local condition -> the gate reports
+#            UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
+#
+# HOW STRICTNESS IS DECIDED — DECLARED first, detected only as a fallback:
+#
+#   1. GATE_STRICT_SKIPS=true|false is the AUTHORITY. .gitea/workflows/gate.yml sets `true` on
+#      the gate step, so on the one CI runner that matters strictness is a DECLARATION IN THE
+#      WORKFLOW, not an inference: the environment that KNOWS it is CI says so, and no
+#      env-var sniffing can get it wrong.
+#   2. `auto` (the default — anywhere the caller said nothing) falls back to GITHUB_ACTIONS /
+#      CI, which are set by the runner ITSELF (Gitea's act_runner, as well as GitHub). We do
+#      NOT sniff hostnames, paths or tool availability: a heuristic that guesses "this looks
+#      like CI" is exactly the unreliable inference this gate must not contain.
+#
+# WHY A WRONG ANSWER CANNOT FAIL OPEN: both branches are non-zero and NEITHER is a pass. A CI
+# run misread as local yields UNDETERMINED (exit 3) instead of BLOCKED (exit 1) — that still
+# fails the job unless its caller explicitly tolerates 3, which gate.yml does only knowingly
+# and only for the one declared-environmental skip (see its comment). A workstation misread as
+# CI yields BLOCKED instead of UNDETERMINED — harsher, never looser. Set GATE_STRICT_SKIPS=true
+# locally to rehearse CI strictness.
+: "${GATE_STRICT_SKIPS:=auto}"
+strict_skips() {
+  case "$GATE_STRICT_SKIPS" in
+    auto) truthy "${GITHUB_ACTIONS:-}" || truthy "${CI:-}" ;;
+    *)    truthy "$GATE_STRICT_SKIPS" ;;
+  esac
+}
+
 mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
 
+# --- Offline (L0/L1) per-check step summary -----------------------------------
+# conformance/lib/run_l1_offline.py writes one Markdown line per check it executed to
+# conformance/_artifacts/offline-summary.md. Splice those into $GITHUB_STEP_SUMMARY so the
+# gate's output still NAMES every check that ran and its verdict.
+#
+# WHY THIS EXISTS: ci/l0.sh used to carry a hand-written stage per whole-repo check, each with
+# its own `summary "- subst-safety: **passed**"` line. Those stages are gone (the tests declare
+# `conformance.home.k3s/scope: whole-repo` instead and the runner unions them in), so the
+# naming is generated for EVERY offline check rather than for the three that had a stage.
+: "${OFFLINE_SUMMARY_FILE:=${CONFORMANCE_DIR}/_artifacts/offline-summary.md}"
+emit_offline_summary() {
+  local f="${1:-$OFFLINE_SUMMARY_FILE}" line
+  [ -f "$f" ] || return 0
+  while IFS= read -r line; do
+    [ -n "$line" ] && summary "  $line"
+  done < "$f"
+  return 0
+}
+
 # --- Touched-requirement subset (R-TEST-03 input) ----------------------------
 # Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
 # (ready to splice into a conformance/run.sh invocation).
@@ -111,7 +174,16 @@ load_touched_requirements() {
     warn "  Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
     warn "  coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
   elif [ "$rc" -ne 0 ]; then
-    warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
+    # Any OTHER non-zero exit means the subset is UNUSABLE, not empty (today: a
+    # path->requirement map that cannot classify a changed path, once that map fails closed).
+    # The FULL layer still runs here as a superset — nothing goes untested — but this stage must
+    # not claim the SCOPED check happened, and ci/gate.sh turns the same exit into a BLOCKED
+    # merge decision. Wording carried over from the unlanded fix/touched-requirements-fail-closed
+    # (47068ee), whose gate-side half this branch lands.
+    warn "touched-requirements.sh exited ${rc} (subset UNUSABLE — see its output above)."
+    warn "  Running the FULL layer as a superset; the SCOPED R-TEST-03 check did NOT happen."
+    warn "  ci/gate.sh BLOCKS the merge on this: fix the path->requirement map in"
+    warn "  ci/touched-requirements.sh so every changed path is classified."
   elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
     log "touched requirements: ${TOUCHED_REQS[*]}"
     REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")

ci/lint.sh

@@ -25,5 +25,72 @@ if [ "${#files[@]}" -eq 0 ]; then
   exit 0
 fi
 
-yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
+# yamllint is single-threaded, so the file list is split into LINT_JOBS CONTIGUOUS slices of the
-ok "yaml/lint passed (${#files[@]} files)"
+# sorted list, one yamllint per slice, run concurrently. Each slice's output goes to its own file
+# and the files are printed in slice order afterwards: the log is the same text, in the same
+# file order, as one serial `yamllint "${files[@]}"`, and a finding is never interleaved with
+# another process's output (yamllint's standard format puts the FILE NAME on its own line above
+# that file's `line:col` findings, so interleaving would misattribute them). The stage's exit
+# code is the highest any slice returned (yamllint: 1 = error, 2 = warning under strict), so a
+# single failing slice fails the stage exactly as the serial call did. LINT_JOBS=1 is serial.
+lint_jobs="${LINT_JOBS:-$(n="$(nproc 2>/dev/null || echo 1)"; [ "$n" -gt 8 ] && n=8; echo "$n")}"
+case "$lint_jobs" in ''|*[!0-9]*|0) err "LINT_JOBS must be a positive integer, got '${lint_jobs}'"; exit 2 ;; esac
+[ "$lint_jobs" -gt "${#files[@]}" ] && lint_jobs="${#files[@]}"
+lint_tmp="$(mktemp -d)"
+trap 'rm -rf "$lint_tmp"' EXIT
+per=$(( (${#files[@]} + lint_jobs - 1) / lint_jobs ))
+pids=()
+for (( i = 0; i < lint_jobs; i++ )); do
+  slice=("${files[@]:$(( i * per )):${per}}")
+  [ "${#slice[@]}" -gt 0 ] || break
+  ( rc=0
+    yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${slice[@]}" \
+      >"${lint_tmp}/${i}.out" 2>&1 || rc=$?
+    echo "$rc" >"${lint_tmp}/${i}.rc" ) &
+  pids+=("$!")
+done
+wait "${pids[@]}" || true
+lint_rc=0
+for (( i = 0; i < ${#pids[@]}; i++ )); do
+  cat "${lint_tmp}/${i}.out"
+  # A slice that left no rc file died before yamllint returned: that is a failure, not a pass.
+  slice_rc="$(cat "${lint_tmp}/${i}.rc" 2>/dev/null || true)"
+  case "$slice_rc" in ''|*[!0-9]*) slice_rc=1 ;; esac
+  [ "$slice_rc" -gt "$lint_rc" ] && lint_rc="$slice_rc"
+done
+if [ "$lint_rc" -ne 0 ]; then
+  err "yaml/lint failed (yamllint exit ${lint_rc}; ${#files[@]} files in ${#pids[@]} parallel slice(s))"
+  exit "$lint_rc"
+fi
+ok "yaml/lint passed (${#files[@]} files, ${#pids[@]} parallel slice(s))"
+
+# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
+# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
+# while the gate stayed green. Valid YAML, so yamllint cannot see it.
+missing=0
+while IFS=: read -r wf _ value; do
+  value="${value#*configurationFile:}"
+  value="$(echo "$value" | sed -E 's/#.*//; s/^[[:space:]"'\'']+//; s/[[:space:]"'\'']+$//')"
+  if [ -n "$value" ] && [ ! -e "$value" ]; then
+    err "${wf}: configurationFile '${value}' does not exist in the repo"
+    missing=1
+  fi
+done < <(grep -nE '^[[:space:]]*configurationFile:' .gitea/workflows/*.y*ml 2>/dev/null || true)
+[ "$missing" -eq 0 ] || exit 1
+ok "workflow configurationFile references resolve"
+
+# A workflow must not use an action that needs a Docker daemon: jobs on this runner get NONE, by
+# decision (decisions/2026-10-29-image-pipeline.md; act_runner maps docker_host to `-`). Such an
+# action is valid YAML and passes every other stage, then dies at runtime with
+# "failed to connect to the docker API at unix:///var/run/docker.sock". renovatebot/github-action
+# did exactly that on every Renovate run until 2026-10-30 (run 343). Run the tool's image as the
+# job `container:` instead (see .gitea/workflows/renovate.yaml). Container (`docker://`) actions
+# and Docker's own daemon-bound actions are rejected for the same reason. Comments are ignored.
+daemon_uses='uses:[[:space:]]*["'\'']?(docker://|renovatebot/github-action|docker/(build-push|setup-buildx|setup-qemu|login)-action)'
+bad=0
+while IFS= read -r hit; do
+  err "${hit%%:*}: needs a Docker daemon, which jobs here do not get: $(echo "${hit#*:*:}" | sed -E 's/^[[:space:]]+//')"
+  bad=1
+done < <(grep -HnE "^[^#]*${daemon_uses}" .gitea/workflows/*.y*ml 2>/dev/null || true)
+[ "$bad" -eq 0 ] || exit 1
+ok "no workflow uses a Docker-daemon-bound action"

ci/render-golden.sh

@@ -1,5 +1,10 @@
 #!/usr/bin/env bash
-# ci/render-golden.sh — Stage: render==golden reproducibility gate.
+# ci/render-golden.sh — Stage: the renderer is trustworthy.
+#   (1) catalog/render.py passes its OWN output-contract self-test, and
+#   (2) every committed apps/ golden reproduces byte-for-byte from the canonical render.
+# Two independent assertions about one subject, catalog/render.py. (1) was added
+# 2026-10-01 because nothing in ci/ or .gitea/ had ever invoked `--selftest`; see the
+# block above the invocation for why it lives here and not in a conformance check.
 #
 # WHY: the committed apps/ tree is a set of GOLDENS produced by ONE canonical
 # invocation of catalog/render.py. If a golden is hand-edited, or a catalog
@@ -54,6 +59,54 @@ if [ ! -f "$RENDER_PY" ]; then
   summary "- render-golden: **SKIPPED** (renderer missing)"
   exit 3
 fi
+# --- The renderer's OWN self-test, before anything is compared ---------------
+# `catalog/render.py --selftest` is ~106 executed assertions on the renderer's output
+# contract (default-deny covers both directions, the Kyverno namespace hook label, a valid
+# auth-mode, the oauth2-proxy sidecar's cookie-name/-domain/-refresh flags, the
+# placeholder-substitution guard, the observability refusals...). Until now NOTHING in ci/
+# or .gitea/ invoked it — measured by grep on 2026-09-30 and recorded as the top follow-up
+# of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md — so every one of those
+# assertions was dev-time only and a merge gate could not tell a sound renderer from a
+# gutted one.
+#
+# WHY HERE, AND NOT IN A CONFORMANCE CHECK. The lead was
+# conformance/tests/l0-static/auditability-proofs-producible/check.sh, which drives
+# `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them".
+# That loop is NOT the right hook: that check serves R-SEC-03 and its subject is the
+# EVIDENCE CHAIN (can a forged or stale run become a proof?). The renderer is not part of
+# that chain, and hanging it there would make the check's name and its requirement mapping
+# a lie. This stage's subject already IS catalog/render.py — it exists to assert that the
+# renderer's output is trustworthy — so the renderer's own contract test belongs in it.
+#
+# It runs BEFORE the catalog/empty-catalog exit-3 below on purpose: the self-test is
+# deterministic and catalog-independent (it renders synthetic ServiceDefinitions in
+# memory), so an empty catalog must not be able to skip it.
+#
+# render-golden and the self-test are INDEPENDENT assertions, which is why both are needed:
+# render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose
+# output was re-rendered, including one that just lost a contract invariant and had all ten
+# goldens regenerated in the same commit. The self-test catches exactly that.
+#
+# FAILURE IS A HARD FAIL, NOT A SKIP: the renderer is source in this repo, so "could not
+# run its self-test" has no legitimate environmental cause once python3 and render.py are
+# both present (both asserted above). ci/gate.sh maps this stage's rc 1 to BLOCKED.
+log "Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)"
+selftest_out="$(mktemp)"
+if "$PYTHON" "$RENDER_PY" --selftest >"$selftest_out" 2>&1; then
+  ok "render.py --selftest passed"
+  summary "- render-selftest: **passed** (\`catalog/render.py --selftest\`)"
+else
+  err "catalog/render.py --selftest FAILED — the renderer violates its own output contract."
+  err "  Every golden under apps/ is produced by this renderer, so the goldens below cannot"
+  err "  be trusted even if they reproduce byte-for-byte. Reproduce with:"
+  err "    ${PYTHON} catalog/render.py --selftest"
+  sed 's/^/    /' "$selftest_out" >&2 || true
+  summary "- render-selftest: **FAILED** (\`catalog/render.py --selftest\` — renderer contract violated)"
+  rm -f "$selftest_out"
+  exit 1
+fi
+rm -f "$selftest_out"
+
 if [ ! -d "$CATALOG_DIR" ] || ! compgen -G "${CATALOG_DIR}/*.yaml" >/dev/null 2>&1; then
   skip "no catalog ServiceDefinitions under '${CATALOG_DIR}' — nothing to render."
   summary "- render-golden: **SKIPPED** (empty catalog)"

ci/render-keycloak-client-urls.py

@@ -0,0 +1,214 @@
+#!/usr/bin/env python3
+# =============================================================================
+# ci/render-keycloak-client-urls.py
+#
+# Render platform/keycloak/client-urls.yaml FROM platform/keycloak/keycloak-realm-import.yaml.
+#
+# WHY THIS EXISTS
+# ---------------
+# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
+# Done=True / Started=False it never re-runs, so a redirect URI changed in git is
+# never applied to the realm. The client-secret half of that failure class is
+# already healed by a level-triggered CronJob (platform/keycloak/client-secret-
+# reconciler.yaml). This renderer supplies the DESIRED STATE that the same CronJob
+# needs to heal the URL half.
+#
+# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
+# automountServiceAccountToken: false. It has NO Kubernetes API access at all, by
+# design (R-SEC-02) — so it cannot read the KeycloakRealmImport CR to learn what
+# the realm is supposed to look like. Its inputs must be MOUNTED. Hence a
+# ConfigMap.
+#
+# A hand-authored ConfigMap would be a SECOND SOURCE OF TRUTH for every public
+# hostname in the realm, free to disagree with the realm import — and "two
+# declarations that can disagree" is the same shape as the bug this whole effort
+# is closing. So the ConfigMap is GENERATED from the realm import and the
+# generation is GATED: conformance/tests/l0-static/keycloak-client-secret-
+# reconciler/check.sh runs this script with --check on every merge, and the merge
+# fails if the committed file is not what this renderer produces. That is the same
+# render==golden idiom catalog/render.py + ci/render-golden.sh already use for
+# apps/.
+#
+# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. If a URL in the realm import is
+# written as a Flux substitution token (https://grafana.${BASE_DOMAIN}/...), the
+# token is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files
+# are rendered by the SAME Flux Kustomization (./platform-config, whose
+# postBuild.substituteFrom reads cluster-vars), so both are substituted in the
+# same pass with the same values and cannot disagree about what BASE_DOMAIN is.
+# This renderer never parses, splits, or reconstructs a hostname, and never reads
+# cluster-vars — so there is no second place for the domain to be decided.
+#
+# USAGE
+#   python3 ci/render-keycloak-client-urls.py            # print the rendered file
+#   python3 ci/render-keycloak-client-urls.py --write     # write it (regenerate)
+#   python3 ci/render-keycloak-client-urls.py --check      # exit 1 if it has drifted
+# =============================================================================
+from __future__ import annotations
+
+import argparse
+import difflib
+import json
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:  # pragma: no cover
+    print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(__file__).resolve().parent.parent
+IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
+OUT_FILE = "platform/keycloak/client-urls.yaml"
+DATA_KEY = "client-urls.json"
+CONFIGMAP_NAME = "keycloak-client-urls"
+NAMESPACE = "keycloak"
+
+# The client fields this renderer exports, and therefore the ONLY fields the
+# reconciler may write. Every one of them is an environment-derived URL: it changes
+# when BASE_DOMAIN changes, and a stale value breaks OIDC login (a wrong
+# redirectUri) or silently widens trust (a stale wildcard left behind).
+#
+# DELIBERATELY ABSENT: `secret` (healed by the secret pass from the SOPS Secret,
+# and it must never appear in a ConfigMap), and every non-URL field —
+# protocolMappers, defaultClientScopes, attributes, flags. Those are not
+# environment-derived, so nothing re-renders them on a domain flip; keeping them
+# out keeps the reconciler's write surface as narrow as the problem.
+LIST_FIELDS = ("redirectUris", "webOrigins")
+STR_FIELDS = ("rootUrl", "baseUrl", "adminUrl")
+
+
+def render(root: Path) -> str:
+    """Return the full text of client-urls.yaml as it should be committed."""
+    docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
+    imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
+    if imp is None:
+        raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
+    clients = (imp.get("spec") or {}).get("realm", {}).get("clients") or []
+    if not clients:
+        # Fail closed. Emitting an empty desired state would make the reconciler
+        # believe every client is unmanaged and silently stop reconciling URLs.
+        raise SystemExit("FATAL: %s declares zero clients — refusing to render an "
+                         "empty desired state" % IMPORT_FILE)
+
+    want: dict[str, dict] = {}
+    for c in clients:
+        cid = c.get("clientId")
+        if not isinstance(cid, str) or not cid:
+            raise SystemExit("FATAL: %s has a client with no clientId" % IMPORT_FILE)
+        entry: dict[str, object] = {}
+        for f in LIST_FIELDS:
+            if f not in c:
+                continue                 # not declared => NOT MANAGED (never forced empty)
+            v = c[f]
+            if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
+                raise SystemExit("FATAL: %s client %r field %s is not a list of strings"
+                                 % (IMPORT_FILE, cid, f))
+            entry[f] = list(v)           # verbatim, order as authored
+        for f in STR_FIELDS:
+            if f not in c:
+                continue
+            v = c[f]
+            if not isinstance(v, str):
+                raise SystemExit("FATAL: %s client %r field %s is not a string"
+                                 % (IMPORT_FILE, cid, f))
+            entry[f] = v
+        if entry:
+            want[cid] = entry
+    if not want:
+        raise SystemExit("FATAL: no client in %s declares any of %s — refusing to render "
+                         "an empty desired state"
+                         % (IMPORT_FILE, ", ".join(LIST_FIELDS + STR_FIELDS)))
+
+    body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
+    indented = "\n".join(("    " + ln) if ln else "" for ln in body.splitlines())
+    return HEADER + indented + "\n"
+
+
+HEADER = """---
+# =============================================================================
+# GENERATED FILE — DO NOT EDIT BY HAND.
+#
+#   source:      platform/keycloak/keycloak-realm-import.yaml (spec.realm.clients)
+#   regenerate:  python3 ci/render-keycloak-client-urls.py --write
+#   gated by:    conformance/tests/l0-static/keycloak-client-secret-reconciler
+#                (runs this renderer with --check on every merge)
+#
+# WHAT THIS IS: the DESIRED STATE for every environment-derived client URL in the
+# `homelab` realm — redirectUris, webOrigins, rootUrl, baseUrl, adminUrl. The
+# client-secret reconciler CronJob mounts it and makes the live realm match it.
+#
+# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
+# once Done=True, so a redirect URI edited in git never reaches the realm. Prod's
+# stored redirect URIs were measured still pointing at the pre-migration hostname
+# while git declared the new one, and every OIDC login would have failed with a
+# redirect_uri mismatch. The reconciler is the level-triggered loop that closes
+# that; this file is what it reconciles TOWARDS.
+#
+# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of every public
+# hostname would be a second source of truth, free to disagree with the realm
+# import. It is rendered from the import and the render is gated, so the two
+# cannot drift. URL strings are copied VERBATIM — if the source uses a
+# ${BASE_DOMAIN} token it appears here unchanged, and both files are substituted
+# by the SAME Flux Kustomization (./platform-config), so the domain is decided in
+# exactly one place.
+#
+# A field that is NOT declared on a client in the realm import is simply ABSENT
+# here, and the reconciler leaves it alone. Absence means UNMANAGED, never
+# "force to empty".
+# =============================================================================
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: %s
+  namespace: %s
+  labels:
+    app.kubernetes.io/name: %s
+    app.kubernetes.io/part-of: keycloak
+data:
+  %s: |
+""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME, DATA_KEY)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    g = ap.add_mutually_exclusive_group()
+    g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
+    g.add_argument("--check", action="store_true",
+                   help="exit 1 if the committed file has drifted from the source")
+    args = ap.parse_args()
+
+    want = render(ROOT)
+    target = ROOT / OUT_FILE
+
+    if args.write:
+        target.write_text(want, encoding="utf-8")
+        print("wrote %s" % OUT_FILE)
+        return 0
+
+    if args.check:
+        if not target.is_file():
+            print("CHECK FAIL: %s does not exist. Run: python3 "
+                  "ci/render-keycloak-client-urls.py --write" % OUT_FILE, file=sys.stderr)
+            return 1
+        have = target.read_text(encoding="utf-8")
+        if have == want:
+            print("client-urls.yaml is in sync with %s" % IMPORT_FILE)
+            return 0
+        print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
+              "realm on a desired state the realm import does not declare.\n"
+              "Regenerate with: python3 ci/render-keycloak-client-urls.py --write\n"
+              % (OUT_FILE, IMPORT_FILE), file=sys.stderr)
+        for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
+                                         fromfile="committed " + OUT_FILE,
+                                         tofile="rendered from " + IMPORT_FILE):
+            sys.stderr.write(line)
+        return 1
+
+    sys.stdout.write(want)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

ci/render-keycloak-passkey-policy.py

@@ -0,0 +1,490 @@
+#!/usr/bin/env python3
+# =============================================================================
+# ci/render-keycloak-passkey-policy.py
+#
+# Render platform/keycloak/passkey-desired-state.yaml FROM
+# platform/keycloak/keycloak-realm-import.yaml.
+#
+# WHY THIS EXISTS
+# ---------------
+# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
+# Done=True / Started=False it never re-runs. Measured on prod 2026-10-18, hours
+# after the passkey declaration merged:
+#
+#     kubectl -n keycloak get keycloakrealmimport homelab -o jsonpath=...status
+#     Done=True  Started=False  HasErrors=False
+#
+# So the WebAuthn passwordless policy and the restructured `browser-mfa` flow that
+# keycloak-realm-import.yaml declares are INERT. They are in git, Flux applied
+# them, and they reach the realm never. platform/keycloak/passkey-reconciler.yaml
+# is the level-triggered loop that closes that; this renderer supplies the DESIRED
+# STATE it converges towards.
+#
+# THE ONE VALUE THAT IS NOT RENDERED: `bind-browser-flow`, git's switch for whether the
+# reconciler may BIND its verified candidate browser flow (one atomic PUT of the
+# realm's browserFlow). It is an operator DECISION -- "I have tested the candidate
+# flow in a browser; make it the login path" -- not a fact derivable from the realm
+# import, so it cannot be generated. It is instead CARRIED THROUGH from the committed
+# file: --write preserves it, --check accepts any value of it that is exactly the
+# string "true" or "false" and rejects everything else (an unquoted YAML true is a
+# bool, which a ConfigMap rejects at apply time -- failing the whole platform-config
+# Kustomization -- so it is refused HERE, at merge time). An absent file or key
+# renders as "false". Going live is therefore a one-line PR flipping that line.
+#
+# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
+# automountServiceAccountToken: false, exactly like its client-secret sibling. It
+# has NO Kubernetes API access at all, by design (R-SEC-02) — so it cannot read the
+# KeycloakRealmImport CR to learn what the realm is supposed to look like. Its input
+# must be MOUNTED. Hence a ConfigMap.
+#
+# THAT IS ALSO WHY THIS IS A RENDERER AND NOT A HAND-WRITTEN ConfigMap. A
+# hand-authored copy of the policy would be a SECOND SOURCE OF TRUTH for the RP ID,
+# the user-verification requirement and the whole flow tree — free to disagree with
+# the realm import. "Two declarations that can disagree" is the same shape as the
+# bug this whole effort is closing. So the ConfigMap is GENERATED from the realm
+# import and the generation is GATED:
+# conformance/tests/l0-static/keycloak-passkey-reconciler/check.sh runs this script
+# with --check on every merge, and the merge fails if the committed file is not what
+# this renderer produces. Same render==golden idiom as
+# ci/render-keycloak-client-urls.py, catalog/render.py and ci/render-golden.sh.
+#
+# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. webAuthnPolicyPasswordlessRpId is
+# written in the realm import as the Flux substitution token ${BASE_DOMAIN}, and it
+# is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files are
+# rendered by the SAME Flux Kustomization (./platform-config, whose
+# postBuild.substituteFrom reads cluster-vars), so both are substituted in the same
+# pass with the same value and cannot disagree about what the base domain is. This
+# renderer never parses, splits or reconstructs a hostname and never reads
+# cluster-vars — there is no second place the domain is decided.
+#
+# THE ONE THING THIS RENDERER DERIVES RATHER THAN COPIES
+# -----------------------------------------------------
+# The enrolment required action, `webauthn-register-passwordless`. It is
+# deliberately NOT declared in the realm import, and the import's own header
+# explains at length why declaring it there would be actively DANGEROUS: Keycloak
+# 26.0.6 treats `requiredActions` in a realm representation as REPLACE, not merge,
+# so declaring one action imports exactly that action plus delete_account and skips
+# DefaultRequiredActions.addActions entirely — deleting CONFIGURE_TOTP,
+# UPDATE_PASSWORD, VERIFY_EMAIL and seven more. Losing CONFIGURE_TOTP is a hard
+# lock-out of every platform-admin, because auth-otp-form in the browser flow still
+# demands a TOTP they could then never enrol.
+#
+# So the required action cannot be copied from the import, because it is correctly
+# absent from it. It is instead DERIVED, as a consequence of what the import DOES
+# declare:
+#
+#   IF the declared browser flow contains a `webauthn-authenticator-passwordless`
+#   execution, THEN passkey logins are offered, THEREFORE the enrolment action must
+#   be registered and enabled — and NOT a defaultAction, because the import's whole
+#   stated posture is "additive and optional" and a defaultAction forces enrolment
+#   on every user at next login.
+#
+# This is a derivation, not a second declaration: there is no value here that a
+# human could edit into disagreement with the import. If the import stops declaring
+# the passkey execution, this renderer stops emitting the required action (and
+# fails, because then the whole desired state is empty of passkey intent).
+#
+# WHAT IS DELIBERATELY NOT EXPORTED
+#   * the TWO-FACTOR webAuthnPolicy* family. Nothing in this realm uses the
+#     `webauthn-authenticator` two-factor authenticator; the passwordless
+#     authenticator reads getWebAuthnPolicyPasswordless() only, so a policy written
+#     into the two-factor family is a silent no-op. Exporting it would hand the
+#     reconciler a write surface with no purpose.
+#   * `clients`, `users`, `groups`, `roles`. Client secrets and client URLs are the
+#     SIBLING reconciler's surface (platform/keycloak/client-secret-reconciler.yaml),
+#     and users/groups/roles are nobody's — prod's real user and the media-users /
+#     platform-admins / viewers groups must survive every run of everything.
+#   * execution `priority`. It is consumed here to ORDER the exported executions
+#     (Keycloak returns them in priority order), but the numbers themselves are not
+#     exported: the reconciler compares the resulting SEQUENCE, which is what
+#     actually determines behaviour. Exporting the integers would make a
+#     priority renumbering that changes no ordering read as drift.
+#
+# USAGE
+#   python3 ci/render-keycloak-passkey-policy.py            # print the rendered file
+#   python3 ci/render-keycloak-passkey-policy.py --write     # write it (regenerate)
+#   python3 ci/render-keycloak-passkey-policy.py --check     # exit 1 if it has drifted
+# =============================================================================
+from __future__ import annotations
+
+import argparse
+import difflib
+import json
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:  # pragma: no cover
+    print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(__file__).resolve().parent.parent
+IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
+OUT_FILE = "platform/keycloak/passkey-desired-state.yaml"
+DATA_KEY = "passkey-desired-state.json"
+CONFIGMAP_NAME = "keycloak-passkey-desired-state"
+NAMESPACE = "keycloak"
+
+# The realm keys this renderer exports, and therefore the ONLY realm keys the
+# reconciler may write. Every one is a webAuthnPolicyPasswordless* field. Kept in
+# lockstep with POLICY_FIELDS in the reconciler script; the L0 check asserts the two
+# tuples are identical, so they cannot drift apart silently.
+POLICY_STR_FIELDS = (
+    "webAuthnPolicyPasswordlessRpId",
+    "webAuthnPolicyPasswordlessRpEntityName",
+    "webAuthnPolicyPasswordlessRequireResidentKey",
+    "webAuthnPolicyPasswordlessUserVerificationRequirement",
+    "webAuthnPolicyPasswordlessAuthenticatorAttachment",
+    "webAuthnPolicyPasswordlessAttestationConveyancePreference",
+)
+POLICY_LIST_FIELDS = (
+    "webAuthnPolicyPasswordlessSignatureAlgorithms",
+    "webAuthnPolicyPasswordlessAcceptableAaguids",
+)
+POLICY_INT_FIELDS = ("webAuthnPolicyPasswordlessCreateTimeout",)
+POLICY_BOOL_FIELDS = ("webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister",)
+POLICY_FIELDS = (POLICY_STR_FIELDS + POLICY_LIST_FIELDS + POLICY_INT_FIELDS
+                 + POLICY_BOOL_FIELDS)
+
+# The field Keycloak 26.0.6 gates the WHOLE passwordless policy on
+# (DefaultExportImportManager.updateRealm: the policy is applied only when
+# RpEntityName is non-null and non-empty). A desired state without it would produce
+# an accepted PUT that changes nothing, so refuse to render one.
+RP_ENTITY_NAME = "webAuthnPolicyPasswordlessRpEntityName"
+
+# The execution whose presence is what makes passkey enrolment necessary.
+PASSKEY_AUTHENTICATOR = "webauthn-authenticator-passwordless"
+ENROL_ACTION = "webauthn-register-passwordless"
+
+
+BIND_KEY = "bind-browser-flow"
+BIND_VALUES = ("true", "false")
+
+
+def committed_bind(root: Path) -> str:
+    """The operator's bind-browser-flow value from the COMMITTED file, or "false".
+
+    This is the only input read from the output file, and it is validated, not trusted:
+    anything other than the exact strings "true" / "false" is a hard failure."""
+    target = root / OUT_FILE
+    if not target.is_file():
+        return "false"
+    docs = [d for d in yaml.safe_load_all(target.read_text(encoding="utf-8")) if d]
+    cm = next((d for d in docs if d.get("kind") == "ConfigMap"), None) or {}
+    data = cm.get("data") or {}
+    if BIND_KEY not in data:
+        return "false"
+    v = data[BIND_KEY]
+    if not isinstance(v, str) or v not in BIND_VALUES:
+        raise SystemExit(
+            "FATAL: %s data.%s is %r. It must be exactly the QUOTED string \"true\" or "
+            "\"false\": an unquoted true/false is a YAML boolean, which a ConfigMap rejects "
+            "at apply time and which would fail the whole platform-config Kustomization"
+            % (OUT_FILE, BIND_KEY, v))
+    return v
+
+
+def render(root: Path) -> str:
+    """Return the full text of passkey-desired-state.yaml as it should be committed."""
+    docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
+    imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
+    if imp is None:
+        raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
+    realm = (imp.get("spec") or {}).get("realm") or {}
+    realm_name = realm.get("realm")
+    if not isinstance(realm_name, str) or not realm_name:
+        raise SystemExit("FATAL: %s declares no spec.realm.realm name" % IMPORT_FILE)
+
+    # ---- the passwordless policy, copied VERBATIM ----------------------------------
+    policy: dict[str, object] = {}
+    for f in POLICY_STR_FIELDS:
+        if f not in realm:
+            continue                      # not declared => NOT MANAGED
+        v = realm[f]
+        # A bare ${BASE_DOMAIN} parses as a plain string; an int or bool here would be
+        # a YAML quoting mistake that must not be silently stringified.
+        if not isinstance(v, str):
+            raise SystemExit("FATAL: %s realm field %s is not a string (got %r) — "
+                             "quote it in the realm import" % (IMPORT_FILE, f, v))
+        policy[f] = v
+    for f in POLICY_LIST_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
+            raise SystemExit("FATAL: %s realm field %s is not a list of strings"
+                             % (IMPORT_FILE, f))
+        policy[f] = list(v)               # verbatim, order as authored
+    for f in POLICY_INT_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, int) or isinstance(v, bool):
+            raise SystemExit("FATAL: %s realm field %s is not an integer"
+                             % (IMPORT_FILE, f))
+        policy[f] = v
+    for f in POLICY_BOOL_FIELDS:
+        if f not in realm:
+            continue
+        v = realm[f]
+        if not isinstance(v, bool):
+            raise SystemExit("FATAL: %s realm field %s is not a boolean"
+                             % (IMPORT_FILE, f))
+        policy[f] = v
+
+    if not policy:
+        # Fail closed. An empty policy would make the reconciler believe every field is
+        # unmanaged and silently stop reconciling — the realm import's own fail-open.
+        raise SystemExit("FATAL: %s declares no webAuthnPolicyPasswordless* field — "
+                         "refusing to render an empty desired state" % IMPORT_FILE)
+    if not policy.get(RP_ENTITY_NAME):
+        raise SystemExit(
+            "FATAL: %s does not declare a non-empty %s. Keycloak 26.0.6 applies the "
+            "passwordless policy ONLY when that field is set, so every realm PUT would "
+            "be accepted and change nothing" % (IMPORT_FILE, RP_ENTITY_NAME))
+
+    # ---- the flow tree, flattened per flow and ORDERED BY PRIORITY -----------------
+    browser_flow = realm.get("browserFlow")
+    if not isinstance(browser_flow, str) or not browser_flow:
+        raise SystemExit("FATAL: %s declares no spec.realm.browserFlow — the flow could "
+                         "be perfect and still not be the one anyone logs in through"
+                         % IMPORT_FILE)
+    raw_flows = realm.get("authenticationFlows") or []
+    if not isinstance(raw_flows, list) or not raw_flows:
+        raise SystemExit("FATAL: %s declares no authenticationFlows" % IMPORT_FILE)
+
+    flows: dict[str, object] = {}
+    for fl in raw_flows:
+        alias = fl.get("alias")
+        if not isinstance(alias, str) or not alias:
+            raise SystemExit("FATAL: %s has an authentication flow with no alias"
+                             % IMPORT_FILE)
+        if alias in flows:
+            raise SystemExit("FATAL: %s declares the flow %r twice"
+                             % (IMPORT_FILE, alias))
+        execs = fl.get("authenticationExecutions") or []
+        if not isinstance(execs, list) or not execs:
+            raise SystemExit("FATAL: %s flow %r declares no authenticationExecutions"
+                             % (IMPORT_FILE, alias))
+        ordered = []
+        for ex in sorted(execs, key=lambda e: (e.get("priority") or 0)):
+            is_flow = bool(ex.get("authenticatorFlow"))
+            name = ex.get("flowAlias") if is_flow else ex.get("authenticator")
+            if not isinstance(name, str) or not name:
+                raise SystemExit(
+                    "FATAL: %s flow %r has an execution with neither an authenticator "
+                    "nor a flowAlias" % (IMPORT_FILE, alias))
+            requirement = ex.get("requirement")
+            if not isinstance(requirement, str) or not requirement:
+                raise SystemExit("FATAL: %s flow %r execution %r declares no requirement"
+                                 % (IMPORT_FILE, alias, name))
+            entry: dict[str, object] = {
+                "kind": "flow" if is_flow else "authenticator",
+                "name": name,
+                "requirement": requirement,
+            }
+            cfg = ex.get("authenticatorConfig")
+            if cfg is not None:
+                if not isinstance(cfg, str) or not cfg:
+                    raise SystemExit("FATAL: %s flow %r execution %r has a non-string "
+                                     "authenticatorConfig" % (IMPORT_FILE, alias, name))
+                entry["configAlias"] = cfg
+            ordered.append(entry)
+        flows[alias] = {"topLevel": bool(fl.get("topLevel")), "executions": ordered}
+
+    if browser_flow not in flows:
+        raise SystemExit("FATAL: %s binds browserFlow to %r but declares no such flow"
+                         % (IMPORT_FILE, browser_flow))
+    # Every referenced subflow must be declared, or the reconciler's flattening walks
+    # into a hole and the audit silently covers less than it claims.
+    for alias, spec in flows.items():
+        for ex in spec["executions"]:
+            if ex["kind"] == "flow" and ex["name"] not in flows:
+                raise SystemExit("FATAL: %s flow %r references subflow %r, which is not "
+                                 "declared" % (IMPORT_FILE, alias, ex["name"]))
+
+    # ---- authenticator configs, copied verbatim ------------------------------------
+    configs: dict[str, dict] = {}
+    for cfg in realm.get("authenticatorConfig") or []:
+        alias = cfg.get("alias")
+        if not isinstance(alias, str) or not alias:
+            raise SystemExit("FATAL: %s has an authenticatorConfig with no alias"
+                             % IMPORT_FILE)
+        body = cfg.get("config") or {}
+        if not isinstance(body, dict) or not body:
+            raise SystemExit("FATAL: %s authenticatorConfig %r has no config body"
+                             % (IMPORT_FILE, alias))
+        # Values are stringified because Keycloak stores authenticator config values as
+        # strings. `negate: "false"` in the import and a stored "false" are the same
+        # declaration, and a YAML-unquoted `false` must not become a Python bool that
+        # then never matches.
+        configs[alias] = {k: str(v) for k, v in sorted(body.items())}
+
+    # Every configAlias a flow references must be defined, else the reconciler reports
+    # "git declares config X but defines no key/value for it" on every run forever.
+    for alias, spec in flows.items():
+        for ex in spec["executions"]:
+            ca = ex.get("configAlias")
+            if ca and ca not in configs:
+                raise SystemExit(
+                    "FATAL: %s flow %r execution %r references authenticatorConfig %r, "
+                    "which is not declared" % (IMPORT_FILE, alias, ex["name"], ca))
+
+    # ---- the enrolment required action, DERIVED (see the header) -------------------
+    offers_passkey = any(
+        ex["kind"] == "authenticator" and ex["name"] == PASSKEY_AUTHENTICATOR
+        for spec in flows.values() for ex in spec["executions"])
+    if not offers_passkey:
+        raise SystemExit(
+            "FATAL: %s declares a webAuthnPolicyPasswordless* policy but no %r execution "
+            "in any flow. The policy decides what a passkey must BE; without that "
+            "execution nothing ever OFFERS one, so the policy is unreachable "
+            "configuration and this desired state would be meaningless"
+            % (IMPORT_FILE, PASSKEY_AUTHENTICATOR))
+
+    want = {
+        "realm": realm_name,
+        "webAuthnPolicyPasswordless": policy,
+        # enabled: passkey enrolment must be possible, since the flow offers passkeys.
+        # defaultAction FALSE: enrolment stays OPT-IN. True would force every user to
+        # enrol a passkey at next login — the platform-wide lock-out the realm import's
+        # "additive and optional" posture exists to avoid. Derived, not copied; see the
+        # header for why the import cannot declare this itself.
+        "requiredAction": {"alias": ENROL_ACTION, "enabled": True,
+                           "defaultAction": False},
+        "browserFlow": browser_flow,
+        "flows": flows,
+        "authenticatorConfig": configs,
+    }
+
+    body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
+    indented = "\n".join(("    " + ln) if ln else "" for ln in body.splitlines())
+    return (HEADER + BIND_BLOCK % (BIND_KEY, committed_bind(root))
+            + "  %s: |\n" % DATA_KEY + indented + "\n")
+
+
+HEADER = """---
+# =============================================================================
+# GENERATED FILE — DO NOT EDIT BY HAND, except the one bind-browser-flow line below.
+#
+#   source:      platform/keycloak/keycloak-realm-import.yaml (spec.realm)
+#   regenerate:  python3 ci/render-keycloak-passkey-policy.py --write
+#   gated by:    conformance/tests/l0-static/keycloak-passkey-reconciler
+#                (runs this renderer with --check on every merge)
+#
+# WHAT THIS IS: the DESIRED STATE for the `homelab` realm's PASSKEY configuration —
+# the WebAuthn passwordless policy, the enrolment required action, and the declared
+# browser authentication flow tree. platform/keycloak/passkey-reconciler.yaml mounts
+# it and (a) converges the policy and the required action onto it, (b) builds the
+# declared flow tree BLUE/GREEN under a new alias and verifies it by readback, never
+# writing the flow users are logging in through, and (c) BINDS that verified flow as
+# the realm's browserFlow ONLY when `bind-browser-flow` below is "true".
+#
+# EVERYTHING HERE IS GENERATED EXCEPT ONE LINE: `bind-browser-flow`. That is an
+# operator decision, carried through by the renderer from this committed file (see
+# ci/render-keycloak-passkey-policy.py). It defaults to "false".
+#
+# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
+# once Done=True, so a realm-level field or a flow edited in git never reaches the
+# realm. Measured on prod 2026-10-18, hours after the passkey declaration merged:
+# Done=True / Started=False, with the whole passkey block inert. The reconciler is
+# the level-triggered loop that closes that; this file is what it reconciles
+# TOWARDS.
+#
+# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of the RP ID, the
+# user-verification requirement and the flow tree would be a second source of truth,
+# free to disagree with the realm import — the same "two declarations that can
+# disagree" shape as the bug being closed. It is rendered from the import and the
+# render is gated, so the two cannot drift.
+#
+# STRINGS ARE VERBATIM: webAuthnPolicyPasswordlessRpId is the ${BASE_DOMAIN}
+# substitution token exactly as the realm import writes it. Both files are rendered
+# by the SAME Flux Kustomization (./platform-config), so the token resolves to the
+# same value in both and the domain is decided in exactly one place.
+#
+# `requiredAction` is the ONE entry that is DERIVED rather than copied: the realm
+# import deliberately declares no `requiredActions` block, because Keycloak 26.0.6
+# treats that as REPLACE and declaring one action would delete CONFIGURE_TOTP and
+# nine more (a hard lock-out of every platform-admin). It is derived instead from
+# the presence of a webauthn-authenticator-passwordless execution in the flow:
+# passkeys are offered, therefore enrolment must be enabled — and defaultAction is
+# false, so enrolment stays opt-in.
+#
+# A field NOT declared on the realm in the import is simply ABSENT here, and the
+# reconciler leaves it alone. Absence means UNMANAGED, never "force to a default".
+# =============================================================================
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: %s
+  namespace: %s
+  labels:
+    app.kubernetes.io/name: %s
+    app.kubernetes.io/part-of: keycloak
+data:
+""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME)
+
+BIND_BLOCK = """  # ---------------------------------------------------------------------------
+  # THE BIND SWITCH -- the ONLY hand-edited line in this file. Exactly "true" or
+  # "false", QUOTED.
+  #
+  #   "false"  the reconciler builds and verifies the candidate browser flow and
+  #            logs `WOULD BIND <alias>`, but NEVER changes the realm's browserFlow.
+  #            Whatever the realm is bound to -- including a rollback -- is left alone.
+  #   "true"   it binds the verified candidate in one atomic PUT, and keeps the
+  #            realm bound to it: a binding changed by hand is changed back.
+  #
+  # GO LIVE: test the candidate in a browser first (platform/keycloak/README.md,
+  # "Passkeys"), then a PR setting this to "true".
+  # ROLL BACK: a PR setting this to "false" FIRST (or the next run rebinds), then
+  # one write setting browserFlow back to the previous alias from the Job log.
+  # ---------------------------------------------------------------------------
+  %s: "%s"
+"""
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    g = ap.add_mutually_exclusive_group()
+    g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
+    g.add_argument("--check", action="store_true",
+                   help="exit 1 if the committed file has drifted from the source")
+    args = ap.parse_args()
+
+    want = render(ROOT)
+    target = ROOT / OUT_FILE
+
+    if args.write:
+        target.write_text(want, encoding="utf-8")
+        print("wrote %s" % OUT_FILE)
+        return 0
+
+    if args.check:
+        if not target.is_file():
+            print("CHECK FAIL: %s does not exist. Run: python3 "
+                  "ci/render-keycloak-passkey-policy.py --write" % OUT_FILE,
+                  file=sys.stderr)
+            return 1
+        have = target.read_text(encoding="utf-8")
+        if have == want:
+            print("passkey-desired-state.yaml is in sync with %s" % IMPORT_FILE)
+            return 0
+        print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
+              "realm on a passkey policy the realm import does not declare, and build "
+              "the login flow from a tree nobody authored.\n"
+              "Regenerate with: python3 ci/render-keycloak-passkey-policy.py --write\n"
+              % (OUT_FILE, IMPORT_FILE), file=sys.stderr)
+        for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
+                                         fromfile="committed " + OUT_FILE,
+                                         tofile="rendered from " + IMPORT_FILE):
+            sys.stderr.write(line)
+        return 1
+
+    sys.stdout.write(want)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

ci/tests/approval-override.test.sh

@@ -0,0 +1,232 @@
+#!/usr/bin/env bash
+# ci/tests/approval-override.test.sh — proves ci/approval-override.py selects the override it
+# claims to select, and REJECTS the ones it claims to reject.
+#
+# This parser decides whether a human review requirement gets waived. Every rule in it exists
+# because the obvious implementation ("does the body contain the directive?") is wrong in a way
+# that is invisible until abused: a quoted directive would re-arm an override, a bare directive
+# would waive review with no stated reason, and an override posted against commit A would go on
+# covering commit B forever. A guard is worth nothing until somebody has watched it refuse, which
+# is the same reasoning as ci/tests/decisions-log.test.sh one layer up.
+#
+# The parser is pure and offline BY DESIGN — authorisation (the live permission lookup) lives in
+# ci/approval-policy.sh, which cannot be tested without Gitea. That split is what makes this file
+# possible, and it is also why a green run here does NOT prove the stage authorises correctly.
+set -uo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+SUT="${CI_DIR}/approval-override.py"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+HEAD_TIME="2026-10-01T12:00:00Z"
+AFTER="2026-10-01T12:30:00Z"
+BEFORE="2026-10-01T11:30:00Z"
+
+# run <comments-json> -> stdout of the parser (stderr is diagnostic only)
+run() {
+  COMMENTS_JSON="$1" REVIEWS_JSON="" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# run_rev <reviews-json> — the same, but through the REVIEWS source.
+run_rev() {
+  COMMENTS_JSON="" REVIEWS_JSON="$1" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# comment <login> <created_at> <body>
+comment() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "created_at": sys.argv[2], "body": sys.argv[3]}]))
+' "$1" "$2" "$3"
+}
+
+expect_none() {
+  local label="$1" out
+  out="$(run "$2")"
+  if [ -z "$out" ]; then ok "$label"; else bad "$label (got: ${out})"; fi
+}
+
+expect_user() {
+  local label="$1" json="$2" want="$3" out got
+  out="$(run "$json")"
+  got="$(cut -f1 <<<"$out")"
+  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted ${want}, got '${got}')"; fi
+}
+
+expect_reason() {
+  local label="$1" json="$2" want="$3" out got
+  out="$(run "$json")"
+  got="$(cut -f4- <<<"$out")"
+  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted '${want}', got '${got}')"; fi
+}
+
+# check <label> <actual> <wanted>
+check() {
+  if [ "$2" = "$3" ]; then ok "$1"; else bad "$1 (wanted '${3}', got '${2}')"; fi
+}
+
+# review <login> <submitted_at> <state> <body>
+review() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "submitted_at": sys.argv[2],
+                   "state": sys.argv[3], "body": sys.argv[4]}]))
+' "$1" "$2" "$3" "$4"
+}
+
+echo "== ci/approval-override.py"
+
+# --- the happy path -------------------------------------------------------------------
+expect_user "a directive with a reason, posted after the head commit, is selected" \
+  "$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" jakman85
+
+expect_reason "the reason is returned verbatim" \
+  "$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" \
+  'sole admin, gate is green'
+
+# --- absence is a normal answer, not an error -----------------------------------------
+expect_none "no comments at all" '[]'
+expect_none "an ordinary comment" "$(comment jakman85 "$AFTER" 'looks good to me')"
+
+# --- a reason is MANDATORY ------------------------------------------------------------
+expect_none "a bare directive (no reason) is refused" \
+  "$(comment jakman85 "$AFTER" '/approve-override')"
+expect_none "a directive followed by only whitespace is refused" \
+  "$(comment jakman85 "$AFTER" '/approve-override    ')"
+
+# --- the directive must START a line --------------------------------------------------
+expect_none "the directive mid-sentence does not count" \
+  "$(comment jakman85 "$AFTER" 'you could just /approve-override this one')"
+expect_none "a QUOTED directive does not re-arm an override" \
+  "$(comment jakman85 "$AFTER" '> /approve-override quoting someone else')"
+expect_user "leading whitespace before the directive is fine" \
+  "$(comment jakman85 "$AFTER" '   /approve-override indented but mine')" jakman85
+expect_user "the directive on a later line of a longer comment counts" \
+  "$(comment jakman85 "$AFTER" 'some preamble
+/approve-override the real reason here')" jakman85
+
+# --- staleness: an override must post-date the commit it waives review on -------------
+expect_none "an override OLDER than the head commit is dropped as stale" \
+  "$(comment jakman85 "$BEFORE" '/approve-override stale, predates the push')"
+expect_none "an override exactly AT the head commit time is dropped (strictly newer)" \
+  "$(comment jakman85 "$HEAD_TIME" '/approve-override same instant')"
+
+# --- malformed input is never a free pass --------------------------------------------
+expect_none "an unreadable timestamp drops the candidate" \
+  "$(comment jakman85 'not-a-date' '/approve-override bad clock')"
+expect_none "a comment with no author is dropped" \
+  '[{"user": {}, "created_at": "2026-10-01T12:30:00Z", "body": "/approve-override no login"}]'
+expect_none "invalid JSON yields no override (and never a traceback into stdout)" \
+  'this is not json'
+expect_none "a JSON object instead of a list yields no override" \
+  '{"user": "nope"}'
+
+# --- latest wins ----------------------------------------------------------------------
+TWO='[
+  {"user": {"login": "early"},  "created_at": "2026-10-01T12:10:00Z", "body": "/approve-override first"},
+  {"user": {"login": "latest"}, "created_at": "2026-10-01T12:40:00Z", "body": "/approve-override second"}
+]'
+expect_user "the LATEST override supersedes an earlier one" "$TWO" latest
+expect_reason "and brings its own reason" "$TWO" second
+
+# --- offsets, not just Z --------------------------------------------------------------
+# Gitea emits RFC3339 WITH AN OFFSET (measured: 2026-10-01T23:09:08-05:00), not a trailing Z.
+# A parser that only handles 'Z' silently drops every real override, so this case is load-bearing.
+expect_user "an RFC3339 offset timestamp (-05:00), as Gitea actually emits, parses" \
+  '[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T08:30:00-05:00", "body": "/approve-override offset form"}]' \
+  jakman85
+expect_none "and an offset timestamp that is genuinely earlier is still stale" \
+  '[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T06:00:00-05:00", "body": "/approve-override offset, stale"}]'
+
+# --- the directive is configurable ----------------------------------------------------
+out="$(run "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')" "$HEAD_TIME" '/admin-waive')"
+if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
+  ok "a custom directive from the declaration is honoured"
+else
+  bad "a custom directive from the declaration is honoured (got '${out}')"
+fi
+expect_none "the default directive does not match a custom-directive comment" \
+  "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')"
+
+# --- no staleness anchor: do not invent one -------------------------------------------
+out="$(COMMENTS_JSON="$(comment jakman85 "$BEFORE" '/approve-override no anchor')" \
+       HEAD_TIME="" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
+  ok "with no HEAD_TIME the staleness test is skipped, not faked (caller blocks instead)"
+else
+  bad "with no HEAD_TIME the candidate should still be returned (got '${out}')"
+fi
+
+# --- the REVIEWS source ---------------------------------------------------------------
+# Load-bearing, not redundant. Measured on PR #38: SETTINGS_API_TOKEN is minted
+# [read:repository, read:admin], /issues/{n}/comments needs the ISSUE scope, so a well-formed
+# override in the comment box came back as an error object and was invisible. /pulls/{n}/reviews
+# IS readable with that token, so a review body is the route that works today.
+check "a review body carries an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  jakman85
+check "the source is reported as 'review'" \
+  "$(cut -f3 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  review
+check "the source is reported as 'comment' for a comment" \
+  "$(cut -f3 <<<"$(run "$(comment jakman85 "$AFTER" '/approve-override via a comment')")")" \
+  comment
+
+out="$(run_rev "$(review jakman85 "$BEFORE" COMMENT '/approve-override stale review')")"
+check "a stale review override is dropped too" "${out:-<none>}" "<none>"
+
+out="$(run_rev "$(review jakman85 "$AFTER" PENDING '/approve-override unsubmitted draft')")"
+check "a PENDING (unsubmitted) review is NOT an act of anything" "${out:-<none>}" "<none>"
+
+check "an APPROVED review carrying the directive still counts as an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" APPROVED '/approve-override belt and braces')")")" \
+  jakman85
+
+out="$(run_rev "$(review jakman85 "$AFTER" COMMENT 'ordinary review prose, no directive')")"
+check "an ordinary review body is not an override" "${out:-<none>}" "<none>"
+
+# --- an unreadable source degrades to 'no candidates', never to a pass ----------------
+# The real-world shape: Gitea answers with {"message": "token does not have ... scope"}.
+ERR_OBJ='{"message":"token does not have at least one of required scope(s): [read:issue]"}'
+out="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an error OBJECT from the comments API yields no override" "${out:-<none>}" "<none>"
+
+out="$(COMMENTS_JSON="$ERR_OBJ" \
+       REVIEWS_JSON="$(review jakman85 "$AFTER" COMMENT '/approve-override reviews still work')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an unreadable COMMENTS source does not suppress a valid REVIEW override" \
+  "$(cut -f1 <<<"$out")" jakman85
+
+# and it must SAY so, or a scope problem reads as 'you never posted one' (it did, on PR #38)
+errout="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+          DIRECTIVE=/approve-override python3 "$SUT" 2>&1 >/dev/null)"
+case "$errout" in
+  *"not a list"*read:issue*) ok "the diagnostic names the scope Gitea complained about" ;;
+  *) bad "the diagnostic should quote Gitea's message (got: ${errout})" ;;
+esac
+
+# --- latest wins ACROSS the two sources ----------------------------------------------
+out="$(COMMENTS_JSON="$(comment early "2026-10-01T12:10:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review later "2026-10-01T12:50:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (review newer)" "$(cut -f1 <<<"$out")" later
+check "and its source is labelled" "$(cut -f3 <<<"$out")" review
+
+out="$(COMMENTS_JSON="$(comment later "2026-10-01T12:50:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review early "2026-10-01T12:10:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (comment newer)" "$(cut -f1 <<<"$out")" later
+check "and that source is labelled too" "$(cut -f3 <<<"$out")" comment
+
+# --- both sources absent entirely -----------------------------------------------------
+out="$(COMMENTS_JSON="" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "no sources at all yields no override, and no crash" "${out:-<none>}" "<none>"
+
+printf '\n  %d passed, %d failed\n' "$PASS" "$FAIL"
+[ "$FAIL" -eq 0 ] || exit 1

ci/tests/decisions-log.test.sh

@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+# ci/tests/decisions-log.test.sh — proves ci/decisions-log.sh actually fails closed.
+#
+# The stage exists because appending to DECISIONS-LOG.md silently re-creates a merge conflict
+# for every other open branch. A guard against a silent regression is itself worth nothing
+# unless somebody has watched it reject the thing it claims to reject, so each case below
+# CONSTRUCTS the violation in a throwaway tree and asserts a non-zero exit — the same reasoning
+# as ci/tests/gate-decision.test.sh, one layer down.
+set -uo pipefail
+REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
+
+# A minimal well-formed tree: the frozen archive, its pin, and one valid entry.
+mk_tree() {
+  local d="$1"; rm -rf "$d"; mkdir -p "$d/ci" "$d/decisions"
+  cp "${REAL_CI_DIR}/decisions-log.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$d/ci/"
+  printf 'frozen history\n' > "$d/DECISIONS-LOG.md"
+  ( cd "$d" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 )
+  mk_entry "$d/decisions/2026-09-29-a-good-entry.md" 2026-09-29
+  # REPO_ROOT in ci/lib.sh is derived from git; give the tree a repo so it resolves here.
+  ( cd "$d" && git init -q . && git add -A && git -c user.email=t@t -c user.name=t commit -qm init )
+}
+
+mk_entry() {
+  cat > "$1" <<ENTRY
+---
+date: $2
+branch: chore/example
+title: An example entry
+---
+
+## What changed
+
+Enough prose here to clear the twenty-character body floor comfortably.
+
+## What I verified
+
+Enough prose here to clear the twenty-character body floor comfortably.
+
+## What I did NOT verify
+
+Enough prose here to clear the twenty-character body floor comfortably.
+ENTRY
+}
+
+# run <dir> -> sets RC and OUT
+run() { OUT="$( cd "$1" && bash ci/decisions-log.sh 2>&1 )"; RC=$?; }
+
+D="${TMP}/t"
+
+echo "== 1. a well-formed tree passes"
+mk_tree "$D"; run "$D"
+[ "$RC" -eq 0 ] && ok "clean tree -> exit 0" || bad "clean tree -> exit ${RC}: ${OUT}"
+
+echo
+echo "== 2. THE REGRESSION THIS STAGE EXISTS FOR: appending to the frozen archive"
+mk_tree "$D"; printf '\n## my new handoff\n' >> "$D/DECISIONS-LOG.md"; run "$D"
+[ "$RC" -ne 0 ] && ok "append to DECISIONS-LOG.md -> non-zero (${RC})" \
+                || bad "append to DECISIONS-LOG.md was ACCEPTED — the guard is inert"
+grep -qi 'frozen archive' <<<"$OUT" && ok "the message names the archive as frozen" \
+                                    || bad "the message does not say the archive is frozen"
+grep -q 'decisions/\$(date +%F)' <<<"$OUT" && ok "the message names the file to write instead" \
+                                           || bad "the message does not name the remedy"
+
+echo
+echo "== 3. a re-pin is a deliberate, accepted act"
+mk_tree "$D"; printf '\nredacted\n' >> "$D/DECISIONS-LOG.md"
+( cd "$D" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 ); run "$D"
+[ "$RC" -eq 0 ] && ok "archive changed AND re-pinned -> exit 0" || bad "re-pin rejected: ${OUT}"
+
+echo
+echo "== 4. a missing pin is unverifiable, not green"
+mk_tree "$D"; rm "$D/decisions/ARCHIVE.sha256"; run "$D"
+[ "$RC" -ne 0 ] && ok "no ARCHIVE.sha256 -> non-zero (${RC})" || bad "missing pin was accepted"
+
+echo
+echo "== 5. entry naming"
+for badname in 2026-9-29-x.md 26-09-29-x.md 2026-09-29-Upper.md 2026-09-29-.md 2026-02-30-x.md; do
+  mk_tree "$D"; mk_entry "$D/decisions/${badname}" 2026-09-29; run "$D"
+  [ "$RC" -ne 0 ] && ok "rejects decisions/${badname}" || bad "ACCEPTED decisions/${badname}"
+done
+mk_tree "$D"; mk_entry "$D/decisions/2026-10-01-fix-thing-2.md" 2026-10-01; run "$D"
+[ "$RC" -eq 0 ] && ok "accepts a digits-and-dashes slug" || bad "rejected a valid slug: ${OUT}"
+
+echo
+echo "== 6. frontmatter"
+mk_tree "$D"; sed -i '1d' "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
+[ "$RC" -ne 0 ] && ok "rejects an entry with no frontmatter" || bad "accepted a bare entry"
+
+for key in date branch title; do
+  mk_tree "$D"; sed -i "/^${key}:/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
+  [ "$RC" -ne 0 ] && ok "rejects frontmatter missing '${key}'" || bad "accepted missing '${key}'"
+done
+
+# The copied-template failure: the body was rewritten, the date never was.
+mk_tree "$D"; mk_entry "$D/decisions/2026-09-30-copied.md" 2026-09-29; run "$D"
+[ "$RC" -ne 0 ] && ok "rejects a frontmatter date that disagrees with the filename" \
+                || bad "accepted a date/filename disagreement"
+
+echo
+echo "== 7. the three required sections, and their bodies"
+for sec in 'What changed' 'What I verified' 'What I did NOT verify'; do
+  mk_tree "$D"; sed -i "/^## ${sec}\$/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
+  [ "$RC" -ne 0 ] && ok "rejects an entry missing '## ${sec}'" || bad "accepted missing '## ${sec}'"
+done
+
+# A heading with nothing under it is the way this floor gets defeated in practice.
+mk_tree "$D"
+python3 - "$D/decisions/2026-09-29-a-good-entry.md" <<'PY'
+import sys,re
+p=sys.argv[1]; s=open(p).read()
+s=re.sub(r'## What I did NOT verify\n\n.*', '## What I did NOT verify\n\nn/a\n', s, flags=re.S)
+open(p,'w').write(s)
+PY
+run "$D"
+[ "$RC" -ne 0 ] && ok "rejects a stub body under a required section" || bad "accepted an 'n/a' section"
+
+echo
+echo "== 8. README.md in decisions/ is not an entry and is not policed as one"
+mk_tree "$D"; printf 'not an entry\n' > "$D/decisions/README.md"; run "$D"
+[ "$RC" -eq 0 ] && ok "decisions/README.md ignored by the entry rules" || bad "README policed: ${OUT}"
+
+echo
+if [ "$FAIL" -eq 0 ]; then echo "decisions-log tests: PASS ($PASS assertions)"; exit 0
+else echo "decisions-log tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/tests/gate-decision.test.sh

@@ -0,0 +1,219 @@
+#!/usr/bin/env bash
+# ci/tests/gate-decision.test.sh — the merge gate's OWN decision table.
+#
+# WHY THIS SUITE EXISTS (2026-09-13). ci/gate.sh used to set BLOCK=1 for a FAILED stage but
+# only `warn` for a SKIPPED one, so it printed the words "— not green" and then concluded
+# `MERGE GATE: PASS` and exited 0. Observed live twice: `prom-rules` skipped for a missing
+# promtool, and `l0` skipped for a missing pinned kyverno CLI — both read by automation as
+# merge-approved. That is fail-open in the one component whose entire job is to fail closed,
+# and nothing tested it: every other ci/tests suite covers the INPUT (touched-requirements),
+# none covered the DECISION.
+#
+# WHAT IS UNDER TEST: the real ci/gate.sh and ci/lib.sh, copied into a throwaway CI_DIR beside
+# STUB stage scripts whose exit codes this suite dictates (STAGE_RC_<stage>) and a stub
+# touched-requirements.sh whose rc/reason it dictates (TR_RC / TR_REASON / TR_OUT). Stubbing the
+# stages is the point: this asserts how the gate REASONS about pass/fail/skip, independently of
+# what any stage happens to do on this machine — so it runs anywhere, with no tools, no cluster,
+# no network, in milliseconds.
+#
+# THE INVARIANT: `MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and
+# was GREEN. A stage that could not run is never a pass, and the exit code says so too.
+#
+# Run:  ci/tests/gate-decision.test.sh     (or `make -C ci test`)
+set -uo pipefail
+
+REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+TMP="$(mktemp -d)"
+trap 'rm -rf "$TMP"' EXIT
+CI="${TMP}/ci"; mkdir -p "$CI"
+REPO="${TMP}/repo"; mkdir -p "$REPO"
+
+cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
+
+# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
+# NOTE: this list must name EVERY stage gate.sh runs. It did not: `gitea-settings` was added to
+# gate.sh and never added here, so its script was absent from the stub CI dir, `run_stage` got
+# exit 127 from the missing file, and the gate recorded a FAILED stage in every single case —
+# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
+# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
+# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
+#
+# AND IT HAPPENED AGAIN, with `approval-policy` (fixed 2026-10-01). Same shape, same cost: the
+# stage was added to gate.sh by PR #17, never added here, so 15 of 50 assertions failed on `main`
+# — measured on 0c7e8dd before touching anything, to be sure it was not the change under test.
+# The note above was written about the previous instance and did not prevent the next one, which
+# says the comment is not the fix. THE FIX IS TO DERIVE THE LIST: it now comes from gate.sh's own
+# `run_stage` calls, so a stage added there can no longer be silently missing here.
+mapfile -t STAGES < <(sed -n 's/^run_stage[[:space:]]\+\([a-z0-9-]\+\).*/\1/p' "${REAL_CI_DIR}/gate.sh")
+[ "${#STAGES[@]}" -ge 8 ] || { echo "could not parse stage names out of gate.sh (got ${#STAGES[@]})" >&2; exit 1; }
+for stage in "${STAGES[@]}"; do
+  cat > "${CI}/${stage}.sh" <<'STUB'
+#!/usr/bin/env bash
+n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
+exit "${!v:-0}"
+STUB
+  chmod +x "${CI}/${stage}.sh"
+done
+
+# Stub decision input. TR_RC (default 0), TR_REASON (written to $TOUCHED_REASON_FILE like the
+# real script does), TR_OUT (the touched requirement list on stdout).
+cat > "${CI}/touched-requirements.sh" <<'STUB'
+#!/usr/bin/env bash
+if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "${TR_REASON:-}" > "$TOUCHED_REASON_FILE"; fi
+[ -n "${TR_OUT:-}" ] && printf '%s\n' "$TR_OUT"
+exit "${TR_RC:-0}"
+STUB
+chmod +x "${CI}/touched-requirements.sh"
+
+# run_gate <VAR=VAL>... — run the gate with a clean, explicit environment. Sets OUT and RC.
+# GATE_STRICT_SKIPS is pinned false unless a case says otherwise, so the suite never depends on
+# whether the machine running it happens to look like CI.
+OUT="" RC=0
+run_gate() {
+  local summary_file="${TMP}/summary.md"; : > "$summary_file"
+  OUT="$(env -i PATH="$PATH" HOME="$HOME" \
+        REPO_ROOT="$REPO" GITHUB_STEP_SUMMARY="$summary_file" \
+        GATE_STRICT_SKIPS=false TR_OUT="R-TEST-03" \
+        "$@" bash "${CI}/gate.sh" 2>&1)"
+  RC=$?
+  SUMMARY="$(cat "$summary_file")"
+}
+
+# assert_verdict <headline> <expected-rc> <label>
+assert_verdict() {
+  local want="$1" want_rc="$2" label="$3"
+  if ! grep -qF "MERGE GATE: ${want}" <<<"$OUT"; then
+    bad "${label}: headline should be '${want}' — got: $(grep -oE 'MERGE GATE: [A-Z ]+' <<<"$OUT" | tail -1)"
+  elif [ "$RC" -ne "$want_rc" ]; then
+    bad "${label}: headline '${want}' correct but exit code ${RC} != ${want_rc}"
+  else
+    ok "${label} -> ${want}, exit ${want_rc}"
+  fi
+}
+# The load-bearing negative: nothing may say PASS, and nothing may exit 0.
+assert_not_pass() {
+  local label="$1"
+  if grep -qF "MERGE GATE: PASS" <<<"$OUT"; then bad "${label}: printed 'MERGE GATE: PASS'"
+  elif [ "$RC" -eq 0 ]; then bad "${label}: exited 0"
+  else ok "${label}: not a PASS, non-zero exit (${RC})"; fi
+}
+
+echo "== 1. the happy path: every stage green"
+run_gate
+assert_verdict PASS 0 "all stages green + a touched subset"
+grep -qF "nothing was declared out of scope" <<<"$SUMMARY" \
+  && ok "PASS summary states nothing was declared out of scope" \
+  || bad "PASS summary does not state the absence of declared gaps"
+
+echo
+echo "== 2. a FAILED stage blocks (unchanged behaviour, locked down)"
+for s in lint l0 render-golden l1 conformance spec-immutability; do
+  run_gate "STAGE_RC_${s//-/_}=1"
+  assert_verdict BLOCKED 1 "${s} FAILED"
+done
+
+echo
+echo "== 3. THE REGRESSION: a SKIPPED stage is not a PASS and not exit 0"
+for s in lint l0 render-golden l1; do
+  run_gate "STAGE_RC_${s//-/_}=3"
+  assert_not_pass "${s} SKIPPED (local)"
+  assert_verdict UNDETERMINED 3 "${s} SKIPPED (local)"
+  grep -qF "$s" <<<"$SUMMARY" && ok "${s}: the step summary names the stage that did not run" \
+                              || bad "${s}: step summary does not name the skipped stage"
+done
+
+echo
+echo "== 4. under STRICT skips (CI declares it) a could-not-run skip BLOCKS, never merely 3"
+for s in lint l0 render-golden l1; do
+  run_gate GATE_STRICT_SKIPS=true "STAGE_RC_${s//-/_}=3"
+  assert_verdict BLOCKED 1 "${s} SKIPPED with GATE_STRICT_SKIPS=true"
+done
+# ...which is what makes gate.yml's `|| [ $? -eq 3 ]` safe: in CI, exit 3 cannot be a missing
+# tool or an unwired seam. Only the declared-environmental skip below can produce it.
+run_gate GATE_STRICT_SKIPS=true STAGE_RC_l0=3 STAGE_RC_lint=3
+assert_verdict BLOCKED 1 "several stages SKIPPED under strict skips"
+
+echo
+echo "== 5. spec-immutability: unverified is never green, but is not upgraded to a block by CI"
+run_gate STAGE_RC_spec_immutability=3
+assert_not_pass "spec-immutability SKIPPED"
+assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED (local)"
+run_gate GATE_STRICT_SKIPS=true STAGE_RC_spec_immutability=3
+assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED under strict skips (deliberate exception)"
+run_gate REQUIRE_SPEC_VERIFICATION=true STAGE_RC_spec_immutability=3
+assert_verdict BLOCKED 1 "spec-immutability SKIPPED + REQUIRE_SPEC_VERIFICATION=true"
+
+echo
+echo "== 6. conformance L2/L3: a DECLARED-absent capability may still pass — and is NAMED"
+run_gate STAGE_RC_conformance=3
+assert_verdict PASS 0 "conformance SKIPPED, no cluster declared"
+grep -qF "conformance-l2-l3" <<<"$OUT" \
+  && ok "the PASS headline names the declared L2/L3 gap" \
+  || bad "the PASS headline hides the declared L2/L3 gap"
+grep -qF "conformance-l2-l3" <<<"$SUMMARY" \
+  && ok "the step summary names the declared L2/L3 gap" \
+  || bad "the step summary hides the declared L2/L3 gap"
+run_gate CLUSTER_AVAILABLE=true STAGE_RC_conformance=3
+assert_verdict BLOCKED 1 "conformance SKIPPED despite CLUSTER_AVAILABLE=true"
+run_gate REQUIRE_CLUSTER_TESTS=true STAGE_RC_conformance=3
+assert_verdict BLOCKED 1 "conformance SKIPPED + REQUIRE_CLUSTER_TESTS=true with a touched subset"
+# A misspelled switch must leave the guard OFF (truthy()), never half-on.
+run_gate CLUSTER_AVAILABLE=ture STAGE_RC_conformance=3
+assert_verdict PASS 0 "CLUSTER_AVAILABLE=ture is NOT 'available' (no cluster was declared)"
+run_gate CLUSTER_AVAILABLE=TRUE STAGE_RC_conformance=3
+assert_verdict BLOCKED 1 "CLUSTER_AVAILABLE=TRUE IS a declaration (skip then blocks)"
+
+echo
+echo "== 7. the undetermined touched subset: unchanged (2026-08-31 false-green paths)"
+run_gate TR_RC=3 TR_REASON=on-base TR_OUT=
+assert_verdict "NOT APPLICABLE" 0 "subset undeterminable, reason on-base, all stages green"
+run_gate TR_RC=3 TR_REASON=no-base TR_OUT=
+assert_verdict BLOCKED 1 "subset undeterminable, reason no-base"
+run_gate TR_RC=3 TR_REASON=diff-failed TR_OUT=
+assert_verdict BLOCKED 1 "subset undeterminable, reason diff-failed"
+# Precedence: a stage that did not run outranks "nothing to gate" — the stronger, non-zero
+# headline must win, or a push-to-main run could hide a missing tool behind NOT APPLICABLE.
+run_gate TR_RC=3 TR_REASON=on-base TR_OUT= STAGE_RC_l1=3
+assert_not_pass "on-base AND l1 SKIPPED"
+assert_verdict UNDETERMINED 3 "on-base AND l1 SKIPPED (UNDETERMINED outranks NOT APPLICABLE)"
+
+echo
+echo "== 8. an UNUSABLE subset blocks (carried over from 47068ee): any other non-zero exit"
+for rc in 1 2 4; do
+  run_gate "TR_RC=${rc}" TR_REASON=unclassified-path TR_OUT=
+  assert_verdict BLOCKED 1 "touched-requirements.sh exited ${rc} (subset unusable, not empty)"
+done
+run_gate TR_RC=1 TR_REASON=unclassified-path TR_OUT=
+grep -qF "UNUSABLE" <<<"$SUMMARY" && ok "the step summary calls the subset UNUSABLE" \
+                                  || bad "the step summary does not report an unusable subset"
+
+echo
+echo "== 9. combinations: a FAILURE outranks a non-run, and the headline is the worst news"
+run_gate STAGE_RC_l0=1 STAGE_RC_l1=3
+assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
+run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
+assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
+
+echo
+echo "== 10. decisions-log: the handoff log's storage format has no non-blocking outcome"
+# A FAIL here means the frozen archive was appended to (re-creating, for every OTHER open
+# branch, the tail-hunk conflict that made 4 of 5 PRs unmergeable on 2026-09-29) or an entry is
+# malformed. A SKIP is impossible by construction — the stage needs no tool and no cluster — so
+# if one is ever reported it is a defect in the stage, and the gate must not soften it into
+# UNDETERMINED. Both outcomes block, and neither is upgradeable by an env knob: there is
+# deliberately no REQUIRE_DECISIONS_LOG, because a guard with an off switch is not a guard.
+run_gate STAGE_RC_decisions_log=1
+assert_verdict BLOCKED 1 "decisions-log FAILED (archive modified / entry malformed)"
+run_gate STAGE_RC_decisions_log=3
+assert_verdict BLOCKED 1 "decisions-log SKIPPED (impossible by construction -> defect, blocks)"
+run_gate STAGE_RC_decisions_log=1 STAGE_RC_l1=3
+assert_verdict BLOCKED 1 "decisions-log FAILED outranks an l1 non-run"
+
+echo
+if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
+else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/tests/touched-requirements-test-attribution.test.sh

@@ -0,0 +1,164 @@
+#!/usr/bin/env bash
+# ci/tests/touched-requirements-test-attribution.test.sh — regression tests for DETECTION
+# RULE 3 in ci/touched-requirements.sh: a change anywhere under `conformance/tests/<test>/`
+# attributes THAT TEST'S OWN `conformance.home.k3s/R-*` labels.
+#
+# THE GAP THESE LOCK DOWN (2026-09-12). The coarse path map keys `conformance/*` to
+# R-TEST-01..04 — the requirements about the HARNESS — and says nothing about what a given
+# test PROVES. So weakening `conformance/tests/.../check.sh` attributed R-TEST-01..04 and
+# nothing else: the merge gate ran the harness's self-tests and NOT the weakened test's own
+# requirement subset. Weakening (or deleting) a test is the one change that most needs the
+# requirement it certifies to be re-verified, and it was the change least likely to select it.
+# The content scan cannot close this: it reads only ADDED/REMOVED lines, so it sees a change
+# to the LABEL BLOCK but not one to the test BODY — exactly the case that matters.
+#
+# Also covers the two path-map entries added in the same change (infra/* -> R-ENV-01, and
+# R-TEST-06 on .gitea/*|ci/*), which is how the two L0 checks that are DELIBERATELY not
+# whole-repo-scoped get selected by the paths that can break them.
+#
+# Same harness as the sibling suites: the real script, pointed at a THROWAWAY git repo via
+# the REPO_ROOT override. No network, no cluster, the working repo is never touched.
+#
+# Run:  ci/tests/touched-requirements-test-attribution.test.sh   (or `make -C ci test`)
+set -euo pipefail
+
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+SCRIPT="${CI_DIR}/touched-requirements.sh"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
+
+assert_present() {
+  if grep -qxF "$1" <<<"$2"; then ok "$3"
+  else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
+}
+assert_absent() {
+  if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))"
+  else ok "$3"; fi
+}
+
+TMP="$(mktemp -d)"
+trap 'rm -rf "$TMP"' EXIT
+
+git -C "$TMP" init -q -b main
+git -C "$TMP" config user.email [email protected]
+git -C "$TMP" config user.name t
+git -C "$TMP" config commit.gpgsign false
+
+# --- Baseline: two conformance tests, each declaring its own requirement labels ----------
+# TEST A: a flat l0-static test. TEST B: a test with a NESTED fixture dir, to prove the
+# walk-up from an arbitrary changed file to the enclosing test dir.
+mkdir -p "$TMP/conformance/tests/l0-static/alpha" \
+         "$TMP/conformance/tests/l2-behavioral/beta/fixtures" \
+         "$TMP/conformance/lib" "$TMP/infra" "$TMP/ci"
+
+cat > "$TMP/conformance/tests/l0-static/alpha/chainsaw-test.yaml" <<'YAML'
+# HEADER PROSE. This test used to also cover conformance.home.k3s/R-STALE-09: "true",
+# which is a REFERENCE in a comment, not a declaration, and must NOT be attributed.
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: alpha
+  labels:
+    conformance.home.k3s/R-ALPHA-01: "true"
+    conformance.home.k3s/R-ALPHA-02: "true"
+  annotations:
+    conformance.home.k3s/layer: "L0"
+spec: {}
+YAML
+# The check BODY deliberately contains NO R-ID text, so anything the scan finds for a
+# body-only edit came from rule 3 and not from the content scan.
+cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
+#!/usr/bin/env bash
+echo "assert something"
+SH
+
+cat > "$TMP/conformance/tests/l2-behavioral/beta/chainsaw-test.yaml" <<'YAML'
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: beta
+  labels:
+    conformance.home.k3s/R-BETA-07: "true"
+spec: {}
+YAML
+echo "seed: 1" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
+
+echo "harness helper" > "$TMP/conformance/lib/helper.sh"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
+
+# ========================================================================================
+echo "== case 1: a BODY-ONLY edit to a test attributes that test's own labels"
+git -C "$TMP" checkout -q -b body-edit
+cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
+#!/usr/bin/env bash
+echo "assert something weaker"
+SH
+git -C "$TMP" add -A && git -C "$TMP" commit -qm weaken
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-ALPHA-01" "$OUT" "body-only test edit attributes the test's own R-ALPHA-01"
+assert_present "R-ALPHA-02" "$OUT" "body-only test edit attributes the test's own R-ALPHA-02"
+assert_present "R-TEST-03"  "$OUT" "the conformance/* path map still applies (R-TEST-03)"
+assert_absent  "R-STALE-09" "$OUT" "an R-ID cited only in the test's HEADER COMMENT is NOT attributed"
+assert_absent  "R-BETA-07"  "$OUT" "an UNRELATED test's labels are NOT attributed"
+
+# ========================================================================================
+echo "== case 2: an edit to a NESTED file walks up to the enclosing test"
+git -C "$TMP" checkout -q -b nested-edit main
+echo "seed: 2" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm fixture
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-BETA-07" "$OUT" "nested fixture edit attributes the enclosing test's R-BETA-07"
+assert_absent  "R-ALPHA-01" "$OUT" "nested fixture edit does NOT attribute a sibling test's labels"
+
+# ========================================================================================
+echo "== case 3: DELETING a test still attributes what it used to certify"
+# Removing the only verifier of a requirement must not be the cheapest way to make the gate
+# stop checking it — so the labels are read from the BASE when HEAD no longer has them.
+git -C "$TMP" checkout -q -b delete-test main
+git -C "$TMP" rm -rq "$TMP/conformance/tests/l0-static/alpha"
+git -C "$TMP" commit -qm "delete the alpha test"
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-ALPHA-01" "$OUT" "deleting a test STILL attributes its R-ALPHA-01 (read from base)"
+assert_present "R-ALPHA-02" "$OUT" "deleting a test STILL attributes its R-ALPHA-02 (read from base)"
+
+# ========================================================================================
+echo "== case 4: a change OUTSIDE conformance/tests/ gains no test labels"
+git -C "$TMP" checkout -q -b lib-edit main
+echo "harness helper v2" > "$TMP/conformance/lib/helper.sh"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm lib
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-TEST-02"  "$OUT" "a conformance/lib edit still attributes the harness requirements"
+assert_absent  "R-ALPHA-01" "$OUT" "a conformance/lib edit attributes NO individual test's labels"
+assert_absent  "R-BETA-07"  "$OUT" "a conformance/lib edit attributes NO individual test's labels (B)"
+
+# ========================================================================================
+echo "== case 5: path-map entries for the two DELIBERATELY path-scoped L0 checks"
+# infra/* -> R-ENV-01 (dev-nodes-provisioned-as-code reads nothing but infra/).
+git -C "$TMP" checkout -q -b infra-edit main
+printf 'resource "x" "y" {}\n' > "$TMP/infra/main.tf"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm infra
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-ENV-01" "$OUT" "an infra/ edit attributes R-ENV-01 (was UNMAPPED: attributed nothing)"
+
+# ci/* -> ... R-TEST-06 (spec-immutability-enforced reads nothing but ci/ + the spec repo).
+git -C "$TMP" checkout -q -b ci-edit main
+printf 'echo hello\n' > "$TMP/ci/thing.sh"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm ci
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-TEST-06"  "$OUT" "a ci/ edit attributes R-TEST-06 (the spec-immutability test's own ID)"
+assert_present "R-GITOPS-01" "$OUT" "a ci/ edit still attributes R-GITOPS-01"
+
+echo
+if [ "$FAIL" -eq 0 ]; then
+  echo "touched-requirements test-attribution tests: PASS (${PASS} assertions)"; exit 0
+fi
+echo "touched-requirements test-attribution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
+exit 1

ci/tests/touched-requirements.test.sh

@@ -93,6 +93,141 @@ printf 'touched set (removal):\n%s\n---\n' "$OUT2"
 assert_absent  "R-DELDOC-01"  "$OUT2" "REMOVED whole-line comment R-ID is NOT attributed"
 assert_present "R-DELREAL-01"  "$OUT2" "REMOVED real-line R-ID IS attributed"
 
+# ---------------------------------------------------------------------------------------
+# PATH MAP — bootstrap/* (added 2026-09-13)
+#
+# WHY THIS IS LOCKED DOWN. path_requirements() keys `bootstrap/*` to R-STATE-03 +
+# R-GITOPS-01, and until 2026-09-13 the comment justifying it described
+# `bootstrap/standup.sh` as a file that carries the placeholder guards. That file has never
+# been on `main` (it landed only on unmerged prod-standup branches), so the comment was
+# stale and the entry looked like dead weight someone would tidy away. It is not dead: it
+# is what SELECTS conformance/tests/l0-static/placeholder-guard-coupling for the change
+# that eventually lands a placeholder guard under bootstrap/ — and that test discovers its
+# guard by content rather than by path, so it activates on exactly that change. Removing
+# the entry would let the change introducing a guard be gated by nothing, which is the
+# 2026-09-05 class. These assertions make the entry's deletion a test failure rather than a
+# silent loss of selection.
+#
+# NOTE the R-ID content scan must not be what produces these: `notes/` is outside the path
+# map precisely so the other cases isolate the scan, and here the file content cites no
+# R-ID at all, so any R-ID in the output came from the PATH.
+git -C "$TMP" checkout -q -b feature3 main
+mkdir -p "$TMP/bootstrap/flux"
+cat > "$TMP/bootstrap/flux/bootstrap.sh" <<'SH'
+#!/usr/bin/env bash
+# a bootstrap-path change that cites no requirement id of its own
+echo "flux bootstrap"
+SH
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch bootstrap/"
+OUT3="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (bootstrap/ path map):\n%s\n---\n' "$OUT3"
+assert_present "R-STATE-03"   "$OUT3" "bootstrap/* attributes R-STATE-03 (selects placeholder-guard-coupling)"
+assert_present "R-GITOPS-01"  "$OUT3" "bootstrap/* attributes R-GITOPS-01 (it IS the GitOps bootstrap path)"
+
+# And the neighbouring entry the same test leans on for its other half: clusters/* must
+# attribute R-ENV-02, which is what selects placeholder-guard-coupling's HALF A (the
+# authored-values sweep over clusters/<env> effective cluster-vars).
+git -C "$TMP" checkout -q -b feature4 main
+mkdir -p "$TMP/clusters/base"
+printf 'kind: ConfigMap\nmetadata:\n  name: cluster-vars\n' > "$TMP/clusters/base/cluster-vars.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch clusters/"
+OUT4="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
+assert_present "R-ENV-02"     "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
+assert_present "R-GITOPS-01"  "$OUT4" "clusters/* attributes R-GITOPS-01"
+
+# ---------------------------------------------------------------------------------------
+# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
+#
+# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
+# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
+# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
+# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
+# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
+# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
+#
+# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
+# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
+#
+# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
+# from the PATH and not from the content scan.
+
+# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
+# under test here, not just the presence of both arms.
+git -C "$TMP" checkout -q -b feature-apps-vms main
+mkdir -p "$TMP/apps/vms/demo"
+printf 'kind: VirtualMachine\nmetadata:\n  name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
+OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
+assert_present "R-PLAT-03"  "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
+assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
+assert_absent  "R-PLAT-02"  "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
+
+# A plain ServiceDefinition under apps/ gets the container-workload set.
+git -C "$TMP" checkout -q -b feature-apps main
+mkdir -p "$TMP/apps/wave-1"
+printf 'kind: ServiceDefinition\nmetadata:\n  name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
+OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
+assert_present "R-PLAT-02"  "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
+assert_present "R-NET-01"   "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
+assert_present "R-NET-06"   "$OUT_APPS" "apps/* attributes the egress-class requirements"
+assert_present "R-IDP-02"   "$OUT_APPS" "apps/* attributes the auth-mode requirements"
+
+# homepage / headlamp ARE the command-centre surface.
+git -C "$TMP" checkout -q -b feature-c2 main
+mkdir -p "$TMP/platform/headlamp"
+printf 'kind: Deployment\nmetadata:\n  name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
+OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
+assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
+assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
+
+# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
+git -C "$TMP" checkout -q -b feature-assembly main
+mkdir -p "$TMP/platform"
+printf 'resources:\n  - cilium\n  - longhorn\n' > "$TMP/platform/kustomization.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
+OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
+assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
+
+# ---------------------------------------------------------------------------------------
+# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
+#
+# The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match `R-C2-01` or
+# `R-C2-02` — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. Those are
+# the only two registry IDs whose workstream segment carries a digit, and the scan was blind
+# to BOTH, in the fail-OPEN direction: a change self-declaring a C2 requirement contributed
+# it to nobody, so the C2 conformance tests were filtered out of the merge gate. Found while
+# wiring conformance/tests/l0-static/c2-root-surface, which closes those two coverage gaps.
+#
+# `notes/` is outside path_requirements() on purpose, so the IDs below can only have come
+# from the CONTENT SCAN. A neighbouring all-letter ID is asserted too, so a regex that
+# somehow matched only the digit form would still fail here.
+git -C "$TMP" checkout -q -b feature5 main
+cat > "$TMP/notes/c2.yaml" <<'YAML'
+kind: Test
+metadata:
+  labels:
+    conformance.home.k3s/R-C2-01: "true"
+    conformance.home.k3s/R-C2-02: "true"
+    conformance.home.k3s/R-OBS-04: "true"
+YAML
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "declare R-C2-01 / R-C2-02"
+OUT5="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (digit in the workstream segment):\n%s\n---\n' "$OUT5"
+assert_present "R-C2-01"  "$OUT5" "an R-ID with a DIGIT in the workstream segment IS attributed (R-C2-01)"
+assert_present "R-C2-02"  "$OUT5" "the second such ID is attributed too (R-C2-02)"
+assert_present "R-OBS-04" "$OUT5" "an all-letter workstream segment still works (control)"
+# The pattern must not become so loose that a non-ID token is read as a requirement: a bare
+# numeric workstream is not a registry shape, and attributing one would WIDEN the gate on
+# noise (over-broad selection, never a false green, but noisy and misleading).
+assert_absent  "R-1-01"   "$OUT5" "a purely numeric workstream segment is not invented from thin air"
+
 echo
 if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
 else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/touched-requirements.sh

@@ -21,8 +21,12 @@
 #      the R-IDs that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011).
 #      Untouched lines and prose are NOT scanned; see changed_diff() for why.
 #   2. A coarse path->requirement fallback for changes that don't self-declare IDs.
+#   3. For a change anywhere under `conformance/tests/<test>/`: THAT TEST'S OWN requirement
+#      labels, read from its chainsaw-test.yaml. See test_requirements_for_path().
 # The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for
-# untagged changes so nothing silently escapes the gate.
+# untagged changes so nothing silently escapes the gate. (3) is the same principle applied
+# to the tests themselves: editing or weakening a test must be gated by the requirements
+# that test CERTIFIES, not merely by R-TEST-0x.
 set -euo pipefail
 CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # shellcheck disable=SC1091
@@ -173,6 +177,23 @@ path_requirements() {
   case "$1" in
     catalog/*)                 echo "R-GITOPS-02" ;;
     schemas/*)                 echo "R-GITOPS-02" ;;
+    # apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
+    # manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
+    # adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
+    # checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
+    # class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
+    # violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
+    # catch it at apply time" is post-merge, and the gate exists to catch it before.
+    #
+    # ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
+    # apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
+    # subject to default-deny (R-NET-01).
+    apps/vms/*)                echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
+    # Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
+    # workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
+    # (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
+    # default-deny (R-NET-01).
+    apps/*)                    echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
     platform/kyverno/*)        echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
     platform/cilium/*)         echo "R-NET-01 R-NET-02 R-NET-05" ;;
     platform/longhorn/*)       echo "R-STATE-01 R-STATE-02" ;;
@@ -180,14 +201,161 @@ path_requirements() {
     platform/externaldns/*)    echo "R-DNS-02" ;;
     platform/velero/*)         echo "R-STATE-03" ;;
     platform/monitoring/*)     echo "R-OBS-01 R-OBS-02" ;;
+    # platform/cnpg was UNMAPPED until 2026-09-06. It is the PostgreSQL DR path for keycloak-db /
+    # nextcloud-db / semaphore-db (barmanObjectStore WAL archiving + ScheduledBackup), so a change
+    # there is an R-IDP-05 / R-STATE-03 change; it now also carries the platform-health alert rules
+    # for that path (R-OBS-04). Without this line an edit under platform/cnpg/ contributed ZERO
+    # requirements unless it happened to retype an R-ID on a non-comment line, and escaped the gate.
+    platform/cnpg/*)           echo "R-IDP-05 R-STATE-03 R-OBS-04" ;;
+    # vpn-egress was UNMAPPED until 2026-09-06. platform/vpn-egress/egress-steering.yaml IS the
+    # R-NET-04 datapath (the rule that selects between the two egress paths) and the R-NET-05/06/07
+    # steering policy — it even self-declares those IDs in its `homelab.keithjackson.dev/requirements`
+    # annotation. But the content scan only reads ADDED/REMOVED lines, so any edit that does not
+    # happen to retype an R-ID (a selector change, an excludedCIDRs change, a comment) contributed
+    # ZERO requirements and escaped the gate entirely. Mapped explicitly; the annotation still wins
+    # when a diff does carry the tags.
+    platform/vpn-egress/*)     echo "R-NET-04 R-NET-05 R-NET-06 R-NET-07" ;;
+    # Keycloak is the whole SSO surface: the realm import (declarative identity config),
+    # the SOPS client secrets, and the reconciler that heals them. Added 2026-09-06 after
+    # the client-secret incident, in which editing platform/keycloak/ attributed NO
+    # requirement, so the L2 keycloak tests were filtered out of the merge gate's touched
+    # subset — the change that could break every OIDC login was gated by nothing.
+    # R-IDP-05 is in scope because platform/keycloak/ also carries the CNPG DB Cluster and
+    # its off-cluster backup credential, not only the realm.
+    platform/keycloak/*)       echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
+    platform/oauth2-proxy/*)   echo "R-IDP-02 R-IDP-06" ;;
     platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
+    # The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
+    # rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
+    # platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
+    # itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
+    # docs/12-command-center-ui.md), so editing them must select the C2 tests.
+    platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
+    # KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
+    platform/kubevirt/*)       echo "R-PLAT-03 R-OBS-01" ;;
+    # The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
+    # TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
+    platform/gateway/*)        echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
+    # CoreDNS is cluster DNS resolution (R-DNS-01).
+    platform/coredns/*)        echo "R-DNS-01" ;;
+    # crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
+    # its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
+    platform/crafty/*)         echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
+    # Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
+    platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
+    # runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
+    # resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
+    # is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
+    # runbook change is an observability change.
+    runbooks/*)                echo "R-OBS-04" ;;
+    # platform-config/ is the env-specific configuration Flux substitutes into and applies.
+    platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
+    # platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
+    # Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
+    # file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
+    # change. That is the most consequential single-line edit in the repo and it attributed nothing.
+    platform/kustomization.yaml) echo "R-GITOPS-01" ;;
+    # .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
+    # here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
+    .sops.yaml)                echo "R-SEC-01 R-GITOPS-01" ;;
     conformance/*)             echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
+    # infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
+    # nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
+    # infra/, yet an infra/ edit attributed ZERO requirements, so the one check that asserts
+    # the nodes are provisioned as code was filtered out of every change that could break it.
+    # That test's invariant IS path-scoped (it is deliberately NOT whole-repo-scoped; see its
+    # chainsaw-test.yaml), and the correct fix for a path-scoped invariant is a path-map entry.
+    infra/*)                   echo "R-ENV-01" ;;
     clusters/*)                echo "R-GITOPS-01 R-ENV-02" ;;
-    .gitea/*|ci/*)             echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
+    # bootstrap/ was UNMAPPED until 2026-09-06. The entry STAYS, but its original rationale
+    # was written against a file that is not here, and that is corrected on 2026-09-13:
+    # it claimed "bootstrap/standup.sh carries the placeholder guards that decide whether
+    # `velero` may be enabled against a real backup endpoint". bootstrap/standup.sh HAS NEVER
+    # BEEN ON `main` — it was not deleted (no deletion commit exists anywhere), it landed only
+    # on unmerged prod-standup branches. `git ls-tree main bootstrap/` lists README.md, flux/,
+    # gitea/, sops/. So R-STATE-03 here is ANTICIPATORY, not descriptive.
+    #
+    # WHY IT IS STILL RIGHT. conformance/tests/l0-static/placeholder-guard-coupling DISCOVERS
+    # its guard (any tracked *.sh defining `<name>_is_placeholder()`) instead of hardcoding a
+    # path, so the moment a standup guard lands under bootstrap/ that test's coupling half
+    # activates — and this entry is what selects the test for the change that lands it.
+    # Deleting the entry would mean the change introducing a placeholder guard is gated by
+    # nothing, which is the 2026-09-05 class (a prefix glob that stopped matching when the
+    # value it guards changed, in a different file) left to reintroduce itself. R-GITOPS-01 is
+    # descriptive today: bootstrap/flux/, gitea/ and sops/ ARE the GitOps bootstrap path.
+    # Locked down by ci/tests/touched-requirements.test.sh's path-map section.
+    bootstrap/*)               echo "R-STATE-03 R-GITOPS-01" ;;
+    # R-TEST-06 added 2026-09-12. The comment in changed_diff()'s rule 3 above records why it
+    # was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE and carried by no
+    # Chainsaw test, so attributing it made the coverage `--require` check false-block. It is now
+    # carried by conformance/tests/l0-static/spec-immutability-enforced — whose entire input
+    # surface is ci/ — so a ci/ change must attribute it, or the test that proves the
+    # spec-immutability mechanism is still load-bearing runs on nothing that can break it.
+    .gitea/*|ci/*)             echo "R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06" ;;
     *) : ;;
   esac
 }
 
+# ---------------------------------------------------------------------------------
+# A CHANGED TEST ATTRIBUTES THE REQUIREMENTS IT CERTIFIES (detection rule 3).
+#
+# THE GAP THIS CLOSES (2026-09-12). The path map keys `conformance/*` to
+# R-TEST-01..04 — the requirements about the HARNESS. It says nothing about what a given
+# test PROVES. So editing `conformance/tests/l2-behavioral/tenant-isolation/check.sh` to
+# weaken or delete an assertion attributed R-TEST-01..04 and nothing else: the merge gate
+# then ran the harness's self-tests and NOT the weakened test's own requirement subset.
+# Weakening a test is the one change that most needs the requirement it certifies to be
+# re-verified, and it was the change least likely to select it.
+#
+# The `conformance.home.k3s/R-*: "true"` labels on a test ARE the authoritative declaration of
+# what it verifies (lib/coverage.py reads the same labels); this reads them directly. Note the
+# content scan in main() only sees ADDED/REMOVED lines, so it catches a change to the LABEL
+# BLOCK but not a change to the test BODY — which is exactly the case that matters here.
+#
+# WHY IT READS HEAD *OR* THE BASE: a change that DELETES a test, or deletes its
+# chainsaw-test.yaml, must still attribute what that test used to certify — otherwise removing
+# the only verifier of a requirement is the cheapest way to make the gate stop checking it.
+# HEAD first (the post-change truth), base as the fallback (the deleted case). Fail-closed:
+# this function can only ADD requirements, never remove one.
+: "${CONF_TEST_LABEL_PREFIX:=conformance.home.k3s}"   # == CONF_LABEL_PREFIX, conformance/lib/common.sh
+
+file_text_head_or_base() {
+  local path="$1"
+  git -C "$REPO_ROOT" show "HEAD:${path}" 2>/dev/null && return 0
+  git -C "$REPO_ROOT" show "${BASE}:${path}" 2>/dev/null && return 0
+  return 0
+}
+
+# test_requirements_for_path <changed-path> -> space-separated R-IDs (possibly empty).
+# Walks UP from the changed file to the nearest ancestor directory carrying a
+# chainsaw-test.yaml, so it works for any file in a test dir (check.sh, mutations.py,
+# fixtures, the chainsaw-test.yaml itself) and for nested test layouts.
+test_requirements_for_path() {
+  local f="$1" dir yaml text
+  case "$f" in conformance/tests/*/*) ;; *) return 0 ;; esac
+  dir="${f%/*}"
+  while :; do
+    case "$dir" in
+      conformance/tests|conformance|.|"") return 0 ;;
+      */*) ;;
+      *) return 0 ;;
+    esac
+    yaml="${dir}/chainsaw-test.yaml"
+    text="$(file_text_head_or_base "$yaml")"
+    if [ -n "$text" ]; then
+      # Whole-line `#` comments are dropped first — same "reference, not declaration" rule the
+      # content scan applies (changed_diff() rule 3): a label shown inside a header example is
+      # documentation. Only a real YAML mapping line counts.
+      printf '%s\n' "$text" \
+        | grep -vE '^[[:space:]]*#' \
+        | sed -nE "s#^[[:space:]]*${CONF_TEST_LABEL_PREFIX}/(R-[A-Z0-9]+-[0-9]+):[[:space:]]*[\"']?true.*#\1#p" \
+        | tr '\n' ' '
+      return 0
+    fi
+    dir="${dir%/*}"
+  done
+}
+
 main() {
   local files reqs="" rc=0 base_sha head_sha local_main merge_base
 
@@ -221,9 +389,27 @@ main() {
   while IFS= read -r f; do
     [ -n "$f" ] || continue
     reqs+=" $(path_requirements "$f")"
+    # Detection rule 3: a change under conformance/tests/<test>/ attributes THAT TEST'S own
+    # requirement labels, so weakening a test is gated by what the test certifies.
+    reqs+=" $(test_requirements_for_path "$f")"
   done <<< "$files"
 
   # Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
+  #
+  # THE ID PATTERN IS `R-[A-Z0-9]+-[0-9]+`, NOT `R-[A-Z]+-[0-9]+`. The WORKSTREAM segment may
+  # contain a DIGIT: `R-C2-01` and `R-C2-02` (command & control UI) are the only two such IDs
+  # in conformance/requirements/registry.yaml today, and the stricter pattern silently failed
+  # to match BOTH of them — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is.
+  # Every other consumer in the repo already allows the digit: test_requirements_for_path()
+  # twenty lines above uses `R-[A-Z0-9]+-[0-9]+`, as do conformance/lib/coverage.py,
+  # conformance/lib/status.py and conformance/lib/run_l1_offline.py. This grep was the lone
+  # dissenter, and it dissented by UNDER-attributing — the fail-OPEN direction. A change that
+  # retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02 from a
+  # path the map keys elsewhere, contributed those IDs to NOBODY, so the C2 tests were
+  # filtered out of the merge gate. Same defect class as the map's `*) : ;;` arm: a miss that
+  # reads as "nothing touched". path_requirements() has no arm for platform/homepage/* or
+  # platform/headlamp/* either, so before this fix the ONLY route to R-C2-01/02 was rule 3
+  # (editing the C2 test itself). Regression-tested in ci/tests/touched-requirements.test.sh.
   # `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
   # `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so
   #   what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
@@ -237,7 +423,7 @@ main() {
     | grep -vE '^(\+\+\+|---)' \
     | sed -E 's/^[+-]//' \
     | grep -vE '^[[:space:]]*#' \
-    | grep -hoE 'R-[A-Z]+-[0-9]+' \
+    | grep -hoE 'R-[A-Z0-9]+-[0-9]+' \
     | tr '\n' ' ' || true)"
 
   # Normalize: split, unique, sort. Emit to stdout.

ci/versions.env

@@ -21,6 +21,15 @@ YAMLLINT_VERSION="1.35.1"
 KUBECTL_VERSION="1.36.4"
 YQ_VERSION="4.44.6"
 
+# Alert-rule unit tests (promtool test rules) — the L0 check
+# conformance/tests/l0-static/alert-rules-unit-tested runs every
+# platform/**/*.promtool-test.yaml through it. MUST stay in lockstep with
+# infra/devcontainer/versions.env (which bakes it into the image) AND with the Prometheus
+# version running on prod, because promtool IS the rule engine: verified 2026-09-12 as
+# v3.14.0-distroless via `kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`.
+# Shipped inside the prometheus release tarball, not as a separate release.
+PROMTOOL_VERSION="3.14.0"
+
 # ServiceDefinition JSON-Schema validation (schemas/validate.sh). Pip-installed;
 # pinned here so CI and a laptop validate against the exact same validator.
 CHECK_JSONSCHEMA_VERSION="0.29.4"

clusters/README.md

@@ -1,5 +1,19 @@
 # clusters/
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
 
 Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),
@@ -97,6 +111,27 @@ Kustomization: apps-wave-1  (./apps/wave-1, dependsOn apps-wave-0)
 Kustomization: apps-wave-N  (./apps/wave-N, dependsOn apps-wave-N-1)
 ```
 
+### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
+
+The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
+controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
+and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
+reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
+before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
+**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
+`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
+have existed for months); it cost 12h on prod's cold start.
+
+So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
+platform-config`), and a rule:
+
+> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
+> **controller at runtime** rather than by a manifest in git.
+
+`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
+`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
+profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
+
 Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
 exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
 (`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own

clusters/base/apps-vms.yaml

@@ -4,8 +4,11 @@
 # NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
 # import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
 # health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
-# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles
+# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
-# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern).
+# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
+# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
+# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
+# adoption concern).
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
@@ -19,6 +22,28 @@ spec:
   wait: false
   dependsOn:
     - name: platform-config
+    # LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
+    # storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
+    # decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
+    # (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
+    # live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
+    # with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
+    # to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
+    # conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
+    #
+    # HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
+    # derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
+    # status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
+    # it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
+    #
+    # This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
+    # deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
+    # dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
+    #
+    # The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
+    # edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
+    # cannot silently drop this tier's KubeVirt/StorageClass gate.
+    - name: platform-config-cdi
   sourceRef:
     kind: GitRepository
     name: flux-system

clusters/base/backup.yaml

@@ -34,6 +34,9 @@ spec:
     substituteFrom:
       - kind: ConfigMap
         name: cluster-vars
-  # SOPS decryption for velero-cloud.sops.yaml (the scoped MinIO `cloud` credential) is layered per
+  # SOPS decryption for the `cloud` INI credentials — velero-cloud.sops.yaml (dev's scoped MinIO key)
-  # environment by the overlay (clusters/dev patches THIS Kustomization by name), identical to how
+  # and velero-cloud-prod.sops.yaml (prod's Cloudflare R2 token) — is layered per environment by the
-  # headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. See clusters/dev/kustomization.yaml.
+  # overlay (clusters/dev and clusters/prod patch THIS Kustomization by name), identical to how
+  # headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. Both Secrets are applied on both
+  # clusters; ${BACKUP_CREDENTIALS_SUFFIX} (substituted from cluster-vars above) decides which one the
+  # velero Deployment actually mounts. See clusters/dev/kustomization.yaml.

clusters/base/buildkit.yaml

@@ -0,0 +1,21 @@
+---
+# Flux Kustomization -> ./platform/buildkit — rootless BuildKit, the image builder CI uses
+# (.gitea/workflows/images.yaml). Its own Kustomization so a stuck Certificate or a failing
+# builder can never stall any other tier (Flux health-checks Certificates under wait:true).
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: buildkit
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: true
+  dependsOn:
+    - name: platform   # cert-manager CRDs (Issuer/Certificate) come from its HelmRelease
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/buildkit

clusters/base/cloudflared.yaml

@@ -0,0 +1,61 @@
+---
+# Flux Kustomization: Cloudflare Tunnel connector — reconciles ./platform/cloudflared.
+#
+# ISOLATION IS THE POINT OF THIS FILE. The Deployment it applies CANNOT become Ready until the
+# operator enrols the tunnel (runbooks/cloudflare-tunnel.md): it ships with REPLACE-ME credentials
+# and a placeholder UUID, and a wrong or revoked credential later produces the same state. So:
+#
+#   * ITS OWN Kustomization, not a line in platform/kustomization.yaml. `platform` is wait:true and
+#     gates platform-config, which gates every apps wave; an unenrolled tunnel there would be a
+#     platform-wide stall ON MERGE, with the cause (a missing manual step) invisible from any Flux
+#     condition. Same reasoning, and same failure shape, as vpn-ingress's unfilled server key.
+#   * wait: false. Its pods are legitimately not-Ready whenever the credentials are absent or the
+#     edge is unreachable. Health is reported by the CloudflaredTunnelDown / ReplicasDegraded
+#     alerts (platform/cloudflared/alerts.yaml), not by this Kustomization's Ready condition —
+#     which reaches Ready on APPLY success.
+#   * NOTHING dependsOn THIS, and nothing should. A missing or invalid credentials Secret can then
+#     only ever affect the tunnel itself. conformance/tests/l0-static/cloudflared-ingress-rules
+#     fails the gate if any Kustomization gains `dependsOn: cloudflared`, or if this one stops
+#     being wait:false.
+#   * No Certificate, no HelmRelease, no CRD: nothing in the path for Flux's health checks to wait
+#     on even if wait were flipped (the Certificate-stall lesson).
+#
+# dependsOn: [platform-config] — A REAL EDGE.
+#   1. CRD ORDERING: PodMonitor and PrometheusRule are monitoring.coreos.com CRs whose CRDs are
+#      installed by kube-prometheus-stack in `platform`; platform-config dependsOn platform, so this
+#      edge carries that guarantee transitively (Flux aborts a whole pass on an unknown kind).
+#   2. THE FAR SIDE OF THE TUNNEL: the Gateway (and so the cilium-gateway-shared Service the ingress
+#      rules point at) is applied by platform-config. A tunnel that comes up before its origin is a
+#      support call.
+#
+# ZERO EFFECT ON LIVE TRAFFIC UNTIL DNS MOVES. Cloudflare only sends a hostname through a tunnel
+# when that hostname's DNS record is a CNAME to <uuid>.cfargotunnel.com. This change creates no
+# such record and leaves external-dns, the Gateway annotations, Pi-hole and CoreDNS untouched.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: cloudflared
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/cloudflared
+  # REQUIRED, not boilerplate: platform/cloudflared/config.yaml renders `${BASE_DOMAIN}` into every
+  # ingress hostname, and alerts.yaml into both runbook_url annotations. Without this block the
+  # token becomes the EMPTY STRING and the Kustomization still reports Ready (substitution fails
+  # OPEN), leaving ingress rules for `plex.` and `*.` — a tunnel that 404s everything at best.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # SOPS decryption is layered per-environment: clusters/dev and clusters/prod each patch THIS
+  # Kustomization by name with a `decryption:` stanza for cloudflared-credentials.sops.yaml.
+  # Without it the Secret applies cleanly full of ENC[...] and cloudflared cannot authenticate.

clusters/base/cluster-vars.yaml

@@ -17,9 +17,63 @@ data:
   CLUSTER_ENV: "base"                       # dev | prod  (overlays set this)
 
   # --- DNS / TLS ---
-  BASE_DOMAIN: "home.keithjackson.dev"      # platform hostname suffix (same both envs; split-horizon)
+  # THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
-  DNS_ZONE: "keithjackson.dev"              # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
+  # the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
-                                            # ExternalDNS domain-filter must be the zone, not the subdomain
+  # ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
+  #
+  # THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
+  # hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
+  # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
+  # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
+  # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
+  # subdomain, so the two clusters can no longer describe the same name.
+  #
+  #   dev   -> home.keithjackson.dev            (UNCHANGED — see below)
+  #   prod  -> prod.home.keithjackson.dev
+  #
+  # DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
+  # `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
+  # relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
+  # the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
+  # (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
+  # GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
+  # it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
+  # tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
+  BASE_DOMAIN: "home.keithjackson.dev"
+
+  # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
+  # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
+  # so it cannot be created, updated or deleted regardless of TXT ownership.
+  #
+  # DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
+  # apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
+  # that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
+  # record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
+  # from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
+  #
+  # PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
+  # (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
+  # chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
+  # — it can never delete another environment's records — and the fallback is to set prod's value
+  # back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
+  EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
+
+  # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
+  # neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
+  # record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
+  # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
+  # one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
+  #
+  # DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
+  # under owner `default`; changing dev's value would make dev's own external-dns stop recognising
+  # them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
+  #
+  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
+  # forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
+  # this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
+  # visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
+  EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
+
   ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
   GATEWAY_VIP: "192.168.1.53"                # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
                                             # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
@@ -42,15 +96,59 @@ data:
   #
   # Substitution coverage (verify before adding a knob here): every path that consumes these declares
   # postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
-  # (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
+  # (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
-  # velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
+  # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
-  # ship the literal string into the cluster.
+  # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
+  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
+  # plex-remote + pihole: PLEX_L4_VIP),
+  # vpn-ingress (WG_CLIENT_CIDR),
+  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
+  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
+  # UNSUBSTITUTED path would ship the literal string into the cluster.
+  #
+  # ADDING A KNOB IS SAFE UNDER StrictPostBuildSubstitutions ONLY BECAUSE OF THE ROOT EDGE. An
+  # UNDEFINED variable is a HARD Kustomization failure, so a manifest that references a knob the
+  # live ConfigMap does not yet carry breaks its whole tier. `clusters/base/platform.yaml` gained
+  # `dependsOn: [flux-system]` for exactly this: dependsOn is REVISION-gated, so no substituting
+  # tier can reconcile revision N until the ROOT Kustomization has APPLIED revision N — i.e. until
+  # the new cluster-vars key EXISTS. Every substituting consumer chains transitively through
+  # `platform`, so one edge covers the graph, and a new knob plus its first reference may ship in
+  # ONE commit. (A hand-run `kubectl apply -k clusters/<env>` bypasses that ordering entirely: apply
+  # the overlay BEFORE the tier that references the knob, or the tier fails until it retries.)
 
   # k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
-  # [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/
+  #
-  # main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot
+  # *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
-  # reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob.
+  # 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
-  K3S_API_HOST: "192.168.1.50"
+  # control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
+  # live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
+  # own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
+  # Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
+  # outage you would notice. A virtual address that follows the control plane is the fix, and it is
+  # what kube-vip was stood up to provide.
+  #
+  #   dev  -> 192.168.1.49   (= kube_vip_address, group_vars/all.yml; base default, below)
+  #   prod -> 192.168.2.9    (= kube_vip_address, group_vars/k3s_prod/vars.yml)
+  #
+  # THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
+  # unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
+  # move. Read it as "the API host Cilium dials", which is what it always meant.
+  #
+  # [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
+  # (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
+  # already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
+  # the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
+  # already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
+  # block at the top of group_vars/all.yml.
+  #
+  # Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
+  # in both envs, so it is NOT a knob.
+  #
+  # TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
+  # both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
+  # https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
+  K3S_API_HOST: "192.168.1.49"
 
   # CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
   # L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
@@ -65,15 +163,131 @@ data:
   # HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
   UNIFI_L4_VIP: "192.168.1.54"               # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
   CRAFTY_L4_VIP: "192.168.1.55"              # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
+  # PiHole's DNS listener: udp/53 + tcp/53 for LAN CLIENTS (apps/wave-0/pihole-dns.yaml). It is a
+  # THIRD pinned L4 VIP for the same reason as the two above — the `shared` Gateway is an L7 HTTPS
+  # terminator and cannot express DNS at all (udp/53 especially). It must NOT be ${GATEWAY_VIP}:
+  # LAN clients are pointed at this address as their resolver, and it has to stay put across
+  # reschedules, so it is pinned by lbipam.cilium.io/ips rather than taken from the pool head.
+  #
+  # WHY A SEPARATE ADDRESS AND NOT THE GATEWAY VIP: every `<svc>.${BASE_DOMAIN}` name resolves TO
+  # the Gateway VIP. Serving DNS from the same address would be legal but conflates "the thing that
+  # answers where services live" with "a service", and it would put LAN resolution behind the
+  # Gateway Service's own lifecycle. Keep them separate.
+  #
+  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
+  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
+  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
+  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
+  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
+  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
+  # measured working) cannot put the server on a client's connection list at all.
+  #
+  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
+  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
+  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
+  #
+  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
+  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
+  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
+  # SECOND CONSUMER since 2026-10-04: catalog/pihole.yaml reads this knob to answer
+  # plex-vip.${BASE_DOMAIN} on the LAN, which is the local half of the single hostname Plex
+  # advertises to plex.tv. Without that per-host answer the zone wildcard sends the name to
+  # ${GATEWAY_VIP}, where nothing listens on 32400. So this value is now typed into THREE places
+  # that must agree: this knob, the WAN port-forward on the SOHO router (off-repo), and PiHole's
+  # answer -- and only the first two were true before.
+
+
+  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
+  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
+  # from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
+  # CIDR, not to Cilium's `world` entity the way the unifi/crafty game-port allows are. It is the
+  # same /24 the node IPs and ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} come out of.
+  #
+  # dev  -> 192.168.1.0/24  (Proxmox LAN; nodes .50/.51/.52)
+  # prod -> 192.168.2.0/24  (prod LAN; nodes .10/.11/.12)
+  LAN_CIDR: "192.168.1.0/24"                 # ingress scope for the PiHole DNS listener
+
+  # --- Remote-access VPN client subnet (platform/vpn-ingress, WireGuard) --------------------------
+  # The tunnel subnet handed to REMOTE VPN clients. `.1` is the server (the hostNetwork DaemonSet,
+  # on every node); `.2`-`.254` are devices, allocated one /32 each in
+  # platform/vpn-ingress/peers.yaml. Consumed twice in the pod's config-assembly script: to derive
+  # the server's own address, and as the `-s` match of the MASQUERADE rule that makes LAN replies
+  # routable.
+  #
+  # IT IS A KNOB RATHER THAN A LITERAL FOR ONE REASON: it is the only address range in this cluster
+  # that must be checked against EVERY OTHER range at once, so it belongs where those are declared.
+  # 10.90.0.0/24 was chosen to collide with none of them:
+  #     10.0.0.0/16     Cilium cluster-pool pod CIDR     (${CLUSTER_POD_CIDR})
+  #     10.43.0.0/16    k3s service CIDR                 (measured live 2026-10-05: kubernetes svc
+  #                                                       clusterIP 10.43.0.1)
+  #     10.72.192.0/18  the PIA tunnel's pushed addresses (platform/vpn-egress)
+  #     192.168.1.0/24  dev LAN / pfSense LAN            (${LAN_CIDR} base default)
+  #     192.168.2.0/24  prod LAN                         (${LAN_CIDR} on prod)
+  #
+  # NO PER-ENVIRONMENT OVERRIDE, and that is deliberate rather than an omission: unlike every VIP
+  # above, this range is NOT drawn from the LAN. It is private to the tunnel, so the same value is
+  # correct in both environments and a prod patch would be a no-op that implies otherwise.
+  #
+  # IF YOU CHANGE IT, every already-issued client profile breaks (its `Address` is from this range)
+  # and must be re-rendered — so treat it as fixed once the first device is enrolled.
+  WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range
+
+  # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
+  # THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
+  # (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
+  # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
+  # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
+  # declared in the ServiceDefinition, because the path is identical in every environment and only
+  # the ADDRESS is not.
+  #
+  # IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
+  # OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
+  # vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
+  # only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
+  # TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
+  # values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
+  # 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
+  # this stays a knob only because the renderer reads it per-environment.
+  #
+  # *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
+  # immutable — measured by server dry-run 2026-10-07:
+  #     spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
+  # so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
+  # deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
+  BULK_NFS_SERVER: "192.168.2.30"
 
   # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
   # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
   # nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
-  # Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR.
+  # Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR —
+  # and, measured 2026-09-05 from a pod INSIDE prod, it CANNOT reach it anyway (prod is on the SOHO
+  # network 192.168.2.0/24, which is pfSense's WAN, while this MinIO sits inside pfSense's LAN:
+  # `curl http://192.168.1.9:9000/minio/health/live` timed out after 15s). Prod uses Cloudflare R2.
   S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
 
+  # S3 region. Every S3 client REQUIRES one; MinIO ignores whatever it is; Cloudflare R2 demands
+  # exactly `auto` (its `head-bucket` reports the physical hint `WNAM`, which is NOT a usable region
+  # string). This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-
+  # accident for MinIO and wrong for anything else, which is why it is a knob now. The base default
+  # is dev's LIVE value, deliberately: dev's Velero has been signing requests with `us-east-1`
+  # against that MinIO since bootstrap, and this repo's rule is that base == the known-good dev
+  # value. Prod overrides to `auto`.
+  S3_BACKUP_REGION: "us-east-1"
+
   # --- storage (ADR-0003) ---
   LONGHORN_DEFAULT_REPLICA_COUNT: "3"       # default StorageClass replica count (3 on 3 nodes)
+  # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
+  # replica is written into. It must be the path the NODE's own data disk is actually mounted at —
+  # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
+  # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
+  # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
+  # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
+  # on it — changing dev's value would orphan every replica, so the dev overlay restates it
+  # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
+  # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
+  # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
+  LONGHORN_DATA_PATH: "/var/lib/longhorn"
 
   # --- observability sizing (docs/10) ---
   MONITORING_PROMETHEUS_RETENTION: "7d"     # dev short | prod long
@@ -113,6 +327,99 @@ data:
   VELERO_BACKUP_SCHEDULE: "0 3 * * *"        # cron; overlays may differ
   VELERO_RETENTION: "168h"                   # dev short | prod long
 
+  # ==========================================================================================
+  # OBJECT-STORE TENANCY — the three knobs that keep two clusters and two backup tools from
+  # writing into each other's data. THESE ARE SAFETY KNOBS, not preferences.
+  # ==========================================================================================
+  # Nothing about the object layout was per-environment before this: Velero's bucket was the
+  # hardcoded literal `velero` with no prefix, and every CNPG Cluster used the hardcoded
+  # `s3://cnpg` with barman's `serverName` defaulting to the CLUSTER NAME — and BOTH environments
+  # have a keycloak-db, a nextcloud-db and a semaphore-db. Point two clusters at one bucket under
+  # that layout and:
+  #   (a) each cluster's Velero ADOPTS the other's backups and TTL-DELETES them on retention; and
+  #   (b) two different Postgres servers stream WAL into ONE Barman prefix, which does not merely
+  #       mix files — it CORRUPTS THE CHAIN and destroys PITR for both. That is the worse half:
+  #       dev's DR restore is PROVEN (R-STATE-03) and this would silently un-prove it.
+  #
+  # WHY THERE IS NO SINGLE `S3_BACKUP_BUCKET`. The two consumers take differently-SHAPED values:
+  # the Velero chart wants `bucket` and `prefix` as separate fields, while CNPG's
+  # barmanObjectStore.destinationPath is ONE full s3:// URL whose first path segment IS the bucket.
+  # CNPG's bucket therefore already lives inside CNPG_BACKUP_PATH; adding a second knob that had to
+  # hold the same bucket name a second time would be a rotation waiting to drift — the exact class
+  # of duplication this repo has been bitten by. Three knobs, each shaped for its own consumer, and
+  # no two of them can contradict each other.
+  #
+  # BASE = DEV'S LIVE LAYOUT, BYTE-FOR-BYTE, AND MUST STAY THAT WAY. Dev's existing backups are at
+  # `s3://velero/` (bucket root, no prefix) and `s3://cnpg/<clusterName>/` on that MinIO. Giving dev
+  # a prefix, or renaming its buckets, would ORPHAN the very backups its proven restore depends on —
+  # the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default` and its LONGHORN_DATA_PATH stays
+  # /var/lib/longhorn. Verified: with an EMPTY prefix the Velero chart omits `objectStorage.prefix`
+  # from the rendered BackupStorageLocation entirely (helm template, chart 12.1.0), so dev's live
+  # BSL is unchanged by this parameterization rather than merely equivalent to it.
+  #
+  # PROD, which has no history to preserve, gets ONE bucket with TWO disjoint top-level prefixes:
+  #
+  #     s3://k3s-prod-backup/velero/…                 Velero (kopia repo + backup metadata)
+  #     s3://k3s-prod-backup/barman/<clusterName>/…   CNPG WAL + base backups
+  #
+  # The operator created exactly one R2 bucket, so velero-vs-barman separation has to come from the
+  # prefix rather than the bucket. The BUCKET ROOT IS DELIBERATELY LEFT EMPTY: if a prefix ever
+  # renders wrong, its objects land somewhere obviously bogus instead of silently colliding with a
+  # real tenant's data. Dev↔prod isolation is doubly independent — different endpoint AND different
+  # bucket — so neither cluster can reach the other's objects even with a mis-substituted prefix.
+  VELERO_BACKUP_BUCKET: "velero"             # dev: the `velero` MinIO bucket | prod: k3s-prod-backup
+  VELERO_BACKUP_PREFIX: ""                   # dev: bucket root (live layout) | prod: "velero"
+  CNPG_BACKUP_PATH: "s3://cnpg"              # dev: the `cnpg` MinIO bucket root | prod: s3://k3s-prod-backup/barman
+
+  # ------------------------------------------------------------------------------------------------
+  # WHICH CREDENTIAL SECRET THE BACKUP CONSUMERS READ (added 2026-09-07 — the per-env credential split)
+  # ------------------------------------------------------------------------------------------------
+  # The three knobs above say WHERE the backups go. This one says WHICH KEY signs the requests, and
+  # it exists because the endpoint moved per-environment while the credential did not.
+  #
+  # THE BUG IT FIXES. The four backup-credential Secrets — velero-cloud.sops.yaml and the three
+  # <db>-backup-credentials.sops.yaml — are SHARED BASE manifests: platform/ and apps/ are rendered
+  # by the same Flux tiers on both clusters, and cluster-vars substitution is the ONLY thing that
+  # differs (R-ENV-02). So writing prod's R2 token into those four files did not GIVE prod a
+  # credential, it REPLACED DEV'S: dev's Velero and all three dev CNPG Clusters would have started
+  # signing MinIO requests with an R2 key id. For Velero that is a broken BSL; for CNPG it is worse,
+  # because `archive_mode=on` means Postgres REFUSES TO RECYCLE a WAL segment it could not archive,
+  # so a 403 loop is a monotonically filling pgdata PVC ending in a PANIC — the same failure prod
+  # spent its whole bring-up in with a placeholder endpoint. One shared Secret also means rotating
+  # either environment's key silently breaks the other; that is the structural reason for the split,
+  # independent of dev's eventual decommission.
+  #
+  # THE MECHANISM. Each credential is now TWO Secret objects, both applied to BOTH clusters (they are
+  # base resources; nothing here forks a manifest per environment). The consumer names one:
+  #
+  #     velero:   credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
+  #     CNPG:     s3Credentials.*.name:  <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
+  #
+  # ONE knob, not four, ON PURPOSE. Four independent secret-name knobs could be updated three-
+  # quarters of the way, leaving one consumer pointed at the OTHER environment's key with nothing
+  # rendering wrong — and StrictPostBuildSubstitutions cannot catch that, because it only fails on an
+  # UNDEFINED variable, never on a defined-but-stale one. With a single suffix the four move together
+  # or not at all. The cost is that the rendered Secret name is not greppable from the consumer; the
+  # names are spelled out in full in each kustomization.yaml beside the resources.
+  #
+  # THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names,
+  # which are the objects that already exist and that the live dev cluster is running. A default
+  # naming a Secret that does not exist would be worse than the shared-Secret state it replaces:
+  # Velero's pod would sit in ContainerCreating forever on a missing `existingSecret` volume, and a
+  # CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — i.e. the disk-full clock
+  # again — so an unresolvable default breaks the referencing cluster HARDER than a wrong key does.
+  # Empty is also the value dev must carry, so base and dev agree, exactly like VELERO_BACKUP_PREFIX.
+  #
+  # An EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions: strict mode fails on a variable that is
+  # not DEFINED, not on one defined as "". VELERO_BACKUP_PREFIX above has shipped empty since the
+  # gate was enabled and proves it. Both overlays state this knob explicitly — dev "" and prod
+  # "-prod" — so neither inherits it silently.
+  #
+  # ADDING A NEW BACKUP CREDENTIAL: create BOTH halves (`x-credentials.sops.yaml` and
+  # `x-credentials-prod.sops.yaml`), list both in the kustomization, and reference
+  # `x-credentials${BACKUP_CREDENTIALS_SUFFIX}`. Never a single shared Secret again.
+  BACKUP_CREDENTIALS_SUFFIX: ""              # dev: "" (MinIO creds) | prod: "-prod" (R2 creds)
+
   # --- placement ---
   PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os"   # base no-op selector; prod may pin roles
   PLATFORM_NODE_SELECTOR_VALUE: "linux"

clusters/base/conformance.yaml

@@ -0,0 +1,31 @@
+---
+# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
+#
+# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
+# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
+# meant to verify is worse than no harness.
+#
+# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
+# waves needs it.
+#
+# NOTE: this reconciles RBAC that is INERT until
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
+# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
+# trigger identity can exist while the policy bounding it does not.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: conformance
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./conformance/incluster

clusters/base/homepage.yaml

@@ -34,8 +34,12 @@ spec:
     kind: GitRepository
     name: flux-system
   path: ./platform/homepage
-  # Homepage references no ${VARS} today; the substitution is retained for parity with the other
+  # LOAD-BEARING, not parity decoration (the comment here used to say the opposite). This component
-  # tiers / future overlay knobs.
+  # references ${BASE_DOMAIN} in TWO places: httproute.yaml's apex hostname, and EVERY tile href in
+  # configmap.yaml. Without this substitution the token is applied LITERALLY — the apex route binds
+  # the host `${BASE_DOMAIN}`, which matches no Gateway listener, and every link on the operator's
+  # front door points at a host that does not exist — while Flux still reports this Kustomization
+  # Ready. It fails OPEN. Asserted by conformance/tests/l0-static/c2-root-surface (5).
   postBuild:
     substituteFrom:
       - kind: ConfigMap

clusters/base/intel-gpu-plugin.yaml

@@ -0,0 +1,104 @@
+---
+# Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin.
+# Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will
+# have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath.
+#
+# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
+#
+# The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and
+# neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human
+# reviewer on the policy's terms. It is still the wrong place, for the reason the repo already
+# wrote down in platform/kustomization.yaml when Cilium was split out of that tier:
+#
+#   * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell
+#     you "the GPU plugin is wedged" from "longhorn is slow".
+#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
+#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device
+#     plugin.
+#   * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node
+#     DaemonSet rollout. This is a 3-node DaemonSet rollout.
+#
+# And the consequence is worse here than for an average component, because of what sits downstream:
+# `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1.
+# A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency
+# not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have
+# acquired the ability to cause a REAL OUTAGE. That trade is the whole argument.
+#
+# The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces
+# `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of
+# path, and authors cannot self-approve. So this change needed the operator's approval either way;
+# touching clusters/** costs nothing extra and buys a blast radius of exactly one component.
+#
+# dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST
+#
+# Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that
+# genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of
+# it. It never uses the network and never talks to the API server; it talks to the local kubelet
+# over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at
+# all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the
+# cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too.
+#
+# Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this
+# reconciles in parallel with the `platform` tier instead of behind all eleven of its components.
+# That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node
+# advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window
+# instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real
+# constraint on the next PR and not a formality.
+#
+# Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this
+# workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled
+# "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose.
+#
+# wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the
+# headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a
+# leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on
+# Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it
+# reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and
+# not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into
+# a one-line answer to "did the plugin come up on all three nodes", which is exactly the question
+# the operator must answer before the consumer PR may merge.
+#
+# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
+# plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding
+# `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app
+# waves that would hand a device plugin the power to stall every app, which is the thing this file
+# exists to prevent. Think before adding that edge.
+#
+# ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting
+# `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT
+# land until this plugin is merged AND confirmed advertising the resource on all three nodes. If
+# the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is
+# wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent
+# no-op into a real outage. This PR's only job is to make the resource exist.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: intel-gpu-plugin
+  namespace: flux-system
+spec:
+  # Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered,
+  # so this is only the drift-correction floor.
+  interval: 10m
+  retryInterval: 2m
+  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
+  # exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls
+  # out that exact mismatch as one of its reasons for leaving the shared tier.
+  timeout: 10m
+  prune: true
+  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
+  wait: true
+  dependsOn:
+    - name: cilium   # the CNI — the only real prerequisite; transitively gives us flux-system too
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/intel-gpu-plugin
+  # Retained for parity with every other tier / future overlay knobs. This component references no
+  # ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike
+  # clusters/base/runbooks.yaml, which omits substitution for exactly that reason).
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it
+  # (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup).

clusters/base/kured.yaml

@@ -0,0 +1,98 @@
+---
+# Flux Kustomization: kured (Kubernetes Reboot Daemon) — reconciles ./platform/kured.
+# Performs the cordon / drain / reboot / uncordon of a node that has a pending OS reboot, strictly
+# one node at a time, inside a nightly window, gated on Longhorn volume health. Design rationale,
+# every tuned value, and the honest list of what is NOT mitigated: platform/kured/helmrelease.yaml.
+#
+# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
+#
+# The same argument the Cilium split-out recorded in platform/kustomization.yaml and intel-gpu-plugin
+# reused, except the consequence is worse here than for either of them:
+#
+#   * `platform` is `wait: true` over twelve-plus components, so its Ready condition cannot tell you
+#     "the reboot daemon is wedged" from "longhorn is slow".
+#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
+#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a DaemonSet.
+#   * That tier's 10m interval / 5m timeout are called out there as WRONG for a 3-node DaemonSet
+#     rollout. This is a 3-node DaemonSet rollout.
+#   * And `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates
+#     apps-wave-1. A reboot daemon that failed to go healthy inside that tier would stall EVERY
+#     APPLICATION in the cluster on "dependency not ready". A component added to make maintenance
+#     safer must not acquire the ability to cause an outage; that trade is the whole argument.
+#
+# dependsOn: [platform] — AND THIS ONE IS A REAL EDGE, NOT CONVENTION
+#
+# Not `cilium` (what intel-gpu-plugin uses) and not `platform-config` (what the leaf UIs use). It is
+# `platform`, for two independent reasons:
+#
+#   1. CRD ORDERING (F-MON1). platform/kured ships a PrometheusRule, and the chart generates a
+#      ServiceMonitor. Both are CRs whose CRDs (monitoring.coreos.com) are installed by
+#      kube-prometheus-stack INSIDE the `platform` tier. Flux dry-runs every object in a pass before
+#      applying any of them, so a pass containing an unknown kind ABORTS — the failure that made the
+#      CDI StorageProfiles a true deadlock on prod on 2026-09-05 (see
+#      clusters/base/platform-config-cdi.yaml). Depending on `platform` guarantees the CRDs exist
+#      before this pass is attempted.
+#   2. FUNCTIONAL PREREQUISITE. kured's reboot gate QUERIES Prometheus
+#      (`prometheusUrl` in the HelmRelease). Prometheus lives in platform/monitoring. If kured ran
+#      first it would get connection errors, treat the gate as unevaluable and skip every reboot —
+#      the correct fail-safe direction, but silent. Longhorn, whose volume health IS the gate, is
+#      also in that tier.
+#
+# It does NOT need platform-config: no Keycloak, no TLS, no split-horizon DNS, no Gateway, no
+# HTTPRoute. kured serves no HTTP to users.
+#
+# wait: true — A HEALTH SIGNAL, NOT A GATE. Following intel-gpu-plugin rather than the leaf-UI
+# template. A leaf UI is legitimately not-Ready for long stretches of a cold start while it waits on
+# Keycloak and DNS, so its health is uninformative; kured has no such dependency — it is a DaemonSet
+# that opens a socket and watches a file, so on healthy nodes it is Ready in seconds and not-Ready
+# genuinely means BROKEN. This turns `flux get kustomization kured` into a one-line answer to "is
+# reboot automation live on all three nodes", which is the question the operator must be able to
+# answer before trusting a kernel upgrade to it.
+#
+# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
+# kured is this Kustomization's own Ready condition. Do not add `dependsOn: [kured]` anywhere: that
+# would hand a node-rebooting DaemonSet the power to stall whatever depends on it, which is precisely
+# what the placement above exists to prevent.
+#
+# prune: true is safe and deliberate. If this Kustomization is deleted, kured's DaemonSet goes with
+# it and reboots simply stop happening — nodes are left uncordoned and running. The one caveat is
+# that pruning mid-reboot would leave a node CORDONED with no daemon to uncordon it; that is a
+# `kubectl uncordon` away and is covered in runbooks/node-reboot-automation.md.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: kured
+  namespace: flux-system
+spec:
+  # A HelmRelease plus five static manifests, nothing to re-render; kustomize-controller is
+  # source-triggered, so this is only the drift-correction floor.
+  interval: 10m
+  retryInterval: 2m
+  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
+  # exceed 5m on a cold start (image pull on three nodes), which is one of the reasons the Cilium
+  # split-out gave for leaving that tier.
+  timeout: 10m
+  prune: true
+  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
+  wait: true
+  dependsOn:
+    - name: platform   # monitoring CRDs + Prometheus + Longhorn; see the header, this is load-bearing
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/kured
+  # REQUIRED, not boilerplate. platform/kured/alerts.yaml renders `${BASE_DOMAIN}` into the
+  # `runbook_url` annotation of all three rules. Without this block that token becomes the EMPTY
+  # STRING and the Kustomization still reports Ready — Flux's substitution fails OPEN — so every
+  # alert would ship a dead link to `https:///runbooks/node-reboot-automation`. The gate stage
+  # `flux-postbuild-substitution` asserts the converse (a rendered token implies this block), and
+  # BASE_DOMAIN is declared in clusters/base/cluster-vars.yaml.
+  #
+  # Nothing else in this path contains `${...}` or a bare `$`: the HelmRelease's alertFilterRegexp is
+  # a plain anchored alternation, and there are no shell snippets anywhere in the component — which
+  # is deliberate, because envsubst would eat a braced shell variable and `$$` would trip the
+  # dollar-collapse assertion in the same check.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars

clusters/base/kustomization.yaml

@@ -12,8 +12,26 @@ resources:
   # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
   # freeze the other eleven platform components. See clusters/base/cilium.yaml.
   - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
+  # Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
+  # platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
+  # that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
+  # components AND would stall platform-config + every apps wave on "dependency not ready" — so a
+  # component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
+  # wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
+  # It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
+  # strictly later. See clusters/base/intel-gpu-plugin.yaml.
+  - intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
+  # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
+  # manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
+  # datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
+  # platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
+  # keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
+  # CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
+  # cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
+  # as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
+  - platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
   # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
   # Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
   # (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
@@ -24,6 +42,7 @@ resources:
   - apps-wave-2.yaml     # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
   # VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
   - apps-vms.yaml        # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
+  - conformance.yaml     # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
   # oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
   # service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
   # `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
@@ -41,8 +60,50 @@ resources:
   # reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same
   # non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform).
   - crafty.yaml          # Flux Kustomization -> ./platform/crafty (dependsOn platform-config)
+  # Alert runbooks — POST-platform tier (dependsOn platform-config, wait:false). Serves the 34
+  # runbooks the alert rules' `runbook_url` annotations point at, at <apex>/runbooks/<name>; before
+  # this tier existed that path was a 404 and every alert handed the operator a dead link. Same
+  # non-gating leaf rationale as headlamp/homepage/crafty. Carries NO postBuild substitution on
+  # purpose — it renders documentation containing `${...}` shell examples that envsubst would empty;
+  # see clusters/base/runbooks.yaml.
+  - runbooks.yaml        # Flux Kustomization -> ./runbooks (dependsOn platform-config)
   # Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup
   # controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
   # Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
   - backup.yaml          # Flux Kustomization -> ./platform/velero (dependsOn platform-config)
+  # WireGuard remote-access VPN (north-south INGRESS, udp/51820) — POST-platform tier
+  # (dependsOn platform-config, wait:false). NOT the same thing as platform/vpn-egress, which sends
+  # pods OUT through PIA; this lets remote CLIENTS IN. Non-gating is load-bearing rather than
+  # conventional here: its DaemonSet cannot start until the operator creates the out-of-band
+  # `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have
+  # stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch.
+  - vpn-ingress.yaml     # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config)
+  # Cloudflare Tunnel connector (north-south INGRESS from the Cloudflare edge) — POST-platform tier
+  # (dependsOn platform-config, wait:false), and NOTHING dependsOn it. Non-gating is load-bearing,
+  # as for vpn-ingress: the Deployment cannot become Ready until the operator enrols the tunnel
+  # (runbooks/cloudflare-tunnel.md), so in a gating tier an unfinished manual step — or a revoked
+  # credential later — would stall every apps wave. Carries cloudflared-credentials.sops.yaml, so
+  # both overlays add a SOPS decryption patch for it (by name). See clusters/base/cloudflared.yaml.
+  - cloudflared.yaml     # Flux Kustomization -> ./platform/cloudflared (dependsOn platform-config)
+  # PM agent — reads Gitea issues and runs one worker agent per assignment that authors a
+  # change, gates it and opens a pull request. POST-APP tier (dependsOn apps-wave-0,
+  # wait:false): what it needs is GITEA, not Keycloak or the Gateway, and it must never gate
+  # anything — it CrashLoops by design while its credentials are still placeholders. Carries
+  # pm-agent-credentials.sops.yaml, so both overlays add a SOPS decryption patch for it (by
+  # name). See clusters/base/pm-agent.yaml and platform/pm-agent/README.md.
+  - pm-agent.yaml        # Flux Kustomization -> ./platform/pm-agent (dependsOn apps-wave-0)
+  # kured (Kubernetes Reboot Daemon) — its OWN Kustomization (dependsOn platform, wait:true), NOT a
+  # line in platform/kustomization.yaml. Same reasoning as the Cilium and intel-gpu-plugin
+  # split-outs, with a sharper consequence: `platform` gates platform-config and every apps wave, so
+  # a wedged reboot DaemonSet placed in that wait:true tier could stall every application in the
+  # cluster — a component added to make maintenance SAFER must not be able to cause an outage.
+  # wait:true here is a HEALTH SIGNAL, not a gate: nothing dependsOn it, and nothing should.
+  # The `platform` edge is load-bearing rather than conventional — it supplies the
+  # monitoring.coreos.com CRDs this component's PrometheusRule/ServiceMonitor need (F-MON1), and
+  # Prometheus itself, which kured QUERIES before every reboot to gate on Longhorn volume health.
+  # It reboots a node only when /var/run/reboot-required exists, which on these nodes only a
+  # deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
+  # See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
+  - kured.yaml           # Flux Kustomization -> ./platform/kured (dependsOn platform)
+  - buildkit.yaml        # Flux Kustomization -> ./platform/buildkit (dependsOn platform)
   - cluster-vars.yaml    # overlay knob surface (base defaults)

clusters/base/platform-config-cdi.yaml

@@ -0,0 +1,135 @@
+---
+# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
+# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
+#
+# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
+#
+# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
+# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
+# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
+# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
+# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
+# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
+# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
+#
+# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
+# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
+# graph order itself, so a cold cluster needs no human in the loop.
+#
+# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
+# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
+# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
+# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
+# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
+# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
+# kind. One edge, no pre-apply.
+#
+# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
+# and everything else waits on: putting CDI's health on that critical path is the mistake the
+# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
+# StorageProfiles that were in the wrong pass.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: platform-config-cdi
+  namespace: flux-system
+spec:
+  interval: 10m
+  # 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
+  # very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
+  # now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
+  # about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
+  # cache is one retryInterval. Keep it short.
+  retryInterval: 2m
+  # 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
+  # image to pull and no controller to become Available, so a slow apply here means a real error, not
+  # a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
+  timeout: 2m
+  # TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
+  # component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
+  # It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
+  # second path also declares. See the transition note at the bottom of this file.
+  prune: true
+  # TRUE, but understand exactly what it buys and what it does NOT.
+  #
+  # CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
+  # provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
+  # `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
+  # it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
+  # is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
+  # If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
+  # `apps-vms` waits on it.
+  #
+  # WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
+  # provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
+  # storageprofile-controller computes that status from our spec override plus inference. So Ready
+  # here means "the spec override is accepted by the API server", not "CDI has folded it into
+  # status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
+  # below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
+  # a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
+  wait: true
+  dependsOn:
+    # platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
+    # causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
+    # ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
+    # flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform-config-cdi
+  # NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
+  # no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
+  # oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
+  # clusters/dev or clusters/prod for this tier.
+  #
+  # substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
+  # substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
+  #   1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
+  #      whose Kustomization declares substituteFrom. Omitting it would move
+  #      cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
+  #      could ship as a literal string, precisely the class of bug that check exists to stop.
+  #   2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
+  #      with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
+  #      empty. Without the stanza there is no substitution at all and the literal ships silently.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
+  # Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
+  # Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
+  # StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
+  # the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
+  # dual-ownership fight — the objects are pruned before they are re-applied under this tier's
+  # `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
+  #
+  # Why that is acceptable here — and it is NOT the platform/cilium case:
+  #   * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
+  #     nothing else in the repo declares a StorageProfile. The stale set is exactly two
+  #     cluster-scoped CDI CRs.
+  #   * The objects do not actually go missing: CDI's storageprofile-controller creates a
+  #     StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
+  #     inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
+  #     absent".
+  #   * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
+  #     existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
+  #     is disturbed and no data path is touched.
+  #   * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
+  #     instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
+  #     permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
+  #     dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
+  #     created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
+  #     could.
+  #
+  # PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
+  # happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
+  # config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
+  # healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
+  # --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
+  # StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
+  #   `flux get kustomization platform-config` must be Ready BEFORE merging.
+  # If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
+  # temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
+  # this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
+  # `prune: true` in a second commit.

clusters/base/pm-agent.yaml

@@ -0,0 +1,57 @@
+---
+# Flux Kustomization: the PM agent service — reconciles ./platform/pm-agent.
+#
+# POST-APP TIER (dependsOn apps-wave-0). What this service actually depends on is GITEA: the
+# dispatcher authenticates to it at startup, reads issue threads from it, and every worker
+# clones and pushes to it. Gitea is in apps-wave-0, so that is the dependency — not
+# platform-config, which brings up the Keycloak realm, the wildcard certificate, the Gateway
+# and the L2 pool. This service has no UI, serves no traffic, takes no ingress and holds no
+# certificate, so none of those are prerequisites and claiming them would be a lie that
+# delays its reconcile behind four things it does not use.
+#
+# NON-GATING (wait:false), and here that is not merely prudent — the pod is EXPECTED to be
+# unhealthy for a while:
+#   * the committed credentials are PLACEHOLDERS, so until the operator replaces them the
+#     dispatcher fails its Gitea authentication at startup and CrashLoops, by design
+#     (platform/pm-agent/pm-agent-credentials.sops.yaml.example);
+#   * the image digest is likewise a placeholder until the operator pushes the image, so
+#     before that the pod cannot even pull.
+# Either of those inside a wait:true tier would stall everything downstream of it. wait:false
+# guarantees this service's health can NEVER gate anything, even if some future resource is
+# mistakenly pointed dependsOn it. Same non-gating leaf rationale as
+# headlamp/homepage/crafty/backup — with a stronger reason than any of them.
+#
+# PRUNE: true. The whole namespace is disposable — the dispatcher keeps no durable state (the
+# Gitea issue thread is its state) and worker Jobs are ephemeral by construction. So removing
+# this service from git really should remove it from the cluster, with nothing to orphan.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: pm-agent
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: apps-wave-0   # Gitea: the repository this service reads issues from and pushes to
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/pm-agent
+  # cluster-vars is substituted for parity with every other tier. NOTE for anyone editing
+  # platform/pm-agent/: envsubst runs over the WHOLE rendered manifest, comments included, so
+  # a `${...}` meant for a shell would be replaced with the empty string before the container
+  # starts (the hazard catalog/README.md documents at length — it silently killed gitea's and
+  # nextcloud's OIDC registration for days). Nothing under platform/pm-agent/ uses a `${}`
+  # token today, and the two scripts that DO contain shell live in the image, not in a
+  # manifest, which is one more reason they are baked in rather than mounted from a ConfigMap.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # SOPS decryption for pm-agent-credentials.sops.yaml is layered per-environment by the
+  # overlay (clusters/prod and clusters/dev patch THIS Kustomization by name), identical to
+  # crafty/headlamp/oauth2-proxy/backup.

clusters/base/runbooks.yaml

@@ -0,0 +1,57 @@
+---
+# Flux Kustomization: alert runbooks — reconciles ./runbooks.
+#
+# WHAT THIS SERVES. Every alert rule in this repo carries
+# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names.
+# Until this tier existed that path returned 404 on the live cluster, so every alert that reached a
+# human handed them a dead link at the moment they needed help. This Kustomization renders the
+# markdown under ./runbooks into a ConfigMap and a two-replica nginx that serves it at that exact
+# prefix. See runbooks/README.md and runbooks/serving/httproute.yaml.
+#
+# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false) — the same placement and
+# the same reasoning as homepage/headlamp/oauth2-proxy. It needs the wildcard TLS cert, the shared
+# Gateway (including the `https-apex` listener) and split-horizon DNS, all of which arrive with
+# `platform` + `platform-config`; and a leaf web server must never be able to stall the platform
+# chain, which is the whole lesson of DECISIONS-LOG's "Headlamp gating stall". wait:false guarantees
+# its health can never gate anything even if something is later mistakenly pointed dependsOn it.
+#
+# NO postBuild.substituteFrom — THE ONE DELIBERATE DIVERGENCE FROM EVERY OTHER TIER, AND THE MOST
+# IMPORTANT LINE IN THIS FILE.
+# This tier renders the runbook PROSE into a ConfigMap, and runbooks are full of shell: `$POD`,
+# `$(kubectl ...)`, and — unavoidably, since they document these very annotations — the literal text
+# `${BASE_DOMAIN}`. Flux's postBuild substitution runs envsubst over the ENTIRE built stream, and it
+# FAILS OPEN: an undefined `${X}` becomes the empty string while the Kustomization still reports
+# Ready (DECISIONS-LOG 2026-09-01 — that is how `k8sServiceHost` was emptied and Cilium broke on 2
+# of 3 dev nodes). Enabled here it would silently mutilate the documentation: every braced expansion
+# in every command example blanked, in a file nobody re-reads after writing. Worse, the L0 check
+# `conformance/tests/l0-static/flux-postbuild-substitution` scans only files reachable through
+# kustomization `resources:`, not `configMapGenerator` inputs, so the damage would be invisible to
+# the gate as well.
+# The fix is to have nothing to substitute. The only knob this tier would have wanted is the apex
+# hostname, and runbooks/serving/httproute.yaml avoids needing it by declaring NO `hostnames:` and
+# inheriting the `https-apex` listener's own hostname instead. So this tier is
+# environment-independent by construction and R-ENV-02 ("authored once, promoted dev -> prod") holds
+# without substitution rather than because of it.
+# IF A FUTURE KNOB EVER GENUINELY NEEDS SUBSTITUTION HERE: do not add substituteFrom to this
+# Kustomization. Split the serving manifests into their own substituted tier and leave the content
+# ConfigMap in an unsubstituted one, or the runbooks lose their shell examples.
+#
+# NO SOPS decryption stanza: this tier carries no encrypted Secret (markdown plus five plain
+# manifests), so — like homepage — neither clusters/dev nor clusters/prod patches it.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: runbooks
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config   # wildcard TLS + Gateway (incl. the https-apex listener) first
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./runbooks

clusters/base/vpn-ingress.yaml

@@ -0,0 +1,59 @@
+---
+# Flux Kustomization: WireGuard remote-access VPN — reconciles ./platform/vpn-ingress.
+#
+# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false). Not a line in
+# platform/kustomization.yaml, for the standard reason plus one specific to this component:
+#
+#   * STANDARD: a leaf must never sit in the gating `platform` tier. While `platform` is not-Ready
+#     NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
+#     (DECISIONS-LOG: "Headlamp gating stall").
+#   * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
+#     operator replaces the `REPLACE-ME` placeholder in platform/vpn-ingress/wg-server-key.sops.yaml
+#     with a real key (runbooks/vpn-remote-access.md step 2). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
+#     the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
+#     cause (a missing manual step) invisible from any Flux condition.
+#
+# WHY dependsOn platform-config AND NOT cilium. This component needs no Gateway, no TLS cert, no
+# Keycloak and no LB-IPAM pool — it has no HTTPRoute and no LoadBalancer Service, and hostNetwork
+# means it does not even need a Cilium endpoint. It could legally reconcile much earlier. It is
+# pinned behind platform-config anyway so that the ONE thing remote clients actually consume
+# through it — split-horizon DNS and the shared Gateway — exists by the time the tunnel does.
+# A tunnel that comes up before the things on the far side of it is a support call, not a feature.
+#
+# wait:false: nothing in the graph dependsOn this, and its pods are legitimately not-Ready whenever
+# the server key is absent or the `wireguard` kernel module has not been loaded on a node. Neither
+# condition should be able to gate anything else, even if a future resource is mistakenly pointed
+# at it.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: vpn-ingress
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/vpn-ingress
+  # REQUIRED, not boilerplate: platform/vpn-ingress/daemonset.yaml renders ${WG_CLIENT_CIDR} into
+  # the config-assembly script. Without this block that token becomes the EMPTY STRING and the
+  # Kustomization still reports Ready (Flux's substitution fails OPEN — ci:
+  # flux-postbuild-substitution), leaving an initContainer that derives a server address of ".1/"
+  # and a MASQUERADE rule with no source. Both fail loudly at pod start, but the Kustomization
+  # would look healthy.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # SOPS decryption is layered per-environment by the overlay (clusters/dev and clusters/prod each
+  # patch THIS Kustomization by name), identical to headlamp/crafty/velero — this path carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml, the WireGuard server private key. The `sops-age`
+  # Secret is created out of band and referenced by NAME only. Without the overlay patch the Secret
+  # applies cleanly full of ENC[...] and only the tunnel is broken, so the patch is as load-bearing
+  # as the encrypted file itself.