feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite

OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."

THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.

SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
  k3s_api_host            unchanged — the FOUNDER's IP: the cold-bootstrap join
                          target and a tls-san. infra/tofu/outputs.tf GENERATES this
                          name, so renaming would ripple into tofu and the runbooks.
  k3s_apiserver_endpoint  NEW = {{ kube_vip_address }} — the HA address; what Cilium
                          consumes. Derived, not restated, so an environment overrides
                          ONE line and the VIP, the SAN and the CNI endpoint move
                          together. Prod needed no new override.

THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.

Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".

VERIFIED BY ME, not just reported:
  renders    dev K3S_API_HOST=192.168.1.49   prod=192.168.2.9  (each environment's VIP)
  cilium     template consumes k3s_apiserver_endpoint, not k3s_api_host
  cold join  still `--server https://{{ k3s_api_host }}:6443` — the founder
  order      site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
  gate       full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.

ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.

APPLY ORDER, and this is the operator's step — nothing was applied:
  PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
  Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
  ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
  incident), it will not proceed past an unready pod, existing datapath survives an
  agent restart, and .9 is already live before the change lands.
  DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
  192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
  ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
  cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.

NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.

Refs R-PLAT-01, R-NET-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>

.gitea/workflows/gate.yml

@@ -1,44 +1,25 @@
 # .gitea/workflows/gate.yml — the ENFORCED PR merge gate (Phase 6, WS-GITOSS).
 #
 # This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
-# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
-# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
-# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
-# below for why the script, not `make`, is invoked here). It runs every stage in order —
-# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
-# R-TEST-03 decision, printing
-# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
-# single required status check on `main` (see branch protection). A non-zero exit fails the job,
-# so a BLOCKED gate blocks the merge.
-#
-# WHY THIS STEP DECLARES GATE_STRICT_SKIPS AND TOLERATES EXIT 3 (2026-09-13). A SKIPPED stage is
-# never green, and since 2026-09-13 gate.sh refuses to print PASS when one did not run (it used to
-# print "— not green" and then "MERGE GATE: PASS", exit 0 — fail-open in the gate itself). Two
-# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
-# environment it is:
-#   * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
-#     installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
-#     "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
-#   * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
-#     not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
-#     is the ONLY skip that can still reach exit 3 under strict skips, which is what makes
-#     `|| [ "$?" -eq 3 ]` a narrow, reviewed tolerance rather than a blanket one. The log still
-#     says UNDETERMINED and names R-TEST-06 as unverified. Wire the spec checkout (or set
-#     REQUIRE_SPEC_VERIFICATION=true) and this tolerance should be deleted.
-# Called DIRECTLY, not through `make`: GNU make reports any recipe failure as its own exit 2, so
-# `make -C ci gate || [ "$?" -eq 3 ]` would see 2 and fail the job. Same reason the
-# spec-immutability pre-step below calls the script directly.
+# Docker executor + DinD). It invokes the AGGREGATE merge gate — `make -C ci gate` — so CI
+# runs BYTE-FOR-BYTE the same command a human runs in the devcontainer (R-TEST-02): the
+# runner just installs the pinned tools from ci/versions.env first. `make -C ci gate`
+# (ci/gate.sh) runs every stage in order — spec-immutability, lint, Kyverno L0,
+# render==golden, L1 offline render — and applies the R-TEST-03 decision, printing
+# "MERGE GATE: PASS" / "MERGE GATE: BLOCKED". It is the single required status check on
+# `main` (see branch protection). A non-zero exit fails the job, so a BLOCKED gate blocks
+# the merge.
 #
 # The cluster-gated L2/L3 conformance stage SKIPS honestly here (exit 3, surfaced not faked
-# — HARD RULE 4) because this runner declares no cluster: nothing here sets CLUSTER_AVAILABLE,
-# so gate.sh reports the touched requirements UNVERIFIED — and NAMES that declared gap in its
-# PASS headline — without blocking Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
+# — HARD RULE 4) because this runner declares no cluster: `make -C ci gate` does not set
+# CLUSTER_AVAILABLE, so gate.sh reports the touched requirements UNVERIFIED without blocking
+# Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
 # no broad kubeconfig) driven by the separate `conformance` job below.
 #
 # NOTE (R-TEST-06 / gate-semantics): spec-immutability is ALSO surfaced as its own explicit
 # early step (below) so its result is a distinct, tolerant status in the Gitea UI; gate.sh
-# re-runs it as its first stage, which is the authoritative one. The gate step below is the
-# same script the laptop runs; the extra pre-step is a read-only git-diff check and is
+# re-runs it as its first stage, which is the authoritative one. `make -C ci gate` here is
+# the byte-identical laptop gate; the extra pre-step is a read-only git-diff check and is
 # purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
 # is retained dormant in merge-gate.yaml.
 ---
@@ -88,17 +69,13 @@ jobs:
       - name: Spec immutability (R-TEST-06, skips if spec repo absent)
         run: ./ci/spec-immutability.sh || [ "$?" -eq 3 ]
 
-      # Aggregate merge gate: the same script a human runs in the devcontainer via
-      # `make -C ci gate`. Runs spec-immutability, lint, Kyverno L0, render==golden and L1
-      # render, then the R-TEST-03 decision. Exits non-zero on a BLOCKED gate, which fails this
-      # job. The cluster-gated L2/L3 stage SKIPS here (no CLUSTER_AVAILABLE) — surfaced as
-      # UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
-      # tolerated exit 3 are explained in the file header; the tolerance covers exactly one
-      # thing, the unverifiable spec repo, and nothing else can produce a 3 here.
-      - name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
-        env:
-          GATE_STRICT_SKIPS: "true"
-        run: ./ci/gate.sh || [ "$?" -eq 3 ]
+      # Aggregate merge gate: byte-for-byte the command a human runs in the devcontainer
+      # (`make -C ci gate` -> ci/gate.sh). Runs spec-immutability, lint, Kyverno L0,
+      # render==golden and L1 render, then the R-TEST-03 decision. Exits non-zero on a
+      # BLOCKED gate, which fails this job. The cluster-gated L2/L3 stage SKIPS here (no
+      # CLUSTER_AVAILABLE) — surfaced as UNVERIFIED, never faked green (see header).
+      - name: Merge gate — make -C ci gate
+        run: make -C ci gate
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
       # ("not supported on GHES" — Gitea does not implement the artifact-v2 backend), which

.gitea/workflows/scripts/install-tools.sh

@@ -39,21 +39,6 @@ install_chainsaw() {
   curl -fsSL "$url" | tar -xz -C "$bindir" chainsaw
 }
 
-# promtool — the alert-rule UNIT-TEST engine the L0 check
-# conformance/tests/l0-static/alert-rules-unit-tested runs. Installed for the l0 stage because
-# without it that check exits 3 (SKIPPED, never green) on the runner and the alert unit tests
-# would be enforced nowhere — the exact "coverage that looks present" failure it exists to end.
-# Pinned from ci/versions.env to the Prometheus version prod runs (promtool IS the rule engine);
-# the check asserts `promtool --version` equals that pin, so a drifted install FAILS, not skips.
-install_promtool() {
-  have promtool && return 0
-  log "promtool ${PROMTOOL_VERSION} (from the prometheus release tarball)"
-  local tarball="prometheus-${PROMTOOL_VERSION}.${os}-${arch}"
-  curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/${tarball}.tar.gz" \
-    | tar -xz -C "$bindir" --strip-components=1 "${tarball}/promtool"
-  chmod +x "${bindir}/promtool"
-}
-
 install_kubectl() {
   have kubectl && return 0
   log "kubectl ${KUBECTL_VERSION}"
@@ -65,10 +50,10 @@ have() { command -v "$1" >/dev/null 2>&1; }
 
 case "$stage" in
   lint)        install_yamllint ;;
-  l0)          install_yamllint; install_check_jsonschema; install_kyverno; install_promtool ;;  # schema (check-jsonschema) + Kyverno L0 + alert unit tests
+  l0)          install_yamllint; install_check_jsonschema; install_kyverno ;;  # schema (check-jsonschema) + Kyverno L0
   l1)          install_chainsaw ;;              # L1 render via conformance runner (Chainsaw)
   conformance) install_chainsaw; install_kubectl ;;   # trigger the in-cluster Job
-  all)         install_yamllint; install_check_jsonschema; install_kyverno; install_promtool; install_chainsaw; install_kubectl ;;
+  all)         install_yamllint; install_check_jsonschema; install_kyverno; install_chainsaw; install_kubectl ;;
   *) echo "unknown stage: $stage" >&2; exit 2 ;;
 esac
 log "stage '${stage}' tools ready"

.standup-dns/dns-before.txt

@@ -1,57 +0,0 @@
-A|*.keithjackson.dev|136.51.115.252
-A|auth.home.keithjackson.dev|192.168.1.53
-A|brickbreak.keithjackson.dev|20.236.242.14
-A|crafty-mc.home.keithjackson.dev|192.168.1.55
-A|crafty.home.keithjackson.dev|192.168.1.53
-A|dev.brickbreak.keithjackson.dev|20.236.242.14
-A|gitea.home.keithjackson.dev|192.168.1.53
-A|grafana.home.keithjackson.dev|192.168.1.53
-A|headlamp.home.keithjackson.dev|192.168.1.53
-A|home.keithjackson.dev|192.168.1.53
-A|jackett.home.keithjackson.dev|192.168.1.53
-A|keycloak.home.keithjackson.dev|192.168.1.53
-A|nextcloud.home.keithjackson.dev|192.168.1.53
-A|pihole.home.keithjackson.dev|192.168.1.53
-A|plex.home.keithjackson.dev|192.168.1.53
-A|qbittorrent.home.keithjackson.dev|192.168.1.53
-A|radarr.home.keithjackson.dev|192.168.1.53
-A|seerr.home.keithjackson.dev|192.168.1.53
-A|semaphore.home.keithjackson.dev|192.168.1.53
-A|sonarr.home.keithjackson.dev|192.168.1.53
-A|unifi.home.keithjackson.dev|192.168.1.54
-TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
-TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
-TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
-TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
-TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
-TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
-TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
-TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
-TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
-TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
-TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
-TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
-TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
-TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
-TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
-TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
-TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
-TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
-TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
-TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
-TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
-TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
-TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
-TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
-TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
-TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
-TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
-TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
-TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
-TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
-TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
-TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
-TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
-TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
-TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
-TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp

.standup-dns/dns-post-l2.txt

@@ -1,95 +0,0 @@
-A|*.keithjackson.dev|136.51.115.252
-A|auth.home.keithjackson.dev|192.168.1.53
-A|auth.prod.home.keithjackson.dev|192.168.2.20
-A|brickbreak.keithjackson.dev|20.236.242.14
-A|conformance-dns.home.keithjackson.dev|192.168.1.56
-A|crafty-mc.home.keithjackson.dev|192.168.1.55
-A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
-A|crafty.home.keithjackson.dev|192.168.1.53
-A|crafty.prod.home.keithjackson.dev|192.168.2.20
-A|dev.brickbreak.keithjackson.dev|20.236.242.14
-A|gitea.home.keithjackson.dev|192.168.1.53
-A|gitea.prod.home.keithjackson.dev|192.168.2.20
-A|grafana.home.keithjackson.dev|192.168.1.53
-A|grafana.prod.home.keithjackson.dev|192.168.2.20
-A|headlamp.home.keithjackson.dev|192.168.1.53
-A|headlamp.prod.home.keithjackson.dev|192.168.2.20
-A|home.keithjackson.dev|192.168.1.53
-A|jackett.home.keithjackson.dev|192.168.1.53
-A|jackett.prod.home.keithjackson.dev|192.168.2.20
-A|keycloak.home.keithjackson.dev|192.168.1.53
-A|keycloak.prod.home.keithjackson.dev|192.168.2.20
-A|nextcloud.home.keithjackson.dev|192.168.1.53
-A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
-A|pihole.home.keithjackson.dev|192.168.1.53
-A|pihole.prod.home.keithjackson.dev|192.168.2.20
-A|plex.home.keithjackson.dev|192.168.1.53
-A|plex.prod.home.keithjackson.dev|192.168.2.20
-A|prod.home.keithjackson.dev|192.168.2.20
-A|qbittorrent.home.keithjackson.dev|192.168.1.53
-A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
-A|radarr.home.keithjackson.dev|192.168.1.53
-A|radarr.prod.home.keithjackson.dev|192.168.2.20
-A|seerr.home.keithjackson.dev|192.168.1.53
-A|seerr.prod.home.keithjackson.dev|192.168.2.20
-A|semaphore.home.keithjackson.dev|192.168.1.53
-A|semaphore.prod.home.keithjackson.dev|192.168.2.20
-A|sonarr.home.keithjackson.dev|192.168.1.53
-A|sonarr.prod.home.keithjackson.dev|192.168.2.20
-A|unifi.home.keithjackson.dev|192.168.1.54
-A|unifi.prod.home.keithjackson.dev|192.168.2.21
-TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
-TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
-TXT|a-conformance-dns.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/chainsaw-growing-grouse/dns-probe-svc
-TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
-TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
-TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
-TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
-TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
-TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
-TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
-TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
-TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
-TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
-TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
-TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
-TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
-TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
-TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
-TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
-TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
-TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
-TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
-TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
-TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
-TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
-TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
-TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
-TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
-TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
-TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
-TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
-TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
-TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
-TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
-TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
-TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
-TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
-TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
-TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
-TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
-TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
-TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
-TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
-TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
-TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
-TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
-TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
-TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
-TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
-TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
-TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
-TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
-TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
-TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
-TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp

.standup-dns/dns-verify.txt

@@ -1,93 +0,0 @@
-A|*.keithjackson.dev|136.51.115.252
-A|auth.home.keithjackson.dev|192.168.1.53
-A|auth.prod.home.keithjackson.dev|192.168.2.20
-A|brickbreak.keithjackson.dev|20.236.242.14
-A|crafty-mc.home.keithjackson.dev|192.168.1.55
-A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
-A|crafty.home.keithjackson.dev|192.168.1.53
-A|crafty.prod.home.keithjackson.dev|192.168.2.20
-A|dev.brickbreak.keithjackson.dev|20.236.242.14
-A|gitea.home.keithjackson.dev|192.168.1.53
-A|gitea.prod.home.keithjackson.dev|192.168.2.20
-A|grafana.home.keithjackson.dev|192.168.1.53
-A|grafana.prod.home.keithjackson.dev|192.168.2.20
-A|headlamp.home.keithjackson.dev|192.168.1.53
-A|headlamp.prod.home.keithjackson.dev|192.168.2.20
-A|home.keithjackson.dev|192.168.1.53
-A|jackett.home.keithjackson.dev|192.168.1.53
-A|jackett.prod.home.keithjackson.dev|192.168.2.20
-A|keycloak.home.keithjackson.dev|192.168.1.53
-A|keycloak.prod.home.keithjackson.dev|192.168.2.20
-A|nextcloud.home.keithjackson.dev|192.168.1.53
-A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
-A|pihole.home.keithjackson.dev|192.168.1.53
-A|pihole.prod.home.keithjackson.dev|192.168.2.20
-A|plex.home.keithjackson.dev|192.168.1.53
-A|plex.prod.home.keithjackson.dev|192.168.2.20
-A|prod.home.keithjackson.dev|192.168.2.20
-A|qbittorrent.home.keithjackson.dev|192.168.1.53
-A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
-A|radarr.home.keithjackson.dev|192.168.1.53
-A|radarr.prod.home.keithjackson.dev|192.168.2.20
-A|seerr.home.keithjackson.dev|192.168.1.53
-A|seerr.prod.home.keithjackson.dev|192.168.2.20
-A|semaphore.home.keithjackson.dev|192.168.1.53
-A|semaphore.prod.home.keithjackson.dev|192.168.2.20
-A|sonarr.home.keithjackson.dev|192.168.1.53
-A|sonarr.prod.home.keithjackson.dev|192.168.2.20
-A|unifi.home.keithjackson.dev|192.168.1.54
-A|unifi.prod.home.keithjackson.dev|192.168.2.21
-TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
-TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
-TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
-TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
-TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
-TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
-TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
-TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
-TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
-TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
-TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
-TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
-TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
-TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
-TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
-TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
-TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
-TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
-TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
-TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
-TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
-TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
-TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
-TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
-TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
-TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
-TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
-TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
-TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
-TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
-TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
-TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
-TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
-TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
-TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
-TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
-TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
-TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
-TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
-TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
-TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
-TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
-TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
-TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
-TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
-TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
-TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
-TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
-TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
-TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
-TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
-TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
-TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
-TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp

README.md

@@ -38,56 +38,22 @@ ci/             the gate as scripts (lint · schema · Kyverno L0 · L1 render
 
 ## Status
 
-Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in
-progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the
-apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration:
-dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
-hardware rather than on Proxmox VMs.
-
-**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring +
-alerts live*:
-
-| criterion | state |
-|---|---|
-| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
-| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
-| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only |
-
-**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 —
-the first requirement in this repo to reach that state. Both of its verifiers are recorded:
-`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`)
-and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn`
-class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy**
-(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
-*different* PersistentVolume — so an in-cluster survival cannot pass as a restore.
-
-**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned
-dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
-exercise rather than something to run in passing.
-
-Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a
-bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
+Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
+open** — `clusters/prod` is a design-only overlay.
 
 | Phase | Delivered |
 |---|---|
 | 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
-| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
-| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
+| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
+| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
 | 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
-| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
+| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
 | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
 | 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
-| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
 
-**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
-2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with
-`./conformance/run.sh --coverage-only`; the full report is
-`conformance/_artifacts/coverage/coverage.md`.
-
-Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and
-*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current
-SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the
-second.
+**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
+(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
+the full report is `conformance/_artifacts/coverage/coverage.md`.
 
 The running narrative — decisions, root causes, incidents, follow-ups — is
 [`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.

apps/vms/unifi/50-service-l4.yaml

@@ -21,61 +21,8 @@
 # k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
 #
 # selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
-# into the guest.
-#
-# -------------------------------------------------------------------------------------------
-# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
-# -------------------------------------------------------------------------------------------
-# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
-# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
-# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
-#
-# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
-#    per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
-#    matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
-#    MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
-#    `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
-#    announcing node that holds no backend has nowhere to send the packet and answers ICMP port
-#    unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
-#    (the leases were 8 days old); the operator's controller UI and device inform path were down.
-#    It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
-#    deleting the Lease forces a re-election — and that re-election is random, so it can pick a
-#    backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
-#    reference implementation this file now follows.
-#
-# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
-#    interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
-#    SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
-#    controller has always seen the pod-side masquerade gateway as the source, never the LAN
-#    client. There is a SECOND translation below the Service, and it already destroyed the very
-#    thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
-#    it is giving up an address the guest could not see in the first place.
-#
-#    Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
-#    MAC (and the device reports its own IP inside the inform payload), not the packet source,
-#    and guest-portal authorization runs on the gateway appliance, not on this controller.
-#
-# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
-# is what makes this survive a reschedule and a rebuild.
-#
-# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
-# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
-# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
-# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
-# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
-# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
-# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
-# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
-# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
-#
-# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
-# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
-# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
-# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
-# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
-# fix would be a NEW restriction that could silently break a remote site's device informing over
-# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
-# change, made deliberately and tested against the device fleet.
+# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
+# VIP pinned to the node running the VM (also what L2 announcement does).
 #
 # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
 # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
@@ -102,7 +49,7 @@ metadata:
     external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Cluster
+  externalTrafficPolicy: Local
   selector:
     kubevirt.io/vm: unifi
   ports:
@@ -125,7 +72,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Cluster
+  externalTrafficPolicy: Local
   selector:
     kubevirt.io/vm: unifi
   ports:

apps/wave-0/gitea.yaml

@@ -170,27 +170,6 @@ spec:
       - port: '3000'
         protocol: TCP
 ---
-# CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): gitea-allow-flux-ingress
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: gitea-allow-flux-ingress
-  namespace: gitea
-  labels:
-    app.kubernetes.io/name: gitea
-spec:
-  endpointSelector:
-    matchLabels:
-      app.kubernetes.io/name: gitea
-  ingress:
-  - fromEndpoints:
-    - matchLabels:
-        k8s:io.kubernetes.pod.namespace: flux-system
-    toPorts:
-    - ports:
-      - port: '3000'
-        protocol: TCP
----
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): gitea-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -238,6 +217,26 @@ spec:
     - name: gitea
       port: 3000
 ---
+# ServiceMonitor (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: gitea
+  namespace: gitea
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: gitea
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: gitea
+  namespaceSelector:
+    matchNames:
+    - gitea
+  endpoints:
+  - port: metrics
+    interval: 30s
+---
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -252,11 +251,11 @@ spec:
   - name: gitea.slis
     rules:
     - alert: gitea-down
-      expr: kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"} == 0 or absent(kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"})
+      expr: up{job="gitea"} == 0
       for: 5m
       labels:
         severity: critical
         service: gitea
       annotations:
         summary: gitea is down
-        description: 'gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'gitea has been unreachable for 5m (SLI: up).'

apps/wave-0/jackett.yaml

@@ -74,7 +74,7 @@ spec:
         homelab.keithjackson.dev/auth-mode: forward-auth
       annotations:
         homelab.keithjackson.dev/auth-mode: forward-auth
-        homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+        homelab.keithjackson.dev/auth-groups: media-users
     spec:
       containers:
       - name: jackett
@@ -83,7 +83,7 @@ spec:
         - containerPort: 9117
         volumeMounts:
         - name: config
-          mountPath: /config
+          mountPath: /data/config
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -97,7 +97,6 @@ spec:
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
-        - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
         - --cookie-domain=.${BASE_DOMAIN}
@@ -267,7 +266,7 @@ metadata:
     app.kubernetes.io/name: jackett
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
-    homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    homelab.keithjackson.dev/auth-groups: media-users
 spec:
   parentRefs:
   - name: shared
@@ -284,6 +283,26 @@ spec:
     - name: jackett
       port: 4180
 ---
+# ServiceMonitor (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: jackett
+  namespace: jackett
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: jackett
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: jackett
+  namespaceSelector:
+    matchNames:
+    - jackett
+  endpoints:
+  - port: metrics
+    interval: 30s
+---
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -298,11 +317,11 @@ spec:
   - name: jackett.slis
     rules:
     - alert: jackett-down
-      expr: kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"} == 0 or absent(kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"})
+      expr: up{job="jackett"} == 0
       for: 5m
       labels:
         severity: critical
         service: jackett
       annotations:
         summary: jackett is down
-        description: 'jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'jackett has been unreachable for 5m (SLI: up).'

apps/wave-0/kustomization.yaml

@@ -11,17 +11,13 @@ resources:
 - jackett-oauth2-proxy-credentials.sops.yaml
 - jackett.yaml
 - nextcloud-db-app.sops.yaml
-- nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
-- pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex.yaml
-- qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
 - qbittorrent.yaml
 - semaphore-db-app.sops.yaml
-- semaphore-db-backup-credentials-prod.sops.yaml
 - semaphore-db-backup-credentials.sops.yaml
 - semaphore-db.yaml

apps/wave-0/nextcloud-db-backup-credentials-prod.sops.yaml

@@ -1,29 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-    name: nextcloud-db-backup-credentials-prod
-    namespace: nextcloud-db
-type: Opaque
-stringData:
-    ACCESS_KEY_ID: ENC[AES256_GCM,data:4lx6et4HcXDegulyIoa7jZR/lVgapGPEkukajfCKjvk=,iv:p6V7tizbikk55tOvpPOsPHTe/p52QLX5A5xrwuB14rA=,tag:oNJuipT/Lb6zynJ2K2Q71A==,type:str]
-    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:V5bHPoPeBOVyLp1n1Jilyso4i5P2aPSH0hrryfweg2q/SctWZxRbLBQ/FLzOigbJsAhlwOGu069mQPma6xjfdw==,iv:8u4UOvI8dC/cm6mEWDlyN5E3SfRZmwKY9uZY/DjPEsQ=,tag:tw7tGs29TmUvMPBCtCszAA==,type:str]
-sops:
-    kms: []
-    gcp_kms: []
-    azure_kv: []
-    hc_vault: []
-    age:
-        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiSUt5N2k2N2lDU2ZsMy9m
-            M1NFS0M2QjFsNFhnR0hCMzFyaTdiQWFZRFJZCmhBTWRXNmxLMGtsc3Rxd2FSMDZp
-            aktjcTE2c1hmU1EzM3RWNksyb3Q1YVUKLS0tIDBFS1I5aDBER2cvcWtLZ3BXTEp5
-            ZGI5ZmZIcStheUNQamxIVVVpc0ZMOHMKlE5aHewtHJpeU24F5sodc9CEm9KTjHQQ
-            8ThEQ/8vyazL9V4ojlICxSHU/SHkHpD1eBaTKcAQd/LZzujIcJ+QfA==
-            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-09-07T06:36:33Z"
-    mac: ENC[AES256_GCM,data:8awz1Jh8t4zjDMyd93/v+6on9iiIaSi01293A7uYhxj9L1+g63zeO+DPmSn6+KbZvHdqU2A7up09H1NzfXAEdGuvVbB36cdGq0zKank3PUi8c7IT6qNuWydxRAcqLnbbD+c30Z6GB+gAuZHstNOe4Rgc4aJvMN6FuP6x6Z0870c=,iv:KGGKJq2QA5xztvoHs/SViXW72DYwT16i+W+dJBUeIJQ=,tag:3UpYqx2+RIK0QdMH/BSIpA==,type:str]
-    pgp: []
-    encrypted_regex: ^(data|stringData)$
-    version: 3.9.4

apps/wave-0/nextcloud-db.yaml

@@ -37,53 +37,16 @@ metadata:
   name: nextcloud-db
   namespace: nextcloud-db
 spec:
-  # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db; see
-  # platform/keycloak/postgres-cluster.yaml for the full rationale. Short version: Longhorn's 3-way
-  # replication saves the DATA on node loss, not the SERVICE, and with `instances: 1` all three CNPG
-  # primaries had drifted onto k3s-prod-2, so one node took Nextcloud down alongside SSO and
-  # Semaphore. A streaming standby lets CNPG promote in seconds and repoint the -rw Service.
-  # Gated on the Barman archive below actually working (ContinuousArchiving=True with a real
-  # firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
-  #
-  # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
-  # pg_basebackup from the running primary, which is neither restarted nor disconnected.
-  instances: 3
+  # Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
+  # keycloak-db). The storage is already 3-way replicated by Longhorn; raise to 3 for Postgres-level
+  # HA as a future hardening.
+  instances: 1
   imageName: ghcr.io/cloudnative-pg/postgresql:16.4
   # Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
   # nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
   inheritedMetadata:
     labels:
       app.kubernetes.io/name: nextcloud-db
-  # HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
-  # primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
-  # recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
-  # version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
-  # after a node loss it may stack two instances on one node — which is how `instances: 3` decays
-  # back into the single-node failure domain this change removes. `required` makes it a constraint;
-  # the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
-  # serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
-  #
-  # APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
-  # switchover. Expect a single brief connection reset for Nextcloud.
-  affinity:
-    enablePodAntiAffinity: true
-    podAntiAffinityType: required
-  primaryUpdateMethod: switchover
-  # SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
-  # rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
-  # last few transactions when a primary dies before its standbys catch up, and `method: any` +
-  # `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
-  # never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
-  # are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
-  # standby into a total write outage. `preferred` relaxes to async instead.
-  #
-  # NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
-  # no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
-  postgresql:
-    synchronous:
-      method: any
-      number: 1
-      dataDurability: preferred
   storage:
     storageClass: longhorn
     size: 5Gi
@@ -105,38 +68,25 @@ spec:
       memory: 256Mi
     limits:
       memory: 512Mi
-  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
+  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
   # Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
-  # backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
-  # destinationPath, so objects land under ${CNPG_BACKUP_PATH}/nextcloud-db/ — dev
-  # s3://cnpg/nextcloud-db/ on the on-Proxmox MinIO, prod
-  # s3://k3s-prod-backup/barman/nextcloud-db/ on Cloudflare R2. Both endpointURL and
-  # destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
-  # Credentials: PER-ENVIRONMENT (2026-09-07). dev reads nextcloud-db-backup-credentials (the
-  # scoped MinIO key), prod reads nextcloud-db-backup-credentials-prod (the Cloudflare R2
-  # token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
-  # generated golden — render.py places the -prod sidecar in this DB's wave); the
-  # cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
-  # one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
-  # ONE shared Secret until the split, so giving prod its R2 token would have left dev
-  # signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
-  # disk-full clock below rather than a merely missing backup.
-  #
-  # archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
-  # Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
-  # the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
-  # endpoint. Treat ContinuousArchiving=False here as an availability incident.
+  # backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
+  # (defaults to the Cluster name), so objects land under s3://cnpg/nextcloud-db/. Path-style HTTP.
+  # Credentials: the scoped key from the in-namespace SOPS Secret nextcloud-db-backup-credentials
+  # (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
+  # endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
+  # substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
   backup:
     retentionPolicy: "30d"
     barmanObjectStore:
-      destinationPath: ${CNPG_BACKUP_PATH}
+      destinationPath: s3://cnpg
       endpointURL: ${S3_BACKUP_ENDPOINT}
       s3Credentials:
         accessKeyId:
-          name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
+          name: nextcloud-db-backup-credentials
           key: ACCESS_KEY_ID
         secretAccessKey:
-          name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
+          name: nextcloud-db-backup-credentials
           key: ACCESS_SECRET_KEY
       wal:
         compression: gzip
@@ -158,243 +108,3 @@ spec:
   cluster:
     name: nextcloud-db
   method: barmanObjectStore
----
-# =============================================================================
-# CNPG METRICS-EXPORTER CONNECT RECONCILER (R-OBS-01)
-#
-# THE BUG THIS CLOSES (true positive, verified live on prod 2026-09-12, and the
-# condition platform/cnpg/alerts.yaml already recorded on BOTH clusters 2026-09-06):
-# the `nextcloud` database's ACL had been narrowed out-of-band so that PUBLIC held
-# nothing, and CNPG's metrics exporter role therefore could not even CONNECT:
-#
-#   nextcloud-db-1/-2/-3, every scrape:
-#     FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
-#     DETAIL: User does not have CONNECT privilege.
-#     msg="Error collecting user query" targetDatabase=nextcloud
-#
-# CONSEQUENCE: `cnpg_last_error` = 1 on all three instances and the ENTIRE
-# cnpg_pg_stat_* / pg_settings / pg_postmaster family MISSING for this database,
-# while the scrape target still reported up=1 — a blind spot that looks like health.
-# Measured before the fix: cnpg_pg_settings_setting had 765 series for keycloak and
-# 765 for semaphore-db and ZERO for nextcloud-db. Nextcloud's database had never
-# been monitored.
-#
-# ROOT CAUSE, AND WHY ONLY THIS DATABASE. All three Cluster specs are identical in
-# their bootstrap; the difference is entirely in the live ACL:
-#
-#   nextcloud : datacl = {nextcloud=CTc/nextcloud}          <-- explicit, no PUBLIC entry
-#   keycloak  : datacl = NULL
-#   semaphore : datacl = NULL
-#
-# A NULL datacl means "PostgreSQL defaults", which include CONNECT and TEMP for
-# PUBLIC — so keycloak and semaphore let the exporter in only by inheriting PUBLIC's
-# implicit CONNECT. On `nextcloud` something ran the equivalent of
-# `REVOKE ALL ON DATABASE nextcloud FROM PUBLIC`, which MATERIALISES datacl and
-# drops the implicit PUBLIC entry with it. That REVOKE is not in this repo and not
-# in DECISIONS-LOG.md — it was applied out of band, which is exactly why a
-# git-declared fix is needed rather than another hand-run psql.
-#
-# WHY THE GRANT IS EXPLICIT RATHER THAN "RE-GRANT CONNECT TO PUBLIC": restoring
-# PUBLIC's CONNECT would undo whatever hardening that REVOKE intended and would
-# leave the exporter depending on an implicit default again — the precise fragility
-# that broke here and that keycloak/semaphore are still relying on. Naming the
-# exporter role makes the dependency explicit and survives a future REVOKE FROM
-# PUBLIC on any of the three databases.
-#
-# WHY A CronJob AND NOT A CNPG FIELD — CNPG 1.30.0's API GENUINELY CANNOT EXPRESS
-# THIS. Checked against the live CRDs on prod:
-#   * `Database` (databases.postgresql.cnpg.io) has no database-ACL field at all.
-#     `allowConnections` is the pg_database.datallowconn FLAG (open/closed to
-#     everyone), not a per-role GRANT; its only grant/revoke verbs are the `usage`
-#     lists on `fdws` and `servers`.
-#   * `spec.managed.roles` and `DatabaseRole` manage ROLE ATTRIBUTES (login,
-#     superuser, connectionLimit, ...) plus `inRoles` MEMBERSHIP. Membership is the
-#     wrong tool: the only local role holding CONNECT here is the OWNER, so
-#     `inRoles: [nextcloud]` would hand the exporter the owner's CREATE and TEMP on
-#     the database and read/write on every application table — the opposite of
-#     least privilege, to buy one CONNECT.
-#   * `bootstrap.initdb.postInitApplicationSQL` runs ONCE, at bootstrap. It cannot
-#     repair these three already-bootstrapped instances (the actual bug), and on a
-#     FRESH cluster the grant is unnecessary anyway (datacl starts NULL). Adding it
-#     would be a second mechanism that fixes nothing this one does not.
-# So the SQL has to be applied by something, and this repo already has the pattern
-# for state Flux cannot reach: the level-triggered CronJob reconciler in
-# platform/keycloak/client-secret-reconciler.yaml, whose reasoning transfers
-# verbatim — the drifted state lives inside PostgreSQL, not in a Kubernetes object,
-# so no admission or mutation policy can see it. A one-shot Job is deliberately NOT
-# used: a completed Job is never re-run, so a DB rebuild or a PITR restore would
-# silently reopen the blind spot. A loop converges instead.
-#
-# PRIVILEGE GRANTED — EXACTLY ONE, AND NOTHING ELSE:
-#   GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
-# Resulting ACL entry is `cnpg_metrics_exporter=c/nextcloud` — the `c` is CONNECT;
-# there is no `C` (CREATE) and no `T` (TEMPORARY). No ownership, no superuser, no
-# write, no table or schema grants. Nothing further is needed because CNPG already
-# creates this role as a member of the built-in `pg_monitor` role (verified on all
-# three prod clusters), and pg_monitor already carries the read access every
-# collector query wants — they read pg_catalog views (pg_stat_database,
-# pg_stat_archiver, pg_settings, pg_postmaster) and no application tables. Proof
-# that CONNECT alone is sufficient: after this grant, `Error collecting user query`
-# went to ZERO on all three pods, i.e. every collector query now succeeds.
-#
-# IT RUNS AS THE DATABASE OWNER, NOT AS A SUPERUSER. `enableSuperuserAccess` is off
-# on this Cluster (there is no nextcloud-db-superuser Secret), and it stays off: the
-# owner of a database holds the grant option on that database's privileges, so
-# `nextcloud` can issue this GRANT itself. Verified live before committing, by
-# running the statement under `SET ROLE nextcloud`. Credentials come from the
-# in-namespace SOPS Secret nextcloud-db-app that already exists for the app.
-#
-# NETWORKING: nextcloud-db is intentionally NOT an app-namespace (see this file's
-# header), so Kyverno generates no default-deny baseline here and this same-namespace
-# pod -> nextcloud-db-rw:5432 connection needs no rendered allow.
-#
-# THE POD LABEL IS DELIBERATELY *NOT* app.kubernetes.io/name=nextcloud-db. That
-# label is what `inheritedMetadata` stamps on the CNPG instance pods and what the
-# rendered nextcloud egress CiliumNetworkPolicy selects; reusing it here would quietly
-# make this reconciler a permitted destination for the Nextcloud app. It gets its own
-# name so that selector keeps meaning "the database".
-apiVersion: batch/v1
-kind: CronJob
-metadata:
-  name: nextcloud-db-exporter-grant
-  namespace: nextcloud-db
-  labels:
-    app.kubernetes.io/name: nextcloud-db-exporter-grant
-    app.kubernetes.io/part-of: nextcloud-db
-spec:
-  # Every 15 minutes, matching the keycloak reconciler's cadence. A converged
-  # database makes this one idempotent GRANT and exits; the cost is negligible, and
-  # after a rebuild or PITR restore monitoring is blind for at most one interval
-  # (well inside the alert's `for: 6h`).
-  schedule: "*/15 * * * *"
-  # A run still waiting on the primary must not be lapped by the next tick.
-  concurrencyPolicy: Forbid
-  startingDeadlineSeconds: 300
-  successfulJobsHistoryLimit: 1
-  failedJobsHistoryLimit: 3
-  jobTemplate:
-    spec:
-      backoffLimit: 2
-      activeDeadlineSeconds: 300
-      ttlSecondsAfterFinished: 3600
-      template:
-        metadata:
-          labels:
-            app.kubernetes.io/name: nextcloud-db-exporter-grant
-            app.kubernetes.io/part-of: nextcloud-db
-        spec:
-          restartPolicy: Never
-          # Talks only to PostgreSQL; its one input is a mounted Secret, so it needs
-          # no Kubernetes API access at all (R-SEC-02, least privilege).
-          automountServiceAccountToken: false
-          securityContext:
-            runAsNonRoot: true
-            # uid/gid 26 is `postgres` in the CNPG PostgreSQL image.
-            runAsUser: 26
-            runAsGroup: 26
-            seccompProfile:
-              type: RuntimeDefault
-          containers:
-            - name: grant
-              # The SAME image the instances above run, pinned to the same tag. psql
-              # is already in it and it is already present on every node, so this
-              # adds no new image and no new supply-chain surface. Keep the tag in
-              # step with spec.imageName above when Renovate bumps it.
-              image: ghcr.io/cloudnative-pg/postgresql:16.4
-              imagePullPolicy: IfNotPresent
-              command:
-                - psql
-                # -w never prompts; -X skips ~/.psqlrc; ON_ERROR_STOP makes the Job
-                # FAIL LOUDLY rather than reporting success on a refused grant.
-                - "-w"
-                - "-X"
-                - "-v"
-                - "ON_ERROR_STOP=1"
-                - "-f"
-                - /etc/grant/grant.sql
-              env:
-                # libpq connection, assembled from the owner Secret. Target the -rw
-                # Service: GRANT is a write and must reach the primary.
-                - name: PGHOST
-                  value: nextcloud-db-rw.nextcloud-db.svc.cluster.local
-                - name: PGPORT
-                  value: "5432"
-                # Connecting to the application database itself is deliberate: the
-                # owner has CONNECT on it, and a successful connection is the same
-                # check the exporter was failing.
-                - name: PGDATABASE
-                  value: nextcloud
-                - name: PGUSER
-                  valueFrom:
-                    secretKeyRef:
-                      name: nextcloud-db-app
-                      key: username
-                # Read by libpq from the environment and never logged; psql echoes
-                # no connection parameters.
-                - name: PGPASSWORD
-                  valueFrom:
-                    secretKeyRef:
-                      name: nextcloud-db-app
-                      key: password
-                - name: PGCONNECT_TIMEOUT
-                  value: "10"
-              volumeMounts:
-                - name: sql
-                  mountPath: /etc/grant
-                  readOnly: true
-              securityContext:
-                allowPrivilegeEscalation: false
-                readOnlyRootFilesystem: true
-                capabilities:
-                  drop:
-                    - ALL
-              resources:
-                requests:
-                  cpu: 10m
-                  memory: 32Mi
-                limits:
-                  memory: 128Mi
-          volumes:
-            - name: sql
-              configMap:
-                name: nextcloud-db-exporter-grant
-                defaultMode: 0444
----
-# The desired state, as SQL. Idempotent: re-granting an existing privilege is a
-# no-op, so a converged run changes nothing. The DO block is the ASSERTION — it
-# turns "the grant did not take" into a failed Job instead of a silent success.
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: nextcloud-db-exporter-grant
-  namespace: nextcloud-db
-  labels:
-    app.kubernetes.io/name: nextcloud-db-exporter-grant
-    app.kubernetes.io/part-of: nextcloud-db
-data:
-  grant.sql: |
-    -- The ONE privilege the CNPG metrics exporter is missing on this database.
-    -- CONNECT only: no CREATE, no TEMPORARY, no ownership, no table/schema grants.
-    -- Read access for the collector's pg_catalog queries already comes from the
-    -- built-in pg_monitor role, which CNPG grants this role when it creates it.
-    GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
-
-    -- Fail the Job if the privilege is somehow still absent, so a broken
-    -- reconciler surfaces as a failed Job rather than as a quiet blind spot.
-    -- NO PL/pgSQL, AND NOT ONE DOLLAR SIGN IN THIS FILE — DELIBERATE, including in these
-    -- comments. This assertion began life as a dollar-quoted DO block, which cannot survive
-    -- being shipped this way: the ConfigMap is rendered through Flux's postBuild
-    -- substitution, and envsubst collapses a doubled dollar into a single one. The block
-    -- reached Postgres with its opening quote halved and every Flux-reconciled run died
-    -- with a syntax error pointing at that character.
-    -- It PASSED when hand-applied with kubectl, because that path runs no envsubst — so
-    -- the defect was invisible until Flux owned the object, and three runs of Error pods
-    -- were the first sign. Escaping by doubling again would work and would also be the
-    -- next person's trap; plain SQL leaves substitution nothing to touch.
-    --
-    -- has_database_privilege() returns boolean; ::int is 1 when granted and 0 when not, so
-    -- the division raises division_by_zero and ON_ERROR_STOP=1 fails the Job. The divisor
-    -- is not a constant, so it cannot be folded away at plan time.
-    \echo 'asserting cnpg_metrics_exporter has CONNECT on database nextcloud'
-    SELECT 1 / has_database_privilege('cnpg_metrics_exporter', 'nextcloud', 'CONNECT')::int
-      AS exporter_can_connect;

apps/wave-0/pihole-dns.yaml

@@ -1,196 +0,0 @@
----
-# =================================================================================================
-# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY.
-# =================================================================================================
-# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole`
-# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS
-# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition
-# schema genuinely cannot express it: schemas/servicedefinition.schema.json is
-# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned
-# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at
-# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and
-# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this
-# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and
-# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog
-# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the
-# schema has no word for.
-#
-# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is
-# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service
-# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09
-# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on
-# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that
-# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0).
-#
-# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via
-# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
-# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the
-# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config
-# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is
-# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it
-# can never be allowed to move.
-#
-# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53
-# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same
-# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the
-# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a
-# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently
-# and confusingly.
-#
-# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
-# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC
-# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself
-# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS,
-# and publishing a resolver's address in a public zone invites exactly the traffic the source-range
-# filter below exists to refuse. Internal name, internal answer, no public record — which is also
-# why this change touches nothing under platform/externaldns/.
-#
-# -------------------------------------------------------------------------------------------------
-# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE.
-# -------------------------------------------------------------------------------------------------
-# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to
-# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured,
-# not assumed (Cilium 1.20.1, prod, 2026-09-14):
-#
-#   * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the
-#     CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does
-#     NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services
-#     landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3,
-#     `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1.
-#   * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the
-#     packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three
-#     nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol.
-#   * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod
-#     rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three
-#     minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a
-#     re-election — and that re-election is random, so it may pick a backend-less node again.
-#
-# A resolver that works one time in three and stays broken after any reschedule is not a resolver.
-# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is
-# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement
-# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all
-# answering.
-#
-# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in
-# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the
-# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client
-# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two
-# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole
-# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts
-# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity —
-# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode:
-# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract
-# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
-#
-# -------------------------------------------------------------------------------------------------
-# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
-# -------------------------------------------------------------------------------------------------
-# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
-# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
-# datapath — it never reaches a backend and never reaches a network policy. This is what makes
-# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
-# range to a /24 that excludes the test client, after which the same query that had been answering
-# stopped being answered at all, and restoring it, after which it answered again.
-#
-# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
-# restriction once the source has been rewritten: measured on this cluster, the forwarded query
-# arrives from the ingress node's internal router address and Cilium resolves that source to the
-# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the
-# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction
-# lives here, where it is actually enforceable, and the policy below restricts the PORTS.
-apiVersion: v1
-kind: Service
-metadata:
-  name: pihole-dns-udp
-  namespace: pihole
-  labels:
-    app.kubernetes.io/name: pihole
-  annotations:
-    lbipam.cilium.io/sharing-key: "pihole-dns"
-    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
-spec:
-  type: LoadBalancer
-  externalTrafficPolicy: Cluster
-  loadBalancerSourceRanges:
-    - ${LAN_CIDR}
-  selector:
-    app.kubernetes.io/name: pihole
-  ports:
-    - { name: dns-udp, port: 53, targetPort: 53, protocol: UDP }
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: pihole-dns-tcp
-  namespace: pihole
-  labels:
-    app.kubernetes.io/name: pihole
-  annotations:
-    lbipam.cilium.io/sharing-key: "pihole-dns"   # same key + same IP => shares the VIP with -udp
-    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
-spec:
-  type: LoadBalancer
-  externalTrafficPolicy: Cluster
-  loadBalancerSourceRanges:
-    - ${LAN_CIDR}
-  selector:
-    app.kubernetes.io/name: pihole
-  ports:
-    - { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP }
----
-# =================================================================================================
-# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports.
-# =================================================================================================
-# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered
-# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the
-# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything
-# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the
-# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule:
-# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that
-# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for
-# it and no policy for it.
-#
-# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is
-# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's
-# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium
-# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was
-# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced
-# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
-# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
-# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
-# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
-#
-# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
-# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
-# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none
-# needs PiHole on :53) on fewer ports, behind a filter those two do not have.
-#
-# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the
-# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to
-# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same
-# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not
-# an internet-facing open resolver. Say so out loud rather than implying the filter is total.
-#
-# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved
-# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the
-# node-to-node forwarding that IS the Cluster-policy datapath both need them.
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: pihole-allow-lan-dns-ingress
-  namespace: pihole
-  labels:
-    app.kubernetes.io/name: pihole
-spec:
-  endpointSelector:
-    matchLabels:
-      app.kubernetes.io/name: pihole
-  ingress:
-    - fromEntities:
-        - world         # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
-        - host          # this node's kubelet probes
-        - remote-node   # the node-to-node hop that forwards a query to wherever the pod runs
-      toPorts:
-        - ports:
-            - { port: "53", protocol: UDP }
-            - { port: "53", protocol: TCP }

apps/wave-0/pihole.yaml

@@ -81,15 +81,6 @@ spec:
         image: pihole/pihole
         ports:
         - containerPort: 80
-        env:
-        - name: FTLCONF_dns_listeningMode
-          value: ALL
-        - name: FTLCONF_dns_upstreams
-          value: 8.8.8.8;8.8.4.4
-        - name: FTLCONF_dns_rateLimit_count
-          value: '0'
-        - name: FTLCONF_misc_dnsmasq_lines
-          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
         volumeMounts:
         - name: config
           mountPath: /data/config
@@ -248,6 +239,26 @@ spec:
     - name: pihole
       port: 4180
 ---
+# ServiceMonitor (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: pihole
+  namespace: pihole
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: pihole
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  namespaceSelector:
+    matchNames:
+    - pihole
+  endpoints:
+  - port: metrics
+    interval: 30s
+---
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -262,11 +273,11 @@ spec:
   - name: pihole.slis
     rules:
     - alert: pihole-down
-      expr: kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
+      expr: up{job="pihole"} == 0
       for: 5m
       labels:
         severity: critical
         service: pihole
       annotations:
         summary: pihole is down
-        description: 'pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'pihole has been unreachable for 5m (SLI: up).'

apps/wave-0/plex.yaml

@@ -35,34 +35,6 @@ spec:
     - protocol: TCP
       port: 53
 ---
-# PersistentVolume plex-media - static external NFS (bulk tier)
-apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: plex-media
-  labels:
-    app.kubernetes.io/name: plex
-spec:
-  capacity:
-    storage: 9Ti
-  accessModes:
-  - ReadOnlyMany
-  persistentVolumeReclaimPolicy: Retain
-  storageClassName: ''
-  mountOptions:
-  - nfsvers=4.1
-  - hard
-  - noatime
-  - timeo=600
-  - retrans=2
-  nfs:
-    server: ${BULK_NFS_SERVER}
-    path: /mnt/Storage/plex
-    readOnly: true
-  claimRef:
-    namespace: plex
-    name: plex-media
----
 # PVC plex-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -79,23 +51,6 @@ spec:
     requests:
       storage: 8Gi
 ---
-# PVC plex-media (R-STATE)
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: plex-media
-  namespace: plex
-  labels:
-    app.kubernetes.io/name: plex
-spec:
-  accessModes:
-  - ReadOnlyMany
-  storageClassName: ''
-  volumeName: plex-media
-  resources:
-    requests:
-      storage: 9Ti
----
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -135,16 +90,10 @@ spec:
         volumeMounts:
         - name: config
           mountPath: /config
-        - name: media
-          mountPath: /media
-          readOnly: true
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: plex-config
-      - name: media
-        persistentVolumeClaim:
-          claimName: plex-media
 ---
 # Service
 apiVersion: v1
@@ -244,6 +193,26 @@ spec:
     - name: plex
       port: 32400
 ---
+# ServiceMonitor (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: plex
+  namespace: plex
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: plex
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  namespaceSelector:
+    matchNames:
+    - plex
+  endpoints:
+  - port: metrics
+    interval: 30s
+---
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -258,11 +227,11 @@ spec:
   - name: plex.slis
     rules:
     - alert: plex-down
-      expr: kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})
+      expr: up{job="plex"} == 0
       for: 5m
       labels:
         severity: critical
         service: plex
       annotations:
         summary: plex is down
-        description: 'plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'plex has been unreachable for 5m (SLI: up).'

apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml

@@ -1,60 +0,0 @@
-# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux.
-#
-# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13):
-#   qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR
-#   `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW
-#   random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
-#   password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's
-#   saved download-client credentials stop working at the next qbittorrent pod restart, silently
-#   breaking Completed Download Handling; (b) a human operator has no stable login at all and must
-#   read the pod log after every restart.
-#
-# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env
-#   (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as
-#   gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores
-#   and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then
-#   verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore
-#   from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME
-#   credential, because the hook re-asserts it on every start rather than once.
-#
-# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is
-#   the name its own temporary-password path uses, so keeping it avoids inventing a second identity.
-# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client
-#   entries store in their own databases (app state on their config PVCs, NOT in git — see
-#   DECISIONS-LOG), so rotating it here means re-pointing both arrs.
-#   Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON
-#   payloads, and a shell/URL-special character in any of those is a bug waiting to happen.
-#
-# NEVER put a value in a comment: SOPS does NOT encrypt comments.
-# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml
-apiVersion: v1
-kind: Secret
-metadata:
-    name: qbittorrent-bootstrap
-    namespace: qbittorrent
-    labels:
-        app.kubernetes.io/name: qbittorrent
-type: Opaque
-stringData:
-    webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str]
-    webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str]
-sops:
-    kms: []
-    gcp_kms: []
-    azure_kv: []
-    hc_vault: []
-    age:
-        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2
-            TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3
-            UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv
-            VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv
-            FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw==
-            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-09-13T05:23:23Z"
-    mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str]
-    pgp: []
-    encrypted_regex: ^(data|stringData)$
-    version: 3.9.4

apps/wave-0/qbittorrent.yaml

@@ -35,33 +35,6 @@ spec:
     - protocol: TCP
       port: 53
 ---
-# PersistentVolume qbittorrent-downloads - static external NFS (bulk tier)
-apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: qbittorrent-downloads
-  labels:
-    app.kubernetes.io/name: qbittorrent
-spec:
-  capacity:
-    storage: 9Ti
-  accessModes:
-  - ReadWriteMany
-  persistentVolumeReclaimPolicy: Retain
-  storageClassName: ''
-  mountOptions:
-  - nfsvers=4.1
-  - hard
-  - noatime
-  - timeo=600
-  - retrans=2
-  nfs:
-    server: ${BULK_NFS_SERVER}
-    path: /mnt/Storage/plex
-  claimRef:
-    namespace: qbittorrent
-    name: qbittorrent-downloads
----
 # PVC qbittorrent-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -78,23 +51,6 @@ spec:
     requests:
       storage: 5Gi
 ---
-# PVC qbittorrent-downloads (R-STATE)
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: qbittorrent-downloads
-  namespace: qbittorrent
-  labels:
-    app.kubernetes.io/name: qbittorrent
-spec:
-  accessModes:
-  - ReadWriteMany
-  storageClassName: ''
-  volumeName: qbittorrent-downloads
-  resources:
-    requests:
-      storage: 9Ti
----
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -118,43 +74,16 @@ spec:
         homelab.keithjackson.dev/auth-mode: forward-auth
       annotations:
         homelab.keithjackson.dev/auth-mode: forward-auth
-        homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+        homelab.keithjackson.dev/auth-groups: media-users
     spec:
       containers:
       - name: qbittorrent
         image: lscr.io/linuxserver/qbittorrent:5.1.2
         ports:
         - containerPort: 8080
-        env:
-        - name: PUID
-          value: '1000'
-        - name: PGID
-          value: '3000'
-        - name: QBT_WEBUI_USERNAME
-          valueFrom:
-            secretKeyRef:
-              name: qbittorrent-bootstrap
-              key: webui-username
-        - name: QBT_WEBUI_PASSWORD
-          valueFrom:
-            secretKeyRef:
-              name: qbittorrent-bootstrap
-              key: webui-password
         volumeMounts:
         - name: config
-          mountPath: /config
-        - name: downloads
-          mountPath: /downloads
-          subPath: downloads
-        lifecycle:
-          postStart:
-            exec:
-              command:
-              - /bin/bash
-              - -c
-              - "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n  echo \"[qbt-webui-cred] $*\"\n  printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n    >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n#     before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n#     racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n  attempt=$((attempt + 1))\n\n  # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n  #     10 = rewritten, anything else = could not write.\n  python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n    sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n    raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n#   WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
-                , \"[Preferences]\"\n\ndef matches(raw):\n    \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n    v = raw.strip().strip('\"')\n    if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n        v = v[len(\"@ByteArray(\"):-1]\n    try:\n        salt_b64, digest_b64 = v.split(\":\", 1)\n        salt = base64.b64decode(salt_b64, validate=True)\n        digest = base64.b64decode(digest_b64, validate=True)\n    except Exception:\n        return False\n    if not salt or not digest:\n        return False\n    return hmac.compare_digest(\n        hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n    lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n    lines = []\n\ndef value_of(key):\n    for i, line in enumerate(lines):\n        if line.startswith(key + \"=\"):\n            return i, line.split(\"=\", 1)[1]\n    return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n    raise SystemExit(0)          # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n    at = out.index(SECTION) + 1\nelse:\n    if out and out[-1].strip():\n        out.append(\"\")\n    out.append(SECTION)\n    at = len(out)\nout[at:at] = desired\n\ntry:\n    os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n    tmp = conf + \".webui-cred.new\"\n    with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n        fh.write(\"\\n\".join(out) + \"\\n\")\n        fh.flush()\n        os.fsync(fh.fileno())\n    # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n    # cost the app the ability to save its own settings, so carry the ownership over.\n    try:\n        st = os.stat(conf)\n        os.chown(tmp, st.st_uid, st.st_gid)\n        os.chmod(tmp, stat.S_IMODE(st.st_mode))\n    except FileNotFoundError:\n        os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n                 int(os.environ.get(\"PGID\", \"1000\")))\n        os.chmod(tmp, 0o664)\n    os.replace(tmp, conf)        # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n    sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n    raise SystemExit(1)\nraise SystemExit(10)\nPY\n  rc=$?\n\n  if [ \"$rc\" = 0 ]; then\n    log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n  elif [ \"$rc\" = 10 ]; then\n    log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n    # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n    # have read the old, credential-less conf and be serving a random temporary password.\n    # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n    # Nothing to do when the service has not started yet - it will read the new file.\n    # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n    # 2026-09-13 failure spent the entire verification budget before the first login and\n    # then fired attempts 2 and 3 with no wait left at all.\n    sv=0\n    while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n      sv=$((sv + 1)); sleep 1\n    done\n    if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n      log \"qbittorrent-nox was already running - restarting it to load the credential\"\n      s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n      sleep 3\n    fi\n  else\n    log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n    log \"minting a random password\
-                \ every start and BOTH arrs will break. Fix and restart.\" >&2\n    exit 0\n  fi\n\n  # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n  #     (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n  #     at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n  #\n  # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n  # conflating them is what made this hook lie once already (see the PORT comment). An\n  # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n  # its remaining login attempts to produce two more identical wrong diagnoses.\n  api_ok=0\n  while [ \"$waited\" -lt 120 ]; do\n    code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n    if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n    waited=$((waited + 1)); sleep 1\n  done\n  if [ \"$api_ok\" != 1 ]; then\n    log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n    log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n    log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n    exit 0\n  fi\n  body=$(curl -sS -m 10 \\\n           --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n           --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n           \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n  if [ \"$body\" = \"Ok.\" ]; then\n    log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n    exit 0\n  fi\n  log \"WebUI rejected the durable credential (login said '$body')\" >&2\n  # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n  # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n  # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n  # make the process RE-READ the file, so restart the supervised service and let the next\n  # attempt re-verify. Without this the retry loop could never change its own outcome.\n  if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n    log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n    s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n    sleep 3\n  fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
+          mountPath: /data/config
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -168,7 +97,6 @@ spec:
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
-        - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
         - --cookie-domain=.${BASE_DOMAIN}
@@ -220,15 +148,10 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
-      securityContext:
-        fsGroup: 3000
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: qbittorrent-config
-      - name: downloads
-        persistentVolumeClaim:
-          claimName: qbittorrent-downloads
 ---
 # Service
 apiVersion: v1
@@ -343,7 +266,7 @@ metadata:
     app.kubernetes.io/name: qbittorrent
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
-    homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    homelab.keithjackson.dev/auth-groups: media-users
 spec:
   parentRefs:
   - name: shared
@@ -360,6 +283,26 @@ spec:
     - name: qbittorrent
       port: 4180
 ---
+# ServiceMonitor (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: qbittorrent
+  namespace: qbittorrent
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: qbittorrent
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: qbittorrent
+  namespaceSelector:
+    matchNames:
+    - qbittorrent
+  endpoints:
+  - port: metrics
+    interval: 30s
+---
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -374,11 +317,11 @@ spec:
   - name: qbittorrent.slis
     rules:
     - alert: qbittorrent-down
-      expr: kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"} == 0 or absent(kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"})
+      expr: up{job="qbittorrent"} == 0
       for: 5m
       labels:
         severity: critical
         service: qbittorrent
       annotations:
         summary: qbittorrent is down
-        description: 'qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'qbittorrent has been unreachable for 5m (SLI: up).'

apps/wave-0/semaphore-db-backup-credentials-prod.sops.yaml

@@ -1,29 +0,0 @@
-apiVersion: v1
-kind: Secret
-metadata:
-    name: semaphore-db-backup-credentials-prod
-    namespace: semaphore-db
-type: Opaque
-stringData:
-    ACCESS_KEY_ID: ENC[AES256_GCM,data:OhWEEYqrvmDB40NmiETgeUisBvaO9ocevCZAsurNijw=,iv:kEjOIg9NHbRYm+Ul1WrBAzb3gLbIv+ipiYCxFK0QoVc=,tag:KAB1JAIw0ATwbjDEKZVT9A==,type:str]
-    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:C0PNKv07LCYpFONLqd55sz+ioyU/i2YJUkbZ8OX/dCErM/DilqchyUAT5EqnT/r3YJmdaA1pX85S3fxlVVHkbg==,iv:4ha28b+xIebvX2JPITmtTovIe4vdmj2XfI3cgkUHH4c=,tag:xbMyflcHojytPzgaoOYcXQ==,type:str]
-sops:
-    kms: []
-    gcp_kms: []
-    azure_kv: []
-    hc_vault: []
-    age:
-        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZVTJ3cXR6cks3Y0RNS2hx
-            K3pjZEduYUxmL0V3SmFUcU9XbGVHSVFITWh3Cjl1eHlTLytoOW5PaVNFVkJoeTBo
-            T1FnRHNDVzJ6dWpuR09WaElvbWF4YU0KLS0tIEJoQ1VsM1cxRlg0R1dtNGdOenNn
-            aVNjMjVBTytLdHlFdmZCS2tPelo1Y2MKYt1G5t+sDiHgcg+Q9ECBzJn2gS/paSEC
-            7aoX60SBrPgHvnGls5lzc3Tin9nH4jWYl6Nu8DAvIE0YPXDFr5a0UA==
-            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-09-07T06:36:33Z"
-    mac: ENC[AES256_GCM,data:mhVP7tjdZK2FR2imvpJbTyT7BigvcIKEpLSnBS1oSITUirB8xZC7aHBfyFuIoKOrpOJgEe40nHUdDYCHRJtb4dLeVja8+7meVaQWzxC8BWVHsksW+73hsqhjAzlHeobx6Irbk4PAMqIY0/jnf4MHryjnRn3dUuwu9U1CqL30JbE=,iv:57oCcp6FW9EAIVKnYGtkaTrnbijyv54T+vmZxMbMqnw=,tag:agrRP/JLypU60KOeN0s/5w==,type:str]
-    pgp: []
-    encrypted_regex: ^(data|stringData)$
-    version: 3.9.4

apps/wave-0/semaphore-db.yaml

@@ -38,53 +38,16 @@ metadata:
   name: semaphore-db
   namespace: semaphore-db
 spec:
-  # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db /
-  # nextcloud-db; see platform/keycloak/postgres-cluster.yaml for the full rationale. Short version:
-  # Longhorn's 3-way replication saves the DATA on node loss, not the SERVICE, and with
-  # `instances: 1` all three CNPG primaries had drifted onto k3s-prod-2, so one node took Semaphore
-  # down alongside SSO and Nextcloud. A streaming standby lets CNPG promote in seconds and repoint
-  # the -rw Service. Gated on the Barman archive below actually working (ContinuousArchiving=True
-  # with a real firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
-  #
-  # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
-  # pg_basebackup from the running primary, which is neither restarted nor disconnected.
-  instances: 3
+  # Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
+  # keycloak-db / nextcloud-db). The storage is already 3-way replicated by Longhorn; raise to 3 for
+  # Postgres-level HA as a future hardening.
+  instances: 1
   imageName: ghcr.io/cloudnative-pg/postgresql:16.4
   # Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
   # semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
   inheritedMetadata:
     labels:
       app.kubernetes.io/name: semaphore-db
-  # HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
-  # primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
-  # recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
-  # version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
-  # after a node loss it may stack two instances on one node — which is how `instances: 3` decays
-  # back into the single-node failure domain this change removes. `required` makes it a constraint;
-  # the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
-  # serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
-  #
-  # APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
-  # switchover. Expect a single brief connection reset for Semaphore.
-  affinity:
-    enablePodAntiAffinity: true
-    podAntiAffinityType: required
-  primaryUpdateMethod: switchover
-  # SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
-  # rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
-  # last few transactions when a primary dies before its standbys catch up, and `method: any` +
-  # `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
-  # never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
-  # are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
-  # standby into a total write outage. `preferred` relaxes to async instead.
-  #
-  # NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
-  # no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
-  postgresql:
-    synchronous:
-      method: any
-      number: 1
-      dataDurability: preferred
   storage:
     storageClass: longhorn
     size: 5Gi
@@ -106,38 +69,25 @@ spec:
       memory: 256Mi
     limits:
       memory: 512Mi
-  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
+  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
   # Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
-  # backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
-  # destinationPath, so objects land under ${CNPG_BACKUP_PATH}/semaphore-db/ — dev
-  # s3://cnpg/semaphore-db/ on the on-Proxmox MinIO, prod
-  # s3://k3s-prod-backup/barman/semaphore-db/ on Cloudflare R2. Both endpointURL and
-  # destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
-  # Credentials: PER-ENVIRONMENT (2026-09-07). dev reads semaphore-db-backup-credentials (the
-  # scoped MinIO key), prod reads semaphore-db-backup-credentials-prod (the Cloudflare R2
-  # token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
-  # generated golden — render.py places the -prod sidecar in this DB's wave); the
-  # cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
-  # one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
-  # ONE shared Secret until the split, so giving prod its R2 token would have left dev
-  # signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
-  # disk-full clock below rather than a merely missing backup.
-  #
-  # archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
-  # Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
-  # the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
-  # endpoint. Treat ContinuousArchiving=False here as an availability incident.
+  # backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
+  # (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP.
+  # Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
+  # (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
+  # endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
+  # substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
   backup:
     retentionPolicy: "30d"
     barmanObjectStore:
-      destinationPath: ${CNPG_BACKUP_PATH}
+      destinationPath: s3://cnpg
       endpointURL: ${S3_BACKUP_ENDPOINT}
       s3Credentials:
         accessKeyId:
-          name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
+          name: semaphore-db-backup-credentials
           key: ACCESS_KEY_ID
         secretAccessKey:
-          name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
+          name: semaphore-db-backup-credentials
           key: ACCESS_SECRET_KEY
       wal:
         compression: gzip

apps/wave-0/tests/app-slis.test.yaml

@@ -1,267 +0,0 @@
-# promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
-# Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
-# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
-# the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
-# tests run against the exact expressions Flux applies, never a copy.
-#
-# Run with: platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml
-#           (the merge gate runs the same harness via
-#            conformance/tests/l0-static/alert-rules-unit-tested)
-#
-# WHY THESE EXIST. Until 2026-09-13 every one of these rules read `up{job="<app>"} == 0` while the
-# rendered ServiceMonitor named a Service port (`metrics`) that no rendered Service defined. The
-# monitor discovered ZERO targets, so no `up{job="<app>"}` series existed and the rule could not
-# fire — on prod, `count by (job) (up)` listed 24 jobs and not one of the twelve appeared.
-# `PrometheusTargetsMissing` could not see it either: a target that was never discovered has no `up`
-# series to be 0. The rules now read kube-state-metrics / kubevirt series that DO exist for these
-# workloads (verified on prod), and every rule below is pinned in BOTH directions:
-#   * QUIET on the healthy state -> it is not permanently firing;
-#   * FIRING when the workload has no ready replica -> it is not inert;
-#   * FIRING when the signal is ABSENT -> the failure mode that produced this whole defect class
-#     (a rule that goes quiet because its series vanished) is itself alerted on.
-#
-# CLOCK/`for` CONVENTION: sample interval 1m, series 21 samples long, evaluated at 10m — comfortably
-# past every rule's `for: 5m` and well inside Prometheus' 5m lookback.
-rule_files:
-  - generated.rules.yaml
-
-evaluation_interval: 1m
-
-tests:
-  # ===========================================================================================
-  # 1. HEALTHY WAVE-0 (what prod looks like right now) -> every rule SILENT.
-  #    This is the "not permanently firing" half. It is also the test that catches a mis-spelled
-  #    label: get `deployment=` wrong and the `== 0` half matches nothing while the `absent()` half
-  #    matches everything, so this test goes red instead of the rule going quietly wrong.
-  # ===========================================================================================
-  - interval: 1m
-    name: wave-0 healthy - every app has a ready replica, the demo VM runs - all quiet
-    input_series:
-      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
-        values: '1+0x20'
-      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running", node="k3s-prod-1"}'
-        values: '1+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: gitea-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: jackett-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: pihole-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: plex-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: qbittorrent-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: vm-demo-down
-        exp_alerts: []
-
-  # ===========================================================================================
-  # 2. EACH APP DOWN IN TURN (0 available replicas) -> ONLY that app's rule fires.
-  #    The per-app isolation matters: a rule that forgot its `namespace=`/`deployment=` matchers
-  #    would light up every app at once and be useless for routing.
-  # ===========================================================================================
-  - interval: 1m
-    name: wave-0 plex has no available replica - only plex-down fires
-    input_series:
-      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
-        values: '0+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
-        values: '1+0x20'
-      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
-        values: '1+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: plex-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: plex
-              namespace: plex
-              deployment: plex
-            exp_annotations:
-              summary: "plex is down"
-              description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: gitea-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: jackett-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: pihole-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: qbittorrent-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: vm-demo-down
-        exp_alerts: []
-    # Before `for: 5m` has elapsed the rule must be PENDING, not firing — proof the 5m is real and
-    # a single scrape blip does not page anyone.
-    promql_expr_test:
-      - expr: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})'
-        eval_time: 2m
-        exp_samples:
-          - labels: 'kube_deployment_status_replicas_available{deployment="plex", namespace="plex"}'
-            value: 0
-
-  - interval: 1m
-    name: wave-0 gitea/jackett/pihole/qbittorrent each down - each own rule fires
-    input_series:
-      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
-        values: '0+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
-        values: '0+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
-        values: '0+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
-        values: '0+0x20'
-      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
-        values: '1+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: gitea-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: gitea
-              namespace: gitea
-              deployment: gitea
-            exp_annotations:
-              summary: "gitea is down"
-              description: "gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: jackett-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: jackett
-              namespace: jackett
-              deployment: jackett
-            exp_annotations:
-              summary: "jackett is down"
-              description: "jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: pihole-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: pihole
-              namespace: pihole
-              deployment: pihole
-            exp_annotations:
-              summary: "pihole is down"
-              description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: qbittorrent-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: qbittorrent
-              namespace: qbittorrent
-              deployment: qbittorrent
-            exp_annotations:
-              summary: "qbittorrent is down"
-              description: "qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: plex-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: vm-demo-down
-        exp_alerts: []
-
-  # ===========================================================================================
-  # 3. THE SIGNAL VANISHES -> the rule FIRES. This is the defect class itself, asserted on.
-  #    The old `up{job=...} == 0` rule had exactly this shape of hole and was silent for it. Here
-  #    the Deployment series for plex is simply not in the input at all, and plex-down must still
-  #    fire (via the `absent()` half), with the labels the selector's equality matchers carry.
-  # ===========================================================================================
-  - interval: 1m
-    name: wave-0 plex Deployment series ABSENT (never scraped / workload gone) - plex-down fires
-    input_series:
-      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
-        values: '1+0x20'
-      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
-        values: '1+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: plex-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: plex
-              namespace: plex
-              deployment: plex
-            exp_annotations:
-              summary: "plex is down"
-              description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: gitea-down
-        exp_alerts: []
-
-  # ===========================================================================================
-  # 4. THE VM: a VMI carries no replica count, so liveness is the EXISTENCE of a running-phase
-  #    series. Two ways to be down, both asserted: the VMI is gone entirely, and the VMI exists in
-  #    some OTHER phase. The second is the one a naive `kubevirt_vmi_info == 1` would miss, and it
-  #    is why the phase value is matched — LOWERCASE `running`, the value KubeVirt actually writes
-  #    (a capital-R matcher selects nothing and would be just as inert).
-  # ===========================================================================================
-  - interval: 1m
-    name: wave-0 demo VM has no running-phase VMI - vm-demo-down fires
-    input_series:
-      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
-        values: '1+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: vm-demo-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: vm-demo
-              namespace: vm-demo
-              name: vm-demo
-              phase: running
-            exp_annotations:
-              summary: "vm-demo is down"
-              description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."
-
-  - interval: 1m
-    name: wave-0 demo VM exists but is Failed, not running - vm-demo-down still fires
-    input_series:
-      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="failed"}'
-        values: '1+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: vm-demo-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: vm-demo
-              namespace: vm-demo
-              name: vm-demo
-              phase: running
-            exp_annotations:
-              summary: "vm-demo is down"
-              description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."

apps/wave-0/vm-demo.yaml

@@ -1,139 +0,0 @@
----
-# Namespace (trust zone; default-deny + policies apply)
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: vm-demo
-  labels:
-    homelab.keithjackson.dev/app-namespace: 'true'
-  annotations:
-    homelab.keithjackson.dev/backup-policy: daily-offcluster
----
-# Default-deny baseline (R-NET-01): deny all, allow only DNS
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: default-deny
-  namespace: vm-demo
-  labels:
-    homelab.keithjackson.dev/baseline: default-deny
-spec:
-  podSelector: {}
-  policyTypes:
-  - Ingress
-  - Egress
-  ingress: []
-  egress:
-  - to:
-    - namespaceSelector:
-        matchLabels:
-          kubernetes.io/metadata.name: kube-system
-    ports:
-    - protocol: UDP
-      port: 53
-    - protocol: TCP
-      port: 53
----
-# CDI DataVolume (standalone; survives rebuild, R-STATE-01)
-apiVersion: cdi.kubevirt.io/v1beta1
-kind: DataVolume
-metadata:
-  name: vm-demo-rootdisk
-  namespace: vm-demo
-spec:
-  source:
-    registry:
-      url: docker://quay.io/containerdisks/fedora:41
-  storage:
-    accessModes:
-    - ReadWriteOnce
-    storageClassName: longhorn
-    resources:
-      requests:
-        storage: 6Gi
----
-# KubeVirt VirtualMachine (R-PLAT-03)
-apiVersion: kubevirt.io/v1
-kind: VirtualMachine
-metadata:
-  name: vm-demo
-  namespace: vm-demo
-  labels:
-    app.kubernetes.io/name: vm-demo
-spec:
-  running: false
-  template:
-    metadata:
-      labels:
-        kubevirt.io/vm: vm-demo
-        homelab.keithjackson.dev/egress-class: internet
-        homelab.keithjackson.dev/auth-mode: none
-    spec:
-      domain:
-        cpu:
-          cores: 1
-        memory:
-          guest: 1536Mi
-        devices:
-          disks:
-          - name: rootdisk
-            disk:
-              bus: virtio
-          - name: cloudinit
-            disk:
-              bus: virtio
-          interfaces:
-          - name: default
-            masquerade: {}
-        resources:
-          requests:
-            memory: 1536Mi
-      networks:
-      - name: default
-        pod: {}
-      volumes:
-      - name: rootdisk
-        dataVolume:
-          name: vm-demo-rootdisk
-      - name: cloudinit
-        cloudInitNoCloud:
-          userData: "#cloud-config\npackage_update: true\npackages:\n  - dnf-automatic\nwrite_files:\n  - path: /etc/dnf/automatic.conf\n    content: |\n      [commands]\n      upgrade_type = default\n      random_sleep = 0\n      download_updates = yes\n      apply_updates = yes\nruncmd:\n  - [ systemctl, enable, --now, dnf-automatic.timer ]\n"
----
-# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): vm-demo-allow-internet-egress
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: vm-demo-allow-internet-egress
-  namespace: vm-demo
-  labels:
-    app.kubernetes.io/name: vm-demo
-spec:
-  endpointSelector:
-    matchLabels:
-      app.kubernetes.io/name: vm-demo
-  egress:
-  - toEntities:
-    - world
----
-# PrometheusRule SLI alerts (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: vm-demo-slis
-  namespace: vm-demo
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: vm-demo
-spec:
-  groups:
-  - name: vm-demo.slis
-    rules:
-    - alert: vm-demo-down
-      expr: absent(kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"})
-      for: 5m
-      labels:
-        severity: critical
-        service: vm-demo
-      annotations:
-        summary: vm-demo is down
-        description: 'vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info).'

apps/wave-1/keycloak.yaml

@@ -1,208 +0,0 @@
----
-# Namespace (trust zone; default-deny + policies apply)
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: keycloak
-  labels:
-    homelab.keithjackson.dev/app-namespace: 'true'
-    gateway-access: shared
-  annotations:
-    homelab.keithjackson.dev/backup-policy: daily-offcluster
----
-# Default-deny baseline (R-NET-01): deny all, allow only DNS
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: default-deny
-  namespace: keycloak
-  labels:
-    homelab.keithjackson.dev/baseline: default-deny
-spec:
-  podSelector: {}
-  policyTypes:
-  - Ingress
-  - Egress
-  ingress: []
-  egress:
-  - to:
-    - namespaceSelector:
-        matchLabels:
-          kubernetes.io/metadata.name: kube-system
-    ports:
-    - protocol: UDP
-      port: 53
-    - protocol: TCP
-      port: 53
----
-# PVC keycloak-db (R-STATE)
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: keycloak-db
-  namespace: keycloak
-  labels:
-    app.kubernetes.io/name: keycloak
-spec:
-  accessModes:
-  - ReadWriteOnce
-  storageClassName: longhorn
-  resources:
-    requests:
-      storage: 10Gi
----
-# Deployment (R-PLAT-02)
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: keycloak
-  namespace: keycloak
-  labels:
-    app.kubernetes.io/name: keycloak
-spec:
-  replicas: 1
-  strategy:
-    type: Recreate
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: keycloak
-  template:
-    metadata:
-      labels:
-        app.kubernetes.io/name: keycloak
-        homelab.keithjackson.dev/egress-class: internet
-        homelab.keithjackson.dev/auth-mode: native
-    spec:
-      containers:
-      - name: keycloak
-        image: quay.io/keycloak/keycloak
-        ports:
-        - containerPort: 8080
-        volumeMounts:
-        - name: db
-          mountPath: /data/db
-      volumes:
-      - name: db
-        persistentVolumeClaim:
-          claimName: keycloak-db
----
-# Service
-apiVersion: v1
-kind: Service
-metadata:
-  name: keycloak
-  namespace: keycloak
-  labels:
-    app.kubernetes.io/name: keycloak
-spec:
-  selector:
-    app.kubernetes.io/name: keycloak
-  ports:
-  - name: port-8080
-    port: 8080
-    targetPort: 8080
----
-# CiliumNetworkPolicy egress allow (R-NET-02): keycloak-allow-keycloak-db-5432
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: keycloak-allow-keycloak-db-5432
-  namespace: keycloak
-  labels:
-    app.kubernetes.io/name: keycloak
-spec:
-  endpointSelector:
-    matchLabels:
-      app.kubernetes.io/name: keycloak
-  egress:
-  - toEndpoints:
-    - matchLabels:
-        app.kubernetes.io/name: keycloak-db
-        k8s:io.kubernetes.pod.namespace: keycloak-db
-    toPorts:
-    - ports:
-      - port: '5432'
-        protocol: TCP
----
-# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): keycloak-allow-gateway-ingress
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: keycloak-allow-gateway-ingress
-  namespace: keycloak
-  labels:
-    app.kubernetes.io/name: keycloak
-spec:
-  endpointSelector:
-    matchLabels:
-      app.kubernetes.io/name: keycloak
-  ingress:
-  - fromEntities:
-    - ingress
-    toPorts:
-    - ports:
-      - port: '8080'
-        protocol: TCP
----
-# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): keycloak-allow-internet-egress
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: keycloak-allow-internet-egress
-  namespace: keycloak
-  labels:
-    app.kubernetes.io/name: keycloak
-spec:
-  endpointSelector:
-    matchLabels:
-      app.kubernetes.io/name: keycloak
-  egress:
-  - toEntities:
-    - world
----
-# HTTPRoute keycloak-8080 (R-TLS/R-DNS)
-apiVersion: gateway.networking.k8s.io/v1
-kind: HTTPRoute
-metadata:
-  name: keycloak-8080
-  namespace: keycloak
-  labels:
-    app.kubernetes.io/name: keycloak
-spec:
-  parentRefs:
-  - name: shared
-    namespace: gateway
-    sectionName: https
-  hostnames:
-  - keycloak.${BASE_DOMAIN}
-  rules:
-  - matches:
-    - path:
-        type: PathPrefix
-        value: /
-    backendRefs:
-    - name: keycloak
-      port: 8080
----
-# PrometheusRule SLI alerts (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: keycloak-slis
-  namespace: keycloak
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: keycloak
-spec:
-  groups:
-  - name: keycloak.slis
-    rules:
-    - alert: keycloak-down
-      expr: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"})
-      for: 5m
-      labels:
-        severity: critical
-        service: keycloak
-      annotations:
-        summary: keycloak is down
-        description: 'keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'

apps/wave-1/nextcloud.yaml

@@ -35,34 +35,6 @@ spec:
     - protocol: TCP
       port: 53
 ---
-# PersistentVolume nextcloud-legacy-archive - static external NFS (bulk tier)
-apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: nextcloud-legacy-archive
-  labels:
-    app.kubernetes.io/name: nextcloud
-spec:
-  capacity:
-    storage: 270Gi
-  accessModes:
-  - ReadOnlyMany
-  persistentVolumeReclaimPolicy: Retain
-  storageClassName: ''
-  mountOptions:
-  - nfsvers=4.1
-  - hard
-  - noatime
-  - timeo=600
-  - retrans=2
-  nfs:
-    server: ${BULK_NFS_SERVER}
-    path: /mnt/Storage/nextcloud
-    readOnly: true
-  claimRef:
-    namespace: nextcloud
-    name: nextcloud-legacy-archive
----
 # PVC nextcloud-data (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -79,23 +51,6 @@ spec:
     requests:
       storage: 20Gi
 ---
-# PVC nextcloud-legacy-archive (R-STATE)
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: nextcloud-legacy-archive
-  namespace: nextcloud
-  labels:
-    app.kubernetes.io/name: nextcloud
-spec:
-  accessModes:
-  - ReadOnlyMany
-  storageClassName: ''
-  volumeName: nextcloud-legacy-archive
-  resources:
-    requests:
-      storage: 270Gi
----
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -164,10 +119,6 @@ spec:
         volumeMounts:
         - name: data
           mountPath: /var/www/html
-        - name: legacy-archive
-          mountPath: /mnt/legacy-nextcloud
-          subPath: nextcloud
-          readOnly: true
         lifecycle:
           postStart:
             exec:
@@ -181,9 +132,6 @@ spec:
       - name: data
         persistentVolumeClaim:
           claimName: nextcloud-data
-      - name: legacy-archive
-        persistentVolumeClaim:
-          claimName: nextcloud-legacy-archive
 ---
 # Service
 apiVersion: v1
@@ -290,6 +238,26 @@ spec:
     - name: nextcloud
       port: 80
 ---
+# ServiceMonitor (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: nextcloud
+  namespace: nextcloud
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: nextcloud
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: nextcloud
+  namespaceSelector:
+    matchNames:
+    - nextcloud
+  endpoints:
+  - port: metrics
+    interval: 30s
+---
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -304,11 +272,11 @@ spec:
   - name: nextcloud.slis
     rules:
     - alert: nextcloud-down
-      expr: kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"} == 0 or absent(kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"})
+      expr: up{job="nextcloud"} == 0
       for: 5m
       labels:
         severity: critical
         service: nextcloud
       annotations:
         summary: nextcloud is down
-        description: 'nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'nextcloud has been unreachable for 5m (SLI: up).'

apps/wave-1/radarr.yaml

@@ -35,33 +35,6 @@ spec:
     - protocol: TCP
       port: 53
 ---
-# PersistentVolume radarr-media - static external NFS (bulk tier)
-apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: radarr-media
-  labels:
-    app.kubernetes.io/name: radarr
-spec:
-  capacity:
-    storage: 9Ti
-  accessModes:
-  - ReadWriteMany
-  persistentVolumeReclaimPolicy: Retain
-  storageClassName: ''
-  mountOptions:
-  - nfsvers=4.1
-  - hard
-  - noatime
-  - timeo=600
-  - retrans=2
-  nfs:
-    server: ${BULK_NFS_SERVER}
-    path: /mnt/Storage/plex
-  claimRef:
-    namespace: radarr
-    name: radarr-media
----
 # PVC radarr-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -78,23 +51,6 @@ spec:
     requests:
       storage: 5Gi
 ---
-# PVC radarr-media (R-STATE)
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: radarr-media
-  namespace: radarr
-  labels:
-    app.kubernetes.io/name: radarr
-spec:
-  accessModes:
-  - ReadWriteMany
-  storageClassName: ''
-  volumeName: radarr-media
-  resources:
-    requests:
-      storage: 9Ti
----
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -118,25 +74,16 @@ spec:
         homelab.keithjackson.dev/auth-mode: forward-auth
       annotations:
         homelab.keithjackson.dev/auth-mode: forward-auth
-        homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+        homelab.keithjackson.dev/auth-groups: media-users
     spec:
       containers:
       - name: radarr
         image: lscr.io/linuxserver/radarr:5.21.1
         ports:
         - containerPort: 7878
-        env:
-        - name: PUID
-          value: '1000'
-        - name: PGID
-          value: '3000'
-        - name: TZ
-          value: Etc/UTC
         volumeMounts:
         - name: config
-          mountPath: /config
-        - name: media
-          mountPath: /media
+          mountPath: /data/config
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -150,7 +97,6 @@ spec:
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
-        - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
         - --cookie-domain=.${BASE_DOMAIN}
@@ -202,15 +148,10 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
-      securityContext:
-        fsGroup: 3000
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: radarr-config
-      - name: media
-        persistentVolumeClaim:
-          claimName: radarr-media
 ---
 # Service
 apiVersion: v1
@@ -347,7 +288,7 @@ metadata:
     app.kubernetes.io/name: radarr
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
-    homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    homelab.keithjackson.dev/auth-groups: media-users
 spec:
   parentRefs:
   - name: shared
@@ -364,6 +305,26 @@ spec:
     - name: radarr
       port: 4180
 ---
+# ServiceMonitor (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: radarr
+  namespace: radarr
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: radarr
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  namespaceSelector:
+    matchNames:
+    - radarr
+  endpoints:
+  - port: metrics
+    interval: 30s
+---
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -378,11 +339,11 @@ spec:
   - name: radarr.slis
     rules:
     - alert: radarr-down
-      expr: kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"})
+      expr: up{job="radarr"} == 0
       for: 5m
       labels:
         severity: critical
         service: radarr
       annotations:
         summary: radarr is down
-        description: 'radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'radarr has been unreachable for 5m (SLI: up).'

apps/wave-1/semaphore-bootstrap-secret.sops.yaml

@@ -5,15 +5,6 @@
 #   credential from its own namespace (a Pod cannot read a Secret in another namespace).
 # access-key-encryption: FIXED base64 32-byte AES key Semaphore uses to encrypt stored SSH keys /
 #   credentials; MUST stay stable across restarts or stored secrets become undecryptable.
-# oidc-client-secret: the Keycloak `semaphore` CONFIDENTIAL client secret, and ONLY that. It MIRRORS
-#   `semaphore-client-secret` in platform/keycloak/keycloak-realm-clients.sops.yaml (the realm import
-#   injects that copy into the client), same cross-namespace mirror pattern as db-password above.
-#   It replaced the former `oidc-providers-json` key (2026-09-07): that key held the WHOLE
-#   SEMAPHORE_OIDC_PROVIDERS blob, which buried the environment-specific realm-issuer and redirect
-#   URLs inside ciphertext where Flux's ${BASE_DOMAIN} postBuild substitution could not reach them,
-#   so prod ran with dev's URLs. Those URLs are PUBLIC and now live in plaintext in
-#   catalog/semaphore.yaml (rendered to apps/wave-1/semaphore.yaml); the kubelet splices THIS value
-#   back into the blob via $(SEMAPHORE_OIDC_CLIENT_SECRET). Only the secret half is encrypted.
 # Recover: infra/devcontainer/dev.sh run sops -d apps/semaphore-bootstrap-secret.sops.yaml
 apiVersion: v1
 kind: Secret
@@ -24,11 +15,11 @@ metadata:
         app.kubernetes.io/name: semaphore
 type: Opaque
 stringData:
-    admin-username: ENC[AES256_GCM,data:ju3wVdjNzRUP/w==,iv:d/5ZxgQZ5bympu3gT2ck3DJy9iUPPBAtZYsJvD3To7I=,tag:0UtBaDVqNSK4RCJrQicchw==,type:str]
-    admin-password: ENC[AES256_GCM,data:pxIwpgh3BoND2TI0mPTixOTJoictzT77dqtuRdMKgY/2WaW4wM1Ra6H6n/dxnyR2,iv:eYbFQGFYVJsnYGwn6cOI3BgAdzMcin9YggAhiuL+xqM=,tag:/sPWGv4gue0vjc2nMk7wWA==,type:str]
-    db-password: ENC[AES256_GCM,data:hBFpWXdpk1R5a23Pntup03t56NBRPS+SZU8CrHVb/ypIaIpgf7kjYtVy6nhTF6u0,iv:eBzoQgH5ZhWWpfKZ6AQmwaSal5QaiuN9gfexQ7GsVFs=,tag:j2k8EwhKDlj/Ud6O39ioVQ==,type:str]
-    access-key-encryption: ENC[AES256_GCM,data:SKtAV4HxJ9pNYDqUK/1uOdI1rtMji3pTfprI2rhQxFQyhc6dFlhqI+OUEcY=,iv:rKCP+FLidcOltOwmMMcO5M8tz4qiifIMI+W5soGO2M8=,tag:HaHzkX/JI+9wmcg2SD9GAQ==,type:str]
-    oidc-client-secret: ENC[AES256_GCM,data:uCo7TNzbM2QKw9ie+xzyUG9MFKco1OAlwr5qPn/AGpNvo1YyXSfbZyyM0CLJ1NS821hGg5Zr/m3R/xpy84s3FQ==,iv:O8a4haskufpmorPaxOkJ1Q99Y5GTlzmDFJAsHS+irLk=,tag:VojJpA9jNMbNQmIe1Qr44w==,type:str]
+    admin-username: ENC[AES256_GCM,data:uD/vO9A9MjerFg==,iv:i/NAqVD6rWg9oWWSGBrRkxzwcPp2IW1HLIKY0pdcPPg=,tag:r27tKLAJ+ydE7Ty3ccMNUw==,type:str]
+    admin-password: ENC[AES256_GCM,data:544nWY1UmBnlsdPJAQZJMXRumu4W9X7UXZvZ6vHN/ArjRgLXKTArOe5JkT/nNrNw,iv:AC1DDbZj2HfpU19EjzA7T+WpIaR4nZc3PNN61y8bAzU=,tag:r28LFi5w6ZeCCaRaX2jRdg==,type:str]
+    db-password: ENC[AES256_GCM,data:ySkdvaGRCW9fMQqChs5l8dY4zCG19i4HOEinRC48mgLuh9WOrjd16T6jwFgiwB55,iv:+1gztJ22HbwgQJiolZMMWOcKIsxrBJpnruJdtxdooOU=,tag:8CfJGwyPTjDGyr0CvalXMQ==,type:str]
+    access-key-encryption: ENC[AES256_GCM,data:1YAMLo1nVkpJWZc+KBZ7B4ia+Y8P+xMyOhHMcEGGJOXd3o+WC5FqirLBVh4=,iv:j28ozeaDhGWDjoaxFOhn2t9YbYE7L3ah/Mfi+i2DWwY=,tag:czG9Wxc/7Nsuh1W5iATOww==,type:str]
+    oidc-providers-json: ENC[AES256_GCM,data:90yt5mIVWLVI2KrlJWpxhL/BMDpEgRhv7+BL+mBRt79diRvU7pN9vtzhZprAKzcU1sEywlT4EwFlsvJpYyo5Qbkp+ZVknbyhAlGvHhJOBHukOPWgf4LignLerO2GzhXjOOzFqWfZ4EyUxNMWTbSe/xc0zIVHDDNQiYplGVRK+E/ukiI7sD/Tn2RrbrdvpC5GyK0NXWRsXn9ffhE3fCoiRHXagj9i/q+uUrQI5EKpPobrDUAGqvKoBXTl/wJ4phnWg9IqtCTE/c3yA7fLLz4j5dCtPZNSVPT4TKDx8Mdv8CfCgziVicjG2s+yHuWpsQn8lf6nvdJmpYPvIqTtLWdxFWuD8Vfo6Qbq2mqPbKR+KmDOGUnjl4gbZdL1SkE9JiqBUobz7UXeLXwYzxMIsinOfYZLcCk7alrD5Yndj9JDxDJZQ54sMMBfVbxDl94W1VHy03RxAR46KKAbPaxOP5wH4Q4z3SDceNwmuGCDRGChRXiHhN5X2X0GqYCbc8E6GdkIMcpmD80e26ydEmAjw7F89mCF4F/yMzJNiG82qaZIn8HeqgGA8urous9r9bYJNlitYTYQ,iv:jbfVdrrZ9fGiOxhQeoTJfFg1QwiOMkoQyaadiVfxX/w=,tag:JkepcRs34EuEgNb4buTTDQ==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -38,14 +29,14 @@ sops:
         - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyTzF6NTBtcFVWK1Vtc0I2
-            VGdySHBsRGgvc1h0VzgvQ01xYVBJV3g2Q1NRCjJyZHVYZGtuamo1MFJHWlBwWkxt
-            cHA4VkY1TkRpN3M4UTdMOWZTbFVGemMKLS0tIGVNdDBrWVJveU56VjVBU2JWdHF4
-            RGtaYStXYUpONTZjZCtmM0NyZ3Rka1UKIQpmQai8dz+uRAsXxyeuGaRlRwsFqIPQ
-            9AfrZJcslYwlaK3ptm1oqaCdrlxcMar2nI7thSqzbLe+e1nzA2z7vQ==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuRDlqRFJ0dk14UHluSGRP
+            S1pESy9PWlpMakU0aTkzNVhqUGdtbWNEdTJjCm5RNG14azJHc28rdEJ4L0RzR0xO
+            MGRocGZ1V1JZTENWQkN3VGZzamp2eTgKLS0tIGt6QkNNaW5hdTBJVHY0SVo5d3JX
+            elp0SWxHcjg0RWF3aW9mWTJHZ0lhcUEKc0rU5VilAVDleCwqq4SOF+Y9sniu1Pka
+            w/MGDA9THSB0b+YgWRWiX/KiimFqp0ZFODOJ9cHVwFGY71+VadPwOg==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-09-07T07:01:50Z"
-    mac: ENC[AES256_GCM,data:Xutm1rQt57J4RILElD8FkNOsQT1HK5jZlPtqqLyZ/VqOoDwo+IS1wp7hROprlUvQC0bUXy79tRRIl5Q0CCOD+fvAGIdmade+vB7qXlYN6Q1441dGhuSTPvckso6FaB2ddRjKhKGLc/Z4bJoz9VaRNNxJNpWXIrENTg+I+QhNhlo=,iv:klJOrEQ6QN5dTGkQfGnbMbgCWsgWwZPb5YXDMeWeE34=,tag:fuJr4t3Tt3dT9u3/0SFo8Q==,type:str]
+    lastmodified: "2026-08-27T22:12:32Z"
+    mac: ENC[AES256_GCM,data:nHftJTGnTr0O5h8yGXwgz3eflggXJfIXM3m8n3bzDJE/P4nvSpZzFeMb+sc2on+AL+t+SeKrIl8blzm33XsQ1lkaY85Zm+bc5cnED1g04BMDVmmK3brh/8w50Bp2wPEVzb4I01MDzhhyIbOYg7pFvZAH3/VuqvM1t6XmfF1NrOA=,iv:4KBa9PH/dfdGwLC16RenZIVB6BtSoGo6x3qt3h1R43I=,tag:KULiA1klP/RZMNigkloGDA==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

apps/wave-1/semaphore.yaml

@@ -118,13 +118,11 @@ spec:
               key: access-key-encryption
         - name: SEMAPHORE_PORT
           value: :3000
-        - name: SEMAPHORE_OIDC_CLIENT_SECRET
+        - name: SEMAPHORE_OIDC_PROVIDERS
           valueFrom:
             secretKeyRef:
               name: semaphore-bootstrap
-              key: oidc-client-secret
-        - name: SEMAPHORE_OIDC_PROVIDERS
-          value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
+              key: oidc-providers-json
         volumeMounts:
         - name: config
           mountPath: /etc/semaphore
@@ -243,6 +241,26 @@ spec:
     - name: semaphore
       port: 3000
 ---
+# ServiceMonitor (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: semaphore
+  namespace: semaphore
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: semaphore
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: semaphore
+  namespaceSelector:
+    matchNames:
+    - semaphore
+  endpoints:
+  - port: metrics
+    interval: 30s
+---
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -257,11 +275,11 @@ spec:
   - name: semaphore.slis
     rules:
     - alert: semaphore-down
-      expr: kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"} == 0 or absent(kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"})
+      expr: up{job="semaphore"} == 0
       for: 5m
       labels:
         severity: critical
         service: semaphore
       annotations:
         summary: semaphore is down
-        description: 'semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'semaphore has been unreachable for 5m (SLI: up).'

apps/wave-1/sonarr.yaml

@@ -35,33 +35,6 @@ spec:
     - protocol: TCP
       port: 53
 ---
-# PersistentVolume sonarr-media - static external NFS (bulk tier)
-apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: sonarr-media
-  labels:
-    app.kubernetes.io/name: sonarr
-spec:
-  capacity:
-    storage: 9Ti
-  accessModes:
-  - ReadWriteMany
-  persistentVolumeReclaimPolicy: Retain
-  storageClassName: ''
-  mountOptions:
-  - nfsvers=4.1
-  - hard
-  - noatime
-  - timeo=600
-  - retrans=2
-  nfs:
-    server: ${BULK_NFS_SERVER}
-    path: /mnt/Storage/plex
-  claimRef:
-    namespace: sonarr
-    name: sonarr-media
----
 # PVC sonarr-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -78,23 +51,6 @@ spec:
     requests:
       storage: 5Gi
 ---
-# PVC sonarr-media (R-STATE)
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: sonarr-media
-  namespace: sonarr
-  labels:
-    app.kubernetes.io/name: sonarr
-spec:
-  accessModes:
-  - ReadWriteMany
-  storageClassName: ''
-  volumeName: sonarr-media
-  resources:
-    requests:
-      storage: 9Ti
----
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -118,25 +74,16 @@ spec:
         homelab.keithjackson.dev/auth-mode: forward-auth
       annotations:
         homelab.keithjackson.dev/auth-mode: forward-auth
-        homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+        homelab.keithjackson.dev/auth-groups: media-users
     spec:
       containers:
       - name: sonarr
         image: lscr.io/linuxserver/sonarr:4.0.15
         ports:
         - containerPort: 8989
-        env:
-        - name: PUID
-          value: '1000'
-        - name: PGID
-          value: '3000'
-        - name: TZ
-          value: Etc/UTC
         volumeMounts:
         - name: config
-          mountPath: /config
-        - name: media
-          mountPath: /media
+          mountPath: /data/config
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -150,7 +97,6 @@ spec:
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
-        - --allowed-group=platform-admins
         - --skip-provider-button=true
         - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
         - --cookie-domain=.${BASE_DOMAIN}
@@ -202,15 +148,10 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
-      securityContext:
-        fsGroup: 3000
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: sonarr-config
-      - name: media
-        persistentVolumeClaim:
-          claimName: sonarr-media
 ---
 # Service
 apiVersion: v1
@@ -347,7 +288,7 @@ metadata:
     app.kubernetes.io/name: sonarr
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
-    homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    homelab.keithjackson.dev/auth-groups: media-users
 spec:
   parentRefs:
   - name: shared
@@ -364,6 +305,26 @@ spec:
     - name: sonarr
       port: 4180
 ---
+# ServiceMonitor (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: sonarr
+  namespace: sonarr
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: sonarr
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  namespaceSelector:
+    matchNames:
+    - sonarr
+  endpoints:
+  - port: metrics
+    interval: 30s
+---
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -378,11 +339,11 @@ spec:
   - name: sonarr.slis
     rules:
     - alert: sonarr-down
-      expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
+      expr: up{job="sonarr"} == 0
       for: 5m
       labels:
         severity: critical
         service: sonarr
       annotations:
         summary: sonarr is down
-        description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'sonarr has been unreachable for 5m (SLI: up).'

apps/wave-1/tests/app-slis.test.yaml

@@ -1,245 +0,0 @@
-# promtool rule-unit tests for the wave-1 per-app `<app>-down` SLI rules.
-# Manifests under test: every PrometheusRule in apps/wave-1/ — nextcloud, radarr, semaphore, sonarr
-# (container/Deployment) and keycloak (StatefulSet, see below). Discovered by the SUITE convention;
-# the flattened manifests are the RENDERED ones Flux applies.
-#
-# Run with: platform/monitoring/tests/run.sh apps/wave-1/tests/app-slis.test.yaml
-#
-# See apps/wave-0/tests/app-slis.test.yaml for the full history: these rules all read
-# `up{job="<app>"} == 0` until 2026-09-13 against a job no scrape target produced, and were
-# structurally inert on prod. Each is now pinned QUIET on the healthy state, FIRING on no-ready-
-# replica, and FIRING when the signal is ABSENT.
-#
-# KEYCLOAK IS THE INTERESTING ONE. Its server is a StatefulSet run by the Keycloak operator, NOT the
-# Deployment this ServiceDefinition would otherwise imply — so its rule reads
-# kube_statefulset_status_replicas_ready (catalog/keycloak.yaml declares
-# `observability.livenessWorkload: {kind: StatefulSet, name: keycloak}`). Had it been left to the
-# default, the `absent()` half would have matched forever and the alert would have been PERMANENTLY
-# FIRING, which is the other failure mode this repo has shipped to prod. Verified on prod
-# 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}
-# = 2, and the only Deployment in that namespace is keycloak-operator.
-rule_files:
-  - generated.rules.yaml
-
-evaluation_interval: 1m
-
-tests:
-  # ===========================================================================================
-  # 1. HEALTHY WAVE-1 -> every rule SILENT ("not permanently firing").
-  # ===========================================================================================
-  - interval: 1m
-    name: wave-1 healthy - every app ready, keycloak StatefulSet at 2 replicas - all quiet
-    input_series:
-      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
-        values: '2+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
-        values: '1+0x20'
-      # The namespace's OTHER workloads are present on purpose: keycloak-operator is a Deployment in
-      # the keycloak namespace, and keycloak-db is a StatefulSet in it. A rule that matched the
-      # namespace only (not the object) would be satisfied — or triggered — by these instead of by
-      # the IdP itself. That is the exact masking KeycloakDown had to be fixed for.
-      - series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
-        values: '1+0x20'
-      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
-        values: '3+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: keycloak-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: nextcloud-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: radarr-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: semaphore-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: sonarr-down
-        exp_alerts: []
-
-  # ===========================================================================================
-  # 2. THE IdP ITSELF IS DOWN while its operator and DB are fine -> ONLY keycloak-down fires.
-  #    This is the anti-masking assertion: the neighbours in that namespace stay ready.
-  # ===========================================================================================
-  - interval: 1m
-    name: wave-1 keycloak StatefulSet has no ready replica (operator + DB fine) - keycloak-down fires
-    input_series:
-      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
-        values: '0+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
-        values: '1+0x20'
-      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
-        values: '3+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
-        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
-        values: '1+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: keycloak-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: keycloak
-              namespace: keycloak
-              statefulset: keycloak
-            exp_annotations:
-              summary: "keycloak is down"
-              description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
-      - eval_time: 10m
-        alertname: nextcloud-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: radarr-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: semaphore-down
-        exp_alerts: []
-      - eval_time: 10m
-        alertname: sonarr-down
-        exp_alerts: []
-
-  # ===========================================================================================
-  # 3. EACH DEPLOYMENT-BACKED APP DOWN -> its own rule fires, keycloak stays quiet.
-  # ===========================================================================================
-  - interval: 1m
-    name: wave-1 nextcloud/radarr/semaphore/sonarr have no available replica - each own rule fires
-    input_series:
-      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
-        values: '2+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
-        values: '0+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
-        values: '0+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
-        values: '0+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
-        values: '0+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: nextcloud-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: nextcloud
-              namespace: nextcloud
-              deployment: nextcloud
-            exp_annotations:
-              summary: "nextcloud is down"
-              description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: radarr-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: radarr
-              namespace: radarr
-              deployment: radarr
-            exp_annotations:
-              summary: "radarr is down"
-              description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: semaphore-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: semaphore
-              namespace: semaphore
-              deployment: semaphore
-            exp_annotations:
-              summary: "semaphore is down"
-              description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: sonarr-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: sonarr
-              namespace: sonarr
-              deployment: sonarr
-            exp_annotations:
-              summary: "sonarr is down"
-              description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: keycloak-down
-        exp_alerts: []
-
-  # ===========================================================================================
-  # 4. THE SIGNAL VANISHES -> the rules FIRE. The defect class itself, asserted on: a rule that can
-  #    only fire on `== 0` goes SILENT when kube-state-metrics or the workload object disappears,
-  #    which is indistinguishable from healthy. Here NOTHING in wave-1 is scraped.
-  # ===========================================================================================
-  - interval: 1m
-    name: wave-1 all liveness series ABSENT (kube-state-metrics gone) - every rule fires
-    input_series:
-      # One unrelated series so the test has an input at all; no wave-1 liveness series exists.
-      - series: 'up{job="kube-state-metrics"}'
-        values: '0+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: keycloak-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: keycloak
-              namespace: keycloak
-              statefulset: keycloak
-            exp_annotations:
-              summary: "keycloak is down"
-              description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
-      - eval_time: 10m
-        alertname: nextcloud-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: nextcloud
-              namespace: nextcloud
-              deployment: nextcloud
-            exp_annotations:
-              summary: "nextcloud is down"
-              description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: radarr-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: radarr
-              namespace: radarr
-              deployment: radarr
-            exp_annotations:
-              summary: "radarr is down"
-              description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: semaphore-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: semaphore
-              namespace: semaphore
-              deployment: semaphore
-            exp_annotations:
-              summary: "semaphore is down"
-              description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-      - eval_time: 10m
-        alertname: sonarr-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: sonarr
-              namespace: sonarr
-              deployment: sonarr
-            exp_annotations:
-              summary: "sonarr is down"
-              description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."

apps/wave-2/seerr.yaml

@@ -233,6 +233,26 @@ spec:
     - name: seerr
       port: 5055
 ---
+# ServiceMonitor (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: seerr
+  namespace: seerr
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: seerr
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: seerr
+  namespaceSelector:
+    matchNames:
+    - seerr
+  endpoints:
+  - port: metrics
+    interval: 30s
+---
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -247,11 +267,11 @@ spec:
   - name: seerr.slis
     rules:
     - alert: seerr-down
-      expr: kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})
+      expr: up{job="seerr"} == 0
       for: 5m
       labels:
         severity: critical
         service: seerr
       annotations:
         summary: seerr is down
-        description: 'seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'seerr has been unreachable for 5m (SLI: up).'

apps/wave-2/tests/app-slis.test.yaml

@@ -1,69 +0,0 @@
-# promtool rule-unit tests for the wave-2 per-app `<app>-down` SLI rule (seerr).
-# Manifest under test: apps/wave-2/seerr.yaml. Discovered by the SUITE convention; the flattened
-# manifest is the RENDERED one Flux applies.
-#
-# Run with: platform/monitoring/tests/run.sh apps/wave-2/tests/app-slis.test.yaml
-#
-# seerr (Jellyseerr/Overseerr) exports NO Prometheus metrics of its own, so its `up` SLI could never
-# have been met by scraping the app — see apps/wave-0/tests/app-slis.test.yaml for the full history
-# of the inert `up{job="seerr"} == 0` rule this replaces. Pinned in both directions plus the
-# signal-vanishes case.
-rule_files:
-  - generated.rules.yaml
-
-evaluation_interval: 1m
-
-tests:
-  - interval: 1m
-    name: wave-2 healthy - seerr has an available replica - quiet
-    input_series:
-      - series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
-        values: '1+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: seerr-down
-        exp_alerts: []
-
-  - interval: 1m
-    name: wave-2 seerr has no available replica - seerr-down fires
-    input_series:
-      - series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
-        values: '0+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: seerr-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: seerr
-              namespace: seerr
-              deployment: seerr
-            exp_annotations:
-              summary: "seerr is down"
-              description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
-    # Under `for: 5m` a 2m-old condition must NOT yet be a firing alert, but the expression must
-    # already be selecting — the distinction an inert rule cannot make.
-    promql_expr_test:
-      - expr: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})'
-        eval_time: 2m
-        exp_samples:
-          - labels: 'kube_deployment_status_replicas_available{deployment="seerr", namespace="seerr"}'
-            value: 0
-
-  - interval: 1m
-    name: wave-2 seerr liveness series ABSENT - seerr-down still fires
-    input_series:
-      - series: 'up{job="kube-state-metrics"}'
-        values: '1+0x20'
-    alert_rule_test:
-      - eval_time: 10m
-        alertname: seerr-down
-        exp_alerts:
-          - exp_labels:
-              severity: critical
-              service: seerr
-              namespace: seerr
-              deployment: seerr
-            exp_annotations:
-              summary: "seerr is down"
-              description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."

catalog/gitea.yaml

@@ -180,34 +180,6 @@ dependencies: []                # default-deny: reaches only DNS; git clients re
 ingressRoutes:
   - host: gitea.${BASE_DOMAIN}
     port: 3000                  # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
-# THIS IS WHAT KEEPS PROD'S GITOPS OUT OF A DEADLOCK — it is not an incidental allowance. Remove it
-# and prod cannot reconcile AT ALL, in a way that is very hard to diagnose from cold.
-#
-# Prod's Flux source is THIS SERVICE, not a public hostname:
-#   clusters/prod/kustomization.yaml -> GitRepository.spec.url =
-#   http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
-# chosen deliberately so prod's GitOps depends on NEITHER public DNS, Cloudflare, prod's own
-# Gateway, NOR ${BASE_DOMAIN} — a future domain change cannot sever prod from its own source. That
-# file documents the other half of this pair at length; the two must be read together.
-#
-# The only other ingress this entry renders is gitea-allow-gateway-ingress, which admits
-# `fromEntities: [ingress]` — the shared Gateway's Envoy proxy — and nothing else. Flux's
-# source-controller dials the ClusterIP DIRECTLY, so under the namespace default-deny (R-NET-01) its
-# clone was DROPPED (`wget` from source-controller timed out; srcReady=False, "unable to list
-# remote"). The deadlock that made this urgent: the HTTPRoute that would have served the new public
-# name after the 2026-09-13 apex flip lives in a DOWNSTREAM tier, which cannot apply while the
-# source is down — source down -> routes never rename -> source stays down. It was broken on
-# 2026-09-13 with a live `kubectl apply` of a hand-written gitea-allow-flux-ingress, i.e. with a
-# cluster object that existed OUTSIDE git. This declaration renders that object, so one
-# `kubectl delete` or one cluster rebuild can no longer put prod back in the deadlock.
-inClusterClients:
-  # `name` is pinned to `flux` (rather than defaulting to the namespace) so the rendered policy name
-  # equals the live object's, `gitea-allow-flux-ingress`, and Flux ADOPTS it in place: no
-  # near-duplicate policy beside it, and no window in which prod's own source has no allow.
-  - name: flux
-    namespace: flux-system      # matched by NAMESPACE alone — the flux controllers' pod labels belong
-                                # to upstream's manifests, not to this platform's contract
-    port: 3000                  # ONLY the Gitea HTTP port — the same single port the Gateway allow opens
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
   groups:

catalog/jackett.yaml

@@ -23,17 +23,6 @@ storage:
   - name: config
     size: 2Gi
     class: longhorn             # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
-    mountPath: /config          # THE APP WRITES /config, NOT the renderer default /data/config.
-                                # LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
-                                # /config), so without this the Longhorn PVC mounted at /data/config sat
-                                # holding nothing but lost+found while the app wrote its entire
-                                # configuration to the container's EPHEMERAL writable layer — destroyed on
-                                # every pod recreation: restart, reschedule, image bump. Measured in-cluster
-                                # 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
-                                # dev=2053 on the node's own disk with [Sentry asp config.xml logs].
-                                # This is the SAME failure catalog/plex.yaml:54 already records fixing.
-                                # The live /config contents were copied onto each PVC before this landed,
-                                # so API keys and credentials other apps reference were preserved.
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
   - host: jackett.${BASE_DOMAIN}
@@ -42,15 +31,6 @@ auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
   groups:
     - media-users               # authenticated AND in an authorized group (R-IDP-06)
-    # platform-admins is a SUPERSET, not a widening. --allowed-group is repeatable and OR-ed, so
-    # listing both means "media consumers OR platform operators". Without it the media UIs are
-    # reachable by NOBODY the moment media-users is empty: on prod the realm's only user is the
-    # operator, who is in platform-admins only, so jackett/qbittorrent/radarr/sonarr answered an
-    # immediate 403 at the oauth2-proxy callback (no login prompt, because the Keycloak SSO session
-    # already existed) while pihole/crafty — which already gate on platform-admins — worked.
-    # This grants no real privilege: platform-admins already maps to cluster-admin via
-    # k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods.
-    - platform-admins           # operators of these services must not be locked out of them
 update:
   channel: stable               # R-UPD-01
   range: ">=0.22.0"             # stay on the Jackett 0.22.x stable line

catalog/keycloak.yaml

@@ -31,15 +31,5 @@ observability:
   metrics: true
   slis:
     - up                      # IdP down/unhealthy is a critical alert (docs/10; R-OBS-01)
-  # WHAT "up" IS for Keycloak. The Keycloak OPERATOR runs the server as a StatefulSet named
-  # `keycloak` (pods keycloak-0/1), not as the Deployment this ServiceDefinition would otherwise
-  # imply — the operator owns the workload (platform/keycloak/keycloak-instance.yaml) and this SD
-  # is the catalog's declaration of intent for it. Without this override the rendered SLI would
-  # read a Deployment series that does not exist and its `absent()` half would fire FOREVER.
-  # Verified on prod 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak",
-  # statefulset="keycloak"} = 2, and the only Deployment in that namespace is keycloak-operator.
-  livenessWorkload:
-    kind: StatefulSet
-    name: keycloak
 backup:
   policy: daily-offcluster    # DR-critical: rebuilds from DB + git realm config (docs/15, docs/04; R-STATE-03)

catalog/nextcloud.yaml

@@ -223,44 +223,6 @@ storage:
     mountPath: /var/www/html    # 3-replica tier; irreplaceable, DR-critical (docs/04; R-STATE-02).
                                 # The whole html dir is persisted so config.php + user data survive a
                                 # rebuild (the default /data/<name> would strand them on ephemeral rootfs).
-  # THE LEGACY NEXTCLOUD DATASET (Storage/nextcloud, 270 G) — MOUNTED READ-ONLY, DELIBERATELY, AND
-  # NOT AS THIS INSTANCE'S DATA DIRECTORY. Read this before changing either decision.
-  #
-  # WHAT IS ACTUALLY IN THERE (measured 2026-09-07, not assumed):
-  #     /mnt/Storage/nextcloud/nextcloud/config          103 M
-  #     /mnt/Storage/nextcloud/nextcloud/config_old      3.4 G
-  #     /mnt/Storage/nextcloud/nextcloud/config_backup   267 G   <- 99% of the dataset
-  # There is NO `data/` directory. This is the OLD, non-k3s Nextcloud's configuration plus a backup
-  # blob — an ARCHIVE, not a live user-data tree.
-  #
-  # WHY READ-ONLY. Two independent reasons, either sufficient:
-  #   1. The old instance is STILL LIVE on it. The export is currently shared rw to 192.168.1.6 and
-  #      192.168.1.5, and the operator has said the existing non-k3s prod must remain active. Two
-  #      Nextcloud instances writing one instance's config directory, each with its own database and
-  #      therefore its own oc_filecache, is a corruption scenario with no clean recovery.
-  #   2. Nothing here needs writing. An archive is read.
-  # "Do not grant more than needed" therefore lands on `ro`. If the operator's intent is in fact a
-  # MIGRATION (read this data in, then decommission the old instance), that is a different and
-  # larger change than a mount, and it should be a deliberate one — flip `readOnly` and update the
-  # export at that point, not before. Enforced by the PV's nfs.readOnly, so it is a real `ro` mount.
-  #
-  # WHY NOT /var/www/html/data. That path is the LONGHORN PVC above and is this instance's own
-  # state; mounting a foreign tree into it would have Nextcloud scan and index an archive it does
-  # not own. A neutral path keeps the two completely separate. To expose it to users, add it as a
-  # Nextcloud External Storage (Local) mount pointing at this path — an in-app, per-user decision.
-  #
-  # subPath skips the redundant single-child directory: the dataset root holds exactly one entry,
-  # `nextcloud/`, so mounting that subtree puts config/ config_old/ config_backup/ at the top.
-  #
-  # UID: nothing to do. The tree is 0755 owned 1000:3000 and the mount is read-only, so www-data
-  # (uid 33) can read and traverse it. (fsGroup would be a no-op on an NFS volume in any case.)
-  - name: legacy-archive
-    class: bulk
-    export: /mnt/Storage/nextcloud   # the ZFS dataset Storage/nextcloud, on ${BULK_NFS_SERVER}
-    subPath: nextcloud
-    mountPath: /mnt/legacy-nextcloud
-    readOnly: true
-    size: 270Gi                      # ADVISORY ONLY on a static NFS PV — the dataset's used size.
 dependencies:
   - to: nextcloud-db            # explicit allow: Nextcloud -> its PostgreSQL (docs/13 graph; R-NET-02).
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no

catalog/pihole.yaml

@@ -13,93 +13,6 @@ storage:
     size: 2Gi
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
-# ------------------------------------------------------------------------------------------------
-# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
-# ------------------------------------------------------------------------------------------------
-# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
-# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
-# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
-# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
-# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
-# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
-# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
-# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
-#
-# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
-# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
-# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
-# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
-# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
-# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
-# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
-# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
-# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
-# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
-# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
-# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
-#
-# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
-# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
-# configured through the UI is lost on the next reschedule — which is exactly what happened to the
-# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
-# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
-# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
-# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
-env:
-  # Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
-  # pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
-  # required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
-  # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
-  # enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
-  # nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
-  - name: FTLCONF_dns_listeningMode
-    value: "ALL"
-  # Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
-  # router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
-  # answers, and forwarding to a device that may forward back here invites a loop.
-  - name: FTLCONF_dns_upstreams
-    value: "8.8.8.8;8.8.4.4"
-  # Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
-  # because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
-  # externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
-  # announcement lease and the pihole pod land on different nodes, which is most of the time — and
-  # Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
-  # INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
-  # apparent client's budget, and a normal evening of browsing from a handful of machines would trip
-  # a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
-  # off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
-  # limit becomes meaningful again — and should be restored — if Cilium is ever switched to
-  # `loadBalancer.mode: dsr`, which preserves the client address.
-  - name: FTLCONF_dns_rateLimit_count
-    value: "0"
-  # THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
-  # every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
-  # so the first line is a wildcard for the whole zone and each later line is a specific override.
-  # PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
-  # line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
-  # changes nothing.
-  #
-  # A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
-  # `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
-  # ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
-  # platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
-  # behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
-  # repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
-  # is the annotation that means "this Service publishes its own hostname at its own address":
-  # apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
-  # platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
-  # resolver address, are the complete set of exceptions today.
-  #
-  # AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
-  # (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
-  # NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
-  #
-  # Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
-  # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
-  # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
-  # while resolution for the zone does not work. Confirmed both ways against the live image.
-  - name: FTLCONF_misc_dnsmasq_lines
-    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
 ingressRoutes:
   - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)

catalog/plex.yaml

@@ -4,11 +4,21 @@
 # auth: native (Plex accounts, NOT Keycloak — docs/13 + docs/15 documented exception).
 # Validates against schemas/servicedefinition.schema.json.
 #
-# BULK MEDIA — NOW ATTACHED (docs/04 bulk tier; the long-standing OQ-2 prerequisite, resolved for
-# the STATIC case). The library is 8.33 T of ZFS on the Proxmox host at /mnt/Storage/plex, and it
-# already exists: nothing is provisioned, an existing dataset is mounted. See
-# docs/BULK-NFS-STORAGE.md for the layout, the export change and the uid analysis. (The bulk-tier
-# requirement ID is deliberately still not typed here — see that document's "gate" note.)
+# BULK MEDIA (spec bulk-tier requirement, docs/04) — OPEN PREREQUISITE, intentionally NOT declared
+# here yet (the requirement ID is cited in the WS-APPS report, not as a token here: this change does
+# NOT exercise the bulk tier, so it must not be selected as a "touched" requirement whose in-cluster
+# conformance test is still a Phase-1 gap).
+#   The Plex library belongs on the `bulk` tier (docs/04 / docs/13 cross-cutting note: metadata DB
+#   on replicated Longhorn, large library on bulk). The `bulk` StorageClass exists
+#   (platform/longhorn/storageclasses.yaml) and the schema+renderer both support `class: bulk`, but
+#   that class is a documented STUB: OQ-2 (external NFS/SMB backing) is UNRESOLVED and the NFS CSI
+#   driver is not installed, so a `bulk` PVC will not bind and would leave the Plex pod Pending.
+#   To onboard the running Plex shell now (compute + config + public exposure), the media volume is
+#   omitted. Once OQ-2 is resolved and `bulk` provisions, attach the library with a one-line add:
+#     storage:
+#       - name: media
+#         class: bulk        # external mount; size omitted (sized outside the cluster)
+#   then re-render. See the WS-APPS report for the full bulk-media finding.
 name: plex
 kind: container
 source:
@@ -60,31 +70,6 @@ storage:
                               # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
                               # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
                               # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
-  # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
-  # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
-  # downloads/ movies/ photos/ tv/).
-  #
-  # WHY THE ROOT AND NOT THREE subPath MOUNTS. Nothing is gained by narrowing: the mount is `ro`,
-  # so no subPath could remove a privilege that matters, while three mounts would triple the NFS
-  # mounts per pod and mean a manifest change every time a top-level library is added.
-  #
-  # WHY READ-ONLY, when the export is `rw` and the other consumers write. Plex is a SERVER: it
-  # indexes and streams. Every write on that tree belongs to Sonarr and Radarr. Marking it here
-  # costs nothing and removes Plex from the set of things that can damage 8.33 T. Enforced by the
-  # PV's nfs.readOnly (a real `ro` mount), not only by the volumeMount flag.
-  #
-  # UID: no change needed. The tree is 0755 owned 1000:3000 throughout, so it is world-readable and
-  # world-traversable, and PLEX_UID/PLEX_GID above already make the process uid 1000 in any case.
-  # (fsGroup would have been a no-op here regardless — the in-tree NFS plugin does no ownership
-  # management. That is the trap the arrs' PUID/PGID exists to avoid; Plex never hits it.)
-  - name: media
-    class: bulk
-    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
-    mountPath: /media
-    readOnly: true
-    size: 9Ti                   # ADVISORY ONLY (nothing enforces a static NFS PV's capacity): the
-                                # dataset's 8.33 T used, rounded up, so `kubectl get pv` reports
-                                # something true. The real limit is the ZFS pool.
 dependencies: []              # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
 ingressRoutes:
   - host: plex.${BASE_DOMAIN}

catalog/qbittorrent.yaml

@@ -4,18 +4,19 @@
 # internal download client | egress VPN + kill-switch | config on Longhorn | auth forward-auth
 # (bare web UI, no native OIDC).
 #
-# BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record).
-#   This entry used to ship config-only, with a comment promising to attach the downloads volume "once
-#   OQ-2 is resolved". OQ-2 was resolved for STATIC pre-existing datasets (schema 1.5.0 / renderer
-#   `class: bulk` + `export` -> a static, pre-bound NFS PV), and plex/sonarr/radarr/nextcloud all took
-#   their media mounts — but qBittorrent's was never added, so the DOWNLOAD CLIENT was the one member
-#   of the chain with no media volume at all. Measured on prod 2026-09-12:
-#       Session\DefaultSavePath=/downloads/   Session\TempPath=/downloads/incomplete/
-#       Downloads\SavePath=/downloads/       `ls /downloads` -> No such file or directory
-#   i.e. the app was configured to write a path that DID NOT EXIST, in a container with no volume there.
-#   Anything it did manage to create would have been invisible to sonarr/radarr, lost on every pod
-#   recreation, and filling the NODE's ephemeral disk in the meantime. The `downloads` entry under
-#   storage: below closes that. See docs/BULK-NFS-STORAGE.md and the bulk-tier L0 conformance test.
+# BULK DOWNLOADS TIER — OPEN PREREQUISITE, intentionally NOT declared here (mirrors plex + sonarr/radarr):
+#   docs/13 row 13 places qBittorrent's downloads/staging volume on the `bulk` tier (large, non-3x-
+#   replicated media handoff to the arr stack). The `bulk` StorageClass exists
+#   (platform/longhorn/storageclasses.yaml) and both the schema and renderer support `class: bulk`, BUT
+#   it is a NON-PROVISIONING STUB pending OQ-2 (its NFS/SMB backing is REPLACE-ME and the NFS CSI driver
+#   is not installed) — a `bulk` PVC would never bind and would leave the pod Pending. Declaring it now
+#   would also pull the bulk-tier requirement into the change's "touched" set, whose in-cluster
+#   conformance test is not yet scaffolded, blocking the merge gate. So the downloads volume is DEFERRED
+#   and this onboarding ships config-only. Once OQ-2 is resolved and `bulk` provisions, attach downloads
+#   with a one-line add under storage:
+#         - name: downloads
+#           class: bulk          # external mount; size omitted (sized outside the cluster)
+#   then re-render. See the WS-APPS report for the full bulk-downloads finding.
 #
 # VPN-class (docs/13 row 13; docs/03 "VPN-class set"): ALL of this pod's public-internet egress
 # (peer/tracker/DHT traffic) MUST leave via the node PIA tunnel, never the WAN. `egress: vpn` renders
@@ -31,411 +32,13 @@ name: qbittorrent
 kind: container
 source:
   image: lscr.io/linuxserver/qbittorrent:5.1.2   # pinned stable tag (R-UPD-01); update.range governs advancement
-# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
-# LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
-# 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
-# uid=911 gid=911, and /mnt/Storage/plex/downloads is 0755 owned 1000:1000. 0755 grants write to the
-# OWNER ONLY, so uid 911 would have mounted the export cleanly and then failed every write with
-# EACCES — the "mounts fine, cannot write" failure the sonarr entry warns about at length.
-env:
-  - name: PUID
-    value: "1000"               # owner uid of /mnt/Storage/plex/downloads (and of every dir under the export)
-  - name: PGID
-    value: "3000"               # the export's dominant group, matching sonarr/radarr: files this writes stay
-                                # readable to the EXISTING non-k3s consumers of the same dataset, and land
-                                # group-owned the same way the arrs' own writes do
-  # --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
-  # Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
-  # NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
-  # NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
-  # password is provided for this session: ..."). Two real consequences, not hypotheticals:
-  #   * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
-  #     tested valid against the then-current temporary password; both would have started failing at
-  #     the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
-  #     report a failed client, not a failed import).
-  #   * The human operator had no stable login: every restart meant reading the pod log.
-  # Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
-  # same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
-  - name: QBT_WEBUI_USERNAME
-    valueFrom:
-      secretKeyRef:
-        name: qbittorrent-bootstrap
-        key: webui-username
-  - name: QBT_WEBUI_PASSWORD
-    valueFrom:
-      secretKeyRef:
-        name: qbittorrent-bootstrap
-        key: webui-password
-lifecycle:
-  # LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
-  #
-  # WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
-  # directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
-  # apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
-  # RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
-  # the container start rather than a clock, because the failure this closes IS a container start:
-  #   * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
-  #     PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
-  #     restore or rebuild in which the app serves a random password and both arrs are broken. The
-  #     hook has no such window: it runs before/with the very start that would otherwise regenerate.
-  #   * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
-  #     PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
-  #     then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
-  #     In-container, before the service starts, none of that exists.
-  #   * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
-  #     privilege than the CronJob alternative (R-SEC/least-privilege).
-  # The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
-  # keeps in its own state), and the schema documents this hook for exactly that use.
-  #
-  # AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
-  # BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
-  # would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
-  #   * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
-  #     credential — because the WebUI is not running yet. That assertion is the most valuable line
-  #     in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
-  #     writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
-  #   * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
-  #     mean a schema version bump plus a renderer change — a blast radius across every service's
-  #     golden — to buy something strictly weaker than what the hook already does.
-  # The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
-  # the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
-  #
-  # DURABILITY, the whole point — all three paths end with the SAME credential:
-  #   pod restart      -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
-  #   config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
-  #                       hook runs and re-asserts over whatever the snapshot contained.
-  #   rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
-  #                       credential into it before the service is serving. Nothing outside git.
-  #
-  # WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
-  # for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
-  # depend on network policy staying correct forever. The WebUI keeps demanding a password; that
-  # password is simply now known, fixed, and reproducible from git.
-  postStart:
-    exec:
-      command:
-        - /bin/bash
-        - -c
-        - |
-          # Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
-          # running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
-          #
-          # THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
-          # apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
-          # this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
-          # enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
-          # CLUSTER — but only there:
-          # envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
-          # apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
-          # development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;
-          # see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
-          # are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
-          # however it is run.
-          set -u
-          CONF=/config/qBittorrent/qBittorrent.conf
-          SVC=/run/service/svc-qbittorrent
-          # The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
-          # ServiceDefinition — which is the single place the renderer takes the containerPort, the
-          # Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
-          # env var for a lookup to read anyway.
-          # It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
-          # that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
-          # so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
-          # matched. Every curl below then got an unparseable URL, returned nothing, and the hook
-          # logged "the WebUI rejected the durable
-          # credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
-          # service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
-          # against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
-          PORT=8080
-          # A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
-          # that only echoes is effectively silent (gitea's hits the same wall and writes a marker
-          # file for it). Every outcome therefore also lands, timestamped, on the persistent config
-          # volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
-          # credential from git was asserted and accepted on this start.
-          log() {
-            echo "[qbt-webui-cred] $*"
-            printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
-              >> /config/qbittorrent-webui-cred.log 2>/dev/null || true
-          }
-
-          # (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
-          #     before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
-          #     racing to create one it would then not overwrite.
-          n=0
-          while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
-
-          waited=0
-          attempt=0
-          while [ "$attempt" -lt 3 ]; do
-            attempt=$((attempt + 1))
-
-            # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
-            #     10 = rewritten, anything else = could not write.
-            python3 - "$CONF" <<'PY'
-          import base64, hashlib, hmac, os, stat, sys
-
-          conf = sys.argv[1]
-          user = os.environ.get("QBT_WEBUI_USERNAME", "")
-          pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
-          if not user or not pw:
-              sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
-              raise SystemExit(1)
-
-          # qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
-          # iterations, 16-byte random salt, 64-byte digest, stored as
-          #   WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
-          # Get any one of them wrong and the app simply rejects every login.
-          ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
-          UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
-
-          def matches(raw):
-              """True iff the stored digest is one OUR password produces (salt is read from it)."""
-              v = raw.strip().strip('"')
-              if v.startswith("@ByteArray(") and v.endswith(")"):
-                  v = v[len("@ByteArray("):-1]
-              try:
-                  salt_b64, digest_b64 = v.split(":", 1)
-                  salt = base64.b64decode(salt_b64, validate=True)
-                  digest = base64.b64decode(digest_b64, validate=True)
-              except Exception:
-                  return False
-              if not salt or not digest:
-                  return False
-              return hmac.compare_digest(
-                  hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
-
-          try:
-              lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
-          except FileNotFoundError:
-              lines = []
-
-          def value_of(key):
-              for i, line in enumerate(lines):
-                  if line.startswith(key + "="):
-                      return i, line.split("=", 1)[1]
-              return -1, None
-
-          ui, uval = value_of(UKEY)
-          pi, pval = value_of(PKEY)
-          if uval == user and pval is not None and matches(pval):
-              raise SystemExit(0)          # converged - leave the file (and its salt) alone
-
-          salt = os.urandom(SALT_LEN)
-          digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
-          stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
-          desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
-
-          out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
-          if SECTION in out:
-              at = out.index(SECTION) + 1
-          else:
-              if out and out[-1].strip():
-                  out.append("")
-              out.append(SECTION)
-              at = len(out)
-          out[at:at] = desired
-
-          try:
-              os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
-              tmp = conf + ".webui-cred.new"
-              with open(tmp, "w", encoding="utf-8") as fh:
-                  fh.write("\n".join(out) + "\n")
-                  fh.flush()
-                  os.fsync(fh.fileno())
-              # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
-              # cost the app the ability to save its own settings, so carry the ownership over.
-              try:
-                  st = os.stat(conf)
-                  os.chown(tmp, st.st_uid, st.st_gid)
-                  os.chmod(tmp, stat.S_IMODE(st.st_mode))
-              except FileNotFoundError:
-                  os.chown(tmp, int(os.environ.get("PUID", "1000")),
-                           int(os.environ.get("PGID", "1000")))
-                  os.chmod(tmp, 0o664)
-              os.replace(tmp, conf)        # atomic: the app never reads a half-written conf
-          except OSError as exc:
-              sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
-              raise SystemExit(1)
-          raise SystemExit(10)
-          PY
-            rc=$?
-
-            if [ "$rc" = 0 ]; then
-              log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
-            elif [ "$rc" = 10 ]; then
-              log "asserted WebUI Username + Password_PBKDF2 into $CONF"
-              # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
-              # have read the old, credential-less conf and be serving a random temporary password.
-              # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
-              # Nothing to do when the service has not started yet - it will read the new file.
-              # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
-              # 2026-09-13 failure spent the entire verification budget before the first login and
-              # then fired attempts 2 and 3 with no wait left at all.
-              sv=0
-              while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
-                sv=$((sv + 1)); sleep 1
-              done
-              if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
-                log "qbittorrent-nox was already running - restarting it to load the credential"
-                s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
-                sleep 3
-              fi
-            else
-              log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
-              log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
-              exit 0
-            fi
-
-            # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
-            #     (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
-            #     at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
-            #
-            # "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
-            # conflating them is what made this hook lie once already (see the PORT comment). An
-            # unreachable WebUI is reported as unreachable, and the hook stops rather than spending
-            # its remaining login attempts to produce two more identical wrong diagnoses.
-            api_ok=0
-            while [ "$waited" -lt 120 ]; do
-              code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
-              if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
-              waited=$((waited + 1)); sleep 1
-            done
-            if [ "$api_ok" != 1 ]; then
-              log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
-              log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
-              log "never submitted. Do not rotate the credential on the strength of this line." >&2
-              exit 0
-            fi
-            body=$(curl -sS -m 10 \
-                     --data-urlencode "username=$QBT_WEBUI_USERNAME" \
-                     --data-urlencode "password=$QBT_WEBUI_PASSWORD" \
-                     "http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
-            if [ "$body" = "Ok." ]; then
-              log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
-              exit 0
-            fi
-            log "WebUI rejected the durable credential (login said '$body')" >&2
-            # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
-            # the running qbittorrent-nox is serving something it read from an EARLIER conf, so
-            # rewriting the same bytes on the next attempt would change nothing. The remedy is to
-            # make the process RE-READ the file, so restart the supervised service and let the next
-            # attempt re-verify. Without this the retry loop could never change its own outcome.
-            if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
-              log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
-              s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
-              sleep 3
-            fi
-          done
-
-          log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
-          # A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
-          # (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
-          # Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
-          # app, and the arrs' own download-client health check surfaces a broken credential.
-          exit 0
-podSecurityContext:
-  # For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
-  # on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
-  # the kubelet applies it, so /config becomes group-writable by the now-uid-1000 process without
-  # depending on the image chowning it. (This image DOES chown: init-qbittorrent-config runs
-  # `lsiown -R abc:abc /config`. Belt and braces — the live /config was 911:911 before this change,
-  # and losing write access to it would lose the WebUI password hash and the session state.)
-  fsGroup: 3000
 exposure: internal              # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
 egress: vpn                     # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
     size: 5Gi
     class: longhorn             # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
-    mountPath: /config          # THE APP WRITES /config, NOT the renderer default /data/config.
-                                # LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
-                                # /config), so without this the Longhorn PVC mounted at /data/config sat
-                                # holding nothing but lost+found while the app wrote its entire
-                                # configuration to the container's EPHEMERAL writable layer — destroyed on
-                                # every pod recreation: restart, reschedule, image bump. Measured in-cluster
-                                # 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
-                                # dev=2053 on the node's own disk with [Sentry asp config.xml logs].
-                                # This is the SAME failure catalog/plex.yaml:54 already records fixing.
-                                # The live /config contents were copied onto each PVC before this landed,
-                                # so API keys and credentials other apps reference were preserved.
-  # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
-  # (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
-  # /downloads, which is precisely where this app is ALREADY configured to write.
-  #
-  # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
-  #   * LEAST PRIVILEGE, AND IT COSTS NOTHING HERE. A download client has no business writing the
-  #     LIBRARY. The arrs need the root mount because THEY hardlink downloads/ -> tv/ or movies/ and
-  #     link() is a same-MOUNT operation (EXDEV otherwise) — their smallest common ancestor genuinely
-  #     IS the export root. qBittorrent performs no such cross-directory link: it writes into the
-  #     downloads tree and nothing else, so narrowing to that subtree removes movies/, tv/ and photos/
-  #     from this pod's mount namespace without removing any operation it actually performs.
-  #   * HARDLINKS STILL WORK, because they are not done here. The arrs still see ONE mount containing
-  #     both /media/downloads and /media/tv, so their link() stays intra-mount. Proven live 2026-09-12:
-  #     a file created from this pod under /downloads was hardlinked from the sonarr pod between
-  #     /media/downloads/... and /media/tv/... with st_nlink 1 -> 2 and no EXDEV.
-  #   * IT IS ONE MOUNT, so the L0 test `bulk-tier-distinct-from-replicated` assertion (6) — no
-  #     workload mounts one export twice — holds. That assertion is about TWO mounts inside ONE pod;
-  #     a single subPath mount is not the EXDEV trap, two are.
-  #   * The narrowing is a CLIENT/kubelet guarantee, stated honestly: the NFS server still exports the
-  #     whole dataset to this node, so the containment is the mount namespace, not the server's ACLs —
-  #     the same class of caveat docs/BULK-NFS-STORAGE.md §5 records for plex's readOnly.
-  #
-  # NOT readOnly — this is the one workload whose whole job is WRITING here.
-  #
-  # KNOWN, DELIBERATE SIDE EFFECT ON SHARED STATE: the image's init-qbittorrent-config runs
-  # `lsiown abc:abc /downloads` (non-recursive) whenever /downloads is a mount, so starting this pod
-  # chgrps /mnt/Storage/plex/downloads from 1000:1000 to 1000:3000 on the SERVER. Benign and verified:
-  # the directory is mode 0755, whose group bits (r-x) are identical to its other bits, so no consumer
-  # — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with
-  # non-cluster consumers, which should never be a surprise.
-  #
-  # PATH AGREEMENT WITH THE ARRS IS STILL *NOT* CLOSED — see DECISIONS-LOG 2026-09-12 "Remote Path
-  # Mappings". This pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
-  # /media/downloads/<name>. Same inode, different absolute path, so Completed Download Handling needs
-  # ONE Remote Path Mapping per arr (remote host `qbittorrent.qbittorrent.svc.cluster.local`, remote
-  # path /downloads/, local path /media/downloads/).
-  #
-  # THOSE MAPPINGS ARE NOW CONFIGURED AND VERIFIED ON PROD (2026-09-12), together with the qBittorrent
-  # download client itself — which did not exist in EITHER arr until then (both
-  # `GET /api/v3/downloadclient` returned `[]`, so completed-download handling had never been wired up
-  # at all). `downloadclient/testall` reports isValid=true in both, and Sonarr's queue was observed
-  # reporting outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it
-  # present — same torrent, same client.
-  #
-  # BUT THEY LIVE IN THE ARRS' OWN DATABASES (app state on their config PVCs), NOT IN GIT: a rebuild
-  # from git alone does NOT restore them and no gate here notices their absence. The alternative —
-  # repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead —
-  # buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was
-  # not chosen because it changes a running download client's save path, which the deferral above
-  # already shows is easy to get wrong. It is nonetheless the RECOMMENDED end state (it makes the
-  # mappings unnecessary and puts the mountPath half of the fix in git) and is awaiting an operator
-  # decision, not a technical blocker.
-  #
-  # RELATED OPEN BLOCKER: this app has no durable WebUI credential (no WebUI\Username /
-  # WebUI\Password_PBKDF2 in qBittorrent.conf), so v5 issues a NEW temporary `admin` password on every
-  # start. The arrs' saved download-client password therefore goes stale on every pod restart. See the
-  # same DECISIONS-LOG entry for the two fixes, neither of which was taken unilaterally.
-  - name: downloads
-    class: bulk
-    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export
-                                # plex (ro) and sonarr/radarr (rw) mount; one dataset, one set of inodes
-    subPath: downloads          # the handoff subtree ONLY: no movies/, no tv/, no photos/
-    mountPath: /downloads       # matches the app's EXISTING Session\DefaultSavePath / TempPath, so no
-                                # change to config-PVC app state is needed to make this take effect
-    size: 9Ti                   # ADVISORY ONLY on a static NFS PV (the ZFS pool is the real limit): the
-                                # dataset's 8.4 T used rounded up, stated the same way plex/sonarr/radarr
-                                # state it so `kubectl get pv` reports something true
-dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02).
-                                # STILL EMPTY WITH THE NFS MOUNT ABOVE, AND THAT IS CORRECT, NOT AN
-                                # OVERSIGHT. An NFS PersistentVolume is mounted by the NODE KERNEL in the
-                                # HOST netns, not by this pod: there is no pod-side socket to 2049, so no
-                                # CiliumNetworkPolicy can permit or deny it and none is needed. Verified by
-                                # construction — sonarr and radarr have mounted this same export read-write
-                                # for weeks and neither has any CNP naming the NFS server or port 2049
-                                # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
-                                # egress gateway off this traffic (its selector is a podSelector), and
-                                # 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
-                                # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
+dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
   - host: qbittorrent.${BASE_DOMAIN}
     port: 8080                  # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
@@ -443,11 +46,6 @@ auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
   groups:
     - media-users               # authenticated AND in an authorized group (R-IDP-06)
-    # See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
-    # platform-admins here is a superset, not a widening (that group already carries cluster-admin
-    # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
-    # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
-    - platform-admins           # operators of these services must not be locked out of them
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the qBittorrent 5.x stable line

catalog/radarr.yaml

@@ -9,23 +9,6 @@ name: radarr
 kind: container
 source:
   image: lscr.io/linuxserver/radarr:5.21.1   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
-# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
-# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
-# which cannot write a 1000:3000-owned 0755 tree.
-env:
-  - name: PUID
-    value: "1000"               # owner uid of every directory under /mnt/Storage/plex
-  - name: PGID
-    value: "3000"               # dominant group there; keeps files this writes readable to the
-                                # EXISTING non-k3s consumers of the same export, which stay live
-  - name: TZ
-    value: Etc/UTC              # same safe default plex uses; the arrs schedule by local time
-podSecurityContext:
-  # For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
-  # does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
-  # what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
-  # /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
-  fsGroup: 3000
 exposure: internal              # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
@@ -33,66 +16,11 @@ storage:
   - name: config
     size: 5Gi
     class: longhorn             # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
-    mountPath: /config          # THE APP WRITES /config, NOT the renderer default /data/config.
-                                # LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
-                                # /config), so without this the Longhorn PVC mounted at /data/config sat
-                                # holding nothing but lost+found while the app wrote its entire
-                                # configuration to the container's EPHEMERAL writable layer — destroyed on
-                                # every pod recreation: restart, reschedule, image bump. Measured in-cluster
-                                # 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
-                                # dev=2053 on the node's own disk with [Sentry asp config.xml logs].
-                                # This is the SAME failure catalog/plex.yaml:54 already records fixing.
-                                # The live /config contents were copied onto each PVC before this landed,
-                                # so API keys and credentials other apps reference were preserved.
-  # THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
-  # /media/movies (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
-  #
-  # *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Radarr imports by HARDLINKING or
-  # atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
-  # and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
-  #     /mnt/Storage/plex/movies/                      the library
-  #     /mnt/Storage/plex/downloads/complete/radarr/  where finished releases land
-  # — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
-  # them as two subPath volumes (subPath: movies at /media/movies, subPath: downloads elsewhere) creates
-  # TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Radarr then
-  # silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
-  # seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
-  # is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
-  # requirement, not a shortcut. (The cost is stated plainly: Radarr can also write /media/tv,
-  # which is sonarr's. Narrowing that would cost hardlinks, which is a worse trade.)
-  #
-  # *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
-  # renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
-  # config volume entirely; the pod would come up with its app database shadowed by an 8 T media
-  # tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
-  # the same export, so when qbittorrent is eventually given this mount too, the arrs and the
-  # download client will already agree on absolute paths — which is the other precondition for
-  # hardlinked imports (otherwise every import needs a Remote Path Mapping).
-  #
-  # SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
-  # is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
-  # with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
-  # Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
-  # documents having already been bitten by. Fixing it means starting from an empty config volume, so
-  # it belongs in its own change with the operator present.
-  #
-  # UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
-  # + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
-  # the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
-  # write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
-  # every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
-  #
-  # AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
-  # ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
-  # lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
-  # that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
-  # the image re-chowning /config for itself.
-  - name: media
-    class: bulk
-    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
-    mountPath: /media           # => /media/movies (Root Folder) and /media/downloads (import source)
-    size: 9Ti                   # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
-                                # rounded up, so `kubectl get pv` reports something true.
+  # NOTE (bulk-media PREREQUISITE): docs/13 row 3 places the media library / downloads volume on the
+  # `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
+  # but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
+  # volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
+  # therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Radarr deployable today.
 dependencies:
   - to: qbittorrent             # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 8080
@@ -107,11 +35,6 @@ auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
   groups:
     - media-users               # authenticated AND in the media group (R-IDP-06)
-    # See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
-    # platform-admins here is a superset, not a widening (that group already carries cluster-admin
-    # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
-    # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
-    - platform-admins           # operators of these services must not be locked out of them
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the Radarr v5 stable line

catalog/render.py

@@ -4,8 +4,7 @@
 Consumes ONE ServiceDefinition (schemas/servicedefinition.schema.json, catalog/*.yaml)
 and emits the concrete manifest set the renderer contract specifies
 (schemas/renderer-contract.md): Namespace + default-deny baseline, Deployment/Service or
-KubeVirt VirtualMachine + CDI DataVolume, PVCs, per-dependency and per-in-cluster-client
-CiliumNetworkPolicy allows,
+KubeVirt VirtualMachine + CDI DataVolume, PVCs, per-dependency CiliumNetworkPolicy allows,
 egress-class steering label, HTTPRoute(s), ServiceMonitor/alerts, Flux image-automation,
 and the auth-mode / backup wiring hints.
 
@@ -54,54 +53,9 @@ LBL = "homelab.keithjackson.dev"
 # Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
 # and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10).
 ZONE = "${BASE_DOMAIN}"
-# The bulk (external NFS) tier's SERVER, emitted as a Flux post-build TOKEN for exactly the same
-# reason as ZONE: dev and prod reach the same Proxmox ZFS host by DIFFERENT addresses (dev's nodes
-# are on the host's own LAN and dial it directly; prod's nodes are on the far side of pfSense and
-# must be given whichever address the still-pending routing decision produces). Hardcoding either
-# one authors a manifest that is silently wrong in the other environment. Declared in
-# clusters/base/cluster-vars.yaml and patched by BOTH overlays; written BARE (it is a real knob,
-# not a shell expansion), so StrictPostBuildSubstitutions fails the tier if an overlay forgets it.
-BULK_NFS_SERVER = "${BULK_NFS_SERVER}"
-# Mount options for every bulk NFS volume. These are a PLATFORM decision, not a per-service one,
-# so they live here rather than on the ServiceDefinition surface:
-#   nfsvers=4.1  NFSv4 carries mount+lock IN the protocol, so it needs ONE port (tcp/2049) and no
-#                rpcbind/111, mountd or statd. That is what makes the prod path a single pfSense
-#                rule instead of four, and it is why v4 is chosen over v3. The server advertises
-#                +3 +4 +4.1 +4.2; 4.1 is the conservative floor every node kernel already speaks.
-#   hard         A `soft` mount turns a server hiccup into a short read / EIO handed to the
-#                application, which for a media library and an archive is silent corruption.
-#                `hard` blocks instead. The cost is explicit: an NFS outage HANGS I/O rather than
-#                failing it, so an outage shows up as stuck pods, not as damaged files. Chosen
-#                deliberately -- hung is recoverable, truncated is not.
-#   noatime      The datasets are relatime, so every read would otherwise generate a metadata
-#                WRITE back to the pool. Plex walking an 8 T library must not do that.
-#   timeo=600/retrans=2  the TCP defaults, stated explicitly so a future reader knows the retry
-#                budget was considered rather than inherited.
-BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retrans=2"]
 GATEWAY_NAME = "shared"
 GATEWAY_NS = "gateway"
 KP_RELEASE = "kube-prometheus-stack"  # ServiceMonitor/PrometheusRule discovery label
-MONITORING_NS = "monitoring"          # where Prometheus runs; the only admitted scrape source
-
-# The kube-state-metrics series a workload's readiness IS (r_prometheusrule). kube-state-metrics
-# is scraped by the bundled kube-prometheus-stack ServiceMonitor, i.e. these exist for EVERY
-# workload in the cluster with no per-service exporter — which is the whole point of using them
-# for `slis: [up]`. Each entry is (metric, label naming the object). VERIFIED on prod 2026-09-13:
-# kube_deployment_status_replicas_available returned a series for all ten deployed app
-# Deployments, and kube_statefulset_status_replicas_ready{namespace="keycloak",
-# statefulset="keycloak"} returned 2.
-LIVENESS_METRICS = {
-    "Deployment": ("kube_deployment_status_replicas_available", "deployment"),
-    "StatefulSet": ("kube_statefulset_status_replicas_ready", "statefulset"),
-    "DaemonSet": ("kube_daemonset_status_number_available", "daemonset"),
-}
-# A KubeVirt VMI is not a replica count: kubevirt_vmi_info carries the phase as a LABEL, so
-# liveness is the EXISTENCE of a running-phase series. The value is lowercase `running` — a
-# capital-R matcher selects nothing and is just as inert (the same trap that made
-# kubevirt_vmi_phase_count{phase="Failed"} a no-op; verified on prod, where the one live VMI
-# reports phase="running").
-VMI_METRIC = "kubevirt_vmi_info"
-VMI_RUNNING_PHASE = "running"
 
 # egress classes and auth modes fixed by docs/03 / docs/13 + the Kyverno enums.
 EGRESS_CLASSES = ("internet", "vpn")
@@ -138,7 +92,6 @@ def apply_defaults(sd: dict) -> dict:
     s.setdefault("storage", [])
     s.setdefault("dependencies", [])
     s.setdefault("ingressRoutes", [])
-    s.setdefault("inClusterClients", [])
 
     # auth.mode default posture: forward-auth for any exposed UI, else none (internal, no
     # UI). An explicit auth block always wins (R-IDP-02).
@@ -156,41 +109,12 @@ def apply_defaults(sd: dict) -> dict:
     obs = dict(s.get("observability") or {})
     obs.setdefault("metrics", True)
     obs.setdefault("slis", [])
-    # metricsEndpoint is NEVER defaulted (schema 1.6.0). `metrics: true` says the platform
-    # collects signals FOR this service; it cannot say the workload SERVES a Prometheus
-    # endpoint, because for most of this catalog it does not (see r_servicemonitor). When the
-    # endpoint IS declared, its name/path/interval carry defaults so the common case is
-    # `metricsEndpoint: {port: 9100}`.
-    ep = obs.get("metricsEndpoint")
-    if ep:
-        obs["metricsEndpoint"] = {
-            **{"name": "metrics", "path": "/metrics", "interval": "30s"},
-            **ep,
-        }
-    # livenessWorkload: WHICH Kubernetes object's readiness IS this service being up. Defaulted
-    # from `kind` — a container renders a Deployment named after the service, a vm renders a
-    # VirtualMachine whose VMI carries the same name — and overridable for the case where the
-    # object that actually runs on the cluster is not the one this SD renders (keycloak: the
-    # Keycloak operator runs a StatefulSet, so a Deployment-shaped rule would `absent()` forever).
-    lw = dict(obs.get("livenessWorkload") or {})
-    lw.setdefault("kind", "VirtualMachineInstance" if s["kind"] == "vm" else "Deployment")
-    lw.setdefault("name", s["name"])
-    obs["livenessWorkload"] = lw
     s["observability"] = obs
 
-    # storage entry class default is longhorn (replicated tier); readOnly default is False
-    # (grant-nothing-extra is expressed by SETTING it, so the default stays back-compatible and
-    # every pre-existing golden renders byte-identical).
-    s["storage"] = [{**{"class": "longhorn", "readOnly": False}, **v} for v in s["storage"]]
+    # storage entry class default is longhorn (replicated tier).
+    s["storage"] = [{**{"class": "longhorn"}, **v} for v in s["storage"]]
     # dependency protocol default is tcp.
     s["dependencies"] = [{**{"protocol": "tcp"}, **d} for d in s["dependencies"]]
-    # in-cluster client protocol default is tcp (as for a dependency); its `name` — which only
-    # ever appears in the rendered POLICY name — defaults to the namespace it admits, so the
-    # common case declares just `namespace` + `port` and reads like a dependency edge.
-    s["inClusterClients"] = [
-        {**{"protocol": "tcp", "name": c.get("namespace", "")}, **c}
-        for c in s["inClusterClients"]
-    ]
     return s
 
 
@@ -276,139 +200,18 @@ def _service_ports(s: dict) -> list:
     return [{"name": f"port-{p}", "port": p, "targetPort": p} for p in ports]
 
 
-def _bulk_access_mode(vol: dict) -> str:
-    """A bulk volume's access mode follows its read/write posture.
-
-    Every bulk volume is `...Many`: the SAME export is consumed by several namespaces at once
-    (plex, sonarr and radarr all sit on /mnt/Storage/plex) and by several nodes at once, which is
-    precisely what ReadWriteOnce cannot express. WHICH `Many` it is says whether this consumer
-    writes -- so `kubectl get pv` reports the intent, not merely the topology.
-    """
-    return "ReadOnlyMany" if vol.get("readOnly") else "ReadWriteMany"
-
-
-def r_bulk_pvs(s: dict) -> list:
-    """One STATIC PersistentVolume per `class: bulk` storage entry (docs/04 bulk tier).
-
-    WHY A PV AT ALL, RATHER THAN AN INLINE `nfs:` POD VOLUME
-    -------------------------------------------------------
-    An inline volume is fewer objects and it would work. It is rejected for four reasons, in
-    descending order of what they would cost to discover later:
-
-      1. RECLAIM POLICY. `persistentVolumeReclaimPolicy: Retain` is a declared, reviewable
-         property of a PV. There is no equivalent for an inline volume -- and the data behind
-         these mounts is 8.33 TiB of media and a 267 GiB archive that ALSO has live non-k3s
-         consumers. A statement in git that nothing in this cluster may reclaim it is worth an
-         object.
-      2. READ-ONLY LIVES IN ONE PLACE. `spec.nfs.readOnly: true` becomes a real `ro` mount option
-         for every pod that ever binds this volume. The same intent expressed on an inline volume
-         is one Deployment edit away from being dropped silently, with no diff anywhere near the
-         storage. (It is still a CLIENT-side guarantee -- the export cannot distinguish plex from
-         sonarr, since both may run on any node -- but it is one that lives in one reviewable
-         place.)
-      3. VISIBILITY. `kubectl get pv,pvc` is where an operator looks for "what external storage is
-         attached, and who holds it". An inline volume is invisible there, and invisible to
-         anything that reasons over PVCs.
-      4. IT IS THE CONTRACT THE REPO ALREADY HAS. Every other storage entry is a PVC; the
-         renderer, the Kyverno ServiceDefinition policy and the backup wiring all key on that. An
-         inline volume would be a second, parallel storage model inside the same renderer.
-
-    CROSS-NAMESPACE. A PV binds 1:1 to a PVC, so "plex, sonarr and radarr share one export" is NOT
-    one shared PV -- it is three PVs, one per namespace, each naming the same server+path. That is
-    the correct model rather than a workaround: the sharing happens on the NFS server, which is
-    built for concurrent clients, and each namespace keeps its own independently reviewable access
-    mode (plex ReadOnlyMany, the arrs ReadWriteMany). One shared PV would have forced a single
-    posture on all three.
-
-    BINDING IS PINNED FROM BOTH ENDS -- `claimRef` here and `volumeName` on the PVC. Either alone
-    leaves a race in which some other PVC of matching size and mode grabs the volume first;
-    together the binding is deterministic. `storageClassName: ""` (explicitly empty, NOT the `bulk`
-    StorageClass) is what tells the binder this volume is statically provisioned: it suppresses the
-    default-StorageClass annotation and guarantees no provisioner is ever asked to create anything.
-    It also sidesteps a trap -- a StorageClass's `provisioner` is IMMUTABLE, so re-pointing the
-    existing `bulk` class (whose stub names an NFS CSI driver that is not installed) at this data
-    would make Flux fail forever on that one object.
-    """
-    out = []
-    for vol in s["storage"]:
-        if vol.get("class") != "bulk":
-            continue
-        name = f"{s['name']}-{vol['name']}"
-        # `export` and `size` are both structurally required of a bulk entry, and the JSON
-        # Schema only enforces the first. A PersistentVolume with no `capacity` is rejected by
-        # the API server, so a missing size must fail HERE with a sentence a human can act on
-        # rather than as a bare KeyError three frames down.
-        for field, why in (
-            ("export", "the absolute NFS export path this volume attaches"),
-            ("size", "the PV capacity (advisory, but the API requires the field)"),
-        ):
-            if not vol.get(field):
-                raise ValueError(
-                    f"{s['name']}: storage entry {vol['name']!r} has class 'bulk' but no "
-                    f"'{field}' — {why}. A bulk volume attaches storage that already exists "
-                    f"outside the cluster, so it cannot be rendered without it."
-                )
-        nfs = {"server": BULK_NFS_SERVER, "path": vol["export"]}
-        if vol.get("readOnly"):
-            nfs["readOnly"] = True
-        spec = {
-            "capacity": {"storage": vol["size"]},
-            "accessModes": [_bulk_access_mode(vol)],
-            # Retain: deleting the PVC must NEVER be able to reach the data. There is nothing to
-            # "reclaim" here in any case -- the volume was not provisioned by the cluster.
-            "persistentVolumeReclaimPolicy": "Retain",
-            "storageClassName": "",
-            "mountOptions": list(BULK_NFS_MOUNT_OPTIONS),
-            "nfs": nfs,
-            "claimRef": {"namespace": s["name"], "name": name},
-        }
-        out.append(
-            {
-                "apiVersion": "v1",
-                "kind": "PersistentVolume",
-                "metadata": {
-                    "name": name,
-                    "labels": {"app.kubernetes.io/name": s["name"]},
-                },
-                "spec": spec,
-            }
-        )
-    return out
-
-
 def r_pvcs(s: dict) -> list:
     """One PVC per storage entry (R-STATE-01/02/04).
 
-    longhorn -> 3-replica replicated tier, dynamically provisioned, ReadWriteOnce.
-    bulk     -> the STATIC external NFS PV rendered by r_bulk_pvs, pre-bound by volumeName.
+    longhorn -> 3-replica replicated tier; bulk -> external media tier (not 3x replicated).
     """
     out = []
     for vol in s["storage"]:
         cls = vol.get("class", "longhorn")
-        if cls == "bulk":
-            # Mirror image of the PV: the same explicitly-empty class (so nothing dynamically
-            # provisions), the same access mode, and volumeName completing the two-sided binding.
-            out.append(
-                {
-                    "apiVersion": "v1",
-                    "kind": "PersistentVolumeClaim",
-                    "metadata": {
-                        "name": f"{s['name']}-{vol['name']}",
-                        "namespace": s["name"],
-                        "labels": {"app.kubernetes.io/name": s["name"]},
-                    },
-                    "spec": {
-                        "accessModes": [_bulk_access_mode(vol)],
-                        "storageClassName": "",
-                        "volumeName": f"{s['name']}-{vol['name']}",
-                        "resources": {"requests": {"storage": vol["size"]}},
-                    },
-                }
-            )
-            continue
+        storage_class = "longhorn" if cls == "longhorn" else "bulk"
         spec = {
             "accessModes": ["ReadWriteOnce"],
-            "storageClassName": "longhorn",
+            "storageClassName": storage_class,
         }
         if vol.get("size"):
             spec["resources"] = {"requests": {"storage": vol["size"]}}
@@ -454,17 +257,7 @@ def r_deployment(s: dict) -> dict:
         # it verbatim for apps that need a fixed on-disk layout (e.g. Gitea keeps all state
         # directly under /data). Omitting it preserves the /data/<name> default (back-compat).
         mount_path = vol.get("mountPath", f"/data/{vol['name']}")
-        mount = {"name": vol["name"], "mountPath": mount_path}
-        # subPath mounts a subtree of the volume instead of its root. Emitted only when declared,
-        # so every pre-existing golden stays byte-identical.
-        if vol.get("subPath"):
-            mount["subPath"] = vol["subPath"]
-        # readOnly on the mount as well as on the PV's nfs source. Belt and braces on purpose: the
-        # PV field is what makes the kernel mount `ro`; this one is what a reviewer reading the
-        # Deployment sees. An edit that drops one still leaves the other.
-        if vol.get("readOnly"):
-            mount["readOnly"] = True
-        volume_mounts.append(mount)
+        volume_mounts.append({"name": vol["name"], "mountPath": mount_path})
         volumes.append(
             {
                 "name": vol["name"],
@@ -529,12 +322,7 @@ def r_deployment(s: dict) -> dict:
 
 def r_service(s: dict):
     ports = _service_ports(s)
-    # A declared metrics endpoint becomes a NAMED Service port, because that name is the only
-    # thing a ServiceMonitor endpoint can select on. Appended BELOW rather than inside
-    # _service_ports so it never leaks into r_gateway_ingress_policy's north-south allow list —
-    # the metrics port is for the monitoring namespace, not for the Gateway.
-    ep = s["observability"].get("metricsEndpoint")
-    if not ports and not ep:
+    if not ports:
         return None
     # forward-auth (Option B): the shared Gateway must reach the oauth2-proxy sidecar, so front it
     # with a `authproxy` :4180 Service port (the HTTPRoute backendRef targets this). The app UI/API
@@ -544,10 +332,6 @@ def r_service(s: dict):
         ports = ports + [
             {"name": "authproxy", "port": OAUTH2_PROXY_PORT, "targetPort": OAUTH2_PROXY_PORT}
         ]
-    if ep:
-        ports = ports + [
-            {"name": ep["name"], "port": ep["port"], "targetPort": ep["port"]}
-        ]
     return {
         "apiVersion": "v1",
         "kind": "Service",
@@ -759,82 +543,6 @@ def r_gateway_ingress_policy(s: dict):
     }
 
 
-def r_in_cluster_client_policies(s: dict) -> list:
-    """East-west allow: let a declared IN-CLUSTER client reach this service DIRECTLY (R-NET-02).
-
-    `ingressRoutes` opens the NORTH-SOUTH path only — r_gateway_ingress_policy admits the
-    reserved `ingress` identity (the Gateway proxy) and nothing else — and `dependencies`
-    compiles edges only BETWEEN CATALOG SERVICES (both halves are rendered from a catalog
-    entry's own declaration). A PLATFORM CONTROLLER that dials this service's ClusterIP
-    directly therefore had no authorable form at all, and under default-deny (R-NET-01) it is
-    DROPPED.
-
-    That gap is not theoretical: prod's `flux-system` source-controller clones the platform's
-    own repo from the in-cluster Gitea Service (clusters/prod/kustomization.yaml pins
-    `url: http://gitea.gitea.svc.cluster.local:3000/...` so prod's GitOps depends on neither
-    public DNS, Cloudflare, its own Gateway nor ${BASE_DOMAIN}). gitea-allow-gateway-ingress
-    admits `fromEntities: [ingress]` only, so the clone was denied, the source could not
-    reconcile — and the HTTPRoute that would have served a public name instead lives in a
-    DOWNSTREAM tier that cannot apply while the source is down. Source down -> routes never
-    rename -> source stays down. It had to be broken with a live `kubectl apply`, i.e. with a
-    cluster object outside git, which one `kubectl delete` or one rebuild would take away.
-
-    Each entry renders ONE least-privilege ingress CiliumNetworkPolicy, shaped exactly like the
-    reciprocal-dependency half (r_reverse_dependency_policies) but selecting the client by
-    NAMESPACE alone, because a platform controller's pod labels are the upstream chart's to
-    change and are not part of this platform's contract: `fromEndpoints` carrying ONLY the
-    reserved `k8s:io.kubernetes.pod.namespace` label, `toPorts` carrying ONLY the declared
-    port/protocol. Never `fromEntities: all`, and never the whole service. Additive: the
-    namespace default-deny baseline keeps denying every undeclared source and port.
-
-    The policy NAME is `<service>-allow-<name>-ingress`, `name` defaulting to the namespace —
-    it is operator-facing (`kubectl get cnp`, a Hubble drop), and letting an author pin it is
-    what allows an allowance that already had to be applied live to be authored under its
-    EXISTING name, so Flux adopts that object in place rather than creating a near-duplicate
-    beside it and leaving the live one unmanaged.
-    """
-    out = []
-    for client in s["inClusterClients"]:
-        proto = client.get("protocol", "tcp").upper()
-        out.append(
-            {
-                "apiVersion": "cilium.io/v2",
-                "kind": "CiliumNetworkPolicy",
-                "metadata": {
-                    "name": f"{s['name']}-allow-{client['name']}-ingress",
-                    "namespace": s["name"],
-                    "labels": {"app.kubernetes.io/name": s["name"]},
-                },
-                "spec": {
-                    "endpointSelector": {
-                        "matchLabels": {"app.kubernetes.io/name": s["name"]}
-                    },
-                    "ingress": [
-                        {
-                            "fromEndpoints": [
-                                {
-                                    "matchLabels": {
-                                        "k8s:io.kubernetes.pod.namespace": client[
-                                            "namespace"
-                                        ]
-                                    }
-                                }
-                            ],
-                            "toPorts": [
-                                {
-                                    "ports": [
-                                        {"port": str(client["port"]), "protocol": proto}
-                                    ]
-                                }
-                            ],
-                        }
-                    ],
-                },
-            }
-        )
-    return out
-
-
 def _route_meta(s: dict, route: dict) -> dict:
     meta = {
         "name": f"{s['name']}-{route['port']}",
@@ -1040,33 +748,7 @@ def _auth_annotations(s: dict) -> dict:
 
 
 def r_servicemonitor(s: dict):
-    """ServiceMonitor — rendered ONLY when the SD says WHERE metrics are served.
-
-    THIS FUNCTION USED TO RENDER A MONITOR UNCONDITIONALLY FROM `metrics: true`, hardcoding
-    `endpoints: [{port: metrics}]`. No Service this renderer emits had ever defined a port of
-    that name, so all twelve monitors discovered ZERO targets: no `up{job="<app>"}` series
-    existed anywhere, `up{job="<app>"} == 0` could not fire, and `PrometheusTargetsMissing`
-    could not see the hole either, because a target that was never discovered has no `up` series
-    to be 0. Verified on prod 2026-09-13: `count by (job) (up)` listed 24 jobs and not one of
-    the twelve appeared; `up{job="plex"}` and `up{job="gitea"}` both returned nothing.
-
-    THE ROOT CAUSE WAS NOT A TYPO — it was that `metrics: true` cannot state a fact the workload
-    does not have. Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and
-    Nextcloud serve no Prometheus endpoint at all without a separate exporter; Gitea can serve
-    one but only once its app.ini enables it. So the port name was never the missing piece. A
-    monitor is now rendered only from `observability.metricsEndpoint`, which names the port the
-    workload actually serves — and that same declaration renders the Service port (r_service) and
-    the monitoring-namespace ingress allow (r_monitoring_ingress_policy), so the three halves of
-    a working scrape cannot be authored apart.
-
-    `metrics: true` still means something without it: logs ship to Loki via the node agent, and
-    liveness comes from kube-state-metrics (r_prometheusrule) — neither needs a per-service CR.
-    """
-    obs = s["observability"]
-    if not obs["metrics"]:
-        return None
-    ep = obs.get("metricsEndpoint")
-    if not ep:
+    if not s["observability"]["metrics"]:
         return None
     return {
         "apiVersion": "monitoring.coreos.com/v1",
@@ -1079,108 +761,7 @@ def r_servicemonitor(s: dict):
         "spec": {
             "selector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
             "namespaceSelector": {"matchNames": [s["name"]]},
-            "endpoints": [
-                {"port": ep["name"], "path": ep["path"], "interval": ep["interval"]}
-            ],
-        },
-    }
-
-
-def r_monitoring_ingress_policy(s: dict):
-    """Let the monitoring namespace reach a declared metrics port (R-NET-02, R-OBS-01).
-
-    renderer-contract.md has always said the default-deny baseline carries "a standing allow ...
-    for observability scraping from the monitoring namespace" — and no renderer output ever did.
-    Every app namespace ships `ingress: []`, so a scrape of an app metrics port would be DROPPED
-    by policy even with the Service port and the monitor endpoint in agreement. That is the same
-    trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
-    for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.
-
-    Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
-    monitoring namespace: an allow for a port nothing serves would be a standing hole for no
-    benefit.
-    """
-    ep = s["observability"].get("metricsEndpoint")
-    if not ep or not s["observability"]["metrics"]:
-        return None
-    return {
-        "apiVersion": "cilium.io/v2",
-        "kind": "CiliumNetworkPolicy",
-        "metadata": {
-            "name": f"{s['name']}-allow-monitoring-scrape",
-            "namespace": s["name"],
-            "labels": {"app.kubernetes.io/name": s["name"]},
-        },
-        "spec": {
-            "endpointSelector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
-            "ingress": [
-                {
-                    "fromEndpoints": [
-                        {"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
-                    ],
-                    "toPorts": [
-                        {"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
-                    ],
-                }
-            ],
-        },
-    }
-
-
-def _up_sli_rule(s: dict) -> dict:
-    """`slis: [up]` -> a `<name>-down` alert against a signal THAT EXISTS on this cluster.
-
-    IT USED TO READ `up{job="<name>"} == 0`. That is the textbook expression and it was inert
-    for all twelve services in the catalog, because `up` only exists for a target Prometheus
-    actually DISCOVERED and the rendered ServiceMonitor discovered none (see r_servicemonitor).
-    Worse than a wrong threshold: an inert rule and a correct rule are indistinguishable on a
-    healthy cluster, and the generic safety net cannot see the difference either — a target that
-    was never discovered has no `up` series for PrometheusTargetsMissing to find at 0.
-
-    Rewriting the SLI against the app's own /metrics was NOT an option for most of this catalog:
-    Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export no
-    Prometheus metrics without a separate exporter sidecar per app. So the `up` SLI is expressed
-    against kube-state-metrics, which IS scraped and DOES carry a series for every workload — the
-    same idiom KeycloakDown in platform/monitoring/alerts.yaml has used since it was fixed.
-
-    The `or absent(...)` half is deliberate and load-bearing: it makes the DISAPPEARANCE of the
-    signal fire the alert instead of silencing it. A rule that can only fire on `== 0` is a rule
-    that goes quiet when the workload object (or kube-state-metrics) vanishes, which is precisely
-    how the old expression failed. It is also why this rule cannot be permanently firing where the
-    workload exists: the series is present and non-zero on a healthy cluster (verified on prod).
-    """
-    lw = s["observability"]["livenessWorkload"]
-    kind, obj = lw["kind"], lw["name"]
-    ns = s["name"]
-    if kind == "VirtualMachineInstance":
-        sel = f'{VMI_METRIC}{{namespace="{ns}", name="{obj}", phase="{VMI_RUNNING_PHASE}"}}'
-        expr = f"absent({sel})"
-        desc = (
-            f"{s['name']}: no KubeVirt VMI in phase {VMI_RUNNING_PHASE} for 5m "
-            f"(SLI: up, via {VMI_METRIC})."
-        )
-    else:
-        try:
-            metric, label = LIVENESS_METRICS[kind]
-        except KeyError:
-            raise ValueError(
-                f"{s['name']}: observability.livenessWorkload.kind {kind!r} has no liveness "
-                f"metric; known kinds: {sorted(LIVENESS_METRICS)} + VirtualMachineInstance"
-            ) from None
-        sel = f'{metric}{{namespace="{ns}", {label}="{obj}"}}'
-        expr = f"{sel} == 0 or absent({sel})"
-        desc = (
-            f"{s['name']}: its {kind} has no available/ready replica for 5m, or the signal is "
-            f"gone entirely (SLI: up, via {metric})."
-        )
-    return {
-        "alert": f"{s['name']}-down".replace("_", "-"),
-        "expr": expr,
-        "for": "5m",
-        "labels": {"severity": "critical", "service": s["name"]},
-        "annotations": {
-            "summary": f"{s['name']} is down",
-            "description": desc,
+            "endpoints": [{"port": "metrics", "interval": "30s"}],
         },
     }
 
@@ -1193,7 +774,18 @@ def r_prometheusrule(s: dict):
     rules = []
     for sli in slis:
         if sli == "up":
-            rules.append(_up_sli_rule(s))
+            rules.append(
+                {
+                    "alert": f"{s['name']}-down".replace("_", "-"),
+                    "expr": f'up{{job="{s["name"]}"}} == 0',
+                    "for": "5m",
+                    "labels": {"severity": "critical", "service": s["name"]},
+                    "annotations": {
+                        "summary": f"{s['name']} is down",
+                        "description": f"{s['name']} has been unreachable for 5m (SLI: up).",
+                    },
+                }
+            )
         else:
             rules.append(
                 {
@@ -1503,14 +1095,6 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
     if s["kind"] == "container":
         # Container PVCs. (For a vm, the CDI DataVolume owns/creates the disk PVC, so we
         # never render a plain PVC for the VM root disk — it would collide with CDI.)
-        # The static bulk PV is rendered BEFORE the PVC that names it. Document order inside one
-        # kustomize build is not an apply-ordering guarantee, but a PVC whose PV does not exist
-        # yet merely stays Pending and binds on the next reconcile, so this is a readability
-        # convention rather than a correctness requirement.
-        for pv in r_bulk_pvs(s):
-            docs.append(
-                (f"PersistentVolume {pv['metadata']['name']} - static external NFS (bulk tier)", pv)
-            )
         for pvc in r_pvcs(s):
             docs.append((f"PVC {pvc['metadata']['name']} (R-STATE)", pvc))
         docs.append(("Deployment (R-PLAT-02)", r_deployment(s)))
@@ -1543,25 +1127,6 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
             )
         )
 
-    for cnp in r_in_cluster_client_policies(s):
-        docs.append(
-            (
-                "CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): "
-                f"{cnp['metadata']['name']}",
-                cnp,
-            )
-        )
-
-    mon_allow = r_monitoring_ingress_policy(s)
-    if mon_allow:
-        docs.append(
-            (
-                "CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): "
-                f"{mon_allow['metadata']['name']}",
-                mon_allow,
-            )
-        )
-
     ie = r_internet_egress(s)
     docs.append(
         (
@@ -1679,25 +1244,6 @@ EXTRA_UNITS: dict = {
             "gitea-actions-runner-secret.sops.yaml",
         ],
     },
-    # PiHole's DNS exposure: the udp/53 + tcp/53 LoadBalancer Services on a pinned VIP, plus the
-    # narrow LAN ingress CiliumNetworkPolicy that admits them (apps/wave-0/pihole-dns.yaml).
-    # `above: False` co-locates it in pihole's OWN wave: these are the pihole pod's own Services and
-    # its own policy, applied in the same wave as the Deployment they select, exactly as the
-    # rendered Service and CNPs in pihole.yaml are. It is NOT a runtime prerequisite of anything, so
-    # there is nothing for a higher wave to gate on.
-    #
-    # It is a non-catalog unit because the ServiceDefinition schema cannot express a LoadBalancer
-    # Service, a pinned lbipam VIP, a sharing key or UDP exposure at all (additionalProperties is
-    # false, and `ingressRoutes` only renders HTTPRoutes on the L7 Gateway) — the same reason
-    # apps/vms/unifi/50-service-l4.yaml and platform/crafty/service-l4.yaml are hand-written. The
-    # split-horizon ANSWERS are in the catalog, as FTLCONF env on catalog/pihole.yaml.
-    "pihole-dns": {
-        "anchor": "pihole",
-        "above": False,
-        "files": [
-            "pihole-dns.yaml",
-        ],
-    },
 }
 
 
@@ -1775,11 +1321,6 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
       - `<svc>-db.yaml`                     -> wave(svc-db) (a strictly-lower wave; CNPG DB)
       - `<svc>-db-app.sops.yaml`            -> wave(svc-db) (CNPG initdb owner secret)
       - `<svc>-db-backup-credentials.sops.yaml` -> wave(svc-db) (CNPG Barman S3 backup secret)
-      - `<svc>-db-backup-credentials-prod.sops.yaml` -> wave(svc-db) (the PROD half of that
-        secret: the two environments carry DIFFERENT object-store credentials, selected in
-        the Cluster CR by `${BACKUP_CREDENTIALS_SUFFIX}`. Both objects ship to both clusters,
-        so both must be placed in the DB wave — the Secret has to exist before the Cluster
-        that names it.)
     Non-catalog units (EXTRA_UNITS, e.g. gitea-actions) place by their declared anchor+offset.
     Only files that actually exist in `present_files` are placed, so the plan is robust to a
     service having no bootstrap secret or no DB. Deterministic (sorted).
@@ -1807,9 +1348,6 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
             add(wdb, f"{db}.yaml")
             add(wdb, f"{db}-app.sops.yaml")
             add(wdb, f"{db}-backup-credentials.sops.yaml")   # CNPG Barman S3 creds (R-STATE-03)
-            # PROD half of the same credential (2026-09-07 per-env split). Placed only when
-            # the file exists, so a DB that has not been split is unaffected.
-            add(wdb, f"{db}-backup-credentials-prod.sops.yaml")
 
     for unit, cfg in EXTRA_UNITS.items():
         anchor = cfg["anchor"]
@@ -2097,101 +1635,6 @@ def _selftest() -> int:
         "a non-exposed service must not render a gateway-ingress allow (least privilege)",
     )
 
-    # 4c) inClusterClients -> one ADDITIVE, least-privilege ingress allow per entry that admits a
-    #     PLATFORM controller's NAMESPACE (not the reserved `ingress` identity, not `all`) to ONE
-    #     declared port. This is what keeps a cluster whose GitOps source is an in-cluster Service
-    #     reconcilable: fromEntities[ingress] does not cover a controller dialling the ClusterIP.
-    icc_docs = render(
-        {
-            "name": "g",
-            "source": {"image": "a:1"},
-            "ingressRoutes": [{"host": "g.home.keithjackson.dev", "port": 3000}],
-            "auth": {"mode": "none"},
-            "inClusterClients": [
-                {"name": "flux", "namespace": "flux-system", "port": 3000}
-            ],
-        }
-    )
-    icc = [
-        d[1]
-        for d in icc_docs
-        if d[1]["kind"] == "CiliumNetworkPolicy"
-        and d[1]["metadata"]["name"] == "g-allow-flux-ingress"
-    ]
-    check(
-        len(icc) == 1,
-        "an inClusterClients entry must render one ingress CNP named "
-        "<service>-allow-<name>-ingress",
-    )
-    if icc:
-        p = icc[0]
-        check(
-            p["metadata"]["namespace"] == "g"
-            and p["spec"]["endpointSelector"]["matchLabels"]
-            == {"app.kubernetes.io/name": "g"},
-            "in-cluster client allow must live in the service's namespace and select the service",
-        )
-        ing = p["spec"]["ingress"]
-        check(
-            ing[0]["fromEndpoints"]
-            == [{"matchLabels": {"k8s:io.kubernetes.pod.namespace": "flux-system"}}],
-            "in-cluster client allow must match the client NAMESPACE only (reserved k8s: label)",
-        )
-        check(
-            "fromEntities" not in ing[0],
-            "in-cluster client allow must not widen to an entity (never fromEntities: all)",
-        )
-        icc_ports = {
-            (pt["port"], pt["protocol"])
-            for r in ing
-            for tp in r["toPorts"]
-            for pt in tp["ports"]
-        }
-        check(
-            icc_ports == {("3000", "TCP")},
-            "in-cluster client allow must be least-privilege (only the declared port/protocol)",
-        )
-    # ADDITIVE: default-deny ingress stays [] and the gateway allow is untouched.
-    icc_dd = next(d[1] for d in icc_docs if d[1]["kind"] == "NetworkPolicy")
-    check(
-        icc_dd["spec"]["ingress"] == [],
-        "in-cluster client allow must be ADDITIVE — default-deny ingress stays [] (R-NET-01)",
-    )
-    check(
-        any(
-            d[1]["metadata"].get("name") == "g-allow-gateway-ingress"
-            for d in icc_docs
-            if d[1]["kind"] == "CiliumNetworkPolicy"
-        ),
-        "in-cluster client allow must not displace the north-south gateway allow",
-    )
-    # `name` is OPTIONAL and defaults to the namespace, so the common case is namespace + port.
-    icc_default = render(
-        {
-            "name": "g",
-            "source": {"image": "a:1"},
-            "inClusterClients": [{"namespace": "monitoring", "port": 9090}],
-        }
-    )
-    check(
-        any(
-            d[1]["metadata"].get("name") == "g-allow-monitoring-ingress"
-            for d in icc_default
-            if d[1]["kind"] == "CiliumNetworkPolicy"
-        ),
-        "inClusterClients.name must default to the admitted namespace",
-    )
-    # A service that declares none must render NO such policy (grant-nothing-extra).
-    check(
-        not any(
-            d[1]["kind"] == "CiliumNetworkPolicy"
-            and d[1]["metadata"]["name"].endswith("-ingress")
-            and "fromEndpoints" in d[1]["spec"].get("ingress", [{}])[0]
-            for d in render({"name": "q", "source": {"image": "a:1"}})
-        ),
-        "a service declaring no inClusterClients must render no in-cluster client allow",
-    )
-
     # 5) vm kind -> VirtualMachine + standalone DataVolume.
     vm_docs = render(
         {
@@ -2387,7 +1830,6 @@ def _selftest() -> int:
     present = {
         "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
         "svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
-        "svc-db-backup-credentials-prod.sops.yaml",
         "gitea.yaml",
         "gitea-actions.yaml", "gitea-actions-runner-secret.sops.yaml",
     }
@@ -2397,10 +1839,8 @@ def _selftest() -> int:
     flat = {f: wv for wv, fs in plan.items() for f in fs}
     check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
     check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
-          and flat.get("svc-db-backup-credentials.sops.yaml") == 0
-          and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
-          "CNPG DB manifest + its owner secret + BOTH halves of its per-env Barman backup secret "
-          "must be in the DB's (lower) wave")
+          and flat.get("svc-db-backup-credentials.sops.yaml") == 0,
+          "CNPG DB manifest + its owner secret + its Barman backup secret must be in the DB's (lower) wave")
     check(flat.get("svc.yaml") == 1 and flat.get("svc-bootstrap-secret.sops.yaml") == 1,
           "a DB-backed app + its bootstrap secret must be in the consumer wave, above the DB")
     check(flat.get("gitea.yaml") == 0, "gitea (leaf) must be wave 0")
@@ -2510,154 +1950,6 @@ def _selftest() -> int:
         "the escaped `$${...}` form must render (and reach the shell as `${...}`)",
     )
 
-    # 9) observability (R-OBS-01): `metrics: true` alone renders NO ServiceMonitor, and the `up`
-    #    SLI reads a signal that exists. This is the regression test for the defect that made all
-    #    twelve `<app>-down` rules structurally inert on prod (see r_servicemonitor).
-    obs_sd = {
-        "name": "obsdemo",
-        "source": {"image": "a:1"},
-        "ingressRoutes": [{"host": "obsdemo.${BASE_DOMAIN}", "port": 8080}],
-        "auth": {"mode": "none"},
-        "observability": {"metrics": True, "slis": ["up"]},
-    }
-    obs_docs = render(obs_sd)
-    check(
-        not [d for d in obs_docs if d[1]["kind"] == "ServiceMonitor"],
-        "metrics: true with NO metricsEndpoint must render no ServiceMonitor — a monitor naming a "
-        "port the Service does not define discovers zero targets and makes the SLI inert",
-    )
-    pr = next(d[1] for d in obs_docs if d[1]["kind"] == "PrometheusRule")
-    up_rule = pr["spec"]["groups"][0]["rules"][0]
-    check(up_rule["alert"] == "obsdemo-down", "slis: [up] must render <name>-down")
-    check(
-        "up{job=" not in up_rule["expr"],
-        "the up SLI must NOT read up{job=...}: no repo-rendered scrape target produces that job, "
-        "so the rule can never fire (verified inert on prod for all twelve apps)",
-    )
-    check(
-        up_rule["expr"]
-        == 'kube_deployment_status_replicas_available{namespace="obsdemo", deployment="obsdemo"} '
-        '== 0 or absent(kube_deployment_status_replicas_available{namespace="obsdemo", '
-        'deployment="obsdemo"})',
-        "container up SLI must read kube_deployment_status_replicas_available with an absent() "
-        f"half; got: {up_rule['expr']}",
-    )
-
-    # 9a) livenessWorkload override: the object that RUNS is not always the one this SD renders.
-    sts_docs = render(
-        dict(
-            obs_sd,
-            name="obssts",
-            observability={
-                "metrics": True,
-                "slis": ["up"],
-                "livenessWorkload": {"kind": "StatefulSet", "name": "obssts"},
-            },
-        )
-    )
-    sts_expr = next(
-        d[1] for d in sts_docs if d[1]["kind"] == "PrometheusRule"
-    )["spec"]["groups"][0]["rules"][0]["expr"]
-    check(
-        "kube_statefulset_status_replicas_ready" in sts_expr,
-        f"livenessWorkload kind StatefulSet must read the statefulset series; got: {sts_expr}",
-    )
-
-    # 9b) vm kind: a VMI has no replica count, so liveness is a running-phase series EXISTING.
-    vm_pr = next(
-        d[1]
-        for d in render(
-            {
-                "name": "obsvm",
-                "kind": "vm",
-                "source": {"vmTemplate": "quay.io/x/y:1"},
-                "storage": [{"name": "root", "size": "6Gi"}],
-                "observability": {"metrics": True, "slis": ["up"]},
-            }
-        )
-        if d[1]["kind"] == "PrometheusRule"
-    )
-    check(
-        vm_pr["spec"]["groups"][0]["rules"][0]["expr"]
-        == 'absent(kubevirt_vmi_info{namespace="obsvm", name="obsvm", phase="running"})',
-        "vm up SLI must read kubevirt_vmi_info with the LOWERCASE running phase: "
-        f"{vm_pr['spec']['groups'][0]['rules'][0]['expr']}",
-    )
-
-    # 9c) metricsEndpoint renders ALL THREE halves of a working scrape, and they AGREE. Any one of
-    #     them missing is an independent way for the scrape to silently discover/deliver nothing.
-    ep_docs = render(
-        dict(
-            obs_sd,
-            name="obsep",
-            observability={
-                "metrics": True,
-                "slis": ["up"],
-                "metricsEndpoint": {"port": 9100},
-            },
-        )
-    )
-    ep_svc = next(d[1] for d in ep_docs if d[1]["kind"] == "Service")
-    ep_sm = next(d[1] for d in ep_docs if d[1]["kind"] == "ServiceMonitor")
-    svc_port_names = {p["name"] for p in ep_svc["spec"]["ports"]}
-    ep_ports = {e["port"] for e in ep_sm["spec"]["endpoints"]}
-    check(
-        ep_ports <= svc_port_names,
-        f"ServiceMonitor endpoint port(s) {sorted(ep_ports)} must all exist on the Service "
-        f"{sorted(svc_port_names)} — this exact disagreement is what made twelve monitors inert",
-    )
-    check(
-        {"name": "metrics", "port": 9100, "targetPort": 9100} in ep_svc["spec"]["ports"],
-        "metricsEndpoint must render the named Service port it declares",
-    )
-    mon_cnp = [
-        d[1]
-        for d in ep_docs
-        if d[1]["kind"] == "CiliumNetworkPolicy"
-        and d[1]["metadata"]["name"] == "obsep-allow-monitoring-scrape"
-    ]
-    check(
-        len(mon_cnp) == 1,
-        "a declared metricsEndpoint must render the monitoring-namespace ingress allow — the "
-        "namespace default-deny has `ingress: []` and would DROP the scrape without it",
-    )
-    if mon_cnp:
-        ing = mon_cnp[0]["spec"]["ingress"][0]
-        check(
-            ing["fromEndpoints"][0]["matchLabels"]["k8s:io.kubernetes.pod.namespace"]
-            == MONITORING_NS
-            and ing["toPorts"][0]["ports"] == [{"port": "9100", "protocol": "TCP"}],
-            "the scrape allow must admit ONLY the monitoring namespace to ONLY the metrics port",
-        )
-    gw_cnp = next(
-        d[1]
-        for d in ep_docs
-        if d[1]["kind"] == "CiliumNetworkPolicy"
-        and d[1]["metadata"]["name"] == "obsep-allow-gateway-ingress"
-    )
-    gw_ports = {p["port"] for p in gw_cnp["spec"]["ingress"][0]["toPorts"][0]["ports"]}
-    check(
-        "9100" not in gw_ports,
-        "the metrics port must NOT be opened to the Gateway (north-south); it is for the "
-        f"monitoring namespace only. Gateway allows: {sorted(gw_ports)}",
-    )
-    # A metricsEndpoint the renderer cannot express must be REFUSED, never rendered inert.
-    try:
-        render(
-            dict(
-                obs_sd,
-                name="obsbad",
-                observability={
-                    "metrics": True,
-                    "slis": ["up"],
-                    "livenessWorkload": {"kind": "CronJob", "name": "obsbad"},
-                },
-            )
-        )
-        check(False, "an unknown livenessWorkload.kind must raise, not render a bogus expression")
-    except ValueError as exc:
-        check("CronJob" in str(exc), "the refusal must name the unsupported kind")
-
     if failures:
         for f in failures:
             sys.stderr.write(f"SELFTEST FAIL: {f}\n")

catalog/semaphore.yaml

@@ -78,34 +78,16 @@ env:
   #     (util ConfigType `oidc_providers`, env-tagged SEMAPHORE_OIDC_PROVIDERS). The value is a single
   #     JSON blob for the `keycloak` provider — provider_url is the homelab realm issuer (auto-discovery),
   #     client_id=semaphore, redirect_url=/api/auth/oidc/keycloak/redirect, claims map to the realm's
-  #     token. The name is NOT a kubelet service-link var (those are SEMAPHORE_SERVICE_HOST /
-  #     SEMAPHORE_PORT_*), so no collision. The LOCAL SEMAPHORE_ADMIN above stays break-glass (docs/15).
-  #
-  #     PER-ENVIRONMENT URLS (2026-09-07 fix). The blob USED to live whole inside the SOPS-encrypted
-  #     semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
-  #     client secret. That hid two environment-specific, NON-SECRET URLs — the realm issuer and the
-  #     redirect URI — inside ciphertext, where Flux's ${BASE_DOMAIN} postBuild substitution CANNOT
-  #     reach them. Both were frozen at the apex `home.keithjackson.dev`, so PROD (BASE_DOMAIN =
-  #     prod.home.keithjackson.dev) pointed its OIDC at DEV's Keycloak — unroutable from prod's
-  #     192.168.2.0/24 (the identical root cause as the Headlamp login timeout; see
-  #     platform/headlamp/README.md). Dev was only ever "right" because dev IS the apex.
-  #
-  #     THE SPLIT: the URLs are public, so the blob is now a PLAINTEXT literal here and gets
-  #     ${BASE_DOMAIN}-substituted per environment like every other hostname. Only the confidential
-  #     client secret stays SOPS-encrypted, in its own key (`oidc-client-secret`), and is spliced back
-  #     in by the KUBELET's dependent-env-var expansion: an env `value` may reference an EARLIER env
-  #     var of the same container as $(NAME), and secretKeyRef-sourced vars are eligible. So
-  #     SEMAPHORE_OIDC_CLIENT_SECRET MUST STAY IMMEDIATELY ABOVE — the kubelet resolves env in list
-  #     order, and a reference to a not-yet-defined name is left as the LITERAL text $(...). It is
-  #     fail-loud in the way that matters: if the Secret key is missing the Pod never starts
-  #     (CreateContainerConfigError), rather than silently authenticating against nothing. ---
-  - name: SEMAPHORE_OIDC_CLIENT_SECRET
+  #     token. It CONTAINS the confidential client secret, so the whole blob is held SOPS-encrypted in the
+  #     semaphore-bootstrap Secret (key oidc-providers-json; same secret value SOPS-held in
+  #     keycloak-realm-clients) and injected here by reference — never in cleartext. The name is NOT a
+  #     kubelet service-link var (those are SEMAPHORE_SERVICE_HOST / SEMAPHORE_PORT_*), so no collision.
+  #     The LOCAL SEMAPHORE_ADMIN above stays as the break-glass path (docs/15). ---
+  - name: SEMAPHORE_OIDC_PROVIDERS
     valueFrom:
       secretKeyRef:
         name: semaphore-bootstrap
-        key: oidc-client-secret
-  - name: SEMAPHORE_OIDC_PROVIDERS
-    value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
+        key: oidc-providers-json
 # The semaphoreui/semaphore image runs as the non-root user UID 1001 / GID 0 (tini -> server-wrapper),
 # and on boot the wrapper writes its generated config.json into the config dir. A mounted Longhorn PVC
 # is root:root, so UID 1001 cannot write it -> the pod panics ("permission denied"). fsGroup=1001 makes
@@ -135,12 +117,10 @@ auth:
   groups:
     - platform-admins           # access scoped to an authorized group (R-IDP-06)
   # WIRED (native OIDC): the Keycloak `semaphore` confidential client (platform/keycloak/keycloak-realm-
-  # import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env — a PLAINTEXT JSON blob
-  # (see `env` above) whose per-environment URLs are ${BASE_DOMAIN}-substituted, with the confidential
-  # client secret spliced in at runtime from the semaphore-bootstrap Secret key `oidc-client-secret`.
-  # The `groups` claim is exposed for authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the
-  # break-glass path so an IdP outage never locks the operator out (docs/15 "Availability caveat").
-  # No secret is embedded here.
+  # import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env (JSON blob held encrypted
+  # in the semaphore-bootstrap Secret, oidc-providers-json). The `groups` claim is exposed for
+  # authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the break-glass path so an IdP outage
+  # never locks the operator out (docs/15 "Availability caveat"). No secret is embedded here.
 update:
   channel: stable               # R-UPD-01
   range: ">=2.19.0 <3.0.0"      # stay on the 2.x stable line

catalog/sonarr.yaml

@@ -9,23 +9,6 @@ name: sonarr
 kind: container
 source:
   image: lscr.io/linuxserver/sonarr:4.0.15   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
-# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
-# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
-# which cannot write a 1000:3000-owned 0755 tree.
-env:
-  - name: PUID
-    value: "1000"               # owner uid of every directory under /mnt/Storage/plex
-  - name: PGID
-    value: "3000"               # dominant group there; keeps files this writes readable to the
-                                # EXISTING non-k3s consumers of the same export, which stay live
-  - name: TZ
-    value: Etc/UTC              # same safe default plex uses; the arrs schedule by local time
-podSecurityContext:
-  # For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
-  # does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
-  # what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
-  # /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
-  fsGroup: 3000
 exposure: internal              # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
@@ -33,66 +16,11 @@ storage:
   - name: config
     size: 5Gi
     class: longhorn             # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
-    mountPath: /config          # THE APP WRITES /config, NOT the renderer default /data/config.
-                                # LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
-                                # /config), so without this the Longhorn PVC mounted at /data/config sat
-                                # holding nothing but lost+found while the app wrote its entire
-                                # configuration to the container's EPHEMERAL writable layer — destroyed on
-                                # every pod recreation: restart, reschedule, image bump. Measured in-cluster
-                                # 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
-                                # dev=2053 on the node's own disk with [Sentry asp config.xml logs].
-                                # This is the SAME failure catalog/plex.yaml:54 already records fixing.
-                                # The live /config contents were copied onto each PVC before this landed,
-                                # so API keys and credentials other apps reference were preserved.
-  # THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
-  # /media/tv (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
-  #
-  # *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Sonarr imports by HARDLINKING or
-  # atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
-  # and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
-  #     /mnt/Storage/plex/tv/                      the library
-  #     /mnt/Storage/plex/downloads/complete/tv-sonarr/  where finished releases land
-  # — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
-  # them as two subPath volumes (subPath: tv at /media/tv, subPath: downloads elsewhere) creates
-  # TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Sonarr then
-  # silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
-  # seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
-  # is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
-  # requirement, not a shortcut. (The cost is stated plainly: Sonarr can also write /media/movies,
-  # which is radarr's. Narrowing that would cost hardlinks, which is a worse trade.)
-  #
-  # *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
-  # renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
-  # config volume entirely; the pod would come up with its app database shadowed by an 8 T media
-  # tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
-  # the same export, so when qbittorrent is eventually given this mount too, the arrs and the
-  # download client will already agree on absolute paths — which is the other precondition for
-  # hardlinked imports (otherwise every import needs a Remote Path Mapping).
-  #
-  # SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
-  # is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
-  # with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
-  # Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
-  # documents having already been bitten by. Fixing it means starting from an empty config volume, so
-  # it belongs in its own change with the operator present.
-  #
-  # UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
-  # + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
-  # the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
-  # write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
-  # every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
-  #
-  # AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
-  # ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
-  # lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
-  # that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
-  # the image re-chowning /config for itself.
-  - name: media
-    class: bulk
-    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
-    mountPath: /media           # => /media/tv (Root Folder) and /media/downloads (import source)
-    size: 9Ti                   # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
-                                # rounded up, so `kubectl get pv` reports something true.
+  # NOTE (bulk-media PREREQUISITE): docs/13 row 2 places the media library / downloads volume on the
+  # `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
+  # but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
+  # volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
+  # therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Sonarr deployable today.
 dependencies:
   - to: qbittorrent             # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 8080
@@ -107,11 +35,6 @@ auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
   groups:
     - media-users               # authenticated AND in the media group (R-IDP-06)
-    # See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
-    # platform-admins here is a superset, not a widening (that group already carries cluster-admin
-    # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
-    # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
-    - platform-admins           # operators of these services must not be locked out of them
 update:
   channel: stable               # R-UPD-01
   range: ">=4.0.0 <5.0.0"       # stay on the Sonarr v4 stable line

ci/README.md

@@ -27,31 +27,9 @@ make -C ci test        # unit tests for these scripts themselves (ci/tests/*.tes
 ```
 
 The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
-(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo
-L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
-least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`.
-
-### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped
-
-R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the
-worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
-coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be
-selected *by* the requirements a change touches — because **the change most likely to break it
-attributes no requirement at all**. Requirement-scoped selection then filters the check out of
-exactly that change.
-
-This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
-3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
-`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that
-failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write
-a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"`
-and the runner unions every declaring test into the run. Adding the annotation is the only step.
-
-`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the
-tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the
-gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably,
-by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the
-full rationale and the incidents behind it.
+(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
+per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
+spec-immutability enforcement). See `conformance/tests/README.md`.
 
 ## Exit-code contract
 
@@ -59,47 +37,6 @@ Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIP
 with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
 the gate stays honest in Phase 0 (no fake green).
 
-### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)
-
-**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It
-used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a
-warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
-`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule
-unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as
-merge-approved. Fail-open in the one component whose whole job is to fail closed.
-
-| verdict | exit | meaning |
-|---|---|---|
-| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. |
-| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. |
-| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. |
-| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. |
-
-Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.
-
-**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so
-`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A
-caller that must distinguish them invokes `./ci/gate.sh` directly.
-
-### Which skips block, and why — a skip is not one thing
-
-| stage | a skip there means | verdict |
-|---|---|---|
-| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
-| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
-| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
-
-That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
-
-`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml`
-sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls
-back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction
-cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI
-strictness.
-
-Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed
-stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate).
-
 ### The touched-requirement subset can also be *undeterminable* (exit 3)
 
 `touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
@@ -150,8 +87,6 @@ ci/
   lint.sh                  Stage: yaml/lint
   l0.sh                    Stage: schema validation + Kyverno L0 gate + L0 static conformance
   l1.sh                    Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
-                           (harness: platform/monitoring/tests/). Catches the two alerting bugs no
-                           other stage can see — permanently-firing and structurally INERT rules.
   conformance.sh           Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
   touched-requirements.sh  compute the R-* subset a change touches (drives the R-TEST-03 filter);
                            exits 3 LOUDLY rather than guessing when the diff base is unreliable
@@ -187,15 +122,3 @@ its owner delivers a runnable entrypoint.
 `CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and
 be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+)
 turns any unverified cluster-needing touched requirement into a hard block.
-
-All four switches are read through one `truthy()` in `lib.sh`, so they spell "on" identically
-(`true|1|yes|y|on`, any case) and — the direction that matters — **anything else is OFF**: a
-misspelled `CLUSTER_AVAILABLE=ture` leaves the capability undeclared instead of silently claiming
-it, which can only make the gate stricter about what it says it verified.
-
-| switch | default | effect |
-|---|---|---|
-| `CLUSTER_AVAILABLE` | `false` | declares a cluster; L2/L3 then run and are required |
-| `REQUIRE_CLUSTER_TESTS` | `false` | Phase 1+: an unverified cluster-needing touched requirement BLOCKS |
-| `REQUIRE_SPEC_VERIFICATION` | `false` | an unverifiable spec repo BLOCKS instead of UNDETERMINED |
-| `GATE_STRICT_SKIPS` | `auto` | `true` = a missing tool / unwired seam BLOCKS (CI); `auto` follows `GITHUB_ACTIONS`/`CI` |

ci/conformance.sh

@@ -47,11 +47,7 @@ fi
 # (conformance/lib/common.sh) now unions a comma/space list of layers, so this is a single
 # invocation with one aggregated report (no per-layer looping needed). Artifacts are then
 # COLLECTED from the fixed location below.
-# --include-scope: union the touched subset with every L2/L3 test DECLARING
-# `conformance.home.k3s/scope: whole-repo` (none today — wired once, here, so a future
-# whole-repo behavioural invariant needs only the annotation; see conformance/lib/common.sh).
-: "${WHOLE_REPO_SCOPE:=whole-repo}"
-"$CONFORMANCE_CMD" --layer l2,l3 --include-scope "$WHOLE_REPO_SCOPE" "${req_args[@]}"
+"$CONFORMANCE_CMD" --layer l2,l3 "${req_args[@]}"
 
 CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
 if [ -f "${CONF_ART}/report.junit.xml" ]; then

ci/gate.sh

@@ -7,75 +7,6 @@
 #   skip (3)  not run (cluster-gated stage w/ no cluster, or unwired seam) —
 #             NOT counted as green (HARD RULE 4: don't fake green)
 #
-# ...AND THREE OUTCOMES FOR THE GATE ITSELF (2026-09-13). Until today a stage that
-# SKIPPED only printed a warning ("— not green") and the gate then printed
-# `MERGE GATE: PASS` and exited 0 anyway. That is fail-open in the one component whose
-# entire job is to fail closed: a pinned tool missing from the runner, or a seam silently
-# becoming unwired, produced exit 0 and read as merge-approved. Observed live:
-# `prom-rules: skipped (missing tool) — not green` followed by `MERGE GATE: PASS`, and
-# `l0: skipped (kyverno CLI not installed)` likewise. The gate's own vocabulary already
-# had the answer — the 2026-08-31 "local false green" introduced UNDECIDED for exactly
-# this shape of non-answer — the offline stages just did not use it.
-#
-#   MERGE GATE: PASS           exit 0   every stage that was SUPPOSED TO RUN ran and was
-#                                       green, and the R-TEST-03 subset was checked.
-#   MERGE GATE: BLOCKED        exit 1   a stage FAILED, or a stage did not run WHERE IT
-#                                       MUST (see the per-stage rules below), or the
-#                                       decision input is unusable.
-#   MERGE GATE: UNDETERMINED   exit 3   nothing failed, but a stage did not run, so
-#                                       greenness was NOT ESTABLISHED. Never a pass;
-#                                       non-zero so no automation can read it as approval.
-#                                       `3` is this repo's existing "not run / not
-#                                       established" code (ci/lib.sh), now spoken by the
-#                                       gate as a whole and not only by its stages.
-#   MERGE GATE: NOT APPLICABLE exit 0   there is no branch under gate at all (HEAD is on
-#                                       base) — a push-to-main CI run. Stages were green;
-#                                       no subset was checked, and the headline says so.
-#
-# NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so
-# `make -C ci gate` collapses 1 and 3 into 2 — a human sees the headline and that is enough.
-# A caller that must tell BLOCKED from UNDETERMINED invokes ./ci/gate.sh directly; to tolerate
-# an UNDETERMINED run it writes `./ci/gate.sh || [ $? -eq 3 ]`, the same explicit idiom
-# .gitea/workflows/gate.yml already used for its spec-immutability pre-step and now uses for
-# the gate itself (with GATE_STRICT_SKIPS=true, so that tolerance can ONLY ever cover the
-# declared-environmental skip below — a missing tool on the runner is exit 1, untolerated).
-# TOLERATING A NON-ANSWER IS A VISIBLE, REVIEWED CHOICE IN THE CALLER. There is deliberately no
-# knob inside this script that turns UNDETERMINED into exit 0: that would be the fail-open this
-# change exists to remove, just relocated.
-#
-# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
-#
-#   lint / l0 / render-golden / l1   A skip here means a PINNED tool is absent or a wired
-#     seam went unwired. On a CI runner the pinned toolchain is installed for us by
-#     .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
-#     environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
-#     it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
-#     in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
-#
-#   spec-immutability (R-TEST-06)    A skip means the sibling spec repo is not checked out, so
-#     the spec COULD NOT BE VERIFIED. Never a pass — unverified is unverified — so it denies a
-#     PASS and yields UNDETERMINED. It is deliberately NOT upgraded to BLOCKED by CI strictness,
-#     because on the live runner it is not a defect anyone can fix from inside this repo: the
-#     workflow does not check the spec repo out at all (SPEC_REPO_SLUG is unwired — the blueprint
-#     lives in merge-gate.yaml), and blocking would leave the only enforced status check
-#     permanently red for a reason no PR could clear. So THIS is the one skip whose non-answer the
-#     CALLER may knowingly tolerate, and gate.yml does so in one visible line. Locally the
-#     remedy exists and the message names it: check the spec repo out beside this one, or point
-#     SPEC_REPO at it (infra/devcontainer/dev.sh mounts it automatically when it is a sibling).
-#     Set REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a
-#     hard block once that checkout is wired.
-#
-#   conformance (L2/L3)              A skip when CLUSTER_AVAILABLE!=true is NOT a defect and
-#     NOT an accident: it is the environment DECLARING that the capability is absent, which
-#     is the documented Phase-0 behaviour. A declared-inapplicable stage was never "supposed
-#     to run", so it does not deny a PASS — the touched requirements it would have covered
-#     are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that
-#     into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as
-#     it already did. That asymmetry is the whole distinction this file now draws:
-#     DECLARED-not-applicable may pass; COULD-NOT-RUN may not. A declared gap is still NAMED in
-#     the PASS headline (see DECLARED_GAPS below), so a PASS never reads as "everything was
-#     verified" when a layer was declared out of scope.
-#
 # Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
 # MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
 # available, the gate reports those requirements UNVERIFIED and — unless
@@ -89,7 +20,6 @@ source "${CI_DIR}/lib.sh"
 set +e   # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
 
 : "${REQUIRE_CLUSTER_TESTS:=false}"
-: "${REQUIRE_SPEC_VERIFICATION:=false}"
 
 declare -A OUTCOME
 run_stage() {
@@ -117,13 +47,6 @@ run_stage conformance   "${CI_DIR}/conformance.sh"
 echo >&2
 log "==== gate decision ===="
 BLOCK=0
-# Stages that did NOT run and are tolerated here — but never as green. A non-empty list
-# means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3.
-NOT_ESTABLISHED=()
-# Stages the environment DECLARED inapplicable (today: L2/L3 with no cluster declared). These
-# do not deny a PASS — nothing was supposed to run — but they are named IN the PASS headline, so
-# "PASS" is never mistaken for "everything was verified".
-DECLARED_GAPS=()
 
 # --- R-TEST-03 input: the touched-requirement subset --------------------------
 # Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
@@ -141,80 +64,37 @@ rm -f "$_reason_file"
 TOUCHED=()
 [ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
 UNDECIDED=0
-SUBSET_BROKEN=0
 if [ "$TOUCHED_RC" -eq 3 ]; then
   UNDECIDED=1
   echo >&2
   err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
   err "  determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
   summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
-elif [ "$TOUCHED_RC" -ne 0 ]; then
-  # Any OTHER non-zero exit is a hard defect in the subset computation itself — an
-  # incomplete path->requirement map (a changed path matches no row) or an invalid row
-  # would report exactly this. The reasoning is taken from the unlanded
-  # fix/touched-requirements-fail-closed (47068ee): gate.sh special-cased ONLY exit 3, so
-  # any other non-zero exit left TOUCHED empty and UNDECIDED=0, and the decision below
-  # printed "MERGE GATE: PASS" having applied no coverage gate at all. Unreachable on
-  # `main` today (touched-requirements.sh exits only 0 or 3) and wired here so it is
-  # already closed the moment a fail-closed path map makes exit 1 reachable. A subset that
-  # cannot be computed is NOT an empty subset.
-  SUBSET_BROKEN=1
-  echo >&2
-  err "R-TEST-03 DECISION INPUT BROKEN: touched-requirements.sh exited ${TOUCHED_RC}"
-  err "  (${TOUCHED_REASON:-unknown}). Its output above names the paths/rows to fix."
-  summary "- **R-TEST-03 subset UNUSABLE** (\`${TOUCHED_REASON:-unknown}\`, exit ${TOUCHED_RC}) — the coverage gate could not be applied."
 elif [ "${#TOUCHED[@]}" -gt 0 ]; then
   log "touched requirements: ${TOUCHED[*]}"
 fi
 
 # spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
-# A skip means the spec repo wasn't available to verify — UNVERIFIED, never green: it
-# denies a PASS (exit 3) and, with REQUIRE_SPEC_VERIFICATION=true, blocks outright. See the
-# file header for why this one is not upgraded to a block by CI detection alone.
+# A skip means the spec repo wasn't available to verify — surfaced, not green.
 case "${OUTCOME[spec-immutability]}" in
   fail) err  "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
-  skip)
-    if truthy "$REQUIRE_SPEC_VERIFICATION"; then
-      err "spec-immutability: skipped (spec repo unavailable) and REQUIRE_SPEC_VERIFICATION=true -> blocks merge"
-      summary "- spec-immutability: **SKIPPED** (spec repo unavailable) — blocks merge (REQUIRE_SPEC_VERIFICATION=true)"
-      BLOCK=1
-    else
-      warn "spec-immutability: skipped (spec repo unavailable) — R-TEST-06 NOT VERIFIED, so"
-      warn "  this run cannot be a PASS. Check the spec repo out beside this one (or set"
-      warn "  SPEC_REPO) to verify it."
-      summary "- spec-immutability: **SKIPPED** (spec repo unavailable — R-TEST-06 not verified; not a pass)"
-      NOT_ESTABLISHED+=("spec-immutability")
-    fi ;;
+  skip) warn "spec-immutability: skipped (spec repo unavailable) — not verified" ;;
   pass) ok   "spec-immutability: green" ;;
 esac
 
-# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
-# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
-# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
+# L0/L1 stages: any failure blocks. A SKIP here means a seam is unwired — in
+# Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
 # no-cluster reproducibility gate (canonical render == committed apps/ goldens):
 # a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
 for s in lint l0 render-golden l1; do
   case "${OUTCOME[$s]}" in
     fail) err  "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
-    skip)
-      if strict_skips; then
-        err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
-        err "  toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
-        err "  seam is on main — so this is a DEFECT, not an environment -> blocks merge"
-        summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
-        BLOCK=1
-      else
-        warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS"
-        summary "- ${s}: **SKIPPED** (unwired seam / missing tool) — not green"
-        NOT_ESTABLISHED+=("$s")
-      fi ;;
+    skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;
     pass) ok   "${s}: green" ;;
   esac
 done
 
-# L2/L3 conformance: cluster-gated. The ONLY stage whose non-execution can still be a PASS,
-# because CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent rather
-# than the stage failing to run where it could have — see the file header.
+# L2/L3 conformance: cluster-gated.
 case "${OUTCOME[conformance]}" in
   fail)
     err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
@@ -225,22 +105,18 @@ case "${OUTCOME[conformance]}" in
       err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
     elif [ "$UNDECIDED" -eq 1 ]; then
       warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
-      DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
     else
       if [ "${#TOUCHED[@]}" -gt 0 ]; then
         reqs=("${TOUCHED[@]}")
         warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
         warn "  (no cluster — Phase 0). These are NOT certified green."
         summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
-        if truthy "$REQUIRE_CLUSTER_TESTS"; then
+        if [ "$REQUIRE_CLUSTER_TESTS" = "true" ]; then
           err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
           BLOCK=1
-        else
-          DECLARED_GAPS+=("conformance-l2-l3(no cluster declared; ${#TOUCHED[@]} touched reqs unverified)")
         fi
       else
         skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
-        DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
       fi
     fi ;;
 esac
@@ -256,34 +132,13 @@ if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
   err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
   BLOCK=1
 fi
-# An incomplete/invalid path map is not an environment problem to skip past: the change under
-# gate contains paths whose requirements are UNKNOWN, so no subset can be certified.
-if [ "$SUBSET_BROKEN" -eq 1 ]; then
-  err "R-TEST-03 subset unusable (${TOUCHED_REASON:-unknown}) -> blocks merge"
-  BLOCK=1
-fi
 
 echo >&2
-# Precedence: BLOCKED (something is wrong) > UNDETERMINED (a stage did not run, so
-# greenness was not established) > NOT APPLICABLE (nothing to gate) > PASS. UNDETERMINED
-# outranks NOT APPLICABLE because "no branch under gate" AND "a stage did not run" is two
-# reasons this is not a green run, and the stronger, non-zero one must be the headline.
 if [ "$BLOCK" -ne 0 ]; then
   err "MERGE GATE: BLOCKED"
   summary ""
   summary "**Result: BLOCKED** — see failures above."
   exit 1
-elif [ "${#NOT_ESTABLISHED[@]}" -gt 0 ]; then
-  err "MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED."
-  err "  Stages that did not run: ${NOT_ESTABLISHED[*]}"
-  err "  Nothing FAILED, and nothing is certified either: a stage that did not run is not"
-  err "  green (HARD RULE 4). This is exit 3, never 0 — no automation may read it as"
-  err "  merge-approved. Fix the environment (install the pinned tool / wire the seam /"
-  err "  check out the spec repo) and run again, or accept it KNOWINGLY and explicitly:"
-  err "    ./ci/gate.sh || [ \$? -eq 3 ]"
-  summary ""
-  summary "**Result: UNDETERMINED** — these stages did not run, so nothing is certified: \`${NOT_ESTABLISHED[*]}\`. Not a pass (exit 3)."
-  exit 3
 elif [ "$UNDECIDED" -eq 1 ]; then
   warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
   warn "  The stages above ran and did not fail, but NO touched-requirement subset was"
@@ -293,16 +148,8 @@ elif [ "$UNDECIDED" -eq 1 ]; then
   summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
   exit 0
 else
-  ok "MERGE GATE: PASS — every stage that was supposed to run RAN and was GREEN, and the"
-  ok "  R-TEST-03 touched-requirement subset was checked."
-  if [ "${#DECLARED_GAPS[@]}" -gt 0 ]; then
-    ok "  NOT verified here, because the environment DECLARED it out of scope (never a silent"
-    ok "  gap — Phase 0 by policy, see the header): ${DECLARED_GAPS[*]}"
-    summary ""
-    summary "**Result: PASS** — every stage that was supposed to run ran green. Declared out of scope (NOT verified): \`${DECLARED_GAPS[*]}\`."
-  else
-    summary ""
-    summary "**Result: PASS** — every stage that was supposed to run ran green; nothing was declared out of scope."
-  fi
+  ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
+  summary ""
+  summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
   exit 0
 fi

ci/l0.sh

@@ -10,18 +10,6 @@
 #   Stage 3b L0     : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
 #                     a per-resource policy cannot express (secret hygiene across every
 #                     tracked file, least-privilege RBAC, spec-immutability enforcement).
-#                     Selection = the TOUCHED requirement subset (R-TEST-03) UNION every L0
-#                     test DECLARING `conformance.home.k3s/scope: whole-repo`. That union
-#                     replaces what used to be a hand-written "UNCONDITIONAL" stage per
-#                     whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e
-#                     bulk-tier): three near-identical 12-line rationales, three hardcoded
-#                     check.sh paths and three copies of the same rc 0/3/* case block, which
-#                     did not scale and — worse — failed SILENTLY, because whoever added the
-#                     next whole-repo invariant had to REMEMBER to hand-write a stage. Now the
-#                     test declares its own scope and the runner enforces it. The reasoning and
-#                     the incidents those stages recorded were moved INTO the tests they
-#                     describe (see each chainsaw-test.yaml's `scope` annotation) and the
-#                     general mechanism is documented in conformance/lib/common.sh.
 #
 # Integration seams are env-overridable (see ci/lib.sh + README). When an owning
 # subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
@@ -95,45 +83,58 @@ fi
 # binding, the spec-immutability gate's own behaviour) live in the conformance
 # harness's tests/l0-static subtree and run with no cluster. This stage executes them
 # through the same single-command runner CI and a laptop both use (R-TEST-02).
-#
-# THE UNION, AND WHY IT IS NOT OPTIONAL (R-TEST-03's blind spot)
-# --------------------------------------------------------------
-# 3b scopes the conformance run to the TOUCHED requirements. That is right for an invariant
-# whose inputs live under the subtree its requirements map to. It is WRONG for one that
-# quantifies over the WHOLE tracked tree, or over subtrees (apps/, infra/, most of platform/)
-# that ci/touched-requirements.sh's coarse path map keys to NOTHING: for those, the change
-# most likely to break the invariant attributes no requirement at all, so requirement-scoped
-# selection filters the check out of EXACTLY that change. `--include-scope whole-repo` unions
-# in every L0 test that DECLARES that scope, so those run on every gate invocation.
-#
-# rc 3 IS NOT A PASS. The runner returns 3 when a scope-declared check SKIPPED (its subject
-# subtree is absent, or it has no check script): UNWIRED, not green, exactly as the deleted
-# per-check stages had it ("check SKIPPED (clusters/ absent) — not a pass"). It is mapped to
-# the gate's own rc 3 below, never to rc 0.
-: "${WHOLE_REPO_SCOPE:=whole-repo}"
-log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0 --include-scope ${WHOLE_REPO_SCOPE})"
+log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
 if ! entrypoint_ready "$CONFORMANCE_CMD"; then
   skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
   summary "- l0-static: **SKIPPED** (conformance runner missing)"
   if [ "$rc" -eq 0 ]; then rc=3; fi
 else
   load_touched_requirements
-  l0_static_rc=0
-  "$CONFORMANCE_CMD" --layer l0 --include-scope "$WHOLE_REPO_SCOPE" \
-    ${REQ_ARGS[@]+"${REQ_ARGS[@]}"} || l0_static_rc=$?
-  case "$l0_static_rc" in
-    0) ok "L0 static conformance checks passed"
-       summary "- l0-static: **passed**" ;;
-    3) skip "an L0 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
-       summary "- l0-static: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
+  if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
+    ok "L0 static conformance checks passed"
+    summary "- l0-static: **passed**"
+  else
+    err "L0 static conformance checks failed"
+    summary "- l0-static: **FAILED**"
+    rc=1
+  fi
+fi
+
+# ---- Stage 3c: Flux post-build substitution safety (UNCONDITIONAL) ----------
+# R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
+# FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
+# reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
+# (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN"). The check asserts every
+# ${...} in every substituted path is DEFINED in cluster-vars or deliberately ESCAPED
+# `$${...}` — i.e. that enabling the StrictPostBuildSubstitutions feature gate cannot fail
+# a legitimate manifest, and that no expansion silently empties today.
+#
+# WHY THIS IS ITS OWN STAGE and not left to Stage 3b: 3b scopes the conformance run to the
+# TOUCHED requirements (R-TEST-03), and ci/touched-requirements.sh's coarse path map keys
+# only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or platform/
+# attributes no requirement, so the requirement-scoped selection would filter this check
+# out of exactly the change that introduces the bug. The invariant is whole-repo (same
+# reasoning as spec-immutability's own gate stage), so it runs on EVERY gate invocation.
+# When 3b's touched set does include R-ENV-02/R-GITOPS-01 the check runs twice; it is a
+# sub-second offline scan with no side effects, so the duplication is not worth avoiding.
+SUBST_CHECK="${SUBST_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/flux-postbuild-substitution/check.sh}"
+log "Stage: Flux postBuild substitution safety (${SUBST_CHECK})"
+if [ ! -f "$SUBST_CHECK" ]; then
+  skip "substitution check not present at '${SUBST_CHECK}' — UNWIRED, not green"
+  summary "- subst-safety: **SKIPPED** (check missing)"
+  if [ "$rc" -eq 0 ]; then rc=3; fi
+else
+  subst_rc=0
+  bash "$SUBST_CHECK" || subst_rc=$?
+  case "$subst_rc" in
+    0) ok "Flux postBuild substitution safety passed"
+       summary "- subst-safety: **passed**" ;;
+    3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
+       summary "- subst-safety: **SKIPPED** (nothing to audit)"
        if [ "$rc" -eq 0 ]; then rc=3; fi ;;
-    *) err "L0 static conformance checks failed"
-       summary "- l0-static: **FAILED**"
-       rc=1 ;;
+    *) err "Flux postBuild substitution safety FAILED"
+       summary "- subst-safety: **FAILED**"; rc=1 ;;
   esac
-  # Per-check step-summary lines, so the gate output still NAMES what ran — the generated
-  # replacement for the one hardcoded `summary` line each deleted stage carried.
-  emit_offline_summary
 fi
 
 case "$rc" in

ci/l1.sh

@@ -29,16 +29,7 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
 # [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
 # (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
 # touched subset, then COLLECT its artifacts into the CI artifact dirs.
-# --include-scope: union the touched subset with every L1 test DECLARING
-# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
-# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
-# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
-# "someone must remember" is the failure mode this whole change removes; see
-# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
-: "${WHOLE_REPO_SCOPE:=whole-repo}"
-l1_rc=0
-"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
-  ${req_args[@]+"${req_args[@]}"} || l1_rc=$?
+"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
 
 CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
 if [ -f "${CONF_ART}/report.junit.xml" ]; then
@@ -48,16 +39,5 @@ if [ -d "${CONF_ART}/coverage" ]; then
   cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi
 
-case "$l1_rc" in
-  0) ok "L1 rendering checks passed"
-     summary "- l1-render: **passed**"
-     emit_offline_summary ;;
-  3) skip "an L1 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
-     summary "- l1-render: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
-     emit_offline_summary
-     exit 3 ;;
-  *) err "L1 rendering checks FAILED (rc=${l1_rc})"
-     summary "- l1-render: **FAILED**"
-     emit_offline_summary
-     exit "$l1_rc" ;;
-esac
+ok "L1 rendering checks passed"
+summary "- l1-render: **passed**"

ci/lib.sh

@@ -72,83 +72,20 @@ entrypoint_ready() {
   return 1
 }
 
-# truthy <value> — the ONE place the repo's "is this switch on" spelling lives, so
-# CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS
-# all agree. Anything not listed is OFF, and that direction matters: a misspelled value
-# (CLUSTER_AVAILABLE=ture, REQUIRE_SPEC_VERIFICATION=yeah) leaves the guard OFF rather than
-# silently declaring a capability "available", so a typo can only make the gate stricter about
-# what it claims to have verified, never looser.
-truthy() {
-  case "${1:-}" in
-    true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;;
+# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
+# A cluster is 'available' only if explicitly declared. We NEVER auto-assume a
+# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
+#   CLUSTER_AVAILABLE=true   -> L2/L3 run and are REQUIRED green
+#   CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
+cluster_available() {
+  case "${CLUSTER_AVAILABLE:-false}" in
+    true|1|yes) return 0 ;;
     *) return 1 ;;
   esac
 }
 
-# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
-# A cluster is 'available' only if explicitly DECLARED. We NEVER auto-assume a
-# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
-#   CLUSTER_AVAILABLE=true   -> L2/L3 run and are REQUIRED green
-#   CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
-# That DECLARATION is what makes an L2/L3 skip different IN KIND from every other skip —
-# see "WHICH SKIPS BLOCK, AND WHY" in ci/gate.sh.
-cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
-
-# --- Skip strictness: is this a CI runner, or a workstation? -------------------
-# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
-#
-#   CI       the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
-#            before the gate runs, and every seam is on `main`. A stage that skips here is a
-#            DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
-#   local    a missing pinned tool is a real, ordinary local condition -> the gate reports
-#            UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
-#
-# HOW STRICTNESS IS DECIDED — DECLARED first, detected only as a fallback:
-#
-#   1. GATE_STRICT_SKIPS=true|false is the AUTHORITY. .gitea/workflows/gate.yml sets `true` on
-#      the gate step, so on the one CI runner that matters strictness is a DECLARATION IN THE
-#      WORKFLOW, not an inference: the environment that KNOWS it is CI says so, and no
-#      env-var sniffing can get it wrong.
-#   2. `auto` (the default — anywhere the caller said nothing) falls back to GITHUB_ACTIONS /
-#      CI, which are set by the runner ITSELF (Gitea's act_runner, as well as GitHub). We do
-#      NOT sniff hostnames, paths or tool availability: a heuristic that guesses "this looks
-#      like CI" is exactly the unreliable inference this gate must not contain.
-#
-# WHY A WRONG ANSWER CANNOT FAIL OPEN: both branches are non-zero and NEITHER is a pass. A CI
-# run misread as local yields UNDETERMINED (exit 3) instead of BLOCKED (exit 1) — that still
-# fails the job unless its caller explicitly tolerates 3, which gate.yml does only knowingly
-# and only for the one declared-environmental skip (see its comment). A workstation misread as
-# CI yields BLOCKED instead of UNDETERMINED — harsher, never looser. Set GATE_STRICT_SKIPS=true
-# locally to rehearse CI strictness.
-: "${GATE_STRICT_SKIPS:=auto}"
-strict_skips() {
-  case "$GATE_STRICT_SKIPS" in
-    auto) truthy "${GITHUB_ACTIONS:-}" || truthy "${CI:-}" ;;
-    *)    truthy "$GATE_STRICT_SKIPS" ;;
-  esac
-}
-
 mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
 
-# --- Offline (L0/L1) per-check step summary -----------------------------------
-# conformance/lib/run_l1_offline.py writes one Markdown line per check it executed to
-# conformance/_artifacts/offline-summary.md. Splice those into $GITHUB_STEP_SUMMARY so the
-# gate's output still NAMES every check that ran and its verdict.
-#
-# WHY THIS EXISTS: ci/l0.sh used to carry a hand-written stage per whole-repo check, each with
-# its own `summary "- subst-safety: **passed**"` line. Those stages are gone (the tests declare
-# `conformance.home.k3s/scope: whole-repo` instead and the runner unions them in), so the
-# naming is generated for EVERY offline check rather than for the three that had a stage.
-: "${OFFLINE_SUMMARY_FILE:=${CONFORMANCE_DIR}/_artifacts/offline-summary.md}"
-emit_offline_summary() {
-  local f="${1:-$OFFLINE_SUMMARY_FILE}" line
-  [ -f "$f" ] || return 0
-  while IFS= read -r line; do
-    [ -n "$line" ] && summary "  $line"
-  done < "$f"
-  return 0
-}
-
 # --- Touched-requirement subset (R-TEST-03 input) ----------------------------
 # Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
 # (ready to splice into a conformance/run.sh invocation).
@@ -174,16 +111,7 @@ load_touched_requirements() {
     warn "  Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
     warn "  coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
   elif [ "$rc" -ne 0 ]; then
-    # Any OTHER non-zero exit means the subset is UNUSABLE, not empty (today: a
-    # path->requirement map that cannot classify a changed path, once that map fails closed).
-    # The FULL layer still runs here as a superset — nothing goes untested — but this stage must
-    # not claim the SCOPED check happened, and ci/gate.sh turns the same exit into a BLOCKED
-    # merge decision. Wording carried over from the unlanded fix/touched-requirements-fail-closed
-    # (47068ee), whose gate-side half this branch lands.
-    warn "touched-requirements.sh exited ${rc} (subset UNUSABLE — see its output above)."
-    warn "  Running the FULL layer as a superset; the SCOPED R-TEST-03 check did NOT happen."
-    warn "  ci/gate.sh BLOCKS the merge on this: fix the path->requirement map in"
-    warn "  ci/touched-requirements.sh so every changed path is classified."
+    warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
   elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
     log "touched requirements: ${TOUCHED_REQS[*]}"
     REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")

ci/render-keycloak-client-urls.py

@@ -1,214 +0,0 @@
-#!/usr/bin/env python3
-# =============================================================================
-# ci/render-keycloak-client-urls.py
-#
-# Render platform/keycloak/client-urls.yaml FROM platform/keycloak/keycloak-realm-import.yaml.
-#
-# WHY THIS EXISTS
-# ---------------
-# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
-# Done=True / Started=False it never re-runs, so a redirect URI changed in git is
-# never applied to the realm. The client-secret half of that failure class is
-# already healed by a level-triggered CronJob (platform/keycloak/client-secret-
-# reconciler.yaml). This renderer supplies the DESIRED STATE that the same CronJob
-# needs to heal the URL half.
-#
-# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
-# automountServiceAccountToken: false. It has NO Kubernetes API access at all, by
-# design (R-SEC-02) — so it cannot read the KeycloakRealmImport CR to learn what
-# the realm is supposed to look like. Its inputs must be MOUNTED. Hence a
-# ConfigMap.
-#
-# A hand-authored ConfigMap would be a SECOND SOURCE OF TRUTH for every public
-# hostname in the realm, free to disagree with the realm import — and "two
-# declarations that can disagree" is the same shape as the bug this whole effort
-# is closing. So the ConfigMap is GENERATED from the realm import and the
-# generation is GATED: conformance/tests/l0-static/keycloak-client-secret-
-# reconciler/check.sh runs this script with --check on every merge, and the merge
-# fails if the committed file is not what this renderer produces. That is the same
-# render==golden idiom catalog/render.py + ci/render-golden.sh already use for
-# apps/.
-#
-# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. If a URL in the realm import is
-# written as a Flux substitution token (https://grafana.${BASE_DOMAIN}/...), the
-# token is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files
-# are rendered by the SAME Flux Kustomization (./platform-config, whose
-# postBuild.substituteFrom reads cluster-vars), so both are substituted in the
-# same pass with the same values and cannot disagree about what BASE_DOMAIN is.
-# This renderer never parses, splits, or reconstructs a hostname, and never reads
-# cluster-vars — so there is no second place for the domain to be decided.
-#
-# USAGE
-#   python3 ci/render-keycloak-client-urls.py            # print the rendered file
-#   python3 ci/render-keycloak-client-urls.py --write     # write it (regenerate)
-#   python3 ci/render-keycloak-client-urls.py --check      # exit 1 if it has drifted
-# =============================================================================
-from __future__ import annotations
-
-import argparse
-import difflib
-import json
-import sys
-from pathlib import Path
-
-try:
-    import yaml
-except ImportError:  # pragma: no cover
-    print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
-          file=sys.stderr)
-    raise SystemExit(1)
-
-ROOT = Path(__file__).resolve().parent.parent
-IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
-OUT_FILE = "platform/keycloak/client-urls.yaml"
-DATA_KEY = "client-urls.json"
-CONFIGMAP_NAME = "keycloak-client-urls"
-NAMESPACE = "keycloak"
-
-# The client fields this renderer exports, and therefore the ONLY fields the
-# reconciler may write. Every one of them is an environment-derived URL: it changes
-# when BASE_DOMAIN changes, and a stale value breaks OIDC login (a wrong
-# redirectUri) or silently widens trust (a stale wildcard left behind).
-#
-# DELIBERATELY ABSENT: `secret` (healed by the secret pass from the SOPS Secret,
-# and it must never appear in a ConfigMap), and every non-URL field —
-# protocolMappers, defaultClientScopes, attributes, flags. Those are not
-# environment-derived, so nothing re-renders them on a domain flip; keeping them
-# out keeps the reconciler's write surface as narrow as the problem.
-LIST_FIELDS = ("redirectUris", "webOrigins")
-STR_FIELDS = ("rootUrl", "baseUrl", "adminUrl")
-
-
-def render(root: Path) -> str:
-    """Return the full text of client-urls.yaml as it should be committed."""
-    docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
-    imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
-    if imp is None:
-        raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
-    clients = (imp.get("spec") or {}).get("realm", {}).get("clients") or []
-    if not clients:
-        # Fail closed. Emitting an empty desired state would make the reconciler
-        # believe every client is unmanaged and silently stop reconciling URLs.
-        raise SystemExit("FATAL: %s declares zero clients — refusing to render an "
-                         "empty desired state" % IMPORT_FILE)
-
-    want: dict[str, dict] = {}
-    for c in clients:
-        cid = c.get("clientId")
-        if not isinstance(cid, str) or not cid:
-            raise SystemExit("FATAL: %s has a client with no clientId" % IMPORT_FILE)
-        entry: dict[str, object] = {}
-        for f in LIST_FIELDS:
-            if f not in c:
-                continue                 # not declared => NOT MANAGED (never forced empty)
-            v = c[f]
-            if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
-                raise SystemExit("FATAL: %s client %r field %s is not a list of strings"
-                                 % (IMPORT_FILE, cid, f))
-            entry[f] = list(v)           # verbatim, order as authored
-        for f in STR_FIELDS:
-            if f not in c:
-                continue
-            v = c[f]
-            if not isinstance(v, str):
-                raise SystemExit("FATAL: %s client %r field %s is not a string"
-                                 % (IMPORT_FILE, cid, f))
-            entry[f] = v
-        if entry:
-            want[cid] = entry
-    if not want:
-        raise SystemExit("FATAL: no client in %s declares any of %s — refusing to render "
-                         "an empty desired state"
-                         % (IMPORT_FILE, ", ".join(LIST_FIELDS + STR_FIELDS)))
-
-    body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
-    indented = "\n".join(("    " + ln) if ln else "" for ln in body.splitlines())
-    return HEADER + indented + "\n"
-
-
-HEADER = """---
-# =============================================================================
-# GENERATED FILE — DO NOT EDIT BY HAND.
-#
-#   source:      platform/keycloak/keycloak-realm-import.yaml (spec.realm.clients)
-#   regenerate:  python3 ci/render-keycloak-client-urls.py --write
-#   gated by:    conformance/tests/l0-static/keycloak-client-secret-reconciler
-#                (runs this renderer with --check on every merge)
-#
-# WHAT THIS IS: the DESIRED STATE for every environment-derived client URL in the
-# `homelab` realm — redirectUris, webOrigins, rootUrl, baseUrl, adminUrl. The
-# client-secret reconciler CronJob mounts it and makes the live realm match it.
-#
-# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
-# once Done=True, so a redirect URI edited in git never reaches the realm. Prod's
-# stored redirect URIs were measured still pointing at the pre-migration hostname
-# while git declared the new one, and every OIDC login would have failed with a
-# redirect_uri mismatch. The reconciler is the level-triggered loop that closes
-# that; this file is what it reconciles TOWARDS.
-#
-# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of every public
-# hostname would be a second source of truth, free to disagree with the realm
-# import. It is rendered from the import and the render is gated, so the two
-# cannot drift. URL strings are copied VERBATIM — if the source uses a
-# ${BASE_DOMAIN} token it appears here unchanged, and both files are substituted
-# by the SAME Flux Kustomization (./platform-config), so the domain is decided in
-# exactly one place.
-#
-# A field that is NOT declared on a client in the realm import is simply ABSENT
-# here, and the reconciler leaves it alone. Absence means UNMANAGED, never
-# "force to empty".
-# =============================================================================
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: %s
-  namespace: %s
-  labels:
-    app.kubernetes.io/name: %s
-    app.kubernetes.io/part-of: keycloak
-data:
-  %s: |
-""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME, DATA_KEY)
-
-
-def main() -> int:
-    ap = argparse.ArgumentParser(description=__doc__)
-    g = ap.add_mutually_exclusive_group()
-    g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
-    g.add_argument("--check", action="store_true",
-                   help="exit 1 if the committed file has drifted from the source")
-    args = ap.parse_args()
-
-    want = render(ROOT)
-    target = ROOT / OUT_FILE
-
-    if args.write:
-        target.write_text(want, encoding="utf-8")
-        print("wrote %s" % OUT_FILE)
-        return 0
-
-    if args.check:
-        if not target.is_file():
-            print("CHECK FAIL: %s does not exist. Run: python3 "
-                  "ci/render-keycloak-client-urls.py --write" % OUT_FILE, file=sys.stderr)
-            return 1
-        have = target.read_text(encoding="utf-8")
-        if have == want:
-            print("client-urls.yaml is in sync with %s" % IMPORT_FILE)
-            return 0
-        print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
-              "realm on a desired state the realm import does not declare.\n"
-              "Regenerate with: python3 ci/render-keycloak-client-urls.py --write\n"
-              % (OUT_FILE, IMPORT_FILE), file=sys.stderr)
-        for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
-                                         fromfile="committed " + OUT_FILE,
-                                         tofile="rendered from " + IMPORT_FILE):
-            sys.stderr.write(line)
-        return 1
-
-    sys.stdout.write(want)
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

ci/tests/gate-decision.test.sh

@@ -1,189 +0,0 @@
-#!/usr/bin/env bash
-# ci/tests/gate-decision.test.sh — the merge gate's OWN decision table.
-#
-# WHY THIS SUITE EXISTS (2026-09-13). ci/gate.sh used to set BLOCK=1 for a FAILED stage but
-# only `warn` for a SKIPPED one, so it printed the words "— not green" and then concluded
-# `MERGE GATE: PASS` and exited 0. Observed live twice: `prom-rules` skipped for a missing
-# promtool, and `l0` skipped for a missing pinned kyverno CLI — both read by automation as
-# merge-approved. That is fail-open in the one component whose entire job is to fail closed,
-# and nothing tested it: every other ci/tests suite covers the INPUT (touched-requirements),
-# none covered the DECISION.
-#
-# WHAT IS UNDER TEST: the real ci/gate.sh and ci/lib.sh, copied into a throwaway CI_DIR beside
-# STUB stage scripts whose exit codes this suite dictates (STAGE_RC_<stage>) and a stub
-# touched-requirements.sh whose rc/reason it dictates (TR_RC / TR_REASON / TR_OUT). Stubbing the
-# stages is the point: this asserts how the gate REASONS about pass/fail/skip, independently of
-# what any stage happens to do on this machine — so it runs anywhere, with no tools, no cluster,
-# no network, in milliseconds.
-#
-# THE INVARIANT: `MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and
-# was GREEN. A stage that could not run is never a pass, and the exit code says so too.
-#
-# Run:  ci/tests/gate-decision.test.sh     (or `make -C ci test`)
-set -uo pipefail
-
-REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
-
-PASS=0 FAIL=0
-ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
-bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
-
-TMP="$(mktemp -d)"
-trap 'rm -rf "$TMP"' EXIT
-CI="${TMP}/ci"; mkdir -p "$CI"
-REPO="${TMP}/repo"; mkdir -p "$REPO"
-
-cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
-
-# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
-for stage in spec-immutability lint l0 render-golden l1 conformance; do
-  cat > "${CI}/${stage}.sh" <<'STUB'
-#!/usr/bin/env bash
-n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
-exit "${!v:-0}"
-STUB
-  chmod +x "${CI}/${stage}.sh"
-done
-
-# Stub decision input. TR_RC (default 0), TR_REASON (written to $TOUCHED_REASON_FILE like the
-# real script does), TR_OUT (the touched requirement list on stdout).
-cat > "${CI}/touched-requirements.sh" <<'STUB'
-#!/usr/bin/env bash
-if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "${TR_REASON:-}" > "$TOUCHED_REASON_FILE"; fi
-[ -n "${TR_OUT:-}" ] && printf '%s\n' "$TR_OUT"
-exit "${TR_RC:-0}"
-STUB
-chmod +x "${CI}/touched-requirements.sh"
-
-# run_gate <VAR=VAL>... — run the gate with a clean, explicit environment. Sets OUT and RC.
-# GATE_STRICT_SKIPS is pinned false unless a case says otherwise, so the suite never depends on
-# whether the machine running it happens to look like CI.
-OUT="" RC=0
-run_gate() {
-  local summary_file="${TMP}/summary.md"; : > "$summary_file"
-  OUT="$(env -i PATH="$PATH" HOME="$HOME" \
-        REPO_ROOT="$REPO" GITHUB_STEP_SUMMARY="$summary_file" \
-        GATE_STRICT_SKIPS=false TR_OUT="R-TEST-03" \
-        "$@" bash "${CI}/gate.sh" 2>&1)"
-  RC=$?
-  SUMMARY="$(cat "$summary_file")"
-}
-
-# assert_verdict <headline> <expected-rc> <label>
-assert_verdict() {
-  local want="$1" want_rc="$2" label="$3"
-  if ! grep -qF "MERGE GATE: ${want}" <<<"$OUT"; then
-    bad "${label}: headline should be '${want}' — got: $(grep -oE 'MERGE GATE: [A-Z ]+' <<<"$OUT" | tail -1)"
-  elif [ "$RC" -ne "$want_rc" ]; then
-    bad "${label}: headline '${want}' correct but exit code ${RC} != ${want_rc}"
-  else
-    ok "${label} -> ${want}, exit ${want_rc}"
-  fi
-}
-# The load-bearing negative: nothing may say PASS, and nothing may exit 0.
-assert_not_pass() {
-  local label="$1"
-  if grep -qF "MERGE GATE: PASS" <<<"$OUT"; then bad "${label}: printed 'MERGE GATE: PASS'"
-  elif [ "$RC" -eq 0 ]; then bad "${label}: exited 0"
-  else ok "${label}: not a PASS, non-zero exit (${RC})"; fi
-}
-
-echo "== 1. the happy path: every stage green"
-run_gate
-assert_verdict PASS 0 "all stages green + a touched subset"
-grep -qF "nothing was declared out of scope" <<<"$SUMMARY" \
-  && ok "PASS summary states nothing was declared out of scope" \
-  || bad "PASS summary does not state the absence of declared gaps"
-
-echo
-echo "== 2. a FAILED stage blocks (unchanged behaviour, locked down)"
-for s in lint l0 render-golden l1 conformance spec-immutability; do
-  run_gate "STAGE_RC_${s//-/_}=1"
-  assert_verdict BLOCKED 1 "${s} FAILED"
-done
-
-echo
-echo "== 3. THE REGRESSION: a SKIPPED stage is not a PASS and not exit 0"
-for s in lint l0 render-golden l1; do
-  run_gate "STAGE_RC_${s//-/_}=3"
-  assert_not_pass "${s} SKIPPED (local)"
-  assert_verdict UNDETERMINED 3 "${s} SKIPPED (local)"
-  grep -qF "$s" <<<"$SUMMARY" && ok "${s}: the step summary names the stage that did not run" \
-                              || bad "${s}: step summary does not name the skipped stage"
-done
-
-echo
-echo "== 4. under STRICT skips (CI declares it) a could-not-run skip BLOCKS, never merely 3"
-for s in lint l0 render-golden l1; do
-  run_gate GATE_STRICT_SKIPS=true "STAGE_RC_${s//-/_}=3"
-  assert_verdict BLOCKED 1 "${s} SKIPPED with GATE_STRICT_SKIPS=true"
-done
-# ...which is what makes gate.yml's `|| [ $? -eq 3 ]` safe: in CI, exit 3 cannot be a missing
-# tool or an unwired seam. Only the declared-environmental skip below can produce it.
-run_gate GATE_STRICT_SKIPS=true STAGE_RC_l0=3 STAGE_RC_lint=3
-assert_verdict BLOCKED 1 "several stages SKIPPED under strict skips"
-
-echo
-echo "== 5. spec-immutability: unverified is never green, but is not upgraded to a block by CI"
-run_gate STAGE_RC_spec_immutability=3
-assert_not_pass "spec-immutability SKIPPED"
-assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED (local)"
-run_gate GATE_STRICT_SKIPS=true STAGE_RC_spec_immutability=3
-assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED under strict skips (deliberate exception)"
-run_gate REQUIRE_SPEC_VERIFICATION=true STAGE_RC_spec_immutability=3
-assert_verdict BLOCKED 1 "spec-immutability SKIPPED + REQUIRE_SPEC_VERIFICATION=true"
-
-echo
-echo "== 6. conformance L2/L3: a DECLARED-absent capability may still pass — and is NAMED"
-run_gate STAGE_RC_conformance=3
-assert_verdict PASS 0 "conformance SKIPPED, no cluster declared"
-grep -qF "conformance-l2-l3" <<<"$OUT" \
-  && ok "the PASS headline names the declared L2/L3 gap" \
-  || bad "the PASS headline hides the declared L2/L3 gap"
-grep -qF "conformance-l2-l3" <<<"$SUMMARY" \
-  && ok "the step summary names the declared L2/L3 gap" \
-  || bad "the step summary hides the declared L2/L3 gap"
-run_gate CLUSTER_AVAILABLE=true STAGE_RC_conformance=3
-assert_verdict BLOCKED 1 "conformance SKIPPED despite CLUSTER_AVAILABLE=true"
-run_gate REQUIRE_CLUSTER_TESTS=true STAGE_RC_conformance=3
-assert_verdict BLOCKED 1 "conformance SKIPPED + REQUIRE_CLUSTER_TESTS=true with a touched subset"
-# A misspelled switch must leave the guard OFF (truthy()), never half-on.
-run_gate CLUSTER_AVAILABLE=ture STAGE_RC_conformance=3
-assert_verdict PASS 0 "CLUSTER_AVAILABLE=ture is NOT 'available' (no cluster was declared)"
-run_gate CLUSTER_AVAILABLE=TRUE STAGE_RC_conformance=3
-assert_verdict BLOCKED 1 "CLUSTER_AVAILABLE=TRUE IS a declaration (skip then blocks)"
-
-echo
-echo "== 7. the undetermined touched subset: unchanged (2026-08-31 false-green paths)"
-run_gate TR_RC=3 TR_REASON=on-base TR_OUT=
-assert_verdict "NOT APPLICABLE" 0 "subset undeterminable, reason on-base, all stages green"
-run_gate TR_RC=3 TR_REASON=no-base TR_OUT=
-assert_verdict BLOCKED 1 "subset undeterminable, reason no-base"
-run_gate TR_RC=3 TR_REASON=diff-failed TR_OUT=
-assert_verdict BLOCKED 1 "subset undeterminable, reason diff-failed"
-# Precedence: a stage that did not run outranks "nothing to gate" — the stronger, non-zero
-# headline must win, or a push-to-main run could hide a missing tool behind NOT APPLICABLE.
-run_gate TR_RC=3 TR_REASON=on-base TR_OUT= STAGE_RC_l1=3
-assert_not_pass "on-base AND l1 SKIPPED"
-assert_verdict UNDETERMINED 3 "on-base AND l1 SKIPPED (UNDETERMINED outranks NOT APPLICABLE)"
-
-echo
-echo "== 8. an UNUSABLE subset blocks (carried over from 47068ee): any other non-zero exit"
-for rc in 1 2 4; do
-  run_gate "TR_RC=${rc}" TR_REASON=unclassified-path TR_OUT=
-  assert_verdict BLOCKED 1 "touched-requirements.sh exited ${rc} (subset unusable, not empty)"
-done
-run_gate TR_RC=1 TR_REASON=unclassified-path TR_OUT=
-grep -qF "UNUSABLE" <<<"$SUMMARY" && ok "the step summary calls the subset UNUSABLE" \
-                                  || bad "the step summary does not report an unusable subset"
-
-echo
-echo "== 9. combinations: a FAILURE outranks a non-run, and the headline is the worst news"
-run_gate STAGE_RC_l0=1 STAGE_RC_l1=3
-assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
-run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
-assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
-
-echo
-if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
-else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/tests/touched-requirements-test-attribution.test.sh

@@ -1,164 +0,0 @@
-#!/usr/bin/env bash
-# ci/tests/touched-requirements-test-attribution.test.sh — regression tests for DETECTION
-# RULE 3 in ci/touched-requirements.sh: a change anywhere under `conformance/tests/<test>/`
-# attributes THAT TEST'S OWN `conformance.home.k3s/R-*` labels.
-#
-# THE GAP THESE LOCK DOWN (2026-09-12). The coarse path map keys `conformance/*` to
-# R-TEST-01..04 — the requirements about the HARNESS — and says nothing about what a given
-# test PROVES. So weakening `conformance/tests/.../check.sh` attributed R-TEST-01..04 and
-# nothing else: the merge gate ran the harness's self-tests and NOT the weakened test's own
-# requirement subset. Weakening (or deleting) a test is the one change that most needs the
-# requirement it certifies to be re-verified, and it was the change least likely to select it.
-# The content scan cannot close this: it reads only ADDED/REMOVED lines, so it sees a change
-# to the LABEL BLOCK but not one to the test BODY — exactly the case that matters.
-#
-# Also covers the two path-map entries added in the same change (infra/* -> R-ENV-01, and
-# R-TEST-06 on .gitea/*|ci/*), which is how the two L0 checks that are DELIBERATELY not
-# whole-repo-scoped get selected by the paths that can break them.
-#
-# Same harness as the sibling suites: the real script, pointed at a THROWAWAY git repo via
-# the REPO_ROOT override. No network, no cluster, the working repo is never touched.
-#
-# Run:  ci/tests/touched-requirements-test-attribution.test.sh   (or `make -C ci test`)
-set -euo pipefail
-
-CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
-SCRIPT="${CI_DIR}/touched-requirements.sh"
-
-PASS=0 FAIL=0
-ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
-bad() { printf '  FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
-
-assert_present() {
-  if grep -qxF "$1" <<<"$2"; then ok "$3"
-  else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
-}
-assert_absent() {
-  if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))"
-  else ok "$3"; fi
-}
-
-TMP="$(mktemp -d)"
-trap 'rm -rf "$TMP"' EXIT
-
-git -C "$TMP" init -q -b main
-git -C "$TMP" config user.email [email protected]
-git -C "$TMP" config user.name t
-git -C "$TMP" config commit.gpgsign false
-
-# --- Baseline: two conformance tests, each declaring its own requirement labels ----------
-# TEST A: a flat l0-static test. TEST B: a test with a NESTED fixture dir, to prove the
-# walk-up from an arbitrary changed file to the enclosing test dir.
-mkdir -p "$TMP/conformance/tests/l0-static/alpha" \
-         "$TMP/conformance/tests/l2-behavioral/beta/fixtures" \
-         "$TMP/conformance/lib" "$TMP/infra" "$TMP/ci"
-
-cat > "$TMP/conformance/tests/l0-static/alpha/chainsaw-test.yaml" <<'YAML'
-# HEADER PROSE. This test used to also cover conformance.home.k3s/R-STALE-09: "true",
-# which is a REFERENCE in a comment, not a declaration, and must NOT be attributed.
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: alpha
-  labels:
-    conformance.home.k3s/R-ALPHA-01: "true"
-    conformance.home.k3s/R-ALPHA-02: "true"
-  annotations:
-    conformance.home.k3s/layer: "L0"
-spec: {}
-YAML
-# The check BODY deliberately contains NO R-ID text, so anything the scan finds for a
-# body-only edit came from rule 3 and not from the content scan.
-cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
-#!/usr/bin/env bash
-echo "assert something"
-SH
-
-cat > "$TMP/conformance/tests/l2-behavioral/beta/chainsaw-test.yaml" <<'YAML'
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: beta
-  labels:
-    conformance.home.k3s/R-BETA-07: "true"
-spec: {}
-YAML
-echo "seed: 1" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
-
-echo "harness helper" > "$TMP/conformance/lib/helper.sh"
-git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
-
-# ========================================================================================
-echo "== case 1: a BODY-ONLY edit to a test attributes that test's own labels"
-git -C "$TMP" checkout -q -b body-edit
-cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
-#!/usr/bin/env bash
-echo "assert something weaker"
-SH
-git -C "$TMP" add -A && git -C "$TMP" commit -qm weaken
-OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
-printf 'touched set:\n%s\n---\n' "$OUT"
-assert_present "R-ALPHA-01" "$OUT" "body-only test edit attributes the test's own R-ALPHA-01"
-assert_present "R-ALPHA-02" "$OUT" "body-only test edit attributes the test's own R-ALPHA-02"
-assert_present "R-TEST-03"  "$OUT" "the conformance/* path map still applies (R-TEST-03)"
-assert_absent  "R-STALE-09" "$OUT" "an R-ID cited only in the test's HEADER COMMENT is NOT attributed"
-assert_absent  "R-BETA-07"  "$OUT" "an UNRELATED test's labels are NOT attributed"
-
-# ========================================================================================
-echo "== case 2: an edit to a NESTED file walks up to the enclosing test"
-git -C "$TMP" checkout -q -b nested-edit main
-echo "seed: 2" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
-git -C "$TMP" add -A && git -C "$TMP" commit -qm fixture
-OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
-printf 'touched set:\n%s\n---\n' "$OUT"
-assert_present "R-BETA-07" "$OUT" "nested fixture edit attributes the enclosing test's R-BETA-07"
-assert_absent  "R-ALPHA-01" "$OUT" "nested fixture edit does NOT attribute a sibling test's labels"
-
-# ========================================================================================
-echo "== case 3: DELETING a test still attributes what it used to certify"
-# Removing the only verifier of a requirement must not be the cheapest way to make the gate
-# stop checking it — so the labels are read from the BASE when HEAD no longer has them.
-git -C "$TMP" checkout -q -b delete-test main
-git -C "$TMP" rm -rq "$TMP/conformance/tests/l0-static/alpha"
-git -C "$TMP" commit -qm "delete the alpha test"
-OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
-printf 'touched set:\n%s\n---\n' "$OUT"
-assert_present "R-ALPHA-01" "$OUT" "deleting a test STILL attributes its R-ALPHA-01 (read from base)"
-assert_present "R-ALPHA-02" "$OUT" "deleting a test STILL attributes its R-ALPHA-02 (read from base)"
-
-# ========================================================================================
-echo "== case 4: a change OUTSIDE conformance/tests/ gains no test labels"
-git -C "$TMP" checkout -q -b lib-edit main
-echo "harness helper v2" > "$TMP/conformance/lib/helper.sh"
-git -C "$TMP" add -A && git -C "$TMP" commit -qm lib
-OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
-printf 'touched set:\n%s\n---\n' "$OUT"
-assert_present "R-TEST-02"  "$OUT" "a conformance/lib edit still attributes the harness requirements"
-assert_absent  "R-ALPHA-01" "$OUT" "a conformance/lib edit attributes NO individual test's labels"
-assert_absent  "R-BETA-07"  "$OUT" "a conformance/lib edit attributes NO individual test's labels (B)"
-
-# ========================================================================================
-echo "== case 5: path-map entries for the two DELIBERATELY path-scoped L0 checks"
-# infra/* -> R-ENV-01 (dev-nodes-provisioned-as-code reads nothing but infra/).
-git -C "$TMP" checkout -q -b infra-edit main
-printf 'resource "x" "y" {}\n' > "$TMP/infra/main.tf"
-git -C "$TMP" add -A && git -C "$TMP" commit -qm infra
-OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
-printf 'touched set:\n%s\n---\n' "$OUT"
-assert_present "R-ENV-01" "$OUT" "an infra/ edit attributes R-ENV-01 (was UNMAPPED: attributed nothing)"
-
-# ci/* -> ... R-TEST-06 (spec-immutability-enforced reads nothing but ci/ + the spec repo).
-git -C "$TMP" checkout -q -b ci-edit main
-printf 'echo hello\n' > "$TMP/ci/thing.sh"
-git -C "$TMP" add -A && git -C "$TMP" commit -qm ci
-OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
-printf 'touched set:\n%s\n---\n' "$OUT"
-assert_present "R-TEST-06"  "$OUT" "a ci/ edit attributes R-TEST-06 (the spec-immutability test's own ID)"
-assert_present "R-GITOPS-01" "$OUT" "a ci/ edit still attributes R-GITOPS-01"
-
-echo
-if [ "$FAIL" -eq 0 ]; then
-  echo "touched-requirements test-attribution tests: PASS (${PASS} assertions)"; exit 0
-fi
-echo "touched-requirements test-attribution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
-exit 1

ci/tests/touched-requirements.test.sh

@@ -93,82 +93,6 @@ printf 'touched set (removal):\n%s\n---\n' "$OUT2"
 assert_absent  "R-DELDOC-01"  "$OUT2" "REMOVED whole-line comment R-ID is NOT attributed"
 assert_present "R-DELREAL-01"  "$OUT2" "REMOVED real-line R-ID IS attributed"
 
-# ---------------------------------------------------------------------------------------
-# PATH MAP — bootstrap/* (added 2026-09-13)
-#
-# WHY THIS IS LOCKED DOWN. path_requirements() keys `bootstrap/*` to R-STATE-03 +
-# R-GITOPS-01, and until 2026-09-13 the comment justifying it described
-# `bootstrap/standup.sh` as a file that carries the placeholder guards. That file has never
-# been on `main` (it landed only on unmerged prod-standup branches), so the comment was
-# stale and the entry looked like dead weight someone would tidy away. It is not dead: it
-# is what SELECTS conformance/tests/l0-static/placeholder-guard-coupling for the change
-# that eventually lands a placeholder guard under bootstrap/ — and that test discovers its
-# guard by content rather than by path, so it activates on exactly that change. Removing
-# the entry would let the change introducing a guard be gated by nothing, which is the
-# 2026-09-05 class. These assertions make the entry's deletion a test failure rather than a
-# silent loss of selection.
-#
-# NOTE the R-ID content scan must not be what produces these: `notes/` is outside the path
-# map precisely so the other cases isolate the scan, and here the file content cites no
-# R-ID at all, so any R-ID in the output came from the PATH.
-git -C "$TMP" checkout -q -b feature3 main
-mkdir -p "$TMP/bootstrap/flux"
-cat > "$TMP/bootstrap/flux/bootstrap.sh" <<'SH'
-#!/usr/bin/env bash
-# a bootstrap-path change that cites no requirement id of its own
-echo "flux bootstrap"
-SH
-git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch bootstrap/"
-OUT3="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
-printf 'touched set (bootstrap/ path map):\n%s\n---\n' "$OUT3"
-assert_present "R-STATE-03"   "$OUT3" "bootstrap/* attributes R-STATE-03 (selects placeholder-guard-coupling)"
-assert_present "R-GITOPS-01"  "$OUT3" "bootstrap/* attributes R-GITOPS-01 (it IS the GitOps bootstrap path)"
-
-# And the neighbouring entry the same test leans on for its other half: clusters/* must
-# attribute R-ENV-02, which is what selects placeholder-guard-coupling's HALF A (the
-# authored-values sweep over clusters/<env> effective cluster-vars).
-git -C "$TMP" checkout -q -b feature4 main
-mkdir -p "$TMP/clusters/base"
-printf 'kind: ConfigMap\nmetadata:\n  name: cluster-vars\n' > "$TMP/clusters/base/cluster-vars.yaml"
-git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch clusters/"
-OUT4="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
-printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
-assert_present "R-ENV-02"     "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
-assert_present "R-GITOPS-01"  "$OUT4" "clusters/* attributes R-GITOPS-01"
-
-# ---------------------------------------------------------------------------------------
-# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
-#
-# The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match `R-C2-01` or
-# `R-C2-02` — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. Those are
-# the only two registry IDs whose workstream segment carries a digit, and the scan was blind
-# to BOTH, in the fail-OPEN direction: a change self-declaring a C2 requirement contributed
-# it to nobody, so the C2 conformance tests were filtered out of the merge gate. Found while
-# wiring conformance/tests/l0-static/c2-root-surface, which closes those two coverage gaps.
-#
-# `notes/` is outside path_requirements() on purpose, so the IDs below can only have come
-# from the CONTENT SCAN. A neighbouring all-letter ID is asserted too, so a regex that
-# somehow matched only the digit form would still fail here.
-git -C "$TMP" checkout -q -b feature5 main
-cat > "$TMP/notes/c2.yaml" <<'YAML'
-kind: Test
-metadata:
-  labels:
-    conformance.home.k3s/R-C2-01: "true"
-    conformance.home.k3s/R-C2-02: "true"
-    conformance.home.k3s/R-OBS-04: "true"
-YAML
-git -C "$TMP" add -A && git -C "$TMP" commit -qm "declare R-C2-01 / R-C2-02"
-OUT5="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
-printf 'touched set (digit in the workstream segment):\n%s\n---\n' "$OUT5"
-assert_present "R-C2-01"  "$OUT5" "an R-ID with a DIGIT in the workstream segment IS attributed (R-C2-01)"
-assert_present "R-C2-02"  "$OUT5" "the second such ID is attributed too (R-C2-02)"
-assert_present "R-OBS-04" "$OUT5" "an all-letter workstream segment still works (control)"
-# The pattern must not become so loose that a non-ID token is read as a requirement: a bare
-# numeric workstream is not a registry shape, and attributing one would WIDEN the gate on
-# noise (over-broad selection, never a false green, but noisy and misleading).
-assert_absent  "R-1-01"   "$OUT5" "a purely numeric workstream segment is not invented from thin air"
-
 echo
 if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
 else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/touched-requirements.sh

@@ -21,12 +21,8 @@
 #      the R-IDs that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011).
 #      Untouched lines and prose are NOT scanned; see changed_diff() for why.
 #   2. A coarse path->requirement fallback for changes that don't self-declare IDs.
-#   3. For a change anywhere under `conformance/tests/<test>/`: THAT TEST'S OWN requirement
-#      labels, read from its chainsaw-test.yaml. See test_requirements_for_path().
 # The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for
-# untagged changes so nothing silently escapes the gate. (3) is the same principle applied
-# to the tests themselves: editing or weakening a test must be gated by the requirements
-# that test CERTIFIES, not merely by R-TEST-0x.
+# untagged changes so nothing silently escapes the gate.
 set -euo pipefail
 CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # shellcheck disable=SC1091
@@ -184,128 +180,14 @@ path_requirements() {
     platform/externaldns/*)    echo "R-DNS-02" ;;
     platform/velero/*)         echo "R-STATE-03" ;;
     platform/monitoring/*)     echo "R-OBS-01 R-OBS-02" ;;
-    # platform/cnpg was UNMAPPED until 2026-09-06. It is the PostgreSQL DR path for keycloak-db /
-    # nextcloud-db / semaphore-db (barmanObjectStore WAL archiving + ScheduledBackup), so a change
-    # there is an R-IDP-05 / R-STATE-03 change; it now also carries the platform-health alert rules
-    # for that path (R-OBS-04). Without this line an edit under platform/cnpg/ contributed ZERO
-    # requirements unless it happened to retype an R-ID on a non-comment line, and escaped the gate.
-    platform/cnpg/*)           echo "R-IDP-05 R-STATE-03 R-OBS-04" ;;
-    # vpn-egress was UNMAPPED until 2026-09-06. platform/vpn-egress/egress-steering.yaml IS the
-    # R-NET-04 datapath (the rule that selects between the two egress paths) and the R-NET-05/06/07
-    # steering policy — it even self-declares those IDs in its `homelab.keithjackson.dev/requirements`
-    # annotation. But the content scan only reads ADDED/REMOVED lines, so any edit that does not
-    # happen to retype an R-ID (a selector change, an excludedCIDRs change, a comment) contributed
-    # ZERO requirements and escaped the gate entirely. Mapped explicitly; the annotation still wins
-    # when a diff does carry the tags.
-    platform/vpn-egress/*)     echo "R-NET-04 R-NET-05 R-NET-06 R-NET-07" ;;
-    # Keycloak is the whole SSO surface: the realm import (declarative identity config),
-    # the SOPS client secrets, and the reconciler that heals them. Added 2026-09-06 after
-    # the client-secret incident, in which editing platform/keycloak/ attributed NO
-    # requirement, so the L2 keycloak tests were filtered out of the merge gate's touched
-    # subset — the change that could break every OIDC login was gated by nothing.
-    # R-IDP-05 is in scope because platform/keycloak/ also carries the CNPG DB Cluster and
-    # its off-cluster backup credential, not only the realm.
-    platform/keycloak/*)       echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
-    platform/oauth2-proxy/*)   echo "R-IDP-02 R-IDP-06" ;;
     platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
     conformance/*)             echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
-    # infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
-    # nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
-    # infra/, yet an infra/ edit attributed ZERO requirements, so the one check that asserts
-    # the nodes are provisioned as code was filtered out of every change that could break it.
-    # That test's invariant IS path-scoped (it is deliberately NOT whole-repo-scoped; see its
-    # chainsaw-test.yaml), and the correct fix for a path-scoped invariant is a path-map entry.
-    infra/*)                   echo "R-ENV-01" ;;
     clusters/*)                echo "R-GITOPS-01 R-ENV-02" ;;
-    # bootstrap/ was UNMAPPED until 2026-09-06. The entry STAYS, but its original rationale
-    # was written against a file that is not here, and that is corrected on 2026-09-13:
-    # it claimed "bootstrap/standup.sh carries the placeholder guards that decide whether
-    # `velero` may be enabled against a real backup endpoint". bootstrap/standup.sh HAS NEVER
-    # BEEN ON `main` — it was not deleted (no deletion commit exists anywhere), it landed only
-    # on unmerged prod-standup branches. `git ls-tree main bootstrap/` lists README.md, flux/,
-    # gitea/, sops/. So R-STATE-03 here is ANTICIPATORY, not descriptive.
-    #
-    # WHY IT IS STILL RIGHT. conformance/tests/l0-static/placeholder-guard-coupling DISCOVERS
-    # its guard (any tracked *.sh defining `<name>_is_placeholder()`) instead of hardcoding a
-    # path, so the moment a standup guard lands under bootstrap/ that test's coupling half
-    # activates — and this entry is what selects the test for the change that lands it.
-    # Deleting the entry would mean the change introducing a placeholder guard is gated by
-    # nothing, which is the 2026-09-05 class (a prefix glob that stopped matching when the
-    # value it guards changed, in a different file) left to reintroduce itself. R-GITOPS-01 is
-    # descriptive today: bootstrap/flux/, gitea/ and sops/ ARE the GitOps bootstrap path.
-    # Locked down by ci/tests/touched-requirements.test.sh's path-map section.
-    bootstrap/*)               echo "R-STATE-03 R-GITOPS-01" ;;
-    # R-TEST-06 added 2026-09-12. The comment in changed_diff()'s rule 3 above records why it
-    # was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE and carried by no
-    # Chainsaw test, so attributing it made the coverage `--require` check false-block. It is now
-    # carried by conformance/tests/l0-static/spec-immutability-enforced — whose entire input
-    # surface is ci/ — so a ci/ change must attribute it, or the test that proves the
-    # spec-immutability mechanism is still load-bearing runs on nothing that can break it.
-    .gitea/*|ci/*)             echo "R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06" ;;
+    .gitea/*|ci/*)             echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
     *) : ;;
   esac
 }
 
-# ---------------------------------------------------------------------------------
-# A CHANGED TEST ATTRIBUTES THE REQUIREMENTS IT CERTIFIES (detection rule 3).
-#
-# THE GAP THIS CLOSES (2026-09-12). The path map keys `conformance/*` to
-# R-TEST-01..04 — the requirements about the HARNESS. It says nothing about what a given
-# test PROVES. So editing `conformance/tests/l2-behavioral/tenant-isolation/check.sh` to
-# weaken or delete an assertion attributed R-TEST-01..04 and nothing else: the merge gate
-# then ran the harness's self-tests and NOT the weakened test's own requirement subset.
-# Weakening a test is the one change that most needs the requirement it certifies to be
-# re-verified, and it was the change least likely to select it.
-#
-# The `conformance.home.k3s/R-*: "true"` labels on a test ARE the authoritative declaration of
-# what it verifies (lib/coverage.py reads the same labels); this reads them directly. Note the
-# content scan in main() only sees ADDED/REMOVED lines, so it catches a change to the LABEL
-# BLOCK but not a change to the test BODY — which is exactly the case that matters here.
-#
-# WHY IT READS HEAD *OR* THE BASE: a change that DELETES a test, or deletes its
-# chainsaw-test.yaml, must still attribute what that test used to certify — otherwise removing
-# the only verifier of a requirement is the cheapest way to make the gate stop checking it.
-# HEAD first (the post-change truth), base as the fallback (the deleted case). Fail-closed:
-# this function can only ADD requirements, never remove one.
-: "${CONF_TEST_LABEL_PREFIX:=conformance.home.k3s}"   # == CONF_LABEL_PREFIX, conformance/lib/common.sh
-
-file_text_head_or_base() {
-  local path="$1"
-  git -C "$REPO_ROOT" show "HEAD:${path}" 2>/dev/null && return 0
-  git -C "$REPO_ROOT" show "${BASE}:${path}" 2>/dev/null && return 0
-  return 0
-}
-
-# test_requirements_for_path <changed-path> -> space-separated R-IDs (possibly empty).
-# Walks UP from the changed file to the nearest ancestor directory carrying a
-# chainsaw-test.yaml, so it works for any file in a test dir (check.sh, mutations.py,
-# fixtures, the chainsaw-test.yaml itself) and for nested test layouts.
-test_requirements_for_path() {
-  local f="$1" dir yaml text
-  case "$f" in conformance/tests/*/*) ;; *) return 0 ;; esac
-  dir="${f%/*}"
-  while :; do
-    case "$dir" in
-      conformance/tests|conformance|.|"") return 0 ;;
-      */*) ;;
-      *) return 0 ;;
-    esac
-    yaml="${dir}/chainsaw-test.yaml"
-    text="$(file_text_head_or_base "$yaml")"
-    if [ -n "$text" ]; then
-      # Whole-line `#` comments are dropped first — same "reference, not declaration" rule the
-      # content scan applies (changed_diff() rule 3): a label shown inside a header example is
-      # documentation. Only a real YAML mapping line counts.
-      printf '%s\n' "$text" \
-        | grep -vE '^[[:space:]]*#' \
-        | sed -nE "s#^[[:space:]]*${CONF_TEST_LABEL_PREFIX}/(R-[A-Z0-9]+-[0-9]+):[[:space:]]*[\"']?true.*#\1#p" \
-        | tr '\n' ' '
-      return 0
-    fi
-    dir="${dir%/*}"
-  done
-}
-
 main() {
   local files reqs="" rc=0 base_sha head_sha local_main merge_base
 
@@ -339,27 +221,9 @@ main() {
   while IFS= read -r f; do
     [ -n "$f" ] || continue
     reqs+=" $(path_requirements "$f")"
-    # Detection rule 3: a change under conformance/tests/<test>/ attributes THAT TEST'S own
-    # requirement labels, so weakening a test is gated by what the test certifies.
-    reqs+=" $(test_requirements_for_path "$f")"
   done <<< "$files"
 
   # Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
-  #
-  # THE ID PATTERN IS `R-[A-Z0-9]+-[0-9]+`, NOT `R-[A-Z]+-[0-9]+`. The WORKSTREAM segment may
-  # contain a DIGIT: `R-C2-01` and `R-C2-02` (command & control UI) are the only two such IDs
-  # in conformance/requirements/registry.yaml today, and the stricter pattern silently failed
-  # to match BOTH of them — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is.
-  # Every other consumer in the repo already allows the digit: test_requirements_for_path()
-  # twenty lines above uses `R-[A-Z0-9]+-[0-9]+`, as do conformance/lib/coverage.py,
-  # conformance/lib/status.py and conformance/lib/run_l1_offline.py. This grep was the lone
-  # dissenter, and it dissented by UNDER-attributing — the fail-OPEN direction. A change that
-  # retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02 from a
-  # path the map keys elsewhere, contributed those IDs to NOBODY, so the C2 tests were
-  # filtered out of the merge gate. Same defect class as the map's `*) : ;;` arm: a miss that
-  # reads as "nothing touched". path_requirements() has no arm for platform/homepage/* or
-  # platform/headlamp/* either, so before this fix the ONLY route to R-C2-01/02 was rule 3
-  # (editing the C2 test itself). Regression-tested in ci/tests/touched-requirements.test.sh.
   # `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
   # `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so
   #   what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
@@ -373,7 +237,7 @@ main() {
     | grep -vE '^(\+\+\+|---)' \
     | sed -E 's/^[+-]//' \
     | grep -vE '^[[:space:]]*#' \
-    | grep -hoE 'R-[A-Z0-9]+-[0-9]+' \
+    | grep -hoE 'R-[A-Z]+-[0-9]+' \
     | tr '\n' ' ' || true)"
 
   # Normalize: split, unique, sort. Emit to stdout.

ci/versions.env

@@ -21,15 +21,6 @@ YAMLLINT_VERSION="1.35.1"
 KUBECTL_VERSION="1.36.4"
 YQ_VERSION="4.44.6"
 
-# Alert-rule unit tests (promtool test rules) — the L0 check
-# conformance/tests/l0-static/alert-rules-unit-tested runs every
-# platform/**/*.promtool-test.yaml through it. MUST stay in lockstep with
-# infra/devcontainer/versions.env (which bakes it into the image) AND with the Prometheus
-# version running on prod, because promtool IS the rule engine: verified 2026-09-12 as
-# v3.14.0-distroless via `kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`.
-# Shipped inside the prometheus release tarball, not as a separate release.
-PROMTOOL_VERSION="3.14.0"
-
 # ServiceDefinition JSON-Schema validation (schemas/validate.sh). Pip-installed;
 # pinned here so CI and a laptop validate against the exact same validator.
 CHECK_JSONSCHEMA_VERSION="0.29.4"

clusters/base/backup.yaml

@@ -34,9 +34,6 @@ spec:
     substituteFrom:
       - kind: ConfigMap
         name: cluster-vars
-  # SOPS decryption for the `cloud` INI credentials — velero-cloud.sops.yaml (dev's scoped MinIO key)
-  # and velero-cloud-prod.sops.yaml (prod's Cloudflare R2 token) — is layered per environment by the
-  # overlay (clusters/dev and clusters/prod patch THIS Kustomization by name), identical to how
-  # headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. Both Secrets are applied on both
-  # clusters; ${BACKUP_CREDENTIALS_SUFFIX} (substituted from cluster-vars above) decides which one the
-  # velero Deployment actually mounts. See clusters/dev/kustomization.yaml.
+  # SOPS decryption for velero-cloud.sops.yaml (the scoped MinIO `cloud` credential) is layered per
+  # environment by the overlay (clusters/dev patches THIS Kustomization by name), identical to how
+  # headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. See clusters/dev/kustomization.yaml.

clusters/base/cluster-vars.yaml

@@ -96,24 +96,9 @@ data:
   #
   # Substitution coverage (verify before adding a knob here): every path that consumes these declares
   # postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
-  # (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
-  # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
-  # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
-  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
-  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
-  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
-  # UNSUBSTITUTED path would ship the literal string into the cluster.
-  #
-  # ADDING A KNOB IS SAFE UNDER StrictPostBuildSubstitutions ONLY BECAUSE OF THE ROOT EDGE. An
-  # UNDEFINED variable is a HARD Kustomization failure, so a manifest that references a knob the
-  # live ConfigMap does not yet carry breaks its whole tier. `clusters/base/platform.yaml` gained
-  # `dependsOn: [flux-system]` for exactly this: dependsOn is REVISION-gated, so no substituting
-  # tier can reconcile revision N until the ROOT Kustomization has APPLIED revision N — i.e. until
-  # the new cluster-vars key EXISTS. Every substituting consumer chains transitively through
-  # `platform`, so one edge covers the graph, and a new knob plus its first reference may ship in
-  # ONE commit. (A hand-run `kubectl apply -k clusters/<env>` bypasses that ordering entirely: apply
-  # the overlay BEFORE the tier that references the knob, or the tier fails until it retries.)
+  # (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
+  # velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
+  # ship the literal string into the cluster.
 
   # k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
   #
@@ -161,71 +146,13 @@ data:
   # HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
   UNIFI_L4_VIP: "192.168.1.54"               # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
   CRAFTY_L4_VIP: "192.168.1.55"              # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
-  # PiHole's DNS listener: udp/53 + tcp/53 for LAN CLIENTS (apps/wave-0/pihole-dns.yaml). It is a
-  # THIRD pinned L4 VIP for the same reason as the two above — the `shared` Gateway is an L7 HTTPS
-  # terminator and cannot express DNS at all (udp/53 especially). It must NOT be ${GATEWAY_VIP}:
-  # LAN clients are pointed at this address as their resolver, and it has to stay put across
-  # reschedules, so it is pinned by lbipam.cilium.io/ips rather than taken from the pool head.
-  #
-  # WHY A SEPARATE ADDRESS AND NOT THE GATEWAY VIP: every `<svc>.${BASE_DOMAIN}` name resolves TO
-  # the Gateway VIP. Serving DNS from the same address would be legal but conflates "the thing that
-  # answers where services live" with "a service", and it would put LAN resolution behind the
-  # Gateway Service's own lifecycle. Keep them separate.
-  #
-  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
-  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
-  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
-
-  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
-  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
-  # from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
-  # CIDR, not to Cilium's `world` entity the way the unifi/crafty game-port allows are. It is the
-  # same /24 the node IPs and ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} come out of.
-  #
-  # dev  -> 192.168.1.0/24  (Proxmox LAN; nodes .50/.51/.52)
-  # prod -> 192.168.2.0/24  (prod LAN; nodes .10/.11/.12)
-  LAN_CIDR: "192.168.1.0/24"                 # ingress scope for the PiHole DNS listener
-
-  # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
-  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
-  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
-  # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
-  # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
-  # declared in the ServiceDefinition, because the path is identical in every environment and only
-  # the ADDRESS is not.
-  #
-  # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
-  # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
-  #   dev  nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
-  #        NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
-  #        /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
-  #        per the convention of the LAN block above.
-  #   prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
-  #        22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
-  #        the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
-  #        DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
-  # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
-  # blast radius of shipping it unset, which is NOT the usual "fails to apply".
-  BULK_NFS_SERVER: "192.168.1.9"
 
   # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
   # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
   # nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
-  # Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR —
-  # and, measured 2026-09-05 from a pod INSIDE prod, it CANNOT reach it anyway (prod is on the SOHO
-  # network 192.168.2.0/24, which is pfSense's WAN, while this MinIO sits inside pfSense's LAN:
-  # `curl http://192.168.1.9:9000/minio/health/live` timed out after 15s). Prod uses Cloudflare R2.
+  # Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR.
   S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
 
-  # S3 region. Every S3 client REQUIRES one; MinIO ignores whatever it is; Cloudflare R2 demands
-  # exactly `auto` (its `head-bucket` reports the physical hint `WNAM`, which is NOT a usable region
-  # string). This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-
-  # accident for MinIO and wrong for anything else, which is why it is a knob now. The base default
-  # is dev's LIVE value, deliberately: dev's Velero has been signing requests with `us-east-1`
-  # against that MinIO since bootstrap, and this repo's rule is that base == the known-good dev
-  # value. Prod overrides to `auto`.
-  S3_BACKUP_REGION: "us-east-1"
-
   # --- storage (ADR-0003) ---
   LONGHORN_DEFAULT_REPLICA_COUNT: "3"       # default StorageClass replica count (3 on 3 nodes)
   # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
@@ -278,99 +205,6 @@ data:
   VELERO_BACKUP_SCHEDULE: "0 3 * * *"        # cron; overlays may differ
   VELERO_RETENTION: "168h"                   # dev short | prod long
 
-  # ==========================================================================================
-  # OBJECT-STORE TENANCY — the three knobs that keep two clusters and two backup tools from
-  # writing into each other's data. THESE ARE SAFETY KNOBS, not preferences.
-  # ==========================================================================================
-  # Nothing about the object layout was per-environment before this: Velero's bucket was the
-  # hardcoded literal `velero` with no prefix, and every CNPG Cluster used the hardcoded
-  # `s3://cnpg` with barman's `serverName` defaulting to the CLUSTER NAME — and BOTH environments
-  # have a keycloak-db, a nextcloud-db and a semaphore-db. Point two clusters at one bucket under
-  # that layout and:
-  #   (a) each cluster's Velero ADOPTS the other's backups and TTL-DELETES them on retention; and
-  #   (b) two different Postgres servers stream WAL into ONE Barman prefix, which does not merely
-  #       mix files — it CORRUPTS THE CHAIN and destroys PITR for both. That is the worse half:
-  #       dev's DR restore is PROVEN (R-STATE-03) and this would silently un-prove it.
-  #
-  # WHY THERE IS NO SINGLE `S3_BACKUP_BUCKET`. The two consumers take differently-SHAPED values:
-  # the Velero chart wants `bucket` and `prefix` as separate fields, while CNPG's
-  # barmanObjectStore.destinationPath is ONE full s3:// URL whose first path segment IS the bucket.
-  # CNPG's bucket therefore already lives inside CNPG_BACKUP_PATH; adding a second knob that had to
-  # hold the same bucket name a second time would be a rotation waiting to drift — the exact class
-  # of duplication this repo has been bitten by. Three knobs, each shaped for its own consumer, and
-  # no two of them can contradict each other.
-  #
-  # BASE = DEV'S LIVE LAYOUT, BYTE-FOR-BYTE, AND MUST STAY THAT WAY. Dev's existing backups are at
-  # `s3://velero/` (bucket root, no prefix) and `s3://cnpg/<clusterName>/` on that MinIO. Giving dev
-  # a prefix, or renaming its buckets, would ORPHAN the very backups its proven restore depends on —
-  # the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default` and its LONGHORN_DATA_PATH stays
-  # /var/lib/longhorn. Verified: with an EMPTY prefix the Velero chart omits `objectStorage.prefix`
-  # from the rendered BackupStorageLocation entirely (helm template, chart 12.1.0), so dev's live
-  # BSL is unchanged by this parameterization rather than merely equivalent to it.
-  #
-  # PROD, which has no history to preserve, gets ONE bucket with TWO disjoint top-level prefixes:
-  #
-  #     s3://k3s-prod-backup/velero/…                 Velero (kopia repo + backup metadata)
-  #     s3://k3s-prod-backup/barman/<clusterName>/…   CNPG WAL + base backups
-  #
-  # The operator created exactly one R2 bucket, so velero-vs-barman separation has to come from the
-  # prefix rather than the bucket. The BUCKET ROOT IS DELIBERATELY LEFT EMPTY: if a prefix ever
-  # renders wrong, its objects land somewhere obviously bogus instead of silently colliding with a
-  # real tenant's data. Dev↔prod isolation is doubly independent — different endpoint AND different
-  # bucket — so neither cluster can reach the other's objects even with a mis-substituted prefix.
-  VELERO_BACKUP_BUCKET: "velero"             # dev: the `velero` MinIO bucket | prod: k3s-prod-backup
-  VELERO_BACKUP_PREFIX: ""                   # dev: bucket root (live layout) | prod: "velero"
-  CNPG_BACKUP_PATH: "s3://cnpg"              # dev: the `cnpg` MinIO bucket root | prod: s3://k3s-prod-backup/barman
-
-  # ------------------------------------------------------------------------------------------------
-  # WHICH CREDENTIAL SECRET THE BACKUP CONSUMERS READ (added 2026-09-07 — the per-env credential split)
-  # ------------------------------------------------------------------------------------------------
-  # The three knobs above say WHERE the backups go. This one says WHICH KEY signs the requests, and
-  # it exists because the endpoint moved per-environment while the credential did not.
-  #
-  # THE BUG IT FIXES. The four backup-credential Secrets — velero-cloud.sops.yaml and the three
-  # <db>-backup-credentials.sops.yaml — are SHARED BASE manifests: platform/ and apps/ are rendered
-  # by the same Flux tiers on both clusters, and cluster-vars substitution is the ONLY thing that
-  # differs (R-ENV-02). So writing prod's R2 token into those four files did not GIVE prod a
-  # credential, it REPLACED DEV'S: dev's Velero and all three dev CNPG Clusters would have started
-  # signing MinIO requests with an R2 key id. For Velero that is a broken BSL; for CNPG it is worse,
-  # because `archive_mode=on` means Postgres REFUSES TO RECYCLE a WAL segment it could not archive,
-  # so a 403 loop is a monotonically filling pgdata PVC ending in a PANIC — the same failure prod
-  # spent its whole bring-up in with a placeholder endpoint. One shared Secret also means rotating
-  # either environment's key silently breaks the other; that is the structural reason for the split,
-  # independent of dev's eventual decommission.
-  #
-  # THE MECHANISM. Each credential is now TWO Secret objects, both applied to BOTH clusters (they are
-  # base resources; nothing here forks a manifest per environment). The consumer names one:
-  #
-  #     velero:   credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
-  #     CNPG:     s3Credentials.*.name:  <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
-  #
-  # ONE knob, not four, ON PURPOSE. Four independent secret-name knobs could be updated three-
-  # quarters of the way, leaving one consumer pointed at the OTHER environment's key with nothing
-  # rendering wrong — and StrictPostBuildSubstitutions cannot catch that, because it only fails on an
-  # UNDEFINED variable, never on a defined-but-stale one. With a single suffix the four move together
-  # or not at all. The cost is that the rendered Secret name is not greppable from the consumer; the
-  # names are spelled out in full in each kustomization.yaml beside the resources.
-  #
-  # THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names,
-  # which are the objects that already exist and that the live dev cluster is running. A default
-  # naming a Secret that does not exist would be worse than the shared-Secret state it replaces:
-  # Velero's pod would sit in ContainerCreating forever on a missing `existingSecret` volume, and a
-  # CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — i.e. the disk-full clock
-  # again — so an unresolvable default breaks the referencing cluster HARDER than a wrong key does.
-  # Empty is also the value dev must carry, so base and dev agree, exactly like VELERO_BACKUP_PREFIX.
-  #
-  # An EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions: strict mode fails on a variable that is
-  # not DEFINED, not on one defined as "". VELERO_BACKUP_PREFIX above has shipped empty since the
-  # gate was enabled and proves it. Both overlays state this knob explicitly — dev "" and prod
-  # "-prod" — so neither inherits it silently.
-  #
-  # ADDING A NEW BACKUP CREDENTIAL: create BOTH halves (`x-credentials.sops.yaml` and
-  # `x-credentials-prod.sops.yaml`), list both in the kustomization, and reference
-  # `x-credentials${BACKUP_CREDENTIALS_SUFFIX}`. Never a single shared Secret again.
-  BACKUP_CREDENTIALS_SUFFIX: ""              # dev: "" (MinIO creds) | prod: "-prod" (R2 creds)
-
   # --- placement ---
   PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os"   # base no-op selector; prod may pin roles
   PLATFORM_NODE_SELECTOR_VALUE: "linux"

clusters/base/homepage.yaml

@@ -34,12 +34,8 @@ spec:
     kind: GitRepository
     name: flux-system
   path: ./platform/homepage
-  # LOAD-BEARING, not parity decoration (the comment here used to say the opposite). This component
-  # references ${BASE_DOMAIN} in TWO places: httproute.yaml's apex hostname, and EVERY tile href in
-  # configmap.yaml. Without this substitution the token is applied LITERALLY — the apex route binds
-  # the host `${BASE_DOMAIN}`, which matches no Gateway listener, and every link on the operator's
-  # front door points at a host that does not exist — while Flux still reports this Kustomization
-  # Ready. It fails OPEN. Asserted by conformance/tests/l0-static/c2-root-surface (5).
+  # Homepage references no ${VARS} today; the substitution is retained for parity with the other
+  # tiers / future overlay knobs.
   postBuild:
     substituteFrom:
       - kind: ConfigMap

clusters/base/kustomization.yaml

@@ -50,13 +50,6 @@ resources:
   # reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same
   # non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform).
   - crafty.yaml          # Flux Kustomization -> ./platform/crafty (dependsOn platform-config)
-  # Alert runbooks — POST-platform tier (dependsOn platform-config, wait:false). Serves the 34
-  # runbooks the alert rules' `runbook_url` annotations point at, at <apex>/runbooks/<name>; before
-  # this tier existed that path was a 404 and every alert handed the operator a dead link. Same
-  # non-gating leaf rationale as headlamp/homepage/crafty. Carries NO postBuild substitution on
-  # purpose — it renders documentation containing `${...}` shell examples that envsubst would empty;
-  # see clusters/base/runbooks.yaml.
-  - runbooks.yaml        # Flux Kustomization -> ./runbooks (dependsOn platform-config)
   # Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup
   # controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
   # Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).

clusters/base/runbooks.yaml

@@ -1,57 +0,0 @@
----
-# Flux Kustomization: alert runbooks — reconciles ./runbooks.
-#
-# WHAT THIS SERVES. Every alert rule in this repo carries
-# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names.
-# Until this tier existed that path returned 404 on the live cluster, so every alert that reached a
-# human handed them a dead link at the moment they needed help. This Kustomization renders the
-# markdown under ./runbooks into a ConfigMap and a two-replica nginx that serves it at that exact
-# prefix. See runbooks/README.md and runbooks/serving/httproute.yaml.
-#
-# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false) — the same placement and
-# the same reasoning as homepage/headlamp/oauth2-proxy. It needs the wildcard TLS cert, the shared
-# Gateway (including the `https-apex` listener) and split-horizon DNS, all of which arrive with
-# `platform` + `platform-config`; and a leaf web server must never be able to stall the platform
-# chain, which is the whole lesson of DECISIONS-LOG's "Headlamp gating stall". wait:false guarantees
-# its health can never gate anything even if something is later mistakenly pointed dependsOn it.
-#
-# NO postBuild.substituteFrom — THE ONE DELIBERATE DIVERGENCE FROM EVERY OTHER TIER, AND THE MOST
-# IMPORTANT LINE IN THIS FILE.
-# This tier renders the runbook PROSE into a ConfigMap, and runbooks are full of shell: `$POD`,
-# `$(kubectl ...)`, and — unavoidably, since they document these very annotations — the literal text
-# `${BASE_DOMAIN}`. Flux's postBuild substitution runs envsubst over the ENTIRE built stream, and it
-# FAILS OPEN: an undefined `${X}` becomes the empty string while the Kustomization still reports
-# Ready (DECISIONS-LOG 2026-09-01 — that is how `k8sServiceHost` was emptied and Cilium broke on 2
-# of 3 dev nodes). Enabled here it would silently mutilate the documentation: every braced expansion
-# in every command example blanked, in a file nobody re-reads after writing. Worse, the L0 check
-# `conformance/tests/l0-static/flux-postbuild-substitution` scans only files reachable through
-# kustomization `resources:`, not `configMapGenerator` inputs, so the damage would be invisible to
-# the gate as well.
-# The fix is to have nothing to substitute. The only knob this tier would have wanted is the apex
-# hostname, and runbooks/serving/httproute.yaml avoids needing it by declaring NO `hostnames:` and
-# inheriting the `https-apex` listener's own hostname instead. So this tier is
-# environment-independent by construction and R-ENV-02 ("authored once, promoted dev -> prod") holds
-# without substitution rather than because of it.
-# IF A FUTURE KNOB EVER GENUINELY NEEDS SUBSTITUTION HERE: do not add substituteFrom to this
-# Kustomization. Split the serving manifests into their own substituted tier and leave the content
-# ConfigMap in an unsubstituted one, or the runbooks lose their shell examples.
-#
-# NO SOPS decryption stanza: this tier carries no encrypted Secret (markdown plus five plain
-# manifests), so — like homepage — neither clusters/dev nor clusters/prod patches it.
-apiVersion: kustomize.toolkit.fluxcd.io/v1
-kind: Kustomization
-metadata:
-  name: runbooks
-  namespace: flux-system
-spec:
-  interval: 10m
-  retryInterval: 2m
-  timeout: 5m
-  prune: true
-  wait: false
-  dependsOn:
-    - name: platform-config   # wildcard TLS + Gateway (incl. the https-apex listener) first
-  sourceRef:
-    kind: GitRepository
-    name: flux-system
-  path: ./runbooks

clusters/dev/kustomization.yaml

@@ -106,11 +106,8 @@ patches:
           secretRef:
             name: sops-age
   # velero (backup controller) has its own post-platform Flux Kustomization (clusters/base/backup.yaml).
-  # It carries BOTH halves of the per-env `cloud` INI credential — velero-cloud.sops.yaml (dev's scoped
-  # MinIO key) and velero-cloud-prod.sops.yaml (prod's R2 token) — so it needs its own SOPS decryption
-  # stanza (same as headlamp/oauth2-proxy/crafty). Which one the Deployment mounts is chosen by
-  # BACKUP_CREDENTIALS_SUFFIX in the cluster-vars patch below, not here. The `sops-age` Secret is
-  # out-of-band (name only).
+  # It carries velero-cloud.sops.yaml (the scoped MinIO `cloud` credential), so it needs its own SOPS
+  # decryption stanza (same as headlamp/oauth2-proxy/crafty). The `sops-age` Secret is out-of-band (name only).
   - patch: |-
       apiVersion: kustomize.toolkit.fluxcd.io/v1
       kind: Kustomization
@@ -225,67 +222,11 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.1.55"
-      # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
-      # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
-      # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
-      # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
-      # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
-      # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
-      - op: replace
-        path: /data/BULK_NFS_SERVER
-        value: "192.168.1.9"
       # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
       # barmanObjectStore and Velero's backupStorageLocation write to.
-      #
-      # DEV IS DELIBERATELY STAYING ON THIS MinIO while prod goes to Cloudflare R2 (2026-09-07).
-      # Prod could not use it at all — it cannot even route to it — but dev's move is a separate,
-      # operator-gated decision, because R2 would start dev on an EMPTY prefix: Velero would write
-      # a fresh kopia repository, CNPG would take new base backups, and the 30-day Barman window
-      # would rebuild from zero. Dev's DR restore is the one PROVEN recovery path in this repo
-      # (R-STATE-03); resetting its PITR window is not a side effect to take while fixing prod.
       - op: replace
         path: /data/S3_BACKUP_ENDPOINT
         value: "http://192.168.1.9:9000"
-      # DO NOT CHANGE THESE FOUR. They are dev's LIVE object layout, restated verbatim for the same
-      # reason as LONGHORN_DATA_PATH and EXTERNALDNS_TXT_OWNER_ID below/above: they equal the base
-      # defaults today, and a silent drift in base would ORPHAN dev's existing backups — the ones
-      # its proven restore depends on — without any manifest looking wrong.
-      #
-      # `us-east-1` is what dev's Velero has signed every request with since bootstrap; MinIO
-      # ignores the value, but changing it is a needless risk against a live, working backup path.
-      # An EMPTY VELERO_BACKUP_PREFIX means the bucket root, which is where dev's backups already
-      # are — and the Velero chart omits `objectStorage.prefix` entirely when it is empty (verified
-      # against chart 12.1.0), so dev's live BackupStorageLocation is byte-unchanged by this.
-      - op: replace
-        path: /data/S3_BACKUP_REGION
-        value: "us-east-1"
-      - op: replace
-        path: /data/VELERO_BACKUP_BUCKET
-        value: "velero"
-      - op: replace
-        path: /data/VELERO_BACKUP_PREFIX
-        value: ""
-      - op: replace
-        path: /data/CNPG_BACKUP_PATH
-        value: "s3://cnpg"
-      # WHICH CREDENTIAL SIGNS THOSE REQUESTS. Empty suffix = the UNSUFFIXED Secrets
-      # (velero-cloud-credentials, <db>-backup-credentials) — the scoped MinIO keys dev's Velero and
-      # its three CNPG Clusters are running RIGHT NOW, byte-identical to what is on the cluster. Prod
-      # renders "-prod" and reads the Cloudflare R2 token from the parallel `-prod` Secrets instead.
-      #
-      # RESTATED EXPLICITLY even though it equals the base default, for the same reason as
-      # EXTERNALDNS_TXT_OWNER_ID and the four backup-layout knobs above: the four Secrets are SHARED
-      # base manifests, so before this split prod's R2 token landed in the objects DEV reads. Dev
-      # signing MinIO with an R2 key id is 403 on every barman-cloud-wal-archive, and archive_mode=on
-      # turns that into a filling pgdata PVC — the disk-full clock, on the one cluster whose DR
-      # restore is PROVEN (R-STATE-03). This line is what keeps dev on its own credential.
-      #
-      # DEV IS NOT BEING BACKED UP GOING FORWARD and is slated for decommission (operator,
-      # 2026-09-07), so this is not protecting a future backup — it is protecting the EXISTING
-      # backups and the WAL chain until dev is retired. Do not "simplify" it away while dev runs.
-      - op: replace
-        path: /data/BACKUP_CREDENTIALS_SUFFIX
-        value: ""
       # dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT

clusters/prod/kustomization.yaml

@@ -185,56 +185,9 @@ patches:
         name: flux-system
         namespace: flux-system
       spec:
-        # CUTOVER DONE 2026-09-06 (operator direction: "move the repo to prod"). Prod now reconciles
-        # from PROD'S OWN in-cluster Gitea. The workstation bootstrap container
-        # (http://192.168.2.132:3000) is retired as prod's source — it was a podman container on the
-        # operator's workstation holding a DHCP lease on WIFI, i.e. production's source of truth was a
-        # laptop. It is kept only as a recovery path (bootstrap/gitea/bootstrap-gitea.sh).
-        #
-        # THIS LINE IS SELF-REFERENTIAL AND THAT IS THE WHOLE DIFFICULTY. The root Kustomization
-        # renders ./clusters/prod, which contains this patch — so a live `kubectl patch` of
-        # GitRepository.spec.url is REVERTED on the next reconcile unless the authored value moves
-        # too. The cutover therefore had to be: (1) push the repo to prod's Gitea, (2) fast-forward
-        # its `main` to the exact commit prod was already running so the source swap changed nothing
-        # else, (3) live-patch the url to open a window, (4) land THIS commit through that window so
-        # the authored value agrees. Dev hit the identical trap on 2026-09-05
-        # (infra/PROD-PROVISIONING.md §10a "D0/D1 as executed").
-        #
-        # Verified before the switch: a throwaway pod INSIDE prod resolved this host through
-        # in-cluster CoreDNS to the Gateway VIP and cloned over the public LE cert with no CA
-        # supplied. The workstation being able to reach it proves nothing about pod egress.
-        # MOVED WITH THE APEX FLIP (2026-09-13) — and it had to move in the SAME fetch.
-        # This URL is a hardcoded literal, NOT templated on ${BASE_DOMAIN}, so the flip did not
-        # carry it. But the flip DOES rename prod's own HTTPRoute hostnames, so:
-        #   - external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev, and
-        #   - platform/coredns/coredns-custom.yaml answers the whole ${BASE_DOMAIN} zone with
-        #     ${GATEWAY_VIP}, so in-cluster resolution moves from prod.home. to home.
-        # Leaving this on the old name would therefore have severed prod from its OWN GitOps
-        # source moments after the flip reconciled — self-reference, the same class of trap as
-        # patching this url live and having the root Kustomization revert it. The fix has to land
-        # THROUGH THE OLD SOURCE, which is why it is in the same push as the flip rather than
-        # after it.
-        # FOLLOW-UP worth doing: point this at the in-cluster Service
-        # (gitea-http.gitea.svc.cluster.local) so prod's GitOps stops depending on public DNS and
-        # on its own Gateway at all. Deferred: it needs the TLS/credential path rethought.
-        # IN-CLUSTER SERVICE, NOT A PUBLIC HOSTNAME (2026-09-13). This is the durable fix for a
-        # deadlock the apex flip walked straight into, and it is worth stating precisely.
-        #
-        # The url used to be a public name. The flip renamed prod's HTTPRoute hostnames, so the
-        # name this GitRepository pointed at stopped serving — but the HTTPRoute that WOULD have
-        # served the new name lives in a DOWNSTREAM tier, which cannot reconcile while the source
-        # is broken. Source down -> routes never rename -> source stays down. Observed live:
-        # `srcReady=False`, "unable to list remote for https://gitea.home.keithjackson.dev".
-        #
-        # Pointing at the Service breaks the cycle permanently: prod's GitOps no longer depends on
-        # public DNS, on Cloudflare, on its own Gateway, or on ${BASE_DOMAIN} at all. A future
-        # domain change cannot sever prod from its own source again.
-        #
-        # Plain http is deliberate and safe here: it is cluster-internal traffic to a ClusterIP,
-        # TLS terminates at the Gateway for external clients only, the GitRepository has no
-        # secretRef (the repo is cloned anonymously), and every secret in the tree is SOPS-encrypted
-        # at rest regardless of transport.
-        url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
+        # TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
+        # above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
+        url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
         ref:
           branch: main
   - target:
@@ -278,24 +231,7 @@ patches:
       # trusts a different Keycloak than the one prod actually runs.
       - op: replace
         path: /data/BASE_DOMAIN
-        # APEX AS OF 2026-09-13. Prod now serves home.keithjackson.dev directly; dev has been
-        # decommissioned and released the name. Prerequisites completed and VERIFIED before this
-        # flip, in this order:
-        #   1. dev's 18 A records released by its own external-dns (policy=sync, sources deleted)
-        #   2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand —
-        #      external-dns v0.21.0 removed only the `a-` prefixed form and then reported "All
-        #      records are already up to date", so it would NEVER have cleared them. They carried
-        #      `owner=default`, and prod runs `owner=prod`, so prod could not have claimed these
-        #      names while they existed. Confirmed gone at the AUTHORITY (betty.ns.cloudflare.com);
-        #      1.1.1.1 served stale cache for minutes afterwards and must not be trusted here.
-        #   3. dev's `*.home.keithjackson.dev` + apex Certificate deleted, so it stops renewing.
-        #      It was issued ~18d ago, so prod's request is #1 this week against LE's duplicate
-        #      limit of 5 per identical name set per week.
-        # NOTE the zone has wildcards `*.keithjackson.dev` AND `*.home.keithjackson.dev` pointing
-        # at the public IP 136.51.115.252 (legacy non-k3s prod). Dev's specific records were
-        # SHADOWING that wildcard; prod's new specific records shadow it again. The legacy stack
-        # lives on `*.keithjackson.dev` and is untouched by any of this.
-        value: "home.keithjackson.dev"
+        value: "prod.home.keithjackson.dev"
       # SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
       # OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
       # TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
@@ -349,16 +285,8 @@ patches:
       # Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
       # size cap before 30d if series volume is high, keeping the disk from filling either way.
       - op: replace
-      # LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for
-      # what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL,
-      # not the head's in-flight segments, and not compaction scratch, and a large compaction can need
-      # roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on
-      # /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every
-      # absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB
-      # at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history,
-      # grow the PVC rather than the fraction of it.
         path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
-        value: "36GiB"
+        value: "45GiB"
       - op: replace
         path: /data/MONITORING_LOKI_PVC_SIZE
         value: "50Gi"
@@ -448,163 +376,10 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.2.22"
-      # PiHole's DNS listener VIP — udp/53 + tcp/53 for LAN clients (apps/wave-0/pihole-dns.yaml).
-      # 192.168.2.23 is the NEXT FREE address in the prod pool 192.168.2.20-.30, verified live
-      # 2026-09-14: the CiliumLoadBalancerIPPool `default-pool` reported IPsTotal 11 / IPsUsed 3 /
-      # IPsAvailable 8, and the only three allocations are .20 (cilium-gateway-shared), .21
-      # (unifi-l4-tcp/-udp) and .22 (crafty-l4-tcp/-udp). ARP-probed from a LAN host in the same
-      # broadcast domain (arping 192.168.2.23, 3 probes, 0 responses), so nothing off-cluster holds
-      # it either, and it is inside the pool block the operator already asserted sits outside the
-      # SOHO router's DHCP scope.
-      - op: replace
-        path: /data/PIHOLE_DNS_VIP
-        value: "192.168.2.23"
-      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
-      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
-      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
-      - op: replace
-        path: /data/LAN_CIDR
-        value: "192.168.2.0/24"
-      # ------------------------------------------------------------------------------------------
-      # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
-      # ------------------------------------------------------------------------------------------
-      # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
-      # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
-      #
-      # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
-      # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
-      # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
-      # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
-      # outcomes give DIFFERENT values here:
-      #     port-forward 192.168.2.102:2049 -> 192.168.1.9:2049   =>  "192.168.2.102"
-      #     a WAN->LAN route to 192.168.1.0/24                    =>  "192.168.1.9"
-      # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
-      # ever became reachable — mounts something unintended. So it stays a placeholder.
-      #
-      # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
-      # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
-      # over v3 was in part this decision's cost: v3 would have needed four ports and three of them
-      # on ephemeral numbers.
-      #
-      # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
-      # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
-      # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
-      # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
-      # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
-      # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
-      #
-      # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
-      # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
-      # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
-      # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
-      # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
-      # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
-      # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
-      # a revision containing the bulk volumes; the dev overlay renders a working value, which is
-      # where these manifests are meant to be proven first.
-      - op: replace
-        path: /data/BULK_NFS_SERVER
-        # 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
-        # host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
-        # reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
-        # keeps the base value 192.168.1.9.
-        #
-        # PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
-        #   - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
-        #   - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
-        #     carries no_root_squash, so a root write would have passed and proved nothing, while
-        #     LSIO images default to PUID 911 and would have gone Ready then failed every import
-        #     with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
-        #   - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
-        #     real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
-        # pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
-        # so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
-        value: "192.168.2.102"
-      # ==========================================================================================
-      # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
-      # ==========================================================================================
-      # Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the network and is NOT an
-      # option for prod, for two independent reasons:
-      #   1. UNREACHABLE. Prod is on the SOHO network 192.168.2.0/24 — pfSense's WAN — while that
-      #      MinIO is inside pfSense's LAN. Measured from a pod INSIDE prod (the workstation proves
-      #      nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` → connection
-      #      timed out after 15s. Reaching it would need a WAN→LAN rule that does not exist.
-      #   2. NOT OFF-CLUSTER DR. It runs on the same Proxmox host as dev's VMs, so it already shares
-      #      a physical failure domain with dev. R2 is genuinely off-site, and its free egress means
-      #      restore REHEARSALS cost nothing — R-STATE-03 has no financial disincentive to stay proven.
-      #
-      # THIS UNBLOCKS AN ACTIVE INCIDENT, not just a roadmap item. All three prod CNPG Clusters ship
-      # `archive_mode=on` with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, so barman-cloud-wal-archive
-      # exits 4 on every segment and Postgres — correctly — REFUSES TO RECYCLE WAL. Verified live
-      # 2026-09-07: keycloak-db / nextcloud-db / semaphore-db all report
-      # ContinuousArchiving=False/ContinuousArchivingFailing, and keycloak-db's 5 GiB PVC is at
-      # 579 MiB used / 4360 MiB free and climbing monotonically. A disk-full PANIC there takes the
-      # cluster's SSO database down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and
-      # forward-auth. Fixing the endpoint lets Postgres drain the whole backlog.
-      #
-      # ENDPOINT IS ACCOUNT-SCOPED, so it stays a prod-overlay value even though it is not a LAN
-      # address: this is the operator's Cloudflare account id, confirmed from inside a prod pod
-      # (`head-bucket` on k3s-prod-backup → BucketRegion WNAM; PUT and DELETE both succeed).
-      # It is NOT put in base, because base is dev's live MinIO and dev is deliberately STAYING
-      # there — moving dev to R2 would reset its PITR window and orphan the backups its PROVEN
-      # restore depends on. Two environments, two targets, isolated by URL *and* by bucket.
+      # Off-cluster DR target. Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the
+      # network — a prod backup written there is not off-cluster DR and would mix two environments'
+      # PITR streams in one bucket. Operator supplies the prod endpoint (full URL incl. scheme+port).
       - op: replace
         path: /data/S3_BACKUP_ENDPOINT
-        value: "https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com"
-      # R2 requires the region string to be exactly `auto`. (`head-bucket` answers
-      # `{"BucketRegion":"WNAM"}` — that is a PHYSICAL LOCATION HINT, not a signing region; using it
-      # here produces AuthorizationHeaderMalformed.) Dev keeps the base `us-east-1` its live MinIO
-      # has always been signed with.
-      - op: replace
-        path: /data/S3_BACKUP_REGION
-        value: "auto"
-      # ------------------------------------------------------------------------------------------
-      # OBJECT TENANCY INSIDE THE ONE BUCKET. The operator created a SINGLE R2 bucket, so velero and
-      # barman cannot be separated by bucket the way they are on dev's MinIO (`velero` and `cnpg`).
-      # They are separated by disjoint TOP-LEVEL PREFIXES instead:
-      #
-      #     s3://k3s-prod-backup/velero/…                 Velero: kopia repo + backup metadata
-      #     s3://k3s-prod-backup/barman/<clusterName>/…   CNPG: WAL + base backups
-      #
-      # The bucket ROOT is left empty on purpose — a misrendered prefix then lands somewhere
-      # obviously wrong instead of silently colliding with a live tenant. Velero's retention TTL
-      # deletes objects under its own prefix only, so it can never reap a WAL segment; and barman's
-      # per-cluster serverName subdirectory keeps the three Postgres servers' chains apart, which
-      # matters because interleaving two servers' WAL into one prefix corrupts the chain outright.
-      # Dev cannot be reached from any of these: different endpoint AND different bucket names.
-      # ------------------------------------------------------------------------------------------
-      - op: replace
-        path: /data/VELERO_BACKUP_BUCKET
-        value: "k3s-prod-backup"
-      - op: replace
-        path: /data/VELERO_BACKUP_PREFIX
-        value: "velero"
-      - op: replace
-        path: /data/CNPG_BACKUP_PATH
-        value: "s3://k3s-prod-backup/barman"
-      # ------------------------------------------------------------------------------------------
-      # WHICH CREDENTIAL SIGNS THE REQUESTS TO THAT BUCKET (per-env credential split, 2026-09-07).
-      # ------------------------------------------------------------------------------------------
-      # The knobs above moved the ENDPOINT and the OBJECT LAYOUT per environment. The credential did
-      # not move with them, and it could not: the four backup-credential Secrets are SHARED BASE
-      # manifests, applied to both clusters by the same Flux tiers. Writing the R2 token into them
-      # gave prod a working key by TAKING DEV'S AWAY — dev would have signed MinIO requests with an
-      # R2 key id, and under archive_mode=on that is a filling pgdata PVC, not a missing backup.
-      #
-      # So each credential is now two objects, and this suffix selects prod's:
-      #     velero-cloud-credentials-prod           the R2 token in the AWS INI `cloud` shape
-      #     <db>-backup-credentials-prod            the same token as ACCESS_KEY_ID / ACCESS_SECRET_KEY
-      # for keycloak-db, nextcloud-db and semaphore-db. Both halves ship to both clusters (they are
-      # base resources); ONLY this value decides which one each consumer reads. It is a SUFFIX rather
-      # than four secret-name knobs deliberately — four could be updated three-quarters of the way,
-      # leaving one consumer on dev's MinIO key with nothing rendering wrong, and strict-mode
-      # substitution cannot catch a defined-but-stale variable.
-      #
-      # THE R2 TOKEN IS THE ONE PROVEN FROM INSIDE PROD (chore/r2-credentials 0a6a862): head-bucket on
-      # k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Rotating it now
-      # touches ONLY the four `-prod` files; dev's MinIO keys are untouched objects. That isolation is
-      # the structural reason for the split, independent of dev's eventual decommission.
-      - op: replace
-        path: /data/BACKUP_CREDENTIALS_SUFFIX
-        value: "-prod"
+        value: "TBD-PROD-S3-BACKUP-ENDPOINT"
       # TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.

conformance/README.md

@@ -23,9 +23,6 @@ behavioral / L3 fault-injection** (this harness).
 ./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
 ./run.sh --require-all                # phase-exit gate: every requirement must be covered
 ./run.sh --record                     # run AND capture the evidence (see "Proof of pass")
-./run.sh --layer l0 --include-scope whole-repo   # union: the tagged subset PLUS every test
-                                      #   declaring `conformance.home.k3s/scope: whole-repo`
-./run.sh --layer l0 --print-selection # print the dirs the selection resolves to; run nothing
 ```
 
 **Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
@@ -35,29 +32,6 @@ behavioral / L3 fault-injection** (this harness).
 - `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
   those touched requirements are covered (a listed safety-critical one must have an L3 test).
 - `--require-all` — **phase-exit gate**: every registry requirement must be covered.
-- `--include-scope whole-repo` — **the escape hatch scoping needs**, because scoping has one
-  blind spot and it is the worst-shaped kind. Some L0 invariants quantify over the WHOLE
-  tracked tree (no plaintext secrets anywhere; no RBAC wildcard anywhere), or over subtrees
-  the coarse path→requirement map keys to **nothing** (`apps/`, `infra/`, most of
-  `platform/`). For those, *the change most likely to break the invariant attributes no
-  requirement at all*, so requirement-scoped selection filters the check out of exactly that
-  change. Such a test **declares** its scope:
-
-  ```yaml
-  metadata:
-    annotations:
-      conformance.home.k3s/scope: "whole-repo"
-  ```
-
-  and `--include-scope whole-repo` unions every declaring test into the selection. The CI
-  stages (`ci/l0.sh`, `ci/l1.sh`, `ci/conformance.sh`) all pass it, so **adding the
-  annotation is the only step** — there is nothing to remember and nothing to hand-write.
-  Two consequences, both deliberate: a declaring test always runs, and a **SKIP of a
-  declaring test is UNWIRED, not green** (the runner exits 3, which `ci/l0.sh` turns into the
-  gate's own rc 3 — never a pass). `tests/l0-static/whole-repo-invariants-wired` proves the
-  mechanism end-to-end and falsifiably, so it cannot rot silently. This replaced a
-  hand-written "UNCONDITIONAL" stage per check in `ci/l0.sh`; see `lib/common.sh` for the
-  full rationale and the incidents behind it.
 - Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01`
   state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map)
   are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as

conformance/incluster/rbac.yaml

@@ -57,13 +57,6 @@ rules:
     verbs: [create, delete, get, list, watch, patch]
   - apiGroups: [""]
     resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
-    # LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment-
-    # agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable,
-    # oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run
-    # time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via
-    # chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings.
-    # If this rule is ever narrowed to the test's own namespace, those bindings fail and the
-    # tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap.
     verbs: [create, delete, get, list, watch, patch]
   - apiGroups: [""]
     resources: [pods/exec]

conformance/lib/common.sh

@@ -47,62 +47,6 @@ conf_dirs_for_requirements() {
   done < <(conf_all_test_files) | sort -u
 }
 
-# --- DECLARED SCOPE: whole-repo invariants -------------------------------------------
-# A test may also declare the SCOPE of the invariant it asserts:
-#
-#     metadata:
-#       annotations:
-#         conformance.home.k3s/scope: "whole-repo"
-#
-# WHY THIS EXISTS — the requirement-scoped-selection blind spot (R-TEST-03)
-# ------------------------------------------------------------------------
-# The merge gate runs the conformance subset tagged with the requirements a change
-# TOUCHES (ci/touched-requirements.sh -> `--requirements`). That is right for an invariant
-# whose input surface is the subtree its requirements map to. It is WRONG — silently, and in
-# the worst direction — for an invariant that quantifies over the WHOLE tracked tree, or over
-# subtrees (apps/, platform/<most>, infra/) that the coarse path->requirement map keys to
-# NOTHING. For those, the change most likely to break the invariant attributes no requirement
-# at all, so the requirement-scoped selection filters the check out of EXACTLY that change.
-# Three real incidents came out of that blind spot; each is recorded in the header of the
-# test that now guards it (flux-postbuild-substitution, keycloak-client-secret-reconciler,
-# bulk-tier-distinct-from-replicated).
-#
-# The repo's previous answer was a hand-written "UNCONDITIONAL" stage per such test in
-# ci/l0.sh — which does not scale and fails silently in the worst way: whoever adds the NEXT
-# whole-repo invariant must REMEMBER to hand-write a stage, and if they forget, the check runs
-# only when some unrelated requirement happens to be touched. The scope annotation replaces
-# that with a DECLARATION the runner enforces: `run.sh --include-scope whole-repo` unions
-# every test carrying it into the selection, so adding the annotation is the only step.
-#
-# TWO things follow from the annotation, and both are deliberate:
-#   1. the test is ALWAYS selected (union with the requirement-scoped subset), and
-#   2. a SKIP of that test is UNWIRED, NOT GREEN (lib/run_l1_offline.py --unwired-scope).
-#      A whole-repo invariant that reports "nothing to audit" is reporting that the subtree
-#      it audits has VANISHED, which is not a pass. This is the rc-3 semantics the bespoke
-#      stages encoded ("check SKIPPED (clusters/ absent) — not a pass"), preserved verbatim.
-export CONF_SCOPE_ANNOTATION="${CONF_LABEL_PREFIX}/scope"
-
-# Directories of tests DECLARING the given scope (e.g. "whole-repo"). The annotation is
-# matched on a real YAML mapping line only: a `#` comment cannot start with the key, so a
-# scope named in a test's header prose is documentation, not a declaration (the same
-# "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs).
-conf_dirs_for_scope() {
-  local scope="$1" f
-  [ -n "$scope" ] || return 0
-  while IFS= read -r f; do
-    [ -n "$f" ] || continue
-    if grep -Eq "^[[:space:]]*${CONF_SCOPE_ANNOTATION}:[[:space:]]*[\"']?${scope}[\"']?[[:space:]]*$" "$f"; then
-      dirname "$f"
-    fi
-  done < <(conf_all_test_files) | sort -u
-}
-
-# Union of conf_dirs_for_scope over a comma/space separated list of scopes.
-conf_dirs_for_scopes() {
-  local scopes="$1" s
-  for s in ${scopes//,/ }; do conf_dirs_for_scope "$s"; done | sort -u
-}
-
 # Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
 # Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
 # their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);

conformance/lib/record_run.py

@@ -467,14 +467,7 @@ def git_sha(repo: Path) -> str:
                              capture_output=True, text=True, check=True).stdout.strip()
     except (subprocess.CalledProcessError, OSError):
         return "unknown"
-    # EXCLUDE conformance/runs/. It is TRACKED, so `--record` writing ledger.jsonl dirtied the
-    # tree and every LATER run in the same session was stamped `-dirty` — recording one test
-    # poisoned the next, and check.sh refuses to honour a dirty proven row. That cost a second
-    # live fault-injection run on prod to work around. The docstring above already states the
-    # intent for the untracked runs/<id>/ half of this same artifact; the ledger is the tracked
-    # half. Mirrors status.py's IMPL_EXCLUDE_PREFIXES, which lists this path for the same reason.
-    dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--",
-                            ".", ":(exclude)conformance/runs/"],
+    dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--"],
                            capture_output=True).returncode != 0
     return f"{sha}-dirty" if dirty else sha
 
@@ -489,36 +482,6 @@ def pinned_chainsaw_version(conf_root: Path) -> str:
     return "unknown"
 
 
-def context_env() -> str:
-    """The env a run TARGETED, derived from the live kube context — never guessed.
-
-    WHY THIS IS NOT A DEFAULT STRING. `--env` used to fall back to the literal "dev"
-    (`os.environ.get("CONF_ENV", "dev")`), and CONF_ENV is set nowhere in this repo, so EVERY
-    recorded proof was stamped `env: dev` regardless of the cluster it ran against. That was
-    harmless while dev was the only cluster. Dev was decommissioned 2026-09-12 and prod is now
-    the only one, so on 2026-09-13 a genuine prod DR-restore proof was recorded as `env: dev` —
-    a proof that lies about where it was proven is worse than no proof, because the ledger's
-    whole purpose is to say what was demonstrated WHERE. Caught before it was committed.
-
-    Derived from `kubectl config current-context` (`k3s-prod` -> `prod`), which cannot drift
-    from the cluster the run actually talked to. Undeterminable is "unknown", NEVER a cluster
-    name: an unreadable context must not silently attribute a proof to an environment.
-    """
-    try:
-        p = subprocess.run(["kubectl", "config", "current-context"],
-                           capture_output=True, text=True, timeout=15)
-    except (OSError, subprocess.TimeoutExpired):
-        return "unknown"
-    if p.returncode != 0:
-        return "unknown"
-    ctx = (p.stdout or "").strip()
-    if not ctx:
-        return "unknown"
-    # contexts in this repo are named k3s-<env>; anything else is recorded verbatim so an
-    # unexpected context is visible in the ledger rather than being coerced into a known name.
-    return ctx[len("k3s-"):] if ctx.startswith("k3s-") else ctx
-
-
 def server_kube_version() -> str:
     """Server gitVersion via `kubectl version` (read-only), else "unknown"."""
     for args in (["kubectl", "version", "-o", "json"],
@@ -913,9 +876,7 @@ def main() -> int:
                     help=f"append-only ledger (default {DEFAULT_LEDGER})")
     ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
     ap.add_argument("--repo", type=Path, default=REPO_ROOT)
-    ap.add_argument("--env", default=os.environ.get("CONF_ENV") or context_env(),
-                    help="env the run targeted; derived from the kube context when "
-                         "unset (never defaulted to a cluster name)")
+    ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
     ap.add_argument("--chainsaw-version", default=None,
                     help="override the versions.env pin (recorded verbatim)")
     ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),

conformance/lib/run_l1_offline.py

@@ -19,30 +19,6 @@ Exit 0 iff every selected offline check passes.
 
 The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
 carries; a dir with neither is reported SKIPPED (never a pass).
-
-DECLARED SCOPE, AND WHY A SKIP IS NOT ALWAYS HARMLESS (--unwired-scope)
-----------------------------------------------------------------------
-Exit 0 here means "every selected check passed"; a check that exits 3 is SKIPPED and, by
-default, does not fail the run (a scaffold with no check script, an audit whose subject
-subtree does not exist yet). That default is WRONG for a WHOLE-REPO invariant. Those are the
-checks ci/l0.sh used to run as hand-written "UNCONDITIONAL" stages, and each of those stages
-mapped rc 3 to the GATE's rc 3 with the words "not a pass":
-
-    3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
-       summary "- subst-safety: **SKIPPED** (nothing to audit)"
-       if [ "$rc" -eq 0 ]; then rc=3; fi
-
-A whole-repo check reporting "nothing to audit" is reporting that the tree it audits has
-VANISHED — which is a louder problem than the check failing, not a quieter one. So:
-
-  --unwired-scope SCOPE   a test DECLARING `<prefix>/scope: SCOPE` whose check SKIPS (rc 3,
-                          or no check script at all) makes this runner exit 3 instead of 0.
-                          Hard failures still win (exit 1); rc 3 never masks one.
-
-Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing failed but a
-scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it
-into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now,
-instead of once per hand-written stage.
 """
 from __future__ import annotations
 
@@ -60,24 +36,6 @@ LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
 CHECK_SCRIPTS = ("render-check.sh", "check.sh")
 
 
-def scope_of(test_yaml: Path, annotation: str) -> str:
-    """The scope a test DECLARES, or "" when it declares none.
-
-    Matched on a real YAML mapping line: a `#` comment cannot begin with the annotation key,
-    so a scope named in the test's header prose is documentation, not a declaration — the
-    same "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs.
-    Mirrors conf_dirs_for_scope() in lib/common.sh; they must agree.
-    """
-    if not test_yaml.exists():
-        return ""
-    rx = re.compile(r"^\s*" + re.escape(annotation) + r"\s*:\s*[\"']?([A-Za-z0-9_.-]+)[\"']?\s*$")
-    for line in test_yaml.read_text().splitlines():
-        m = rx.match(line)
-        if m:
-            return m.group(1)
-    return ""
-
-
 def layer_of(d: Path) -> str:
     """Layer for the JUnit properties, from the subtree the test lives in."""
     rel = str(d)
@@ -101,38 +59,25 @@ def main() -> int:
     ap = argparse.ArgumentParser(description=__doc__)
     ap.add_argument("--junit", type=Path, required=True)
     ap.add_argument("--label-prefix", default="conformance.home.k3s")
-    ap.add_argument("--scope-annotation", default="conformance.home.k3s/scope",
-                    help="annotation key a test uses to DECLARE its invariant's scope")
-    ap.add_argument("--unwired-scope", action="append", default=[], metavar="SCOPE",
-                    help="a test declaring this scope that SKIPS is UNWIRED, not green "
-                         "(makes this runner exit 3); repeatable")
-    ap.add_argument("--summary-out", type=Path, default=None,
-                    help="write one Markdown line per case here, for the CI step summary")
     ap.add_argument("dirs", nargs="+", type=Path)
     args = ap.parse_args()
 
-    strict_scopes = set(args.unwired_scope)
-
-    cases = []  # (name, reqs, dir, rc, output, scope)
+    cases = []  # (name, reqs, dir, rc, output)
     for d in args.dirs:
         check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
         name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
-        scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation)
         if check is None:
             cases.append((name, reqs, d, 3,
-                          f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
-                          scope))
+                          f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}"))
             continue
         proc = subprocess.run(
             ["bash", str(check)], capture_output=True, text=True
         )
         out = (proc.stdout + proc.stderr).strip()
-        cases.append((name, reqs, d, proc.returncode, out, scope))
+        cases.append((name, reqs, d, proc.returncode, out))
 
     failures = sum(1 for c in cases if c[3] not in (0, 3))
     skipped = sum(1 for c in cases if c[3] == 3)
-    # A SKIP of a scope-declared check is UNWIRED, not green (see the module docstring).
-    unwired = [c for c in cases if c[3] == 3 and c[5] in strict_scopes]
 
     suite = ET.Element(
         "testsuite",
@@ -144,8 +89,7 @@ def main() -> int:
             "skipped": str(skipped),
         },
     )
-    summary_lines = []
-    for name, reqs, d, rc, out, scope in cases:
+    for name, reqs, d, rc, out in cases:
         tc = ET.SubElement(
             suite,
             "testcase",
@@ -156,33 +100,17 @@ def main() -> int:
         ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
         ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)})
         ET.SubElement(props, "property", {"name": "path", "value": str(d)})
-        ET.SubElement(props, "property", {"name": "scope", "value": scope})
         if rc == 3:
             ET.SubElement(tc, "skipped", {"message": out or "no offline check script"})
         elif rc != 0:
             fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"})
             fail.text = out
         status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
-        mark = f" scope={scope}" if scope else ""
         print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
-              f"({','.join(reqs) or 'untagged'}){mark}",
+              f"({','.join(reqs) or 'untagged'})",
               file=sys.stderr)
         if rc not in (0, 3):
             print(out, file=sys.stderr)
-        # One step-summary line per case, naming what ran — the generated replacement for the
-        # hand-written `summary "- subst-safety: **passed**"` line each deleted stage carried.
-        tag = f" ({scope})" if scope else ""
-        if rc == 0:
-            verdict = "**passed**"
-        elif rc == 3:
-            verdict = ("**SKIPPED** (UNWIRED, not green)" if scope in strict_scopes
-                       else "**SKIPPED** (nothing to audit)")
-        else:
-            verdict = f"**FAILED** (exit {rc})"
-        summary_lines.append(f"- {layer_of(d).lower()}/{name}{tag}: {verdict}")
-        if rc == 3 and scope in strict_scopes:
-            print(f"[offline] {name}: SKIPPED, and it declares scope={scope!r} — "
-                  f"UNWIRED, NOT GREEN. {out}", file=sys.stderr)
 
     root = ET.Element(
         "testsuites",
@@ -198,19 +126,9 @@ def main() -> int:
     args.junit.parent.mkdir(parents=True, exist_ok=True)
     xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent="  ")
     args.junit.write_text(xml)
-    if args.summary_out is not None:
-        args.summary_out.parent.mkdir(parents=True, exist_ok=True)
-        args.summary_out.write_text("\n".join(summary_lines) + ("\n" if summary_lines else ""))
-
-    print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s), "
-          f"{skipped} skip(s), {len(unwired)} unwired) -> {args.junit}", file=sys.stderr)
-    if failures:
-        return 1
-    if unwired:
-        print("[offline] UNWIRED (declared scope, SKIPPED — not a pass): "
-              + ", ".join(c[0] for c in unwired), file=sys.stderr)
-        return 3
-    return 0
+    print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
+          f"{args.junit}", file=sys.stderr)
+    return 1 if failures else 0
 
 
 if __name__ == "__main__":

conformance/lib/status.py

@@ -297,109 +297,6 @@ _EXECUTABLE_RE = re.compile(
 _BLOCKED_RE = re.compile(
     r"^\s*" + re.escape(LABEL_PREFIX) + r"/blocked-by:\s*(.+)$", re.M)
 
-# ---------------------------------------------------------- scaffold-marker detection
-# WHY THIS IS NOT A WHOLE-FILE `"TODO" in text`
-# ---------------------------------------------------
-# It used to be, and that mis-reported a SAFETY-CRITICAL requirement. A test whose SUBJECT
-# is placeholder detection has to name the idioms it hunts for:
-# `l0-static/placeholder-guard-coupling/check.sh` carries
-# `SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", ...]` — its detection vocabulary, as data.
-# A whole-file substring test read that as the test declaring itself unfinished, classified
-# a 32-assertion executable check as a scaffold, raised a permanent `--strict` annotation
-# conflict, and (because the same file is tagged R-STATE-03) reported R-STATE-03 as
-# `exec=1 scaffold=1`. A previous engineer worked around it by rewording prose.
-#
-# The rule below is derived from the corpus (43 tests), not invented. Every genuine
-# scaffold in this tree — `l2-behavioral/onboarding-completeness`, `l3-fault-injection/
-# alerting`, `l3-fault-injection/gitops-drift`, and no others — declares itself in BOTH of
-# two structural positions, and no executable test does either:
-#
-#   1. a `# STATUS: ... scaffold` line in the file's documentation header. This is the
-#      repo's DEDICATED declaration slot (tests/README.md): 3 of 43 tests say
-#      `STATUS: scaffold` and they are exactly the three scaffolds; 19 others declare
-#      `STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`; the rest declare nothing.
-#   2. a TODO/FIXME marker in the BODY that CARRIES A MESSAGE — `echo "TODO: assert PVC
-#      bound ..."`, `TODO(T-4.1): hermetic SMTP capture sink`. A marker that carries a
-#      message is a note about work that is missing. A marker that is the ENTIRE content of
-#      a quoted literal (`"TODO",` in a vocabulary list) carries no message: it is the
-#      marker as a *datum*, which is what a placeholder DETECTOR necessarily contains.
-#
-# Both are still evaluated over the body of `chainsaw-test.yaml` AND of any
-# `render-check.sh` / `check.sh`, and either one still OUTRANKS an
-# `executable: "true"` annotation (see classify_test) — the invariant that an annotation
-# must never outrank the body is unchanged. Ambiguity still only ever DOWNGRADES: an
-# unrecognised marker shape, a marker in body code outside a quoted literal, or a marker
-# carrying any trailing text all classify as scaffold.
-#
-# KNOWN, ACCEPTED IMPRECISION (stated rather than hidden): a test that embeds a marker with
-# text attached but as data — e.g. `re.compile(r"TODO|FIXME")`, or a single string holding
-# several idioms — is still called a scaffold. That is the conservative direction, and the
-# author's remedy is the same structural discipline the corpus already uses: keep each
-# idiom a standalone literal in a list.
-
-# Vocabulary: only the idioms that DECLARE unfinished work. Deliberately excludes `TBD`,
-# `PLACEHOLDER`, `XXX`, `REPLACE-ME` — tests in this tree legitimately carry those in
-# executable code as data (`l0-static/egress-paths-selected-by-rules/check.sh` compares
-# authored values against `"TBD"`; `l0-static/no-plaintext-secrets/check.sh` matches
-# `placeholder.*`), and none of them is used here to mark a placeholder body.
-# Case-insensitive: no English word contains either token, and the corpus carries no
-# lower-case occurrence, so folding case only ever ADDS detection (`echo "todo: ..."`).
-_MARKER_RE = re.compile(r"\b(?:TODO|FIXME)\b", re.I)
-
-# The dedicated declaration slot, on a comment line, case-insensitive. The declared state
-# must be the FIRST token after `STATUS:` — a `STATUS:` line declares exactly one state,
-# and all three real scaffolds read `# STATUS: scaffold — <what is missing>`. Anchoring it
-# there is what keeps `# STATUS: EXECUTABLE ... this is not a scaffold` from matching; a
-# loose `.*scaffold` would turn any prose that merely uses the word into a declaration.
-_STATUS_SCAFFOLD_RE = re.compile(
-    r"^[ \t]*#[ \t]*STATUS:[ \t]*(?:scaffold|placeholder)\b", re.M | re.I)
-
-# A quoted literal whose entire content is one marker word: the marker as data.
-_MARKER_AS_DATUM_RE = re.compile(r"(['\"])[ \t]*(?:TODO|FIXME)[ \t]*\1", re.I)
-
-
-def _split_doc_header(text: str) -> tuple[str, str]:
-    """Split a test file into (documentation header, body).
-
-    Both file kinds in this tree open with a banner comment block that is PROSE about the
-    test: the Given/When/Then, the incident being closed, the mandatory "what this does NOT
-    prove" list (tests/README.md, L0 authoring rule 1). Prose necessarily discusses
-    placeholder idioms when the test hunts them, so an incidental mention there is
-    documentation, not a placeholder body.
-
-    The body starts at the first line that is neither blank nor a `#` comment:
-      *.sh    -> `set -euo pipefail` (the shebang is a comment)
-      *.yaml  -> `apiVersion: chainsaw.kyverno.io/v1alpha1`
-    Comments BELOW that point are body, deliberately: a scaffold marks a missing fixture
-    with an inline comment (`l3-fault-injection/alerting`: `file: fixtures/smtp-sink.yaml
-    # TODO(T-4.1): hermetic SMTP capture sink`), and that must keep counting.
-    """
-    lines = text.splitlines(keepends=True)
-    for i, ln in enumerate(lines):
-        s = ln.strip()
-        if s and not s.startswith("#"):
-            return "".join(lines[:i]), "".join(lines[i:])
-    return text, ""          # no executable line at all: it is all header, body is empty
-
-
-def scaffold_marker(text: str) -> str:
-    """Why `text` declares itself a placeholder body, or "" if it does not.
-
-    Returns a short human-readable reason (used verbatim in the hygiene report) so the
-    defect is actionable rather than a bare path.
-    """
-    head, body = _split_doc_header(text)
-    m = _STATUS_SCAFFOLD_RE.search(head) or _STATUS_SCAFFOLD_RE.search(body)
-    if m:
-        return m.group(0).strip().lstrip("#").strip()[:80]
-    # Blank out markers held as pure data before looking for a marker that has a message.
-    m2 = _MARKER_RE.search(_MARKER_AS_DATUM_RE.sub('""', body))
-    if m2:
-        line = body[:m2.start()].count("\n")
-        snippet = body.splitlines()[line].strip()
-        return f"placeholder marker in body: {snippet[:80]}"
-    return ""
-
 
 def classify_test(t: dict) -> dict:
     """Decide EXECUTABLE vs SCAFFOLD for one discovered test, conservatively.
@@ -407,18 +304,14 @@ def classify_test(t: dict) -> dict:
     A tagged test is only worth counting if it can actually assert something. Rules, in
     order, chosen so ambiguity NEVER upgrades a test:
 
-      scaffold             the test DECLARES itself a placeholder — a `# STATUS: scaffold`
-                           header line, or a TODO/FIXME marker carrying a message in the
-                           body (see `scaffold_marker` for the rule and the corpus it was
-                           derived from) — *even if* it is annotated executable. An
-                           annotation must never outrank the body. That mismatch is also
-                           reported as a hygiene defect.
-      executable           annotated `conformance.home.k3s/executable: "true"` and no
-                           scaffold declaration.
-      executable-untagged  no scaffold declaration and a real runnable body, but the
-                           annotation is missing (true of the L1 render tests, which
-                           execute a render-check.sh + golden diff, of the L0 static
-                           checks, and of the META self-check).
+      scaffold             the body contains TODO markers (a placeholder), *even if* it
+                           is annotated executable — an annotation must never outrank the
+                           body. That mismatch is also reported as a hygiene defect.
+      executable           annotated `conformance.home.k3s/executable: "true"` and no TODOs.
+      executable-untagged  no TODOs and a real runnable body, but the annotation is
+                           missing (true of the L1 render tests, which execute a
+                           render-check.sh + golden diff, of the L0 static checks, and
+                           of the META self-check).
                            Counted as executable, and flagged as an annotation gap.
       scaffold             anything else.
     """
@@ -429,26 +322,16 @@ def classify_test(t: dict) -> dict:
     # downgrades the test to a scaffold — an annotation must never outrank the body,
     # whichever file the body lives in.
     extra = ""
-    scripts: list[tuple[str, str]] = []
     for script in ("render-check.sh", "check.sh"):
         sp = p.parent / script
         if sp.is_file():
-            body = sp.read_text()
-            extra += body
-            scripts.append((script, body))
+            extra += sp.read_text()
     declared = bool(_EXECUTABLE_RE.search(text))
-    # Each file is classified on ITS OWN structure — the yaml and the script have different
-    # header blocks, so they must not be concatenated before the header/body split.
-    reason = scaffold_marker(text)
-    for script, body in scripts:
-        if reason:
-            break
-        r = scaffold_marker(body)
-        reason = f"{script}: {r}" if r else ""
+    has_todo = "TODO" in text or "TODO" in extra
     bm = _BLOCKED_RE.search(text)
     has_body = bool(extra) or bool(re.search(r"^\s+- (script|assert|apply):", text, re.M))
 
-    if reason:
+    if has_todo:
         state = "scaffold"
     elif declared:
         state = "executable"
@@ -465,8 +348,7 @@ def classify_test(t: dict) -> dict:
         "name": t["name"], "path": rel, "layer": t["layer"],
         "state": state,
         "declared_executable": declared,
-        "annotation_conflict": bool(declared and reason),
-        "scaffold_reason": reason,
+        "annotation_conflict": bool(declared and has_todo),
         "blocked_by": bm.group(1).strip().strip('"') if bm else "",
     }
 
@@ -758,10 +640,6 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
     # --- hygiene defects this tool owns -------------------------------------------
     hygiene = {
         "annotation_conflicts": [t["path"] for t in classified if t["annotation_conflict"]],
-        # Same set, with WHY each one was called a placeholder body, so the fix is obvious
-        # from the report instead of requiring the reader to re-derive the rule.
-        "annotation_conflict_reasons": {t["path"]: t["scaffold_reason"]
-                                       for t in classified if t["annotation_conflict"]},
         "executable_untagged": [t["path"] for t in classified
                                 if t["state"] == "executable-untagged"],
         "registry_not_in_spec": [],
@@ -1032,10 +910,8 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
     else:
         o.append("- registry ↔ spec parity: **NOT CHECKED** (spec tree not available; "
                  "this is a skip, not a pass)")
-    o.append("- tests annotated executable but declaring a placeholder body: "
-             + (", ".join(f"`{k}` ({v})"
-                          for k, v in sorted(h["annotation_conflict_reasons"].items()))
-                if h["annotation_conflicts"] else "none"))
+    o.append(f"- tests annotated executable but containing TODO bodies: "
+             f"{h['annotation_conflicts'] or 'none'}")
     o.append(f"- executable tests missing the `{LABEL_PREFIX}/executable` annotation: "
              f"{h['executable_untagged'] or 'none'}")
 
@@ -1054,12 +930,9 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
              "reconciled |")
     o.append("| verifying test | `conformance.home.k3s/R-*` label on a Chainsaw test "
              "(the authoritative tag, same source as `coverage.md`) | exact |")
-    o.append("| executable vs scaffold | a `# STATUS: scaffold` header line, or a "
-             "TODO/FIXME marker CARRYING A MESSAGE in the body (a marker held as a bare "
-             "quoted datum — a placeholder detector's own vocabulary — is not one); "
-             "either outranks the `executable` annotation, and a real body without the "
-             "annotation still counts and is flagged | conservative — ambiguity never "
-             "upgrades |")
+    o.append("| executable vs scaffold | TODO markers in the test body outrank the "
+             "`executable` annotation; a real body without the annotation still counts "
+             "and is flagged | conservative — ambiguity never upgrades |")
     o.append("| **test actually passed** | the committed `runs/ledger.jsonl` row for "
              "every verifying test: JUnit result **plus** that test's own pass sentinel "
              "token grepped from the run log | exact where a run was recorded, refused "
@@ -1254,70 +1127,6 @@ def _selftest() -> int:
               classify_test({"name": "s", "path": str(f), "layer": "L2"})["state"]
               == "scaffold")
 
-        # === THE BUG THIS DETECTOR WAS REWRITTEN FOR ==============================
-        # A whole-file `"TODO" in text` mis-classified `l0-static/placeholder-guard-
-        # coupling` — a 32-assertion executable check whose SUBJECT is placeholder
-        # detection, so it must name the idioms it hunts — as a scaffold, and through it
-        # reported the SAFETY-CRITICAL R-STATE-03 as `exec=1 scaffold=1`. These
-        # assertions pin BOTH directions: a marker held as DATA must not downgrade, and a
-        # real placeholder body must still be caught. Both halves are load-bearing; the
-        # second is what stops this fix from degrading into "always executable".
-        yml = ("metadata:\n  annotations:\n"
-               "    conformance.home.k3s/executable: \"true\"\n"
-               "spec:\n  steps:\n    - script:\n"
-               "        content: bash ./check.sh\n")
-        f.write_text(yml)
-        detector = td2 / "check.sh"
-        # Shaped exactly like the real check: a prose header that discusses the idioms,
-        # then executable code holding them as a vocabulary list.
-        detector.write_text(
-            "#!/usr/bin/env bash\n"
-            "# L0 static check — placeholder vocabulary\n"
-            "#   * SEED — loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER)\n"
-            "# STATUS: EXECUTABLE. A pure offline assertion script.\n"
-            "set -euo pipefail\n"
-            "python3 - <<'EOF'\n"
-            "SEED = [\"TBD\", \"TODO\", \"FIXME\", \"PLACEHOLDER\"]\n"
-            "assert any(w in open('/etc/hostname').read().upper() for w in SEED) is False\n"
-            "EOF\n")
-        c = classify_test({"name": "s", "path": str(f), "layer": "L0"})
-        check("a marker in HEADER PROSE does not downgrade an executable check",
-              c["state"] == "executable" and not c["annotation_conflict"])
-        check("a marker held as a bare quoted DATUM does not downgrade (the R-STATE-03 bug)",
-              c["scaffold_reason"] == "")
-
-        # ... and the protection is intact: a real placeholder body is still caught, in
-        # each of the three shapes the corpus actually uses.
-        detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
-                            "echo \"TODO: assert the PVC is bound\"\n")
-        c = classify_test({"name": "s", "path": str(f), "layer": "L0"})
-        check("a TODO carrying a message in check.sh IS still a scaffold",
-              c["state"] == "scaffold" and c["annotation_conflict"])
-        check("the scaffold reason names the file and the offending line",
-              c["scaffold_reason"].startswith("check.sh:")
-              and "TODO: assert the PVC is bound" in c["scaffold_reason"])
-        detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
-                            "# TODO(T-4.1) write the hermetic sink fixture\ntrue\n")
-        check("a TODO(T-x.y) roadmap marker in an inline body comment IS still a scaffold",
-              classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"]
-              == "scaffold")
-        detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
-                            "echo \"FIXME the extraction before trusting this\"\n")
-        check("an unpunctuated FIXME carrying trailing prose IS still a scaffold",
-              classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"]
-              == "scaffold")
-        # The dedicated declaration slot outranks a body that would otherwise look real —
-        # this is how all three genuine scaffolds in the tree declare themselves.
-        detector.unlink()
-        f.write_text("# L3 FAULT-INJECTION SCAFFOLD — x\n"
-                     "# STATUS: scaffold — needs T-1.4 (Flux bootstrapped).\n" + yml)
-        c = classify_test({"name": "s", "path": str(f), "layer": "L3"})
-        check("a `# STATUS: scaffold` header declaration IS a scaffold, body notwithstanding",
-              c["state"] == "scaffold" and c["annotation_conflict"])
-        check("`# STATUS: EXECUTABLE` is not read as a scaffold declaration",
-              scaffold_marker("# STATUS: EXECUTABLE. `check.sh` is a pure offline "
-                              "assertion script; covers no scaffold.\nset -e\n") == "")
-
     # ---------------------------------------------------------------- PROOF / ledger
     # These are the assertions that stand between this report and a fake green.
     check("empty ledger parses to no rows (pre-ledger behaviour is preserved)",
@@ -1532,11 +1341,8 @@ def main() -> int:
                              f"{h['registry_not_in_spec']} {h['spec_not_in_registry']}\n")
             rc = 1
         if h["annotation_conflicts"]:
-            sys.stderr.write(
-                "[status] STRICT: tests annotated executable that declare a placeholder "
-                "body: " + "; ".join(f"{k} ({v})" for k, v in
-                                     sorted(h["annotation_conflict_reasons"].items()))
-                + "\n")
+            sys.stderr.write(f"[status] STRICT: tests annotated executable with TODO "
+                             f"bodies: {h['annotation_conflicts']}\n")
             rc = 1
         if rep["safety_critical_without_executable_l3"]:
             sys.stderr.write(f"[status] STRICT: safety-critical requirement(s) whose L3 "

conformance/run.sh

@@ -10,12 +10,6 @@
 #   ./run.sh --requirements R-NET-01,R-NET-06   # merge-gate subset (tagged)
 #   ./run.sh --layer l2               # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
 #   ./run.sh --requirements R-NET-01,R-NET-02 --layer l2   # intersect: tagged AND in layer
-#   ./run.sh --layer l0 --requirements R-X --include-scope whole-repo
-#                                     # UNION: the tagged subset PLUS every test that
-#                                     #   DECLARES `conformance.home.k3s/scope: whole-repo`.
-#                                     #   A whole-repo invariant must run on EVERY gate
-#                                     #   invocation — see lib/common.sh's scope block for
-#                                     #   why, and why a SKIP of one is exit 3, not green.
 #   ./run.sh --coverage-only          # regenerate coverage report only (no cluster)
 #   ./run.sh --status                 # DERIVED STATUS report only (no cluster): per
 #                                     #   requirement, is it implemented, is it actually
@@ -24,11 +18,6 @@
 #                                     #   (runs/ledger.jsonl) actually proves it — see
 #                                     #   lib/status.py. No ledger row => not-derivable.
 #   ./run.sh --lint                   # structural-validate test defs (no cluster)
-#   ./run.sh --print-selection        # print the test dirs the CURRENT selection resolves
-#                                     #   to, one per line, and exit. Runs no check. This is
-#                                     #   how a test can PROVE the selection includes it
-#                                     #   (tests/l0-static/whole-repo-invariants-wired) without
-#                                     #   executing the suite and recursing into itself.
 #   ./run.sh --strict-coverage        # hygiene gate: fail on tag/annotation drift
 #   ./run.sh --requirements R-NET-01  # ALSO asserts those IDs are covered (merge gate)
 #   ./run.sh --require-all            # phase-exit: every requirement must be covered
@@ -90,7 +79,6 @@ ORIG_ARGS=("$@")
 
 LAYER="all"
 REQS=""
-INCLUDE_SCOPES=""   # --include-scope: scopes unioned into the selection (see lib/common.sh)
 MODE="run"          # run | coverage-only | status | lint
 STRICT_COVERAGE=0
 REQUIRE_ALL=0
@@ -134,11 +122,9 @@ while [ $# -gt 0 ]; do
   case "$1" in
     --layer)          LAYER="$2"; shift 2 ;;
     --requirements)   REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
-    --include-scope)  INCLUDE_SCOPES="${INCLUDE_SCOPES:+${INCLUDE_SCOPES},}$2"; shift 2 ;;
     --coverage-only)  MODE="coverage-only"; shift ;;
     --status)         MODE="status"; shift ;;
     --lint)           MODE="lint"; shift ;;
-    --print-selection) MODE="print-selection"; shift ;;
     --strict-coverage) STRICT_COVERAGE=1; shift ;;
     --require-all)    REQUIRE_ALL=1; shift ;;
     --no-cluster)     FORCE_NO_CLUSTER=1; shift ;;
@@ -201,7 +187,7 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
   # CONF_LEDGER exists so the harness's own tests can exercise this path without writing
   # to the real ledger; operators should never set it.
   "${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
-    --run-dir "${RUN_DIR}" ${CONF_ENV:+--env "$CONF_ENV"} \
+    --run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
     --ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
   log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
   log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
@@ -239,7 +225,7 @@ gen_status() {
     --registry "${CONF_ROOT}/requirements/registry.yaml" \
     --tests "${CONF_ROOT}/tests" \
     --out "${STATUS_OUT}" \
-    ${CONF_ENV:+--env "$CONF_ENV"} \
+    --env "${CONF_ENV:-dev}" \
     "$@"
 }
 
@@ -265,38 +251,11 @@ lint_tests() {
 }
 
 # --- Select which test directories Chainsaw should run --------------------------------
-# Requirement-scoped selection, then the DECLARED-SCOPE union on top:
-#
-#   (--requirements ∩ --layer)  ∪  (--include-scope ∩ --layer)
-#
-# The left half is R-TEST-03's touched subset. The right half is the fix for its blind
-# spot: an invariant that quantifies over the whole tracked tree cannot be selected BY the
-# requirements a change touches, because the change most likely to break it touches none of
-# them. See lib/common.sh's scope block for the incidents. The union is fail-closed by
-# construction — it can only ADD tests to a run, never remove one.
 select_dirs() {
-  {
-    requirement_scoped_dirs
-    # The scope union is still LAYER-bounded: `--layer l0` must not pull a whole-repo-scoped
-    # L2 test into an offline run. With no --layer (all), nothing is bounded away.
-    if [ -n "${INCLUDE_SCOPES}" ]; then
-      if [ "${LAYER}" != all ]; then
-        comm -12 \
-          <(conf_dirs_for_scopes "${INCLUDE_SCOPES}") \
-          <(conf_dirs_for_layer "${LAYER}")
-      else
-        conf_dirs_for_scopes "${INCLUDE_SCOPES}"
-      fi
-    fi
-  } | sort -u
-}
-
-# The pre-existing requirement/layer selection, unchanged:
-# --requirements AND --layer together => intersection (tests carrying the tag
-# AND living in that layer subtree). Either alone keeps its prior meaning, so
-# this is backward compatible (LAYER defaults to "all"). Both helpers emit
-# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
-requirement_scoped_dirs() {
+  # --requirements AND --layer together => intersection (tests carrying the tag
+  # AND living in that layer subtree). Either alone keeps its prior meaning, so
+  # this is backward compatible (LAYER defaults to "all"). Both helpers emit
+  # sorted, de-duplicated dirs, so `comm -12` yields the intersection.
   if [ -n "${REQS}" ] && [ "${LAYER}" != all ]; then
     comm -12 \
       <(conf_dirs_for_requirements "${REQS}") \
@@ -437,12 +396,6 @@ layers_are_offline_only() {
   return 0
 }
 
-# Markdown one-liner per executed offline check, for the CI step summary. Written next to
-# the JUnit; ci/l0.sh + ci/l1.sh splice it into $GITHUB_STEP_SUMMARY so the gate output still
-# NAMES every check that ran (that is what the deleted per-check gate stages did, one hard-coded
-# `summary` line each — now generated for all of them, whole-repo-scoped ones marked as such).
-: "${OFFLINE_SUMMARY_OUT:=${ARTIFACTS}/offline-summary.md}"
-
 run_offline_checks() {
   local -a dirs
   mapfile -t dirs < <(offline_selected_dirs)
@@ -450,30 +403,14 @@ run_offline_checks() {
     log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
     return 0
   fi
-  # Declared scopes are passed through so the runner knows WHICH cases must not be allowed to
-  # report a SKIP as green (rc 3). See lib/common.sh's scope block and run_l1_offline.py.
-  local -a scope_args=() s
-  for s in ${INCLUDE_SCOPES//,/ }; do scope_args+=(--unwired-scope "$s"); done
   log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
-  [ -n "${INCLUDE_SCOPES}" ] && log "  unconditional (declared scope ${INCLUDE_SCOPES}): $(conf_dirs_for_scopes "${INCLUDE_SCOPES}" | xargs -n1 basename 2>/dev/null | tr '\n' ' ')"
   "${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
-    --junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" \
-    --scope-annotation "${CONF_SCOPE_ANNOTATION}" \
-    --summary-out "${OFFLINE_SUMMARY_OUT}" \
-    ${scope_args[@]+"${scope_args[@]}"} "${dirs[@]}"
+    --junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
 }
 
 ###############################################################################
 # Main
 ###############################################################################
-# --print-selection is deliberately handled FIRST: it resolves the selection and exits,
-# writing nothing and running no check. A test that asserts "the gate's selection cannot
-# filter me out" needs to ask that question without executing the suite it is part of.
-if [ "${MODE}" = "print-selection" ]; then
-  select_dirs
-  exit 0
-fi
-
 log "Coverage report first (always)"
 gen_coverage
 
@@ -503,26 +440,17 @@ esac
 # L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
 # checks and treat their result as authoritative REGARDLESS of cluster availability. An
 # offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
-# rc 3 is the harness-wide SKIPPED/not-run code (ci/README.md): NOT a pass, NOT a fail. The
-# offline runner returns it when a test carrying a --unwired-scope scope SKIPPED — a whole-repo
-# invariant reporting "nothing to audit" means the subtree it audits is gone, which the gate
-# must see as UNWIRED. It is propagated verbatim so ci/l0.sh can set the gate's own rc=3,
-# exactly as the deleted per-check stages did.
 OFFLINE_RC=0
 run_offline_checks || OFFLINE_RC=$?
 if layers_are_offline_only; then
   # Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
-  case "${OFFLINE_RC}" in
-    0) log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md" ;;
-    3) log "Offline ${LAYER}: a scope-declared check SKIPPED — UNWIRED, not green (rc=3). JUnit: ${JUNIT_OUT}" ;;
-    *) log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}" ;;
-  esac
+  if [ "${OFFLINE_RC}" -eq 0 ]; then
+    log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
+  else
+    log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
+  fi
   exit "${OFFLINE_RC}"
 fi
-if [ "${OFFLINE_RC}" -eq 3 ]; then
-  log "Offline L0/L1: a scope-declared check SKIPPED — UNWIRED, not green (rc=3)."
-  exit 3
-fi
 if [ "${OFFLINE_RC}" -ne 0 ]; then
   log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
   exit "${OFFLINE_RC}"

conformance/runs/20260913T014321Z-19fc8bc/report.junit.xml

@@ -1,12 +0,0 @@
-<?xml version="1.0" ?>
-<testsuites name="home-k3s-conformance" tests="1" failures="0" errors="0" skipped="0">
-  <testsuite name="home-k3s-conformance L0/L1 (offline)" tests="1" failures="0" errors="0" skipped="0">
-    <testcase classname="L1:R-NET-05,R-NET-06,R-NET-07" name="egress-class-render">
-      <properties>
-        <property name="requirements" value="R-NET-05,R-NET-06,R-NET-07"/>
-        <property name="layer" value="L1"/>
-        <property name="path" value="/work/.claude/conf-clone/conformance/tests/l1-rendering/egress-class-render"/>
-      </properties>
-    </testcase>
-  </testsuite>
-</testsuites>

conformance/runs/20260913T014321Z-19fc8bc/run.log

@@ -1,7 +0,0 @@
-[run.sh] Coverage report first (always)
-[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
-[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
-[run.sh] Running 1 offline L0/L1 check(s) (no cluster; docs/14).
-[offline] L1  PASS egress-class-render (R-NET-05,R-NET-06,R-NET-07)
-[offline] wrote JUnit (1 case(s), 0 failure(s)) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml
-[run.sh] Done (offline l1). JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260913T014437Z-19fc8bc/report.junit.xml

@@ -1,4 +0,0 @@
-<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch" tests="1" failures="0" errors="0" id="0" time="">
-    <testcase name="vpn-killswitch" classname="" time="174.632779" />
-  </testsuite>
-</testsuites>

conformance/runs/20260913T014437Z-19fc8bc/run.log

@@ -1,237 +0,0 @@
-[run.sh] Coverage report first (always)
-[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
-[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
-[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-NET-06').
-[run.sh] LIGHT pass: 1 test(s), parallel=2 (pinned v0.2.12)
-Version: 0.2.12
-Loading config (/work/.claude/conf-clone/conformance/config/chainsaw-config.yaml)...
-- Using test file: chainsaw-test
-- TestDirs [/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch]
-- SkipDelete false
-- FailFast false
-- ReportFormat 'JUNIT-TEST'
-- ReportName 'chainsaw-report-light'
-- ReportPath '/work/.claude/conf-clone/conformance/_artifacts'
-- Namespace ''
-- FullName false
-- IncludeTestRegex ''
-- ExcludeTestRegex ''
-- ApplyTimeout 1m30s
-- AssertTimeout 3m0s
-- CleanupTimeout 2m0s
-- DeleteTimeout 2m0s
-- ErrorTimeout 1m0s
-- ExecTimeout 3m0s
-- DeletionPropagationPolicy Background
-- Parallel 2
-- Template true
-- NoCluster false
-- PauseOnFailure false
-Loading tests...
-- vpn-killswitch (/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch)
-Loading values...
-Running tests...
-=== RUN   chainsaw
-=== PAUSE chainsaw
-=== CONT  chainsaw
-=== RUN   chainsaw/vpn-killswitch
-=== PAUSE chainsaw/vpn-killswitch
-=== CONT  chainsaw/vpn-killswitch
-    l.go:52: | 01:44:38 | vpn-killswitch | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-credible-bird
-    l.go:52: | 01:44:38 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | TRY       | BEGIN |
-    l.go:52: | 01:44:38 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        kubectl label namespace "$NAMESPACE" \
-          pod-security.kubernetes.io/enforce=privileged \
-          pod-security.kubernetes.io/warn=privileged \
-          pod-security.kubernetes.io/audit=privileged \
-          --overwrite
-        echo "NS_PRIVILEGED_OK"
-    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | LOG   |
-        === STDOUT
-        namespace/chainsaw-credible-bird labeled
-        NS_PRIVILEGED_OK
-    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | DONE  |
-    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | RUN   | v1/Pod @ chainsaw-credible-bird/egress-internet
-    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CREATE    | OK    | v1/Pod @ chainsaw-credible-bird/egress-internet
-    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | DONE  | v1/Pod @ chainsaw-credible-bird/egress-internet
-    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | RUN   | v1/Pod @ chainsaw-credible-bird/egress-vpn
-    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CREATE    | OK    | v1/Pod @ chainsaw-credible-bird/egress-vpn
-    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | DONE  | v1/Pod @ chainsaw-credible-bird/egress-vpn
-    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | RUN   | v1/Pod @ chainsaw-credible-bird/egress-internet
-    l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | DONE  | v1/Pod @ chainsaw-credible-bird/egress-internet
-    l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | RUN   | v1/Pod @ chainsaw-credible-bird/egress-vpn
-    l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | DONE  | v1/Pod @ chainsaw-credible-bird/egress-vpn
-    l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        ns="$NAMESPACE"
-        get_ip() {
-          kubectl -n "$ns" exec "$1" -c curl -- \
-            curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
-        }
-        wan="" vpn=""
-        for i in $(seq 1 12); do
-          [ -n "$wan" ] || wan="$(get_ip egress-internet)"
-          [ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
-          if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
-          [ "$wan" = "$vpn" ] && vpn=""   # not steered yet; re-poll
-          sleep 5
-        done
-        echo "wan=$wan vpn=$vpn"
-        [ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
-        [ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
-        [ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
-        echo "TUNNEL_HEALTHY_OK"
-    l.go:52: | 01:44:43 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | LOG   |
-        === STDOUT
-        wan=136.51.115.252 vpn=158.173.241.11
-        TUNNEL_HEALTHY_OK
-    l.go:52: | 01:44:43 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | DONE  |
-    l.go:52: | 01:44:43 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | TRY       | END   |
-    l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | TRY       | BEGIN |
-    l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | APPLY     | RUN   | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
-    l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CREATE    | OK    | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
-    l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | APPLY     | DONE  | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
-    l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | ASSERT    | RUN   | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
-    l.go:52: | 01:44:48 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | ASSERT    | DONE  | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
-    l.go:52: | 01:44:48 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -u
-        ns="$NAMESPACE"
-        wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
-                 curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
-        for i in $(seq 1 8); do
-          out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
-                   curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
-          if [ -n "$out" ]; then
-            if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
-              echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
-            fi
-            echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
-          fi
-          echo "attempt $i: vpn-class blackholed (no response), good"
-          sleep 3
-        done
-        echo "NO_LEAK_OK"
-    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | LOG   |
-        === STDOUT
-        attempt 1: vpn-class blackholed (no response), good
-        attempt 2: vpn-class blackholed (no response), good
-        attempt 3: vpn-class blackholed (no response), good
-        attempt 4: vpn-class blackholed (no response), good
-        attempt 5: vpn-class blackholed (no response), good
-        attempt 6: vpn-class blackholed (no response), good
-        attempt 7: vpn-class blackholed (no response), good
-        attempt 8: vpn-class blackholed (no response), good
-        NO_LEAK_OK
-    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | DONE  |
-    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        ns="$NAMESPACE"
-        out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
-                 curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
-        echo "internet_ip=$out"
-        [ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
-        echo "INTERNET_UNAFFECTED_OK"
-    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | LOG   |
-        === STDOUT
-        internet_ip=136.51.115.252
-        INTERNET_UNAFFECTED_OK
-    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | DONE  |
-    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | TRY       | END   |
-    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | FINALLY   | BEGIN |
-    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        ns="$NAMESPACE"
-        # Distinct Job name (tunnel-restart) so it never collides with the
-        # tunnel-stop Job. Left-aligned manifest so the block scalar yields
-        # valid YAML at column 0 at runtime.
-        cat <<'JOB' | kubectl -n "$ns" apply -f -
-        apiVersion: batch/v1
-        kind: Job
-        metadata:
-          name: tunnel-restart
-          labels: { app: vpn-killswitch-fault }
-        spec:
-          backoffLimit: 3
-          activeDeadlineSeconds: 90
-          ttlSecondsAfterFinished: 600
-          template:
-            metadata:
-              labels: { app: vpn-killswitch-fault }
-            spec:
-              restartPolicy: Never
-              hostPID: true
-              nodeSelector:
-                homelab.keithjackson.dev/egress-gateway: "true"
-              tolerations:
-                - operator: Exists
-              containers:
-                - name: nsenter
-                  image: busybox:1.37.0
-                  securityContext:
-                    privileged: true
-                  command: ["nsenter","-t","1","-m","-u","-i","-n","-p","--","systemctl","start","vpn-egress-tunnel"]
-        JOB
-        # Block teardown until the restart actually completed on the host — a
-        # leaked-down tunnel is unacceptable. `wait` exits non-zero on failure
-        # or timeout, which (set -e) fails the step and surfaces the problem.
-        kubectl -n "$ns" wait --for=condition=complete --timeout=110s job/tunnel-restart
-        echo "TUNNEL_SELF_HEAL_OK"
-    l.go:52: | 01:46:21 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | LOG   |
-        === STDOUT
-        job.batch/tunnel-restart created
-        job.batch/tunnel-restart condition met
-        TUNNEL_SELF_HEAL_OK
-    l.go:52: | 01:46:21 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | DONE  |
-    l.go:52: | 01:46:21 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | FINALLY   | END   |
-    l.go:52: | 01:46:21 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY       | BEGIN |
-    l.go:52: | 01:46:21 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        ns="$NAMESPACE"
-        wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
-                 curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
-        vpn=""
-        for i in $(seq 1 24); do
-          vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
-                   curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
-          if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
-          vpn=""
-          sleep 5
-        done
-        echo "recovered_vpn=$vpn wan=$wan"
-        [ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
-        [ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
-        echo "TUNNEL_RECOVERED_OK"
-    l.go:52: | 01:46:25 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT    | LOG   |
-        === STDOUT
-        recovered_vpn=158.173.241.11 wan=136.51.115.252
-        TUNNEL_RECOVERED_OK
-    l.go:52: | 01:46:25 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT    | DONE  |
-    l.go:52: | 01:46:25 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY       | END   |
-    l.go:52: | 01:46:25 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CLEANUP   | BEGIN |
-    l.go:52: | 01:46:25 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | DELETE    | OK    | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
-    l.go:52: | 01:46:25 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CLEANUP   | END   |
-    l.go:52: | 01:46:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CLEANUP   | BEGIN |
-    l.go:52: | 01:46:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | DELETE    | OK    | v1/Pod @ chainsaw-credible-bird/egress-vpn
-    l.go:52: | 01:46:56 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | DELETE    | OK    | v1/Pod @ chainsaw-credible-bird/egress-internet
-    l.go:52: | 01:47:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CLEANUP   | END   |
-    l.go:52: | 01:47:26 | vpn-killswitch | @chainsaw                                 | CLEANUP   | BEGIN |
-    l.go:52: | 01:47:26 | vpn-killswitch | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-credible-bird
-    l.go:52: | 01:47:33 | vpn-killswitch | @chainsaw                                 | CLEANUP   | END   |
---- PASS: chainsaw (0.00s)
-    --- PASS: chainsaw/vpn-killswitch (174.63s)
-PASS
-Tests Summary...
-- Passed  tests 1
-- Failed  tests 0
-- Skipped tests 0
-Done.
-[run.sh] HEAVY pass: no heavy tests selected.
-merge_junit: merged 1 report(s) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
-[run.sh] Done. JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260913T014923Z-9d0814a/report.junit.xml

@@ -1,4 +0,0 @@
-<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch" tests="1" failures="0" errors="0" id="0" time="">
-    <testcase name="vpn-killswitch" classname="" time="173.473422" />
-  </testsuite>
-</testsuites>

conformance/runs/20260913T014923Z-9d0814a/run.log

@@ -1,237 +0,0 @@
-[run.sh] Coverage report first (always)
-[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
-[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
-[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-NET-06').
-[run.sh] LIGHT pass: 1 test(s), parallel=2 (pinned v0.2.12)
-Version: 0.2.12
-Loading config (/work/.claude/conf-clone/conformance/config/chainsaw-config.yaml)...
-- Using test file: chainsaw-test
-- TestDirs [/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch]
-- SkipDelete false
-- FailFast false
-- ReportFormat 'JUNIT-TEST'
-- ReportName 'chainsaw-report-light'
-- ReportPath '/work/.claude/conf-clone/conformance/_artifacts'
-- Namespace ''
-- FullName false
-- IncludeTestRegex ''
-- ExcludeTestRegex ''
-- ApplyTimeout 1m30s
-- AssertTimeout 3m0s
-- CleanupTimeout 2m0s
-- DeleteTimeout 2m0s
-- ErrorTimeout 1m0s
-- ExecTimeout 3m0s
-- DeletionPropagationPolicy Background
-- Parallel 2
-- Template true
-- NoCluster false
-- PauseOnFailure false
-Loading tests...
-- vpn-killswitch (/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch)
-Loading values...
-Running tests...
-=== RUN   chainsaw
-=== PAUSE chainsaw
-=== CONT  chainsaw
-=== RUN   chainsaw/vpn-killswitch
-=== PAUSE chainsaw/vpn-killswitch
-=== CONT  chainsaw/vpn-killswitch
-    l.go:52: | 01:49:24 | vpn-killswitch | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-beloved-toad
-    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | TRY       | BEGIN |
-    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        kubectl label namespace "$NAMESPACE" \
-          pod-security.kubernetes.io/enforce=privileged \
-          pod-security.kubernetes.io/warn=privileged \
-          pod-security.kubernetes.io/audit=privileged \
-          --overwrite
-        echo "NS_PRIVILEGED_OK"
-    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | LOG   |
-        === STDOUT
-        namespace/chainsaw-beloved-toad labeled
-        NS_PRIVILEGED_OK
-    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | DONE  |
-    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | RUN   | v1/Pod @ chainsaw-beloved-toad/egress-internet
-    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CREATE    | OK    | v1/Pod @ chainsaw-beloved-toad/egress-internet
-    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | DONE  | v1/Pod @ chainsaw-beloved-toad/egress-internet
-    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | RUN   | v1/Pod @ chainsaw-beloved-toad/egress-vpn
-    l.go:52: | 01:49:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CREATE    | OK    | v1/Pod @ chainsaw-beloved-toad/egress-vpn
-    l.go:52: | 01:49:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | DONE  | v1/Pod @ chainsaw-beloved-toad/egress-vpn
-    l.go:52: | 01:49:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | RUN   | v1/Pod @ chainsaw-beloved-toad/egress-internet
-    l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | DONE  | v1/Pod @ chainsaw-beloved-toad/egress-internet
-    l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | RUN   | v1/Pod @ chainsaw-beloved-toad/egress-vpn
-    l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | DONE  | v1/Pod @ chainsaw-beloved-toad/egress-vpn
-    l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        ns="$NAMESPACE"
-        get_ip() {
-          kubectl -n "$ns" exec "$1" -c curl -- \
-            curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
-        }
-        wan="" vpn=""
-        for i in $(seq 1 12); do
-          [ -n "$wan" ] || wan="$(get_ip egress-internet)"
-          [ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
-          if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
-          [ "$wan" = "$vpn" ] && vpn=""   # not steered yet; re-poll
-          sleep 5
-        done
-        echo "wan=$wan vpn=$vpn"
-        [ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
-        [ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
-        [ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
-        echo "TUNNEL_HEALTHY_OK"
-    l.go:52: | 01:49:27 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | LOG   |
-        === STDOUT
-        wan=136.51.115.252 vpn=158.173.241.11
-        TUNNEL_HEALTHY_OK
-    l.go:52: | 01:49:27 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | DONE  |
-    l.go:52: | 01:49:27 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | TRY       | END   |
-    l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | TRY       | BEGIN |
-    l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | APPLY     | RUN   | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
-    l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CREATE    | OK    | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
-    l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | APPLY     | DONE  | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
-    l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | ASSERT    | RUN   | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
-    l.go:52: | 01:49:31 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | ASSERT    | DONE  | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
-    l.go:52: | 01:49:31 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -u
-        ns="$NAMESPACE"
-        wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
-                 curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
-        for i in $(seq 1 8); do
-          out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
-                   curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
-          if [ -n "$out" ]; then
-            if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
-              echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
-            fi
-            echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
-          fi
-          echo "attempt $i: vpn-class blackholed (no response), good"
-          sleep 3
-        done
-        echo "NO_LEAK_OK"
-    l.go:52: | 01:51:01 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | LOG   |
-        === STDOUT
-        attempt 1: vpn-class blackholed (no response), good
-        attempt 2: vpn-class blackholed (no response), good
-        attempt 3: vpn-class blackholed (no response), good
-        attempt 4: vpn-class blackholed (no response), good
-        attempt 5: vpn-class blackholed (no response), good
-        attempt 6: vpn-class blackholed (no response), good
-        attempt 7: vpn-class blackholed (no response), good
-        attempt 8: vpn-class blackholed (no response), good
-        NO_LEAK_OK
-    l.go:52: | 01:51:01 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | DONE  |
-    l.go:52: | 01:51:01 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        ns="$NAMESPACE"
-        out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
-                 curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
-        echo "internet_ip=$out"
-        [ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
-        echo "INTERNET_UNAFFECTED_OK"
-    l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | LOG   |
-        === STDOUT
-        internet_ip=136.51.115.252
-        INTERNET_UNAFFECTED_OK
-    l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | DONE  |
-    l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | TRY       | END   |
-    l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | FINALLY   | BEGIN |
-    l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        ns="$NAMESPACE"
-        # Distinct Job name (tunnel-restart) so it never collides with the
-        # tunnel-stop Job. Left-aligned manifest so the block scalar yields
-        # valid YAML at column 0 at runtime.
-        cat <<'JOB' | kubectl -n "$ns" apply -f -
-        apiVersion: batch/v1
-        kind: Job
-        metadata:
-          name: tunnel-restart
-          labels: { app: vpn-killswitch-fault }
-        spec:
-          backoffLimit: 3
-          activeDeadlineSeconds: 90
-          ttlSecondsAfterFinished: 600
-          template:
-            metadata:
-              labels: { app: vpn-killswitch-fault }
-            spec:
-              restartPolicy: Never
-              hostPID: true
-              nodeSelector:
-                homelab.keithjackson.dev/egress-gateway: "true"
-              tolerations:
-                - operator: Exists
-              containers:
-                - name: nsenter
-                  image: busybox:1.37.0
-                  securityContext:
-                    privileged: true
-                  command: ["nsenter","-t","1","-m","-u","-i","-n","-p","--","systemctl","start","vpn-egress-tunnel"]
-        JOB
-        # Block teardown until the restart actually completed on the host — a
-        # leaked-down tunnel is unacceptable. `wait` exits non-zero on failure
-        # or timeout, which (set -e) fails the step and surfaces the problem.
-        kubectl -n "$ns" wait --for=condition=complete --timeout=110s job/tunnel-restart
-        echo "TUNNEL_SELF_HEAL_OK"
-    l.go:52: | 01:51:05 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | LOG   |
-        === STDOUT
-        job.batch/tunnel-restart created
-        job.batch/tunnel-restart condition met
-        TUNNEL_SELF_HEAL_OK
-    l.go:52: | 01:51:05 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | DONE  |
-    l.go:52: | 01:51:05 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | FINALLY   | END   |
-    l.go:52: | 01:51:05 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY       | BEGIN |
-    l.go:52: | 01:51:05 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        ns="$NAMESPACE"
-        wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
-                 curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
-        vpn=""
-        for i in $(seq 1 24); do
-          vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
-                   curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
-          if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
-          vpn=""
-          sleep 5
-        done
-        echo "recovered_vpn=$vpn wan=$wan"
-        [ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
-        [ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
-        echo "TUNNEL_RECOVERED_OK"
-    l.go:52: | 01:51:08 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT    | LOG   |
-        === STDOUT
-        recovered_vpn=158.173.241.11 wan=136.51.115.252
-        TUNNEL_RECOVERED_OK
-    l.go:52: | 01:51:08 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT    | DONE  |
-    l.go:52: | 01:51:08 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY       | END   |
-    l.go:52: | 01:51:08 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CLEANUP   | BEGIN |
-    l.go:52: | 01:51:08 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | DELETE    | OK    | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
-    l.go:52: | 01:51:08 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CLEANUP   | END   |
-    l.go:52: | 01:51:08 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CLEANUP   | BEGIN |
-    l.go:52: | 01:51:08 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | DELETE    | OK    | v1/Pod @ chainsaw-beloved-toad/egress-vpn
-    l.go:52: | 01:51:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | DELETE    | OK    | v1/Pod @ chainsaw-beloved-toad/egress-internet
-    l.go:52: | 01:52:10 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CLEANUP   | END   |
-    l.go:52: | 01:52:10 | vpn-killswitch | @chainsaw                                 | CLEANUP   | BEGIN |
-    l.go:52: | 01:52:10 | vpn-killswitch | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-beloved-toad
-    l.go:52: | 01:52:18 | vpn-killswitch | @chainsaw                                 | CLEANUP   | END   |
---- PASS: chainsaw (0.00s)
-    --- PASS: chainsaw/vpn-killswitch (173.47s)
-PASS
-Tests Summary...
-- Passed  tests 1
-- Failed  tests 0
-- Skipped tests 0
-Done.
-[run.sh] HEAVY pass: no heavy tests selected.
-merge_junit: merged 1 report(s) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
-[run.sh] Done. JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260914T021505Z-24b8111/report.junit.xml

@@ -1,4 +0,0 @@
-<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
-    <testcase name="dr-restore" classname="" time="150.698290" />
-  </testsuite>
-</testsuites>

conformance/runs/20260914T021505Z-24b8111/run.log

@@ -1,672 +0,0 @@
-[run.sh] Coverage report first (always)
-[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
-[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
-[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
-[run.sh] LIGHT pass: no tests selected.
-[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
-Version: 0.2.12
-Loading config (/work/conformance/config/chainsaw-config.yaml)...
-- Using test file: chainsaw-test
-- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
-- SkipDelete false
-- FailFast false
-- ReportFormat 'JUNIT-TEST'
-- ReportName 'chainsaw-report-heavy'
-- ReportPath '/work/conformance/_artifacts'
-- Namespace ''
-- FullName false
-- IncludeTestRegex ''
-- ExcludeTestRegex ''
-- ApplyTimeout 1m30s
-- AssertTimeout 3m0s
-- CleanupTimeout 2m0s
-- DeleteTimeout 2m0s
-- ErrorTimeout 1m0s
-- ExecTimeout 3m0s
-- DeletionPropagationPolicy Background
-- Parallel 1
-- Template true
-- NoCluster false
-- PauseOnFailure false
-Loading tests...
-- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
-Loading values...
-Running tests...
-=== RUN   chainsaw
-=== PAUSE chainsaw
-=== CONT  chainsaw
-=== RUN   chainsaw/dr-restore
-=== PAUSE chainsaw/dr-restore
-=== CONT  chainsaw/dr-restore
-    l.go:52: | 02:15:05 | dr-restore | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-glorious-sheep
-    l.go:52: | 02:15:05 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | BEGIN |
-    l.go:52: | 02:15:05 | dr-restore | assert-backup-target-is-off-cluster       | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        
-        # ---- helpers -------------------------------------------------------
-        # Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
-        # fail loudly; an absent API is a genuine precondition gap and may skip.
-        die_or_skip() {
-          case "$2" in
-            *orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
-              echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
-              echo "$2" >&2
-              exit 1 ;;
-            *)
-              echo "SKIP: $1 not available on this cluster -- $2"
-              echo DR_RESTORE_SKIPPED_OK
-              exit 0 ;;
-          esac
-        }
-        
-        ip2int() {
-          _q="$(echo "$1" | tr '.' ' ')"
-          set -- $_q
-          [ $# -eq 4 ] || { echo -1; return 0; }
-          echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
-        }
-        
-        # in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
-        in_cidr() {
-          case "$2" in *"/"*) ;; *) return 1 ;; esac
-          _net="${2%/*}"; _len="${2#*/}"
-          case "$_len" in ''|*[!0-9]*) return 1 ;; esac
-          [ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
-          _i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
-          [ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
-          [ "$_len" -eq 0 ] && return 0
-          _d=1; _s=0
-          while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
-          [ $(( _i / _d )) -eq $(( _n / _d )) ]
-        }
-        
-        # ---- Gate 1: the replicated storage tier the DR-critical data lives on --
-        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
-          echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
-          echo DR_RESTORE_SKIPPED_OK
-          exit 0
-        fi
-        
-        # ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
-        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
-          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
-          2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
-        if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
-          echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
-          echo DR_RESTORE_SKIPPED_OK
-          exit 0
-        fi
-        echo "backup storage locations:"
-        echo "$BSL_LINES"
-        
-        VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
-        
-        # ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
-        # File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
-        # deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
-        # ready node-agent no volume data can be captured, so the drill is not runnable.
-        VREADY=0; NAREADY=0
-        for i in $(seq 1 12); do
-          VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
-          NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
-          echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
-          if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
-          sleep 10
-        done
-        if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
-          echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
-          echo DR_RESTORE_SKIPPED_OK
-          exit 0
-        fi
-        
-        # ---- Pick the BSL this drill will use: prefer the default, and it MUST be
-        #      Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
-        #      gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
-        #      wait for first-validation on a cold start), it does not skip. --------
-        BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
-        for i in $(seq 1 12); do
-          BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
-            -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
-          CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
-          [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
-          if [ -n "$CAND" ]; then
-            BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
-            BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
-            BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
-            BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
-            break
-          fi
-          echo "poll $i: no Available BackupStorageLocation yet"
-          sleep 10
-        done
-        if [ -z "$BSL_NAME" ]; then
-          echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
-          echo "      The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
-          echo "$BSL_LINES" >&2
-          exit 1
-        fi
-        echo "using BSL ${BSL_NS}/${BSL_NAME}  bucket=${BSL_BUCKET}  endpoint=${BSL_URL}"
-        
-        # ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
-        if [ -z "$BSL_URL" ]; then
-          echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
-          echo "      the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
-          echo "      test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
-          exit 1
-        fi
-        HOST="${BSL_URL#*://}"      # strip scheme
-        HOST="${HOST%%/*}"          # strip path
-        HOST="${HOST%%:*}"          # strip port
-        echo "backup endpoint host: $HOST"
-        
-        case "$HOST" in
-          localhost|*.svc|*.svc.cluster.local|*.cluster.local)
-            echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
-            echo "      An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
-            exit 1 ;;
-        esac
-        
-        # Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
-        # off-cluster property -- that is a failure, never a silent pass.
-        case "$HOST" in
-          *[!0-9.]*)
-            # Endpoint given as a name. Try the resolvers that may exist in the
-            # runner image, in order; if none is available or none resolves, the
-            # test FAILS below rather than guessing.
-            EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
-            [ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
-            [ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
-          *) EPIP="$HOST" ;;
-        esac
-        if [ -z "$EPIP" ]; then
-          echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
-          echo "      the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
-          exit 1
-        fi
-        echo "backup endpoint resolves to: $EPIP"
-        
-        # ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
-        case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
-        
-        NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
-        for a in $NODE_IPS; do
-          if [ "$a" = "$EPIP" ]; then
-            echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
-            exit 1
-          fi
-        done
-        echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
-        
-        SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
-        for a in $SVC_IPS; do
-          if [ "$a" = "$EPIP" ]; then
-            echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
-            exit 1
-          fi
-        done
-        echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
-        
-        POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
-        for a in $POD_IPS; do
-          if [ "$a" = "$EPIP" ]; then
-            echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
-            exit 1
-          fi
-        done
-        echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
-        
-        POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
-        for c in $POD_CIDRS; do
-          case "$c" in *:*) continue ;; esac      # IPv6 prefixes are out of scope here
-          if in_cidr "$EPIP" "$c"; then
-            echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
-            exit 1
-          fi
-        done
-        echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
-        
-        # Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
-        # runner may not read): the checks above are the authoritative ones.
-        SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
-        if [ -n "$SVC_CIDRS" ]; then
-          for c in $SVC_CIDRS; do
-            case "$c" in *:*) continue ;; esac
-            if in_cidr "$EPIP" "$c"; then
-              echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
-              exit 1
-            fi
-          done
-          echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
-        else
-          echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
-        fi
-        
-        echo "OFF_CLUSTER_TARGET_OK  bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
-    l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | LOG   |
-        === STDOUT
-        backup storage locations:
-        velero default Available aws true https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com k3s-prod-backup
-        poll 1: velero readyReplicas=1 node-agent numberReady=3
-        using BSL velero/default  bucket=k3s-prod-backup  endpoint=https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
-        backup endpoint host: 7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
-        backup endpoint resolves to: 172.64.66.1
-        endpoint is not any node address (checked 6 node addresses)
-        endpoint is not any Service ClusterIP (checked 82)
-        endpoint is not any Pod IP (checked 197)
-        endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
-        endpoint is outside every ServiceCIDR (10.43.0.0/16 )
-        OFF_CLUSTER_TARGET_OK  bsl=velero/default endpoint=172.64.66.1 bucket=k3s-prod-backup
-    l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | DONE  |
-    l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | END   |
-    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | TRY       | BEGIN |
-    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        SCHED="$(kubectl get schedules.velero.io -A \
-          -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
-        if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
-          # Distinguish "Velero absent" (skip, in lock-step with step 1) from
-          # "Velero present but nothing scheduled" (a real R-STATE-03 defect).
-          if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
-            if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
-              echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
-              echo DR_SCHEDULE_SKIPPED_OK
-              exit 0
-            fi
-            echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
-            echo "      R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
-            exit 1
-          fi
-          echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
-          echo DR_SCHEDULE_SKIPPED_OK
-          exit 0
-        fi
-        echo "velero schedules:"
-        echo "$SCHED"
-        # A Schedule that exists but is paused/disabled backs nothing up.
-        if echo "$SCHED" | grep -q 'phase=Enabled'; then
-          echo BACKUPS_ARE_SCHEDULED_OK
-        else
-          echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
-          exit 1
-        fi
-    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | LOG   |
-        === STDOUT
-        velero schedules:
-        velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T04:01:06Z
-        velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T03:00:03Z
-        BACKUPS_ARE_SCHEDULED_OK
-    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | DONE  |
-    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | TRY       | END   |
-    l.go:52: | 02:15:08 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | BEGIN |
-    l.go:52: | 02:15:08 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -eu
-        NS="$NAMESPACE"
-        SRC="${NS}-dr-src"          # scratch source namespace (destroyed by design)
-        DST="${NS}-dr-dst"          # clean restore target (must not pre-exist)
-        BK="${NS}-dr-backup"
-        RS="${NS}-dr-restore"
-        OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
-        
-        # ---- Re-derive the preconditions (steps do not share shell state). Every
-        #      skip below mirrors step 1 exactly, so the whole test skips or runs
-        #      as a unit -- it can never half-run. -----------------------------
-        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
-          echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
-        fi
-        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
-          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
-        CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
-        [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
-        if [ -z "$CAND" ]; then
-          echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
-          echo DR_RESTORE_SKIPPED_OK; exit 0
-        fi
-        VNS="$(echo "$CAND" | cut -d' ' -f1)"
-        BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
-        if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
-          echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
-          echo DR_RESTORE_SKIPPED_OK; exit 0
-        fi
-        echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
-        
-        # A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
-        # be free. If either is lying around from an aborted run, remove it before we
-        # claim "clean" -- and fail loudly if it will not go, because a half-deleted
-        # namespace would silently invalidate the destroy proof below.
-        kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
-        
-        # ============================ GIVEN ==============================
-        # A DR-critical stateful workload with known data on the replicated tier.
-        CANARY="dr-canary-$(date +%s%N)"
-        kubectl create namespace "$SRC"
-        kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
-        
-        kubectl -n "$SRC" apply -f - <<EOF
-        apiVersion: v1
-        kind: PersistentVolumeClaim
-        metadata:
-          name: dr-data
-          labels:
-            conformance.home.k3s/created-by: ${NS}
-        spec:
-          accessModes: [ReadWriteOnce]
-          storageClassName: longhorn
-          resources:
-            requests:
-              storage: 1Gi
-        EOF
-        
-        kubectl -n "$SRC" apply -f - <<EOF
-        apiVersion: v1
-        kind: Pod
-        metadata:
-          name: dr-workload
-          labels:
-            app: dr-workload
-            conformance.home.k3s/created-by: ${NS}
-        spec:
-          restartPolicy: Never
-          containers:
-            - name: app
-              image: busybox:1.36.1
-              command: ["sh", "-c", "sleep 7200"]
-              volumeMounts:
-                - name: data
-                  mountPath: /data
-          volumes:
-            - name: data
-              persistentVolumeClaim:
-                claimName: dr-data
-        EOF
-        
-        kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
-        printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
-        echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
-        
-        ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
-        [ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
-        echo "original in-cluster PersistentVolume: $ORIG_PV"
-        
-        # ======================= WHEN (part 1): back up off-cluster =======
-        # Created as a Velero Backup CR -- exactly what `velero backup create` does.
-        # defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
-        # the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
-        # ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
-        # own GC still removes this drill's objects from the bucket.
-        kubectl apply -f - <<EOF
-        apiVersion: velero.io/v1
-        kind: Backup
-        metadata:
-          name: ${BK}
-          namespace: ${VNS}
-          labels:
-            conformance.home.k3s/created-by: ${NS}
-        spec:
-          includedNamespaces: [${SRC}]
-          storageLocation: ${BSL_NAME}
-          snapshotVolumes: false
-          defaultVolumesToFsBackup: true
-          ttl: 2h0m0s
-        EOF
-        
-        BPHASE=""
-        for i in $(seq 1 90); do
-          BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
-          echo "poll $i: backup phase=${BPHASE:-<pending>}"
-          case "${BPHASE:-}" in
-            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
-          esac
-          sleep 10
-        done
-        if [ "${BPHASE:-}" != "Completed" ]; then
-          echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
-          kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
-          exit 1
-        fi
-        echo "backup $BK Completed to ${BSL_NAME}"
-        
-        # ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
-        #      kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
-        PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
-                 -l velero.io/backup-name="$BK" \
-                 -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
-        echo "pod volume backups:"
-        echo "$PVB"
-        if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
-          echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
-          echo "      Resource-only backups cannot restore DR-critical DATA." >&2
-          exit 1
-        fi
-        # A line is "<name> <phase> <snapshotID>"; require Completed AND a
-        # non-empty snapshot ID (an empty ID means nothing reached the repository).
-        GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
-        if [ "${GOOD_PVB:-0}" -lt 1 ]; then
-          echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
-          echo "      the volume data did not reach the off-cluster repository." >&2
-          exit 1
-        fi
-        echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
-        
-        # ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
-        # Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
-        # underlying PersistentVolume (and with it the Longhorn replicas). After this
-        # there is no in-cluster source the restore could quietly read from.
-        kubectl delete namespace "$SRC" --wait=true --timeout=420s
-        for i in $(seq 1 30); do
-          if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
-          echo "poll $i: waiting for namespace $SRC to disappear"
-          sleep 10
-        done
-        if kubectl get namespace "$SRC" >/dev/null 2>&1; then
-          echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
-          exit 1
-        fi
-        echo "source namespace destroyed"
-        
-        # The longhorn StorageClass reclaims Delete, so the PV should already be gone;
-        # force it if a Retain policy (or a stuck finalizer) left it behind, because a
-        # surviving PV would make the restore prove nothing about off-cluster recovery.
-        kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
-        for i in $(seq 1 30); do
-          if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
-          echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
-          sleep 10
-        done
-        if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
-          echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
-          echo "      An in-cluster copy of the data still exists, so a successful restore" >&2
-          echo "      would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
-          exit 1
-        fi
-        echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
-        
-        # Best-effort corroboration (the least-privilege runner may not read longhorn.io):
-        # the Longhorn Volume CR and its replicas should be gone too.
-        if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
-          echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
-          exit 1
-        fi
-        echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
-        
-        # ============================ RESTORE ============================
-        # Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
-        # the catalog's "restored into a clean state".
-        kubectl apply -f - <<EOF
-        apiVersion: velero.io/v1
-        kind: Restore
-        metadata:
-          name: ${RS}
-          namespace: ${VNS}
-          labels:
-            conformance.home.k3s/created-by: ${NS}
-        spec:
-          backupName: ${BK}
-          includedNamespaces: [${SRC}]
-          namespaceMapping:
-            ${SRC}: ${DST}
-          existingResourcePolicy: none
-        EOF
-        
-        RPHASE=""
-        for i in $(seq 1 90); do
-          RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
-          echo "poll $i: restore phase=${RPHASE:-<pending>}"
-          case "${RPHASE:-}" in
-            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
-          esac
-          sleep 10
-        done
-        if [ "${RPHASE:-}" != "Completed" ]; then
-          echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
-          kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
-          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
-          exit 1
-        fi
-        echo "restore $RS Completed into $DST"
-        
-        # ============================= THEN ==============================
-        # The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
-        # `restore-wait` init container that blocks until the node-agent has written the
-        # volume contents back, so Ready here already implies the data landed.
-        if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
-          echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
-          kubectl -n "$DST" describe pod dr-workload >&2 || true
-          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
-          exit 1
-        fi
-        
-        NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
-        echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
-        [ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
-        if [ "$NEW_PV" = "$ORIG_PV" ]; then
-          echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
-          echo "      That is an in-cluster survival, not an off-cluster restore." >&2
-          exit 1
-        fi
-        echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
-        
-        GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
-        echo "recovered='$GOT' expected='$CANARY'"
-        if [ "$GOT" != "$CANARY" ]; then
-          echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
-          exit 1
-        fi
-        
-        echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
-    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
-        === STDOUT
-        velero ns=velero bsl=default src=chainsaw-glorious-sheep-dr-src dst=chainsaw-glorious-sheep-dr-dst
-        namespace/chainsaw-glorious-sheep-dr-src created
-        namespace/chainsaw-glorious-sheep-dr-src labeled
-        persistentvolumeclaim/dr-data created
-        pod/dr-workload created
-        pod/dr-workload condition met
-        seeded canary: dr-canary-1789352108935754215
-        original in-cluster PersistentVolume: pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9
-        backup.velero.io/chainsaw-glorious-sheep-dr-backup created
-        poll 1: backup phase=InProgress
-        poll 2: backup phase=InProgress
-        poll 3: backup phase=InProgress
-        poll 4: backup phase=Completed
-        backup chainsaw-glorious-sheep-dr-backup Completed to default
-        pod volume backups:
-        chainsaw-glorious-sheep-dr-backup-2gjrq Completed 979eed437233b45c7f85ba80074ab56b
-        volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
-        namespace "chainsaw-glorious-sheep-dr-src" deleted
-        source namespace destroyed
-        original PersistentVolume pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 is GONE -- no in-cluster copy of the data remains
-        (longhorn Volume CR pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 not present -- either deleted or not readable by this identity)
-        restore.velero.io/chainsaw-glorious-sheep-dr-restore created
-        poll 1: restore phase=InProgress
-        poll 2: restore phase=InProgress
-        poll 3: restore phase=InProgress
-        poll 4: restore phase=Completed
-        restore chainsaw-glorious-sheep-dr-restore Completed into chainsaw-glorious-sheep-dr-dst
-        pod/dr-workload condition met
-        restored PVC bound to PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
-        restored onto a NEW volume (pvc-f0c061e1-c080-49f2-a381-23da5c886df2 != pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9) -- the data came from outside the cluster
-        recovered='dr-canary-1789352108935754215' expected='dr-canary-1789352108935754215'
-        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
-        === STDERR
-        Defaulted container "app" out of: app, restore-wait (init)
-    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
-    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | END   |
-    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | BEGIN |
-    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
-        === COMMAND
-        /usr/bin/sh -c set -u
-        NS="$NAMESPACE"
-        SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
-        
-        VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
-        
-        # Velero objects first: a DeleteBackupRequest makes Velero remove the backup
-        # DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
-        if [ -n "$VNS" ]; then
-          kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
-          if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
-            kubectl apply -f - <<EOF || true
-        apiVersion: velero.io/v1
-        kind: DeleteBackupRequest
-        metadata:
-          name: ${BK}-delete
-          namespace: ${VNS}
-          labels:
-            conformance.home.k3s/created-by: ${NS}
-        spec:
-          backupName: ${BK}
-        EOF
-            for i in $(seq 1 18); do
-              kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
-              sleep 10
-            done
-            # Backstop if Velero could not process the request (the 2h ttl also GCs it).
-            kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
-            kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
-          fi
-          echo "velero objects for $NS cleaned up (ns $VNS)"
-        fi
-        
-        # Scratch namespaces (both, in case the run aborted at any point).
-        for n in "$DST" "$SRC"; do
-          kubectl delete namespace "$n" --ignore-not-found --wait=false || true
-        done
-        # Anything left labelled as ours (defensive; the two names above are the only
-        # namespaces this test ever creates).
-        kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
-        
-        # Released PVs from either namespace's PVCs would otherwise leak storage.
-        for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
-          kubectl delete pv "$pv" --ignore-not-found --wait=false || true
-          echo "released leftover PersistentVolume: $pv"
-        done
-        
-        echo "dr-restore teardown complete (best-effort; every object was created by this test)"
-    l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
-        === STDOUT
-        restore.velero.io "chainsaw-glorious-sheep-dr-restore" deleted from velero namespace
-        deletebackuprequest.velero.io/chainsaw-glorious-sheep-dr-backup-delete created
-        velero objects for chainsaw-glorious-sheep cleaned up (ns velero)
-        namespace "chainsaw-glorious-sheep-dr-dst" deleted
-        namespace "chainsaw-glorious-sheep-dr-dst" deleted
-        persistentvolume "pvc-f0c061e1-c080-49f2-a381-23da5c886df2" deleted
-        released leftover PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
-        dr-restore teardown complete (best-effort; every object was created by this test)
-    l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
-    l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | END   |
-    l.go:52: | 02:17:30 | dr-restore | @chainsaw                                 | CLEANUP   | BEGIN |
-    l.go:52: | 02:17:30 | dr-restore | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-glorious-sheep
-    l.go:52: | 02:17:36 | dr-restore | @chainsaw                                 | CLEANUP   | END   |
---- PASS: chainsaw (0.00s)
-    --- PASS: chainsaw/dr-restore (150.70s)
-PASS
-Tests Summary...
-- Passed  tests 1
-- Failed  tests 0
-- Skipped tests 0
-Done.
-merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
-[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260914T021902Z-bd17b65/report.junit.xml

@@ -1,13 +0,0 @@
-<?xml version="1.0" ?>
-<testsuites name="home-k3s-conformance" tests="1" failures="0" errors="0" skipped="0">
-  <testsuite name="home-k3s-conformance L0/L1 (offline)" tests="1" failures="0" errors="0" skipped="0">
-    <testcase classname="L0:R-ENV-02,R-STATE-03" name="placeholder-guard-coupling">
-      <properties>
-        <property name="requirements" value="R-ENV-02,R-STATE-03"/>
-        <property name="layer" value="L0"/>
-        <property name="path" value="/work/conformance/tests/l0-static/placeholder-guard-coupling"/>
-        <property name="scope" value=""/>
-      </properties>
-    </testcase>
-  </testsuite>
-</testsuites>

conformance/runs/20260914T021902Z-bd17b65/run.log

@@ -1,7 +0,0 @@
-[run.sh] Coverage report first (always)
-[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
-[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
-[run.sh] Running 1 offline L0/L1 check(s) (no cluster; docs/14).
-[offline] L0  PASS placeholder-guard-coupling (R-ENV-02,R-STATE-03)
-[offline] wrote JUnit (1 case(s), 0 failure(s), 0 skip(s), 0 unwired) -> /work/conformance/_artifacts/report.junit.xml
-[run.sh] Done (offline l0). JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/ledger.jsonl

@@ -1,7 +1,2 @@
 {"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "57d4df8c4f992732a49f3e2daddcace866778ab0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260901T235117Z-e9a4b66", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-01T23:51:17Z"}
 {"chainsaw_version": "v0.2.12", "env": "dev", "git_sha": "fdc3265585f4b775c0a6ae3fb33c837b0c643655", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260902T001907Z-fdc3265", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-02T00:19:07Z"}
-{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L1", "note": "L1 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-NET-05", "R-NET-06", "R-NET-07"], "run_id": "20260913T014321Z-19fc8bc", "sentinel": null, "test": "egress-class-render", "ts_utc": "2026-09-13T01:43:21Z"}
-{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6-dirty", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014437Z-19fc8bc", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:44:37Z"}
-{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
-{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
-{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}

conformance/tests/README.md

@@ -93,52 +93,9 @@ and coverage extraction exact and robust.
   break the thing and assert fail-closed / no-data-loss, with `finally` restoring state.
 - **Never weaken a test to go green (R-TEST-06).** If the spec can't be met, escalate to
   the PM — do not relax the assertion or the requirement.
-- **Never write a per-environment value into a test (R-ENV-02).** No hostname suffix, no
-  Gateway VIP, no LB-pool range. One authored suite must run green against dev *and* prod,
-  so a test **asks the cluster it is pointed at** what it is. The mechanism is a chainsaw
-  binding over the same ConfigMap Flux substitutes from:
-
-  ```yaml
-  spec:
-    bindings:
-      - name: baseDomain
-        value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
-  ```
-
-  Declare it at **test level** (step `outputs`/`bindings` are step-scoped), then consume it:
-  `($baseDomain)` for a pod `env:` value, `(join('.', ['myhost', $baseDomain]))` where a
-  full hostname is needed — including inside a `fixtures/*.yaml` applied with `apply: file:`,
-  which chainsaw templates with the same bindings. Prefer reading the *actual* observed value
-  over restating a range (e.g. assert the DNS answer equals `GATEWAY_VIP`, or equals the LB IP
-  the Service was assigned) — it is both stricter and environment-independent. Never take the
-  environment from a CLI flag or an exported variable: a flag someone forgets is how a suite
-  ends up silently asserting the other cluster's expectations. L1 goldens are the one
-  exception: they are rendering fixtures and carry the literal `${BASE_DOMAIN}` token.
-
-### How a scaffold declares itself (and how `status.py` tells one from a verifier)
 
 Scaffolds in this tree carry `TODO(T-x.y)` markers where a fixture/assertion lands as a
-later roadmap phase delivers the capability under test. `lib/status.py` classifies each
-test EXECUTABLE or SCAFFOLD from the test's own files — `chainsaw-test.yaml`,
-`render-check.sh`, `check.sh` — and either signal below **outranks**
-`conformance.home.k3s/executable: "true"`, so a placeholder body can never be reported as
-an executable test. Declare a scaffold in **both** positions:
-
-1. **`# STATUS: scaffold — <what is missing>`** in the file's header comment block. The
-   declared state must be the first token after `STATUS:` (executable tests say
-   `# STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`).
-2. **A `TODO`/`FIXME` marker carrying a message, in the body** — the file from its first
-   non-comment line onward. `echo "TODO(T-4.2): inject a fault to fire a rule"`,
-   or an inline comment on a missing fixture:
-   `file: fixtures/smtp-sink.yaml   # TODO(T-4.1): hermetic SMTP capture sink`.
-
-What is **not** a scaffold marker, and why the distinction is load-bearing: a marker
-mentioned in the **header prose** (the Given/When/Then and the mandatory "what this does
-NOT prove" list), and a marker that is the **entire content of a quoted literal** — a
-placeholder *detector*'s own vocabulary, e.g. `SEED = ["TBD", "TODO", "FIXME", ...]` in
-`l0-static/placeholder-guard-coupling/check.sh`. A whole-file `"TODO" in text` test used to
-classify that 32-assertion check as a scaffold and, through it, report the safety-critical
-R-STATE-03 as `exec=1 scaffold=1`. If your check must hold a marker as data, keep each
-idiom a **standalone quoted literal in a list**; a marker with text attached (a regex
-alternation, several idioms in one string) still classifies as a scaffold, because
-ambiguity here only ever downgrades.
+later roadmap phase delivers the capability under test. `lib/status.py` treats such a
+marker — in the `chainsaw-test.yaml`, in `render-check.sh`, or in `check.sh` — as
+outranking any `conformance.home.k3s/executable: "true"` annotation, so a placeholder body
+can never be reported as an executable test.

conformance/tests/l0-static/alert-metrics-have-scrape-targets/chainsaw-test.yaml

@@ -1,128 +0,0 @@
-# =============================================================================
-# L0 STATIC — every metric an alert rule queries is CLAIMED by a scrape target this repo configures
-# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
-#
-# THE DEFECT THIS CLOSES. An alert rule whose metric is never scraped is not a broken alert — it is
-# an ABSENT alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB with no
-# longhorn_* series returns no vector, so the rule is permanently quiet, permanently "healthy", and
-# indistinguishable from a correct rule on a healthy cluster. It survives review (the PromQL is
-# right), it survives `promtool test rules` (the unit test supplies the series itself), and it
-# survives every dashboard (nobody looks for a panel that was never built).
-#
-# MEASURED ON PROD, 2026-09-13, before this change — 13 rules across three families were inert:
-#   count(longhorn_volume_robustness)                        => (empty)   4 rules
-#   count({__name__=~"certmanager.*"})                       => (empty)   3 rules
-#   count(hubble_drop_total)                                 => (empty)   1 rule
-#   count by (job) (up)                                      => 20 jobs, none Longhorn/cert-manager/Hubble
-# So storage health was unmonitored on a cluster whose prod-1 had just had a failing SSD
-# (DECISIONS-LOG), TLS expiry was unmonitored with the live `wildcard-home` wildcard 89 days from
-# expiry, and the platform had no policy-drop visibility at all. The Flux family (gotk_*, 5 rules)
-# was the same defect and was fixed separately in a79693d.
-#
-# GIVEN every alerting rule in every PrometheusRule in the repo (platform/**, apps/wave-*),
-# WHEN  each rule's PromQL is parsed for the metric names it selects,
-# THEN  (1) every metric name is CLAIMED by exactly one declared scrape owner, and each owner's
-#           claim is PROVEN AGAINST THE REPO — the file that enables that scrape must still contain
-#           the enabling declaration, so deleting `metrics.serviceMonitor.enabled` from
-#           platform/longhorn/helmrelease.yaml fails this check instead of silently re-inerting four
-#           storage alerts;
-#       (2) an owner's NEGATIVE conditions hold too, where the enablement can be defeated without
-#           being deleted — cert-manager's chart calls `fail` if podmonitor and servicemonitor are
-#           both set, Longhorn's own NetworkPolicies drop the scrape unless
-#           networkpolicy-metrics.yaml is wired, and kube_node_labels does not exist at all unless
-#           kube-state-metrics' metricLabelsAllowlist names the label;
-#       (3) a metric with NO owner FAILS, unless it is declared in KNOWN_UNWIRED_METRICS — a
-#           shrink-only register (hubble_drop_total today, with the CNI-rollout reason recorded);
-#       (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
-#           check above can never see this failure: where the repo authors a Service and a
-#           ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
-#           actually defines. All 12 apps/* `<app>-down` SLI rules used to fail this (the monitors
-#           named port `metrics`; no app Service defined one) and were registered in
-#           KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series did not exist.
-#           THAT REGISTER IS NOW EMPTY BECAUSE THE TWELVE RULES WERE FIXED, not waived: they read
-#           kube-state-metrics readiness of their own workload object instead (2026-09-13), the dead
-#           ServiceMonitors are gone, and so is this axis' only subject. The monitor side therefore
-#           moved into its own check — tests/l0-static/servicemonitor-ports-resolve — which needs no
-#           alert rule to reach a monitor, resolves by selector across the whole tree instead of by
-#           same-file adjacency, and proves its resolver on fixtures so it cannot pass merely for
-#           lack of subjects;
-#   AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
-#           so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
-#           invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
-#
-# WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
-# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would have made the gate red
-# until exporters existed for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends
-# — none of which speak Prometheus natively — so the gate would have been disabled, and a disabled
-# gate protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
-# (The ratchet did its job: the twelve were closed a day later without an exporter, by reading a
-# signal that already existed. The register is empty now, and that is the outcome it was for.)
-# So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
-# metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
-# register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot
-# into fiction.
-#
-# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, the same dependency as
-# catalog/render.py and dev-nodes-provisioned-as-code); conformance/run.sh --layer l0 and ci/l0.sh
-# run it directly, and this Chainsaw Test wraps the SAME script.
-#
-# NOT proven (all named in check.sh's header, which carries the full list). The one worth stating
-# here because it bit this very repo: an owner claims metrics by PREFIX, so this proves the SCRAPE
-# TARGET exists, never that the exporter emits that exact SERIES NAME.
-# `kubevirt_vmi_phase_count{phase="Failed"}` sat inside the scraped kubevirt_ family on an UP target
-# and did not exist at all on the deployed KubeVirt (corrected here to
-# `kubevirt_vmi_info{phase="failed"}`, lowercase — a capital-F matcher is equally inert). Catching
-# that needs the exporter's own metric list, i.e. a live read; it belongs to a future L1/L3 check
-# that diffs alert metrics against /api/v1/label/__name__/values.
-#
-# Also not proven: that a target is UP. This is static — it asserts the
-# enabling declaration is in git, never that the endpoint answers, that the chart still honours the
-# value key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
-# recorded in the commit message (longhorn_volume_robustness 44 series, certmanager_* 104 series,
-# all six new targets up==1). It also cannot check a monitor whose Service comes from a Helm chart
-# rather than from this repo — the port name lives outside the tree — which is why (4) is scoped to
-# same-file Service+ServiceMonitor pairs. And it does not judge thresholds or reachability-to-human
-# (R-OBS-02 / the L3 alerting test).
-# =============================================================================
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: alert-metrics-have-scrape-targets
-  labels:
-    conformance.home.k3s/R-OBS-04: "true"
-  annotations:
-    conformance.home.k3s/requirements: "R-OBS-04"
-    conformance.home.k3s/layer: "L0"
-    # WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree AND over
-    # every scrape-enabling declaration that feeds them, which live in eight different components
-    # (platform/monitoring, platform/longhorn, platform/cert-manager, platform/velero,
-    # platform/kubevirt, platform/keycloak, platform/cnpg, apps/wave-*). No single requirement
-    # path-maps to that set.
-    #
-    # WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh maps apps/* to NOTHING and
-    # platform/monitoring/* to R-OBS-01/R-OBS-02 — not R-OBS-04. A change that adds an alert under
-    # apps/ or platform/kubevirt/, or one that flips a chart's `serviceMonitor.enabled` back to
-    # false under platform/longhorn/, therefore attributes a requirement set that would FILTER THIS
-    # CHECK OUT of exactly the change that re-inerts the alerts. Declaring the scope makes
-    # ci/l0.sh Stage 3b union it into every run instead, and makes an honest SKIP loud (rc 3 =
-    # UNWIRED, never green) rather than silently green.
-    conformance.home.k3s/scope: "whole-repo"
-    conformance.home.k3s/safety-critical: "false"
-    conformance.home.k3s/executable: "true"
-    conformance.home.k3s/intent: >-
-      Every metric name queried by every alert rule is claimed by a scrape target whose enabling
-      declaration is still present in this repo, every `up{job=...}` selector names a job a repo
-      monitor can actually produce, and the two gaps that remain are enumerated in registers that
-      can only shrink — so an alert can never again be authored, reviewed and merged against a
-      metric nothing scrapes.
-spec:
-  steps:
-    - name: audit-every-alert-metric-against-the-configured-scrape-targets
-      description: >-
-        Parse every metric name out of every alerting rule, resolve each against the declared
-        scrape-owner table, prove each owner's enabling declaration (and its negative conditions)
-        still hold in git, audit `up{job=...}` against same-file Service/ServiceMonitor port
-        agreement, and ratchet both registers of declared gaps.
-      try:
-        - script:
-            content: bash ./check.sh

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -1,728 +0,0 @@
-#!/usr/bin/env bash
-# =============================================================================
-# L0 static check (offline, no cluster) — alert-metrics-have-scrape-targets
-# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
-#         VPN)." — spec docs/10-observability-and-alerting.md)
-#
-# WHY THIS EXISTS — AN ALERT ON AN UNSCRAPED METRIC IS AN ABSENT ALERT THAT READS AS A PRESENT ONE
-# -----------------------------------------------------------------------------------------------
-# `longhorn_volume_robustness == 2` evaluated against a TSDB that holds no longhorn_* series
-# returns no vector. The rule is therefore permanently quiet, permanently "healthy", and
-# INDISTINGUISHABLE from a correct rule on a healthy cluster. It passes review (the PromQL is
-# right). It passes `promtool test rules` (the unit test supplies the series itself, so a unit
-# test proves nothing about whether anything scrapes them). It shows up on no dashboard, because
-# nobody looks for a panel that was never built.
-#
-# Measured on prod 2026-09-13, BEFORE the change that added this check — 13 rules were inert:
-#   count(longhorn_volume_robustness)            => (empty)    LonghornVolumeDegraded/Faulted/
-#                                                              CapacityHigh, LonghornNodeStorageHigh
-#   count({__name__=~"certmanager.*"})           => (empty)    CertificateExpiringSoon/
-#                                                              ExpiryImminent/NotReady
-#   count(hubble_drop_total)                     => (empty)    HubbleDeniedFlowSpike
-#   count by (job) (up)                          => 20 jobs, none Longhorn, cert-manager or Hubble
-# Storage health was unmonitored on the cluster whose prod-1 had just had a failing SSD; TLS
-# expiry was unmonitored with the live `wildcard-home` wildcard 89 days out; there was no
-# policy-drop visibility at all. The gotk_* family (5 more rules) was the same defect, fixed
-# separately in a79693d.
-#
-# WHAT MAKES THIS CLASS DIFFERENT from the two failure modes alert-rules-unit-tested catches: a
-# permanently-firing rule is loud, and a rule that behaves wrongly against synthetic series is
-# caught by promtool. This one is caught by NEITHER, because the missing thing is not in the rule
-# file at all — it is a `serviceMonitor.enabled` that was never set in a completely different
-# component's HelmRelease. So the audit has to cross from the alert to the scrape config, which is
-# what the owner table below does.
-#
-# GIVEN every alerting rule in every PrometheusRule in the repo,
-# WHEN  each rule's PromQL is parsed for the metric names it selects,
-# THEN  (1) every metric name is CLAIMED by exactly one declared owner in SCRAPE_OWNERS, and the
-#           owner's `require` regexes still match its file — so deleting the enablement from
-#           platform/longhorn/helmrelease.yaml FAILS here instead of silently re-inerting four
-#           storage alerts;
-#       (2) each owner's `forbid` regexes do NOT match, for the cases where the enablement can be
-#           defeated without being deleted (cert-manager's chart `fail`s if podmonitor and
-#           servicemonitor are both enabled; an `enabled: false` inside the exporter's own block);
-#       (3) a metric with no owner FAILS unless declared in KNOWN_UNWIRED_METRICS (shrink-only);
-#       (4) `up{job="X"}` is audited separately — `up` itself always exists, so (1) is blind to
-#           this — by requiring that where this repo authors a Service and a ServiceMonitor in the
-#           SAME file, the monitor's endpoint port is a port that Service actually defines;
-#           violations must be declared in KNOWN_UNWIRED_JOBS (shrink-only);
-#   AND (5) the check is LOAD-BEARING: this test declares `conformance.home.k3s/scope: whole-repo`
-#           so ci/l0.sh unions it into every gate run, and ci/gate.sh blocks on an l0 failure.
-#
-# WHY THE REGISTERS ARE RATCHETS (the decision and its reasons) — the same three options as
-# alert-rules-unit-tested faced:
-#   * FAIL outright on all 13: correct in principle, but 12 of them are app SLI rules that need
-#     Prometheus exporters for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr and Seerr —
-#     none of which speak Prometheus natively. The gate would be red for every change until that
-#     work lands, so it would get disabled, and a disabled gate protects nothing.
-#   * merely REPORT: that is precisely how these rules stayed inert across phases 4-6.
-#   * RATCHET (chosen): each gap is declared here, printed with counts on every run, and CANNOT
-#     GROW. A new alert on an unclaimed metric FAILS. A new same-file Service+ServiceMonitor pair
-#     with a mismatched port FAILS. A STALE entry — the gap was closed, or the alert deleted —
-#     also FAILS, so neither register can rot into fiction.
-#
-# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
-#   - It does not prove a target is UP, that an endpoint answers, that the chart still honours the
-#     values key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
-#     recorded in the commit message: with the enablements applied, longhorn_volume_robustness
-#     returned 44 series, {__name__=~"certmanager.*"} returned 104, and all six new targets read
-#     up==1. A live probe needs a cluster, which L0 by definition does not have.
-#   - It cannot audit a ServiceMonitor whose Service is created by a Helm chart rather than by this
-#     repo: the port name lives outside the tree. That is why (4) is scoped to same-file
-#     Service+ServiceMonitor pairs — the apps/* ServiceDefinition pattern — and why the
-#     platform/* monitors (oauth2-proxy, kubevirt, keycloak, flux) are out of its reach. Their
-#     liveness is covered by the fact that their metrics have owners under (1).
-#   - IT PROVES THE SCRAPE TARGET, NOT THE SERIES NAME. An owner claims a metric by PREFIX, so a
-#     metric in a scraped family whose exact name the exporter does not emit still passes. This is
-#     a real residual hole and it was live in this repo: `kubevirt_vmi_phase_count{phase="Failed"}`
-#     (KubeVirtVMLaunchFailing) sat inside the scraped `kubevirt_` family on an UP target, and the
-#     series simply does not exist on the deployed KubeVirt — verified on prod 2026-09-13, where the
-#     only kubevirt_vmi_* names are `info` and `number_of_outdated`, and the only phase-bearing
-#     names are the phase_transition_time histograms. Corrected in the same change to
-#     `kubevirt_vmi_info{phase="failed"}` (note the LOWERCASE value KubeVirt writes: a capital-F
-#     matcher selects nothing and is just as inert). Catching that class needs the exporter's own
-#     metric list, which is a live read, not a static one — it belongs to a future L1/L3 check that
-#     diffs every alert metric against `/api/v1/label/__name__/values`. The same limit applies to
-#     every LABEL VALUE an alert matches on.
-#   - The metric-name parser is a PromQL approximation, not a parser: it strips string literals,
-#     label-matcher bodies and aggregation label lists, then treats remaining bare identifiers as
-#     metric selectors. A metric referenced ONLY through a `{__name__=~"..."}` regex selector would
-#     be missed. No rule in this repo does that, and assertion (0) fails if the parser stops
-#     finding metrics at all.
-#   - It does not judge thresholds, and does not prove an alert reaches a human (R-OBS-02 and the
-#     L3 alerting test own that).
-#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
-#
-# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, and because this test
-# declares the whole-repo scope the runner turns it into rc 3 = UNWIRED, never green) only if the
-# repo contains no PrometheusRule at all — i.e. there is genuinely nothing to audit.
-# =============================================================================
-set -euo pipefail
-DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
-PYTHON="${PYTHON:-python3}"
-
-# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
-if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then
-  echo "check SKIPPED: no PrometheusRule found under platform/ or apps/ — no alert rules to audit" >&2
-  exit 3
-fi
-
-cd "${ROOT}"
-"${PYTHON}" - "${ROOT}" <<'PY'
-"""Static audit: every alert rule's metrics are claimed by a scrape target this repo configures.
-
-PyYAML is used deliberately rather than the line-based parsing the neighbouring checks favour:
-alert `expr:` values are multi-line block scalars, and a line-based reader of PromQL would be the
-kind of approximate parser that silently stops matching — the exact failure class this check
-exists to prevent. Same dependency as catalog/render.py and dev-nodes-provisioned-as-code.
-"""
-from __future__ import annotations
-
-import re
-import sys
-from pathlib import Path
-
-try:
-    import yaml
-except ImportError:                                  # pragma: no cover
-    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
-          file=sys.stderr)
-    raise SystemExit(1)
-
-ROOT = Path(sys.argv[1])
-sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
-
-FAILURES: list[str] = []
-CHECKS = 0
-
-
-def check(cond, msg):
-    global CHECKS
-    CHECKS += 1
-    if not cond:
-        FAILURES.append(msg)
-
-
-def read(p: Path) -> str:
-    try:
-        return p.read_text()
-    except (OSError, UnicodeDecodeError):
-        return ""
-
-
-# =============================================================================================
-# THE DECLARED SCRAPE-OWNER TABLE (assertions 1 and 2).
-#
-# Each entry claims a set of metric names (exact names and/or prefixes) and names the file whose
-# declaration is what makes those metrics exist in the TSDB, plus the regexes that prove it. This
-# is the crossing from "an alert queries X" to "something in this repo scrapes X", and it is the
-# part that makes the check a REGRESSION guard rather than a one-time audit: every `require`
-# regex is re-evaluated against the working tree on every gate run.
-#
-# `require`  — ALL must match the file. The enablement itself.
-# `forbid`   — NONE may match. For enablements that can be defeated without being deleted.
-#
-# Longest prefix wins, so kube_velero_schedule_ is resolved before kube_.
-# =============================================================================================
-KPS = "platform/monitoring/kube-prometheus-stack.yaml"
-
-SCRAPE_OWNERS: list[dict] = [
-    {
-        "owner": "kube-prometheus-stack: bundled node-exporter DaemonSet + its ServiceMonitor",
-        "prefixes": ("node_",),
-        "file": KPS,
-        # The subchart block must be present and configured here (it is where the exporter's
-        # resources live). The chart default is enabled:true, so the regression to guard is a
-        # DISABLE, not a missing enable.
-        "require": [r"^\s*prometheus-node-exporter:\s*$"],
-        "forbid": [r"prometheus-node-exporter:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
-        "why": "node_* powers all seven platform-health.nodes rules including the two IO-pressure "
-               "rules that exist because of prod-1's etcd WAL starvation",
-    },
-    {
-        "owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)",
-        # container_* is cAdvisor, which the kubelet ServiceMonitor scrapes on the SAME target as
-        # kubelet_* via its /metrics/cadvisor path (this entry's title always said so; the prefix
-        # was added 2026-09-14 when ContainerIOStallHigh became the first rule to query it).
-        # Verified on prod: container_pressure_io_stalled_seconds_total, 405 series across the
-        # three nodes, job="kubelet", metrics_path="/metrics/cadvisor".
-        "prefixes": ("kubelet_", "container_"),
-        "file": KPS,
-        # Chart default kubelet.enabled:true. There is no explicit enable to require, so the only
-        # honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were.
-        "require": [r"kubeProxy:"],   # sentinel: the exporter-toggle block is still authored here
-        "forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
-        "why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three "
-               "pvc-capacity-alerts rules; container_pressure_io_stalled_seconds_total is the "
-               "only signal on this cluster that can name WHICH workload an I/O stall is hitting",
-    },
-    {
-        "owner": "kube-prometheus-stack: the kube-apiserver ServiceMonitor (chart default, NOT "
-                 "disabled the way kubeEtcd/kubeProxy are)",
-        # DELIBERATELY EXACT NAMES, NOT AN `etcd_` PREFIX. On this cluster the only etcd-shaped
-        # metrics that exist are the APISERVER'S CLIENT-SIDE view of its own local member. etcd's
-        # server-side disk metrics (etcd_disk_wal_fsync_duration_seconds,
-        # etcd_disk_backend_commit_duration_seconds) are NOT scraped — kubeEtcd.enabled:false for
-        # k3s' embedded etcd — and verified absent on prod 2026-09-14: a query for
-        # {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing at all. A blanket
-        # `etcd_` claim would let a future rule on those names pass this check while being inert,
-        # which is precisely the hole this test exists to close. Widen only with a real scrape.
-        "exact": (
-            "etcd_request_duration_seconds_bucket",
-            "etcd_request_duration_seconds_count",
-        ),
-        "file": KPS,
-        "require": [r"kubeProxy:"],   # sentinel: the exporter-toggle block is still authored here
-        "forbid": [r"\n\s*kubeApiServer:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
-        "why": "etcd_request_duration_seconds_* is the ONLY per-node view of the etcd storage path "
-               "this cluster has, and NodeEtcdRequestsSlow is the only alert that can see the "
-               "condition that took k3s-prod-1 NotReady on 2026-09-07 before the node drops",
-    },
-    {
-        "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules",
-        "prefixes": ("kube_velero_schedule_",),
-        "file": KPS,
-        "require": [
-            r"customResourceState:\s*\n\s*enabled:\s*true",
-            r"metricNamePrefix:\s*kube_velero_schedule",
-            # The RBAC is MANDATORY, not cosmetic: without velero.io read access KSM emits the
-            # custom-resource-state metrics as an EMPTY SET and logs a forbidden error — silent,
-            # and the inert-alert failure mode all over again (measured; see the block's header).
-            r'apiGroups:\s*\["velero\.io"\]',
-            r'resources:\s*\["schedules"\]',
-        ],
-        "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
-        "why": "kube_velero_schedule_* is the only thing that can see a Schedule that has NEVER "
-               "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
-               "Velero itself publishes nothing until a backup is attempted",
-    },
-    {
-        "owner": "kube-prometheus-stack: bundled kube-state-metrics",
-        "prefixes": ("kube_",),
-        "file": KPS,
-        "require": [r"^\s*kube-state-metrics:\s*$"],
-        "forbid": [r"kube-state-metrics:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
-        # kube_node_labels is a SPECIAL CASE and is checked on its own below: KSM exposes no
-        # custom node labels by default, so the metric does not exist unless metricLabelsAllowlist
-        # names the label. The prefix claim alone would pass while VPNTunnelDown's join matched
-        # nothing.
-        "why": "kube_* powers the Keycloak, KubeVirt-crashloop, CNPG-colocation and Velero-schedule "
-               "rules",
-    },
-    {
-        "owner": "kube-state-metrics metricLabelsAllowlist (the egress-gateway node label)",
-        "exact": ("kube_node_labels",),
-        "file": KPS,
-        "require": [
-            r"metricLabelsAllowlist:",
-            r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
-        ],
-        "forbid": [],
-        "why": "KSM exposes NO custom labels by default. Without the allowlist entry "
-               "kube_node_labels carries no egress-gateway label and VPNTunnelDown's join silently "
-               "matches nothing — a rule that is present, evaluating, and structurally inert",
-    },
-    {
-        "owner": "CNPG: enablePodMonitor on the Cluster CRs (the operator renders one PodMonitor each)",
-        "prefixes": ("cnpg_",),
-        "file": "platform/keycloak/postgres-cluster.yaml",
-        "require": [r"^\s*enablePodMonitor:\s*true\s*$"],
-        "forbid": [],
-        "why": "cnpg_* powers all six platform-health.cnpg rules, including the WAL-archive backlog "
-               "and no-recoverability-point rules that guard R-STATE-03",
-    },
-    {
-        "owner": "Flux controllers: platform/monitoring/flux-podmonitor.yaml (merged a79693d)",
-        "prefixes": ("gotk_",),
-        "file": "platform/monitoring/flux-podmonitor.yaml",
-        "require": [r"kind:\s*PodMonitor", r"port:\s*http-prom"],
-        "forbid": [],
-        "why": "gotk_* powers FluxReconciliationFailure, FluxResourceSuspended and the three "
-               "automated-update rules — on a platform whose entire delivery model is Flux",
-    },
-    {
-        "owner": "cert-manager: prometheus.servicemonitor.enabled in the HelmRelease",
-        "prefixes": ("certmanager_",),
-        "file": "platform/cert-manager/helmrelease.yaml",
-        "require": [r"prometheus:\s*\n\s*servicemonitor:\s*\n\s*enabled:\s*true"],
-        # The chart calls `fail` when both monitor kinds are enabled, which would break the whole
-        # HelmRelease rather than just the scrape. Measured against v1.21.2.
-        "forbid": [r"podmonitor:\s*\n\s*enabled:\s*true"],
-        "why": "certmanager_* powers CertificateExpiringSoon / ExpiryImminent / NotReady — the only "
-               "warning that an ACME DNS-01 renewal of the live wildcard has quietly failed",
-    },
-    {
-        "owner": "Longhorn: metrics.serviceMonitor.enabled in the HelmRelease",
-        "prefixes": ("longhorn_",),
-        "file": "platform/longhorn/helmrelease.yaml",
-        "require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
-        "forbid": [],
-        "why": "longhorn_* powers the four storage rules on a cluster whose recurring failure mode "
-               "is a disk",
-        # The ServiceMonitor ALONE is not sufficient here — see the companion assertion below.
-        "companion": {
-            "file": "platform/longhorn/kustomization.yaml",
-            "require": [r"^\s*-\s*networkpolicy-metrics\.yaml\s*$"],
-            "why": "Longhorn 1.12.1 ships its own effectively-unconditional NetworkPolicies "
-                   "(guarded on networkPolicies.restrictInternalTraffic, default true — NOT on "
-                   "networkPolicies.enabled) whose longhorn-manager policy admits only other "
-                   "Longhorn components in the same namespace. MEASURED on prod: with the "
-                   "ServiceMonitor applied and no allowance, all three targets were DISCOVERED and "
-                   "every scrape failed with 'context deadline exceeded' after 10.0s — a drop, not "
-                   "a refusal — and the TSDB stayed empty. So without networkpolicy-metrics.yaml "
-                   "the four storage alerts are inert even though the monitor exists",
-        },
-    },
-    {
-        "owner": "KubeVirt/CDI: platform/kubevirt/servicemonitor.yaml",
-        "prefixes": ("kubevirt_",),
-        "file": "platform/kubevirt/servicemonitor.yaml",
-        "require": [r"kind:\s*ServiceMonitor", r"port:\s*metrics"],
-        "forbid": [],
-        "why": "kubevirt_* powers the four platform-health.kubevirt rules (note the CDI rules use "
-               "the kubevirt_cdi_* names, not bare cdi_*)",
-    },
-    {
-        "owner": "Velero: metrics.serviceMonitor.enabled in the HelmRelease",
-        "prefixes": ("velero_",),
-        "file": "platform/velero/helmrelease.yaml",
-        "require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
-        "forbid": [],
-        "why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
-               "that failed to notice prod running with zero volume backups",
-    },
-]
-
-# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
-# is structurally blind to `up{job="X"}` naming a job nothing produces. That axis is assertion (4).
-UNIVERSAL = {"up"}
-
-# =============================================================================================
-# REGISTER 1 — KNOWN_UNWIRED_METRICS. Shrink-only. A metric here is queried by an alert and
-# claimed by NOTHING, so that alert is inert and the platform is blind to what it describes.
-# Removing an entry means wiring the scrape. An entry that stops being referenced by any alert,
-# or that gains an owner, FAILS as stale.
-# =============================================================================================
-KNOWN_UNWIRED_METRICS: dict[str, str] = {
-    "hubble_drop_total":
-        "HubbleDeniedFlowSpike — the platform has NO policy-drop / NetworkPolicy-regression "
-        "visibility. Unlike Longhorn and cert-manager (whose charts already serve /metrics and "
-        "only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is switched on "
-        "in Cilium's own config: platform/cilium/helmrelease.yaml sets hubble.{enabled,relay,ui} "
-        "but not hubble.metrics.enabled, and not prometheus.enabled for the agent. Turning them on "
-        "OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all three nodes "
-        "— the single most dangerous rollout on this cluster (the 2026-09-01 k8sServiceHost outage; "
-        "cilium was moved to its own Flux Kustomization precisely to isolate that blast radius) — "
-        "and requires a matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same "
-        "commit or a rebuilt node's bootstrap CNI config silently disagrees with the reconciled "
-        "one. Deliberately DEFERRED to its own change with a one-node canary; see the rule's "
-        "header in platform/monitoring/alerts.yaml.",
-}
-
-# =============================================================================================
-# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only, and NOW EMPTY.
-#
-# Each entry was a `up{job="X"}` selector whose monitor named an endpoint port the Service it
-# selects did not define, so the monitor discovered ZERO targets, no `up{job="X"}` series existed,
-# and `up{job="X"} == 0` could never fire. The nastiest variant of the class: `up` exists in
-# abundance, so nothing about the metric name looks wrong, and `PrometheusTargetsMissing` cannot see
-# it either — a target that was never discovered has no up series to be 0.
-#
-# IT HELD ALL TWELVE apps/* SLI RULES, AND IT IS EMPTY BECAUSE THEY WERE FIXED, not because the gap
-# was waived (2026-09-13, branch fix/app-sli-rules-discover-no-targets). The register's own note
-# said closing it needed "a Prometheus exporter per app". That was the wrong conclusion from a true
-# premise: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud
-# genuinely export no Prometheus metrics, so instead the `up` SLI was rewritten against a signal that
-# DOES exist for every workload — kube-state-metrics readiness of the service's own workload object
-# (`kube_deployment_status_replicas_available` / `kube_statefulset_status_replicas_ready`), and
-# `kubevirt_vmi_info{phase="running"}` for the VM. No exporter, no new scrape target; each rule is
-# unit-tested firing AND quiet (apps/wave-*/tests/app-slis.test.yaml) and each expression was
-# evaluated live on prod, where eleven of the twelve return no series on a healthy cluster and the
-# twelfth (vm-demo) is not deployed at all.
-#
-# The rendered ServiceMonitors went with them: a monitor for a workload that serves no metrics is
-# dead configuration that reads as coverage. So assertion (4) below now has NOTHING to audit under
-# apps/ — which is why the monitor-side axis moved into its own check,
-# tests/l0-static/servicemonitor-ports-resolve. THAT is where a ServiceMonitor naming a port no
-# Service defines fails, with no alert rule required, resolving by selector across the whole tree
-# rather than by same-file adjacency, and with its resolver proven on fixtures so it cannot pass for
-# lack of subjects. Keeping this register non-empty as "documentation" of a closed gap is exactly the
-# fiction the ratchet forbids: the stale-entry assertion below would fail, and rightly.
-# =============================================================================================
-KNOWN_UNWIRED_JOBS: dict[str, str] = {}
-
-# =============================================================================================
-# (0) PARSE EVERY ALERT RULE
-# =============================================================================================
-# PromQL metric-name approximation. Documented as a gap in the header: string literals,
-# label-matcher bodies and aggregation/vector-matching label lists are stripped first, then every
-# remaining bare identifier that is not a PromQL keyword or function is treated as a metric
-# selector. Assertion (0b) fails loudly if this stops finding metrics at all, which is the way an
-# approximate parser silently stops guarding.
-PROMQL_KEYWORDS = set("""
-abs absent absent_over_time ceil changes clamp clamp_max clamp_min day_of_month day_of_week
-day_of_year days_in_month delta deriv exp floor histogram_quantile histogram_count histogram_sum
-histogram_fraction histogram_stddev histogram_stdvar holt_winters hour idelta increase irate
-label_join label_replace ln log2 log10 minute month predict_linear rate resets round scalar sgn
-sort sort_desc sort_by_label sort_by_label_desc sqrt time timestamp vector year avg count
-count_values group max min quantile stddev stdvar sum topk bottomk limitk limit_ratio
-last_over_time avg_over_time min_over_time max_over_time sum_over_time count_over_time
-quantile_over_time stddev_over_time stdvar_over_time present_over_time mad_over_time
-double_exponential_smoothing info ts_of_max_over_time ts_of_min_over_time ts_of_last_over_time
-by without on ignoring group_left group_right offset bool and or unless if atan2 start end
-inf nan NaN Inf
-""".split())
-
-IDENT_RE = re.compile(r"(?<![\w:.])([a-zA-Z_:][a-zA-Z0-9_:]*)")
-UP_JOB_RE = re.compile(r"""\bup\s*\{[^}]*\bjob\s*=\s*["']([^"']+)["']""")
-
-
-def metric_names(expr: str) -> list[str]:
-    e = re.sub(r'"(?:[^"\\]|\\.)*"', '""', expr)
-    e = re.sub(r"'(?:[^'\\]|\\.)*'", "''", e)
-    e = re.sub(r"`(?:[^`\\]|\\.)*`", "``", e)
-    e = re.sub(r"\{[^{}]*\}", " ", e)               # label matchers
-    for _ in range(6):                               # nested aggregations
-        e = re.sub(r"\b(?:by|without|on|ignoring|group_left|group_right)\s*\([^()]*\)", " ", e)
-    out, seen = [], set()
-    for m in IDENT_RE.finditer(e):
-        n = m.group(1)
-        if n in PROMQL_KEYWORDS or re.fullmatch(r"\d+[smhdwy]?", n) or n in seen:
-            continue
-        seen.add(n)
-        out.append(n)
-    return out
-
-
-def is_golden_fixture(p: Path) -> bool:
-    """conformance/tests/**/golden/** holds EXPECTED-OUTPUT fixtures, not deployed manifests.
-
-    conformance/tests/l1-rendering/egress-class-render/golden/*.expected.yaml each contain a
-    rendered `<app>-down` PrometheusRule. They are the assertion data of another test, never
-    reconciled into a cluster, so auditing their scrape targets would be auditing a copy — the
-    same "reference, not declaration" distinction ci/touched-requirements.sh draws for ci/tests/
-    fixture paths. The templates they are rendered FROM are audited through apps/ and catalog/.
-    """
-    parts = p.parts
-    return "conformance" in parts and "golden" in parts
-
-
-rule_files = sorted(
-    p for p in ROOT.rglob("*.yaml")
-    if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
-)
-check(len(rule_files) > 0,
-      "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
-      "this check would silently audit nothing")
-
-# metric -> [ "<relpath>:<alert>", ... ] ; and the same shape for up{job="X"} selectors
-refs: dict[str, list[str]] = {}
-job_refs: dict[str, list[str]] = {}
-n_rules = 0
-for f in rule_files:
-    rel = str(f.relative_to(ROOT))
-    try:
-        docs = list(yaml.safe_load_all(read(f)))
-    except yaml.YAMLError as exc:
-        check(False, f"{rel} is not parseable YAML ({exc}); every alert in it is unaudited")
-        continue
-    for d in docs:
-        if not isinstance(d, dict) or d.get("kind") != "PrometheusRule":
-            continue
-        for grp in (d.get("spec") or {}).get("groups") or []:
-            for r in grp.get("rules") or []:
-                if "alert" not in r:
-                    continue                         # recording rules: not this check's subject
-                n_rules += 1
-                site = f"{rel}:{r['alert']}"
-                expr = str(r.get("expr", ""))
-                for n in metric_names(expr):
-                    refs.setdefault(n, []).append(site)
-                for j in UP_JOB_RE.findall(expr):
-                    job_refs.setdefault(j, []).append(site)
-
-check(n_rules > 0,
-      "no alerting rule parsed out of any PrometheusRule — the parse contract broke and this "
-      "check would pass vacuously while every alert's metric went unaudited")
-check(len(refs) >= 20,
-      f"only {len(refs)} distinct metric name(s) parsed out of {n_rules} alerting rule(s). The "
-      f"PromQL approximation has stopped matching — which is exactly how an approximate parser "
-      f"turns into a guard that passes while checking nothing. Fix metric_names() or replace it "
-      f"with a real parser")
-
-# =============================================================================================
-# (1)(2) RESOLVE EVERY METRIC TO AN OWNER, AND PROVE THE OWNER'S CLAIM AGAINST THE REPO
-# =============================================================================================
-def resolve(metric: str) -> dict | None:
-    """Longest-prefix / exact-name match against SCRAPE_OWNERS."""
-    best, best_len = None, -1
-    for o in SCRAPE_OWNERS:
-        if metric in o.get("exact", ()):            # exact names win outright
-            return o
-        for pfx in o.get("prefixes", ()):
-            if metric.startswith(pfx) and len(pfx) > best_len:
-                best, best_len = o, len(pfx)
-    return best
-
-
-# Every owner's FILE must exist, whether or not a metric currently lands on it: an owner pointing
-# at a deleted file is a claim nothing backs.
-for o in SCRAPE_OWNERS:
-    check((ROOT / o["file"]).is_file(),
-          f"SCRAPE_OWNERS names {o['file']} as the scrape owner for "
-          f"{o.get('prefixes') or o.get('exact')}, and that file does not exist in this repo — the "
-          f"claim is unbacked and the alerts that depend on it are inert")
-
-claimed: dict[str, dict] = {}
-unowned: list[str] = []
-for metric in sorted(refs):
-    if metric in UNIVERSAL:
-        continue
-    o = resolve(metric)
-    if o is None:
-        if metric in KNOWN_UNWIRED_METRICS:
-            continue                                 # declared gap; ratcheted below
-        unowned.append(metric)
-        check(False,
-              f"alert metric `{metric}` (queried by {', '.join(sorted(set(refs[metric])))}) is "
-              f"claimed by NO entry in SCRAPE_OWNERS. Nothing in this repo configures a scrape "
-              f"that produces it, so every rule above is STRUCTURALLY INERT: it evaluates to no "
-              f"vector, stays silent forever, and is indistinguishable from a healthy result. "
-              f"Either wire the scrape and add the owner entry, or — if it cannot be wired yet — "
-              f"declare it in KNOWN_UNWIRED_METRICS with the reason, so the gap is reported "
-              f"instead of silent")
-        continue
-    check(metric not in KNOWN_UNWIRED_METRICS,
-          f"`{metric}` is listed in KNOWN_UNWIRED_METRICS but is now claimed by "
-          f"{o['owner']} — the register is shrink-only: delete the entry so the next real gap is "
-          f"the one that shows up")
-    claimed[metric] = o
-
-# The owners that are actually load-bearing right now get their declarations verified.
-for o in SCRAPE_OWNERS:
-    if o not in claimed.values():
-        continue                                     # nothing queries it today; nothing to guard
-    body = read(ROOT / o["file"])
-    for rx in o.get("require", []):
-        check(re.search(rx, body, re.M) is not None,
-              f"{o['file']} no longer matches /{rx}/, which is what makes {o['owner']} scrape. "
-              f"Removing it re-inerts every alert on "
-              f"{o.get('prefixes') or o.get('exact')} — {o['why']}")
-    for rx in o.get("forbid", []):
-        check(re.search(rx, body, re.M) is None,
-              f"{o['file']} matches the FORBIDDEN pattern /{rx}/, which defeats {o['owner']} "
-              f"even though the enablement is still authored. {o['why']}")
-    comp = o.get("companion")
-    if comp:
-        cbody = read(ROOT / comp["file"])
-        check((ROOT / comp["file"]).is_file(),
-              f"{comp['file']} is missing — {comp['why']}")
-        for rx in comp.get("require", []):
-            check(re.search(rx, cbody, re.M) is not None,
-                  f"{comp['file']} no longer matches /{rx}/. The monitor alone is NOT enough here: "
-                  f"{comp['why']}")
-
-# The Longhorn NetworkPolicy must actually grant the path it exists to grant, and no more: the
-# whole point of splitting it out instead of flipping networkPolicies.restrictInternalTraffic off.
-NP = ROOT / "platform/longhorn/networkpolicy-metrics.yaml"
-if any(m.startswith("longhorn_") for m in refs):
-    np = read(NP)
-    check(np != "", f"{NP.relative_to(ROOT)} is missing; the Longhorn scrape is dropped and its "
-                    f"four alerts are inert")
-    check(re.search(r"kind:\s*NetworkPolicy", np) is not None,
-          "platform/longhorn/networkpolicy-metrics.yaml is not a NetworkPolicy")
-    check(re.search(r"app:\s*longhorn-manager", np) is not None,
-          "the Longhorn metrics NetworkPolicy does not select app=longhorn-manager, so it does not "
-          "open the pods that serve /metrics")
-    check(re.search(r"kubernetes\.io/metadata\.name:\s*monitoring", np) is not None,
-          "the Longhorn metrics NetworkPolicy does not admit namespace monitoring, so Prometheus "
-          "still cannot reach longhorn-manager and the storage alerts stay inert")
-    check(re.search(r"port:\s*9500", np) is not None,
-          "the Longhorn metrics NetworkPolicy does not name port 9500 (longhorn-manager's `manager` "
-          "port, which serves /metrics)")
-    # LEAST PRIVILEGE (R-SEC-02): ingress only, one port, and NOT the webhook/recovery ports.
-    check(re.search(r"policyTypes:\s*\n\s*-\s*Ingress\s*$", np, re.M) is not None
-          and "Egress" not in np,
-          "the Longhorn metrics NetworkPolicy is not Ingress-only — it must grant exactly the "
-          "scrape path and nothing else (R-SEC-02)")
-    # Comments are stripped first: the file's header deliberately NAMES 9502/9503 to record that
-    # they stay closed, and a substring search over prose would fail on the documentation of the
-    # very property it is asserting.
-    np_spec = "\n".join(re.sub(r"#.*$", "", ln) for ln in np.splitlines())
-    for closed in ("9502", "9503"):
-        check(closed not in np_spec,
-              f"the Longhorn metrics NetworkPolicy names port {closed} (admission webhook / "
-              f"recovery backend). Only 9500 serves metrics; opening the others to ns monitoring "
-              f"is reach this observability path does not need (R-SEC-02)")
-
-# stale KNOWN_UNWIRED_METRICS entries
-for m in sorted(KNOWN_UNWIRED_METRICS):
-    check(m in refs,
-          f"KNOWN_UNWIRED_METRICS declares `{m}`, which NO alert rule in this repo queries any "
-          f"more — a stale exemption hides the next real gap; delete the entry")
-
-# =============================================================================================
-# (4) up{job="X"} — the axis assertion (1) is structurally blind to.
-#
-# Scoped to files where THIS REPO authors both the Service and the ServiceMonitor (the apps/*
-# ServiceDefinition pattern), because only then is the port name inside the tree. A monitor whose
-# Service comes from a Helm chart cannot be audited statically and is out of scope (header gap).
-# =============================================================================================
-def yaml_docs(path: Path) -> list[dict]:
-    try:
-        return [d for d in yaml.safe_load_all(read(path)) if isinstance(d, dict)]
-    except yaml.YAMLError:
-        return []
-
-
-# job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
-same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
-for p in sorted(ROOT.rglob("*.yaml")):
-    if ".git" in p.parts:
-        continue
-    txt = read(p)
-    if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:
-        continue
-    docs = yaml_docs(p)
-    monitors = [d for d in docs if d.get("kind") == "ServiceMonitor"]
-    services = [d for d in docs if d.get("kind") == "Service"]
-    if not monitors or not services:
-        continue
-    svc_ports: set[str] = set()
-    for s in services:
-        for port in (s.get("spec") or {}).get("ports") or []:
-            if isinstance(port, dict) and port.get("name"):
-                svc_ports.add(str(port["name"]))
-    for m in monitors:
-        ep_ports = {str(e["port"]) for e in (m.get("spec") or {}).get("endpoints") or []
-                    if isinstance(e, dict) and e.get("port")}
-        name = (m.get("metadata") or {}).get("name")
-        if name:
-            same_file_monitors[str(name)] = (str(p.relative_to(ROOT)), ep_ports, svc_ports)
-
-for job in sorted(job_refs):
-    sites = ", ".join(sorted(set(job_refs[job])))
-    mon = same_file_monitors.get(job)
-    if mon is None:
-        # No same-file Service+ServiceMonitor pair is named for this job. It may well be produced
-        # by a platform monitor against a chart Service (oauth2-proxy, kubevirt, velero, flux all
-        # are), which this check cannot see — so this is NOT a failure, only unauditable.
-        continue
-    rel, ep_ports, svc_ports = mon
-    missing = sorted(ep_ports - svc_ports)
-    if missing:
-        check(job in KNOWN_UNWIRED_JOBS,
-              f"`up{{job=\"{job}\"}}` ({sites}) can never produce a series: {rel}'s ServiceMonitor "
-              f"scrapes port(s) {missing}, and the Service(s) in that same file define only "
-              f"{sorted(svc_ports) or ['(no named ports)']}. The monitor therefore discovers ZERO "
-              f"targets, there is no `up{{job=\"{job}\"}}` to compare against 0, and the rule is "
-              f"structurally inert — note PrometheusTargetsMissing cannot see this either, because "
-              f"a target that was never discovered has no up series to be 0. Give the Service a "
-              f"named port the workload actually serves metrics on (and an exporter if it has "
-              f"none), or declare it in KNOWN_UNWIRED_JOBS")
-    else:
-        check(job not in KNOWN_UNWIRED_JOBS,
-              f"`{job}` is listed in KNOWN_UNWIRED_JOBS but {rel}'s ServiceMonitor port(s) "
-              f"{sorted(ep_ports)} now all exist on its Service — the register is shrink-only: "
-              f"delete the entry")
-
-# stale KNOWN_UNWIRED_JOBS entries
-for job in sorted(KNOWN_UNWIRED_JOBS):
-    check(job in job_refs,
-          f"KNOWN_UNWIRED_JOBS declares `{job}`, which no alert rule selects via up{{job=...}} any "
-          f"more — a stale exemption hides the next real gap; delete the entry")
-
-# =============================================================================================
-# (5) THE CHECK IS LOAD-BEARING. Asserted against the DECLARATIVE mechanism, in both halves —
-# neither alone is sufficient. The per-test hand-written ci/l0.sh stage is deliberately NOT used
-# (whole-repo-invariants-wired FAILS the gate if one reappears); the property is DECLARED here and
-# HONOURED by ci/l0.sh's --include-scope union.
-# =============================================================================================
-SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo"
-SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "alert-metrics-have-scrape-targets"
-            / "chainsaw-test.yaml")
-check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
-      "alert-metrics-have-scrape-targets/chainsaw-test.yaml does not declare "
-      "conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
-      "change happens to touch R-OBS-04. ci/touched-requirements.sh maps apps/* to NOTHING and "
-      "platform/monitoring/* to R-OBS-01/R-OBS-02 — so adding an alert under apps/ or flipping a "
-      "chart's serviceMonitor.enabled back to false would be filtered out of exactly the change "
-      "that re-inerts the alerts")
-L0 = read(ROOT / "ci" / "l0.sh")
-GATE = read(ROOT / "ci" / "gate.sh")
-check("--include-scope" in L0,
-      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
-      "purely requirement-scoped run again and the scope annotation above is inert decoration")
-check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None
-      and re.search(r"fail\).*BLOCK=1", GATE) is not None,
-      "ci/gate.sh does not set the merge block when the l0 stage fails")
-
-# ======================================================================== report
-owners_used = {id(o): o for o in claimed.values()}
-print(f"Alert metric inventory: {n_rules} alerting rule(s) across {len(rule_files)} "
-      f"PrometheusRule file(s) -> {len(refs)} distinct metric name(s)")
-print(f"  claimed by a declared scrape owner: {len(claimed)}/{len(refs) - len(UNIVERSAL & set(refs))}"
-      f"   owners exercised: {len(owners_used)}/{len(SCRAPE_OWNERS)}")
-for o in SCRAPE_OWNERS:
-    ms = sorted(m for m, oo in claimed.items() if oo is o)
-    mark = "  " if ms else "· "
-    print(f"  {mark}{o['file']:<46} {len(ms):>2} metric(s)  {o['owner'][:64]}")
-print(f"\n  DECLARED GAPS (shrink-only registers):")
-print(f"    KNOWN_UNWIRED_METRICS: {len(KNOWN_UNWIRED_METRICS)} — "
-      f"{', '.join(sorted(KNOWN_UNWIRED_METRICS))}")
-print(f"    KNOWN_UNWIRED_JOBS:    {len(KNOWN_UNWIRED_JOBS)} inert up{{job=...}} SLI rule(s) — "
-      f"{', '.join(sorted(KNOWN_UNWIRED_JOBS))}")
-if unowned:
-    print(f"\n  UNCLAIMED (inert, undeclared): {', '.join(unowned)}", file=sys.stderr)
-
-print()
-if not FAILURES:
-    print(f"check OK: alert-metrics-have-scrape-targets — {CHECKS} assertions (R-OBS-04)")
-    raise SystemExit(0)
-print(f"CHECK FAIL (alert-metrics-have-scrape-targets): {len(FAILURES)} failed, "
-      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
-for f in FAILURES:
-    print(f"  FAIL {f}", file=sys.stderr)
-raise SystemExit(1)
-PY

conformance/tests/l0-static/alert-rules-unit-tested/chainsaw-test.yaml

@@ -1,95 +0,0 @@
-# =============================================================================
-# L0 STATIC — the alert rules are UNIT-TESTED, by a pinned engine, and the gaps are declared
-# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
-#
-# R-OBS-04 ("the platform alerts on its own health") is implemented by PrometheusRule CRs, and
-# this repo has shipped both ways they fail SILENTLY:
-#   * PERMANENTLY FIRING — the CNPG primaries alert (fixed 2026-09-12), VPNTunnelDown before it;
-#   * STRUCTURALLY INERT — VeleroNoRecentBackup, whose `time() - <absent series>` yields no
-#     series, so a schedule that had NEVER backed up could not raise it. Prod ran with zero
-#     volume backups and nothing alerted, masked by a resources-only schedule completing
-#     cleanly every night (2026-09-12).
-# An inert rule and a correct rule are indistinguishable while the platform is healthy. Only
-# `promtool test rules` separates them, by constructing the bad state on purpose, offline.
-#
-# The unit tests were then written — and RUN BY NOTHING. promtool was not in the pinned
-# toolchain, no gate stage evaluated the files, and the two harnesses that grew up independently
-# each fetched an engine off the network mid-run if promtool was missing (a GitHub release
-# download in one, a prometheus container image in the other). Coverage that looks present and
-# is enforced nowhere is the failure mode this test ends.
-#
-# GIVEN the repository as authored,
-# WHEN  the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
-#       COMMITTED PrometheusRule CRs into the rules file promtool wants — never a copy) is run
-#       over every test file in the repo with the PINNED promtool,
-# THEN  promtool is pinned in BOTH versions.env files in lockstep at the Prometheus version prod
-#       actually runs, baked into the image and installed on the CI runner; the promtool in use
-#       reports exactly that version; no harness acquires an engine at run time; there is
-#       exactly ONE harness; every rule-unit test file PASSES; the harness is proven able to
-#       FAIL by a mutated synthetic test on every run; and every alert rule with NO unit test is
-#       ENUMERATED in a shrink-only register, so the gap is reported rather than hidden.
-#
-# MISSING TESTS: RATCHET, NOT SKIP, NOT BLANKET-FAIL. 48 of 55 alert rules have no unit test
-# today. Failing on all of them would make the gate red until ~48 tests exist (so it would be
-# disabled, and a disabled gate protects nothing); merely reporting is precisely how the CNPG
-# tests came to be enforced by nothing. So the gap is DECLARED in check.sh, printed with counts
-# every run, and CANNOT GROW: a new PrometheusRule, or a new alert in an already-tested file,
-# FAILS without a test, and a stale register entry FAILS too. A missing test is never an exit-3
-# skip — "SKIPPED is not green" is preserved for the one thing that is genuinely an environment
-# gap: promtool being absent.
-#
-# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (standard library + the harness
-# it delegates to); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
-# Chainsaw Test wraps the SAME script.
-#
-# NOT proven (named in check.sh's header): that an alert reaches a human (Alertmanager routing
-# and SMTP are L3 / R-OBS-02), that a threshold is the right one, that a rule file is actually
-# reconciled into a cluster, or that an author's assertions are meaningful — coverage of one
-# alert is detected by its name appearing in a covering test file, and only the mutation
-# self-test proves the harness CAN fail. R-OBS-04 is tagged for the static slice only: the rules
-# the repo ships behave, under the engine prod runs, as their own unit tests say.
-# =============================================================================
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: alert-rules-unit-tested
-  labels:
-    conformance.home.k3s/R-OBS-04: "true"
-    conformance.home.k3s/R-TEST-02: "true"
-  annotations:
-    conformance.home.k3s/requirements: "R-OBS-04, R-TEST-02"
-    conformance.home.k3s/layer: "L0"
-    # WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree
-    # (apps/wave-0..2, platform/cnpg, platform/kubevirt, platform/monitoring), and the register
-    # it maintains is a property of the WHOLE repo, not of any one path.
-    #
-    # WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map keys
-    # apps/* to NOTHING and platform/monitoring/* to R-OBS-01/02 — not R-OBS-04. So adding a new
-    # untested PrometheusRule under apps/, or editing one under platform/kubevirt/, attributes
-    # no requirement this test carries, and a requirement-scoped selection would filter the
-    # check out of exactly the change that adds the untested alert. The same blind spot that let
-    # `prod-baremetal-not-dev-surface` be filtered out of the run that gated its own breakage
-    # (DECISIONS-LOG 2026-09-12).
-    #
-    # Declaring the scope also makes an honest SKIP loud: lib/run_l1_offline.py turns a rc-3
-    # skip of a scope-declaring check into the runner's rc 3, and ci/l0.sh into the gate's rc 3.
-    # So a gate run on a toolchain WITHOUT promtool reports UNWIRED instead of green — which is
-    # the whole reason this check exists rather than another hand-written ci/l0.sh stage.
-    conformance.home.k3s/scope: "whole-repo"
-    conformance.home.k3s/safety-critical: "false"
-    conformance.home.k3s/executable: "true"
-    conformance.home.k3s/intent: >-
-      Every committed alert rule that has unit tests passes them under the pinned promtool that
-      matches prod's Prometheus, exactly one harness runs them, nothing fetches a rule engine at
-      run time, the harness is proven able to fail, and every alert rule without a unit test is
-      enumerated in a register that can only shrink.
-spec:
-  steps:
-    - name: run-every-alert-rule-unit-test-and-report-the-gaps
-      description: >-
-        Assert the promtool pin is in lockstep and baked, run the single rule-unit harness over
-        every test file in the repo, prove it can fail via a mutated synthetic test, and
-        enumerate every PrometheusRule with no unit tests against the shrink-only register.
-      try:
-        - script:
-            content: bash ./check.sh

conformance/tests/l0-static/alert-rules-unit-tested/check.sh

@@ -1,545 +0,0 @@
-#!/usr/bin/env bash
-# =============================================================================
-# L0 static check (offline, no cluster) — alert-rules-unit-tested
-# Serves: R-OBS-04 ("Platform alerts on its own health (nodes, storage, GitOps, certs,
-#                    VPN)." — spec docs/10-observability-and-alerting.md)
-#         R-TEST-02 ("Tests are reproducible: deterministic, hermetic, single-command."
-#                    — spec docs/14-testing-and-conformance.md)
-#
-# WHY THIS EXISTS — AN UNTESTED ALERT IS NOT A MONITORING CONTROL
-# --------------------------------------------------------------
-# This repo has shipped BOTH halves of the alerting failure space to prod:
-#   * a PERMANENTLY-FIRING `critical` (the CNPG alert fixed 2026-09-12; VPNTunnelDown before
-#     it), and
-#   * a STRUCTURALLY INERT rule — VeleroNoRecentBackup, whose `time() - <absent series>`
-#     yields no series, so a schedule that had NEVER backed up could not raise it. Prod ran
-#     with no volume backups at all and nothing alerted (2026-09-12).
-# Neither is visible from the expression, from `kubectl apply`, or from a healthy cluster: an
-# inert rule and a correct rule are indistinguishable while everything is fine. `promtool test
-# rules` is the only tool that separates them, because the BAD state can be constructed on
-# purpose. Unit tests for those rules were then WRITTEN — and evaluated by nothing: promtool
-# was not in the pinned toolchain, no gate stage ran the files, and each of the two harnesses
-# that existed acquired an engine off the network mid-run if one was missing. Coverage that
-# looks present and is never enforced is the failure mode this check ends.
-#
-# GIVEN the repository as authored,
-# WHEN  the ONE rule-unit harness (platform/monitoring/tests/run.sh, which flattens the
-#       COMMITTED PrometheusRule CRs — never a copy) is asked which manifests are under test
-#       and then run over every test file in the repo,
-# THEN  (1) promtool is a PINNED member of the toolchain: PROMTOOL_VERSION is declared in BOTH
-#           infra/devcontainer/versions.env and ci/versions.env and the two AGREE (the repo's
-#           lockstep rule), the Containerfile takes it as a build ARG, installs it from the
-#           pinned prometheus release and records it for provenance, dev.sh passes it, and the
-#           CI runner installs it for the l0 stage;
-#       (2) the promtool ACTUALLY IN USE reports exactly that pinned version — a unit test is
-#           evidence only if the engine matches the one prod evaluates rules with;
-#       (3) NO harness acquires a rule engine at RUN TIME — neither `curl | tar -xz` of a
-#           GitHub release nor a `podman/docker run` of a prometheus image. Both existed on
-#           2026-09-12 and both are removed; this is the regression guard;
-#       (4) there is exactly ONE harness. Two were written independently within an hour of
-#           each other; the one nothing runs is the one that rots;
-#       (5) every rule-unit test file in the repo PASSES, run through that harness (which
-#           `promtool check rules` first, so a non-parsing rules file cannot make unit tests
-#           pass vacuously by finding no rules at all);
-#       (6) the harness is NOT VACUOUS: a synthetic PrometheusRule + companion test is run
-#           through the SAME harness twice — intact (must PASS) and with one expectation
-#           mutated (must FAIL) — so a harness that has silently stopped evaluating anything
-#           is caught on every run, not only when someone thinks to try it;
-#   AND (7) the alert rules with NO unit tests are ENUMERATED and RATCHETED: each is declared
-#           below, printed with counts on every run, and the register can only SHRINK.
-#
-# WHY A MISSING TEST IS A RATCHET AND NOT AN OUTRIGHT FAIL (the decision, and its reasons)
-# ---------------------------------------------------------------------------------------
-# 48 of the repo's 55 alert rules have no unit test today. Three options were on the table:
-#   * FAIL on any missing test — correct in principle, but it makes the merge gate red for
-#     every change until ~48 unit tests exist, so in practice the gate gets disabled, and a
-#     disabled gate protects nothing.
-#   * merely REPORT — the gap prints and nothing holds it. That is exactly how the CNPG unit
-#     tests came to exist and be enforced by nothing; "it is in the output somewhere" is not
-#     enforcement.
-#   * RATCHET (chosen) — the gap is DECLARED IN THE REPO as a reviewable register, printed with
-#     counts on every gate run, and it CANNOT GROW: a new PrometheusRule, or a new alert in a
-#     file that already has tests, FAILS unless it ships a unit test. Removing an entry requires
-#     writing the test, and a STALE entry (file gone, or now covered) also FAILS, so the
-#     register cannot rot into fiction.
-# "SKIPPED is not green" is honoured: a missing test is NEVER an exit-3 skip here. This check
-# exits 0 only with the gap fully enumerated, and exits 1 the moment the debt grows, the
-# register drifts, or a test that exists stops passing.
-# The one exit-3 path is promtool itself being ABSENT — an environment gap, not a code property
-# (the same convention as ci/l0.sh's kyverno handling), and this test DECLARES
-# `conformance.home.k3s/scope: whole-repo`, which makes lib/run_l1_offline.py turn that skip
-# into the runner's rc 3 and ci/l0.sh into the gate's rc 3: UNWIRED, never green. promtool
-# PRESENT AT THE WRONG VERSION is a FAILURE, not a skip: that case still prints SUCCESS while
-# evaluating the rules with an engine prod does not run.
-#
-# WHAT THIS DOES *NOT* PROVE (honest gaps):
-#   - It does not prove an alert reaches a human. promtool evaluates rules against synthetic
-#     series in-process; Alertmanager routing, SMTP delivery and inhibition are L3
-#     (tests/l3-fault-injection/alerting) and R-OBS-02, and are not claimed here.
-#   - It does not prove the THRESHOLDS are right, only that a rule behaves as its own unit
-#     test says.
-#   - It does not prove a rule file is RECONCILED into a cluster. An alert file no
-#     kustomization includes is a separate "looks applied and isn't" failure, not audited here.
-#   - Coverage of an individual alert is detected by its NAME appearing in a test file that
-#     covers its manifest. A test file that merely mentions a name without asserting anything
-#     counts as covered; only the mutation self-test (6) proves the harness CAN fail, not that
-#     every author wrote a meaningful assertion. Review still owns test quality.
-#   - It does not read the live cluster, so "the pin matches prod's Prometheus" is asserted
-#     against the repo's pin, which was established by a live read recorded in
-#     infra/devcontainer/versions.env (2026-09-12: v3.14.0-distroless).
-# =============================================================================
-set -euo pipefail
-DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
-PYTHON="${PYTHON:-python3}"
-HARNESS="${ALERT_RULE_HARNESS:-${ROOT}/platform/monitoring/tests/run.sh}"
-
-# Honest SKIP (never green) if the thing under test is not present at all. Because this test
-# declares scope: whole-repo, the runner turns this into rc 3 for the whole L0 stage — a
-# PrometheusRule-less repo is a vanished tree, not a pass.
-if ! grep -rqls --include='*.yaml' -e '^[[:space:]]*kind:[[:space:]]*PrometheusRule[[:space:]]*$' \
-       "${ROOT}/platform" "${ROOT}/apps" 2>/dev/null; then
-  echo "check SKIPPED: no PrometheusRule under platform/ or apps/ — nothing to test" >&2
-  exit 3
-fi
-
-# promtool resolution. PROMTOOL= overrides (the same override the harness honours); otherwise
-# PATH. Absent -> every OFFLINE assertion still runs and the check exits 3, so a toolchain gap
-# is reported as a gap and never as a pass.
-PROMTOOL_BIN="${PROMTOOL:-$(command -v promtool || true)}"
-if [ -n "${PROMTOOL_BIN}" ] && [ -x "${PROMTOOL_BIN}" ]; then
-  PROMTOOL_BIN="$(cd "$(dirname "${PROMTOOL_BIN}")" && pwd)/$(basename "${PROMTOOL_BIN}")"
-else
-  PROMTOOL_BIN=""
-fi
-
-cd "${ROOT}"
-"${PYTHON}" - "${ROOT}" "${HARNESS}" "${PROMTOOL_BIN}" <<'PY'
-"""Assert the alert-rule unit tests are pinned, enforced, non-vacuous, and that the alert
-rules with no tests are enumerated and ratcheted. Standard library + the harness itself."""
-from __future__ import annotations
-
-import os
-import re
-import shutil
-import subprocess
-import sys
-import tempfile
-from collections import defaultdict
-from pathlib import Path
-
-ROOT = Path(sys.argv[1])
-HARNESS = Path(sys.argv[2])
-PROMTOOL = sys.argv[3]
-sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
-
-FAILURES: list[str] = []
-CHECKS = 0
-
-
-def check(cond, msg):
-    global CHECKS
-    CHECKS += 1
-    if not cond:
-        FAILURES.append(msg)
-
-
-# =============================================================================
-# THE DECLARED REGISTER (assertion 7). Shrink-only; see the header for why this is a ratchet.
-#
-# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. A file listed here that later
-# gains coverage, or stops carrying a PrometheusRule, FAILS as stale.
-#
-# THE TWELVE apps/* ENTRIES ARE GONE (2026-09-13). They were the single `<app>-down` liveness rules
-# generated from one ServiceDefinition template, and the register's description of them as one
-# template was exactly why leaving them untested was so expensive: ONE wrong expression was wrong
-# twelve times, and it was — `up{job="<app>"} == 0` against a job no scrape target produced, inert on
-# prod for all twelve. They are now covered by apps/wave-0/tests/app-slis.test.yaml,
-# apps/wave-1/tests/app-slis.test.yaml and apps/wave-2/tests/app-slis.test.yaml, each rule pinned
-# BOTH ways (quiet on the healthy state, firing on no-ready-replica, and firing when the signal
-# VANISHES — the failure mode that produced the class). platform/kubevirt/alerts.yaml (4 rules) is
-# now the only entry left and is where the next tests belong.
-# =============================================================================
-UNTESTED_FILES = {
-    "platform/kubevirt/alerts.yaml",
-}
-
-# UNTESTED_ALERTS — individual alerts inside a file that IS covered by a test file, but whose
-# own rule nothing asserts on. This is the half that stops a new alert being slipped into an
-# already-"tested" file untested: anything not listed here must appear in a covering test.
-# The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
-# covers the Velero/backup rules only. It SHRANK by seven on 2026-09-13, when
-# platform/monitoring/tests/longhorn.test.yaml took on the whole platform-health.longhorn group
-# plus the three kubelet-based PVC-fill rules it has to be measured against (the per-volume
-# Longhorn "capacity" rule was deleted as a permanently-firing duplicate of them, so proving the
-# redundancy meant asserting on the rules that survive). Only
-# PersistentVolumeFillingUpPredicted is still unasserted in pvc-capacity-alerts.yaml.
-# It SHRANK BY TWO MORE on 2026-09-14: platform/monitoring/tests/node-io-cause.test.yaml pins the
-# eight new platform-health.node-io-cause rules, and its healthy fixture asserts NodeIOPressureHigh
-# and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
-# that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
-# detectors are no longer unasserted.
-UNTESTED_ALERTS = {
-    ("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
-    ("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
-    ("platform/cnpg/alerts.yaml", "CNPGNoRecentBaseBackup"),
-    ("platform/cnpg/alerts.yaml", "CNPGWALVolumeNotRecycling"),
-    ("platform/cnpg/alerts.yaml", "CNPGMetricsCollectorErroring"),
-    ("platform/monitoring/alerts.yaml", "NodeExporterDown"),
-    ("platform/monitoring/alerts.yaml", "NodeMemoryPressureHigh"),
-    ("platform/monitoring/alerts.yaml", "NodeDiskFillingUp"),
-    ("platform/monitoring/alerts.yaml", "NodeCPUPressureHigh"),
-    ("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
-    ("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
-    ("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
-    ("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
-    ("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
-    ("platform/monitoring/alerts.yaml", "CertificateNotReady"),
-    ("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
-    ("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
-    ("platform/monitoring/alerts.yaml", "KeycloakDown"),
-    ("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
-    ("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
-    ("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
-    ("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
-    ("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
-    ("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
-    ("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
-    ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
-}
-
-# --------------------------------------------------------------- (1) the pins, in lockstep
-def env_pin(rel: str, key: str) -> str | None:
-    f = ROOT / rel
-    if not f.is_file():
-        return None
-    m = re.search(rf'^{re.escape(key)}=["\']?([^"\'\s#]+)', f.read_text(), re.M)
-    return m.group(1) if m else None
-
-
-DEV_PIN = env_pin("infra/devcontainer/versions.env", "PROMTOOL_VERSION")
-CI_PIN = env_pin("ci/versions.env", "PROMTOOL_VERSION")
-check(DEV_PIN is not None,
-      "infra/devcontainer/versions.env declares no PROMTOOL_VERSION — promtool is not part of the "
-      "baked toolchain, so the alert unit tests are evaluated by whatever the host happens to "
-      "have, or by nothing at all (which is where this repo was on 2026-09-12)")
-check(CI_PIN is not None,
-      "ci/versions.env declares no PROMTOOL_VERSION — the gate's own version file does not pin the "
-      "rule engine, so CI and the devcontainer can silently diverge")
-check(DEV_PIN is None or DEV_PIN == CI_PIN,
-      f"PROMTOOL_VERSION is NOT in lockstep: infra/devcontainer/versions.env={DEV_PIN!r} vs "
-      f"ci/versions.env={CI_PIN!r}. The repo's rule is that the conformance-tool versions in the "
-      f"two files stay identical, so a laptop and CI run the same engine")
-
-CF = ROOT / "infra/devcontainer/Containerfile"
-cf_text = CF.read_text() if CF.is_file() else ""
-check(re.search(r"^ARG PROMTOOL_VERSION\s*$", cf_text, re.M),
-      "infra/devcontainer/Containerfile has no `ARG PROMTOOL_VERSION` — the pin is declared but "
-      "the image build does not consume it")
-check("prometheus/prometheus/releases/download" in cf_text and "promtool" in cf_text,
-      "infra/devcontainer/Containerfile does not install promtool from the pinned prometheus "
-      "release tarball (promtool ships inside it)")
-check('echo "promtool=${PROMTOOL_VERSION}"' in cf_text,
-      "infra/devcontainer/Containerfile does not record promtool in /etc/dev-toolchain-versions — "
-      "that provenance line is how an operator tells a stale image from a current one")
-
-DEVSH = ROOT / "infra/devcontainer/dev.sh"
-check(DEVSH.is_file() and "--build-arg PROMTOOL_VERSION=" in DEVSH.read_text(),
-      "infra/devcontainer/dev.sh does not pass --build-arg PROMTOOL_VERSION — the ARG would be "
-      "empty and the build would fetch a bogus URL")
-
-INSTALL = ROOT / ".gitea/workflows/scripts/install-tools.sh"
-if INSTALL.is_file():
-    itext = INSTALL.read_text()
-    check("PROMTOOL_VERSION" in itext
-          and re.search(r"^\s*l0\)[^\n]*install_promtool", itext, re.M),
-          ".gitea/workflows/scripts/install-tools.sh does not install promtool for the l0 stage — "
-          "on the CI runner this check would exit 3 (UNWIRED) forever and the alert unit tests "
-          "would be enforced nowhere, which is the exact gap it exists to close")
-
-# --------------------------------------------- (3)(4) one harness, and it fetches no engine
-check(HARNESS.is_file() and os.access(HARNESS, os.X_OK),
-      f"the rule-unit harness is missing or not executable at {HARNESS.relative_to(ROOT)} — "
-      f"nothing would run the alert unit tests")
-
-# Any script that invokes `promtool test rules` is a rule-unit harness. There must be exactly
-# one: two harnesses were written independently on 2026-09-12 (a cnpg-only script and this one),
-# and the one nothing runs is the one that rots out of sync with the rules.
-harnesses: list[str] = []
-fetchers: list[str] = []
-FETCH_RE = re.compile(
-    r"(curl|wget)[^\n]*prometheus/prometheus/releases"          # GitHub release download
-    r"|(podman|docker)\s+run[^\n]*prometheus",                  # container-image pull
-    re.I)
-# Two exclusions, both the repo's existing "reference, not declaration" rule (see
-# ci/touched-requirements.sh changed_diff() rules 2-3):
-#   * THIS CHECK'S OWN DIRECTORY. It names every forbidden pattern in order to forbid it, so
-#     scanning itself would make both guards permanently red — the self-referential trap.
-#   * THE PINNED TOOLCHAIN INSTALLERS. Downloading the pinned promtool is their JOB, at a
-#     reviewed seam (an image build, or the runner's tool-install step), from a version in
-#     versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
-#     version, the network and the machine are decided silently at test time, and a missing
-#     toolchain becomes a green run instead of an exit 3.
-SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
-INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
-for p in sorted(ROOT.rglob("*.sh")):
-    rel = str(p.relative_to(ROOT))
-    if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
-        continue
-    try:
-        text = p.read_text()
-    except (OSError, UnicodeDecodeError):
-        continue
-    body = "\n".join(l for l in text.splitlines() if not l.lstrip().startswith("#"))
-    if "promtool" in body and re.search(r"test\s+rules", body):
-        harnesses.append(rel)
-    if FETCH_RE.search(body) and not rel.startswith(INSTALLERS):
-        fetchers.append(rel)
-
-check(harnesses == [str(HARNESS.relative_to(ROOT))],
-      f"expected exactly ONE rule-unit harness ({HARNESS.relative_to(ROOT)}) but found "
-      f"{harnesses}. A second harness means two behaviours and two things to keep in step; the "
-      f"one the gate does not run is the one that rots. Fold it in instead")
-check(not fetchers,
-      f"{fetchers} acquire a rule ENGINE at run time (a prometheus release download or a "
-      f"prometheus container image). promtool must come from the pinned toolchain "
-      f"(PROMTOOL_VERSION): a run-time fetch decides the version, the network and the machine "
-      f"silently at test time, makes the tests non-hermetic (R-TEST-02), and quietly turns 'the "
-      f"toolchain has no promtool' into a green run — which is how it stayed missing")
-
-# ------------------------------------------------------------------ discover the alert files
-# Alert files: tracked YAML carrying a PrometheusRule. conformance/ is excluded deliberately —
-# the PrometheusRules under tests/l1-rendering/*/golden/ are EXPECTED-RENDER fixtures for the
-# render checks, not rules any cluster evaluates. Everything a cluster actually gets lives under
-# apps/, platform/, platform-config/ or clusters/.
-def tracked_yaml() -> list[Path]:
-    out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z", "*.yaml", "*.yml"],
-                         capture_output=True, text=True)
-    if out.returncode == 0 and out.stdout:
-        return [ROOT / p for p in out.stdout.split("\0") if p]
-    # No usable index (the gate container mounts a worktree whose .git points outside the
-    # mount). Fall back to a filesystem walk rather than auditing nothing.
-    return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
-
-
-KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
-ALERT_RE = re.compile(r"^\s*-\s*alert:\s*(\S+)", re.M)
-
-alert_files: dict[str, list[str]] = {}
-for p in tracked_yaml():
-    rel = str(p.relative_to(ROOT))
-    if rel.startswith("conformance/") or not p.is_file():
-        continue
-    try:
-        text = p.read_text()
-    except (OSError, UnicodeDecodeError):
-        continue
-    if KIND_RE.search(text):
-        alert_files[rel] = ALERT_RE.findall(text)
-
-check(len(alert_files) > 0,
-      "no PrometheusRule found outside conformance/ — discovery or the repo layout changed, and "
-      "this check would silently audit nothing")
-
-# ------------------- ask the HARNESS which manifests are under test (never re-derive it here)
-# Two implementations of "which rules does this test cover" would eventually disagree, and the
-# disagreement shows up as a file the gate believes is tested and the harness never touches.
-listing = subprocess.run(["bash", str(HARNESS), "--list"], capture_output=True, text=True,
-                         cwd=str(ROOT))
-check(listing.returncode == 0,
-      f"`{HARNESS.relative_to(ROOT)} --list` failed (rc={listing.returncode}): "
-      f"{(listing.stderr or listing.stdout).strip()[-600:]}")
-covering: dict[str, list[str]] = defaultdict(list)   # manifest -> [test files]
-for line in listing.stdout.splitlines():
-    if "\t" in line:
-        tfile, manifest = line.split("\t", 1)
-        covering[manifest.strip()].append(tfile.strip())
-
-for manifest in sorted(covering):
-    check(manifest in alert_files,
-          f"the harness reports {manifest} as under test, but it is not one of the "
-          f"PrometheusRule files found in the repo — the mapping and the tree disagree")
-
-# --------------------------------------------------------- (2) the engine is the pinned one
-version_line = ""
-if PROMTOOL:
-    vp = subprocess.run([PROMTOOL, "--version"], capture_output=True, text=True)
-    out = (vp.stdout + vp.stderr).strip()
-    version_line = out.splitlines()[0] if out else ""
-    m = re.search(r"version\s+(\d+\.\d+\.\d+)", version_line)
-    found = m.group(1) if m else None
-    check(found is not None,
-          f"could not parse a version from `promtool --version`: {version_line!r}")
-    check(found is None or DEV_PIN is None or found == DEV_PIN,
-          f"promtool reports {found}, but the toolchain pins PROMTOOL_VERSION={DEV_PIN} (the "
-          f"Prometheus version prod runs). promtool IS the rule engine: a unit test green under "
-          f"one version and red under another is worse than no test. Run the gate in the pinned "
-          f"devcontainer (rebuild it: ./infra/devcontainer/dev.sh build), or set PROMTOOL= to a "
-          f"v{DEV_PIN} binary. A FAILURE, not a skip, because the wrong engine still prints "
-          f"SUCCESS")
-
-# ------------------------------------------------- (5) every rule-unit test file must pass
-if PROMTOOL:
-    env = dict(os.environ, PROMTOOL=PROMTOOL)
-    run = subprocess.run(["bash", str(HARNESS)], capture_output=True, text=True,
-                         cwd=str(ROOT), env=env)
-    tailed = "\n        ".join((run.stdout + run.stderr).strip().splitlines()[-40:])
-    check(run.returncode == 0,
-          f"`{HARNESS.relative_to(ROOT)}` FAILED (rc={run.returncode}) — the committed alert "
-          f"rules do not behave as their own unit tests say:\n        {tailed}")
-
-    # ------------------------------------------- (6) the harness is proven non-vacuous
-    # A synthetic PrometheusRule + companion test through the SAME harness, outside the repo so
-    # discovery never picks it up. Intact must PASS; a mutated expectation must FAIL. Without
-    # this, a harness that stopped evaluating anything (an empty extraction, a rule file written
-    # under the wrong name) would report green forever — which is the disease being treated.
-    work = tempfile.mkdtemp(prefix="alert-rules-selftest.")
-    try:
-        (Path(work) / "synth.yaml").write_text("""apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: selftest
-spec:
-  groups:
-    - name: selftest
-      rules:
-        - alert: SelfTestCanary
-          expr: selftest_metric > 0
-          for: 5m
-          annotations:
-            runbook_url: https://${BASE_DOMAIN}/runbooks/selftest
-""")
-        test_yaml = """rule_files:
-  - generated.rules.yaml
-evaluation_interval: 1m
-tests:
-  - interval: 1m
-    name: canary fires only after `for` elapses
-    input_series:
-      - series: 'selftest_metric'
-        values: '1x10'
-    promql_expr_test:
-      - expr: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
-        eval_time: EVAL_TIME
-        exp_samples:
-          - labels: 'ALERTS{alertname="SelfTestCanary",alertstate="firing"}'
-            value: 1
-"""
-        synth_test = Path(work) / "synth.promtool-test.yaml"
-        synth_test.write_text(test_yaml.replace("EVAL_TIME", "6m"))
-        good = subprocess.run(["bash", str(HARNESS), str(synth_test)],
-                              capture_output=True, text=True, cwd=str(ROOT), env=env)
-        check(good.returncode == 0,
-              "self-test: the INTACT synthetic unit test did not pass, so the harness itself is "
-              "broken and every green result above is meaningless:\n        "
-              + "\n        ".join((good.stdout + good.stderr).strip().splitlines()[-20:]))
-        # Mutation: assert firing at 2m, which `for: 5m` forbids.
-        synth_test.write_text(test_yaml.replace("EVAL_TIME", "2m"))
-        bad = subprocess.run(["bash", str(HARNESS), str(synth_test)],
-                             capture_output=True, text=True, cwd=str(ROOT), env=env)
-        check(bad.returncode != 0,
-              "self-test: a MUTATED expectation (firing before `for:` elapsed) still PASSED — "
-              "the harness is not actually evaluating anything, so every green result above is "
-              "vacuous")
-    finally:
-        shutil.rmtree(work, ignore_errors=True)
-
-# --------------------------------------------------------- (7) the gap inventory + ratchet
-INVENTORY: list[str] = []
-GAP_FILES: list[str] = []
-alerts_total = covered_total = 0
-
-for rel in sorted(alert_files):
-    names = alert_files[rel]
-    alerts_total += len(names)
-    check(len(names) > 0,
-          f"{rel} declares a PrometheusRule but no `alert:` rule was parsed out of it — the "
-          f"parser or the file shape changed, and any unit test over it would pass vacuously")
-    tests = covering.get(rel, [])
-    if not tests:
-        GAP_FILES.append(rel)
-        INVENTORY.append(f"  {rel:<46} alerts={len(names):<3} tested=0/{len(names):<3} NO TEST FILE")
-        check(rel in UNTESTED_FILES,
-              f"{rel} carries {len(names)} alert rule(s) and NO test file covers it. A new "
-              f"PrometheusRule must ship promtool unit tests: this repo has shipped both a "
-              f"permanently-firing critical and a structurally inert alert, and neither was "
-              f"visible without them. Add `{rel[:-5]}.promtool-test.yaml` (companion convention) "
-              f"or a <component>/tests/*.test.yaml suite. If this is deliberate debt it must be "
-              f"DECLARED in UNTESTED_FILES in this check — the register is what keeps the gap "
-              f"visible instead of silent")
-        continue
-
-    check(rel not in UNTESTED_FILES,
-          f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
-          f"shrink-only: delete the line so the next missing test is caught")
-    text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
-    covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
-    covered_total += len(covered)
-    for n in names:
-        if n in covered:
-            check((rel, n) not in UNTESTED_ALERTS,
-                  f"{rel}: alert {n} is listed in UNTESTED_ALERTS but IS asserted on now — "
-                  f"remove the stale entry (the register is shrink-only)")
-        else:
-            check((rel, n) in UNTESTED_ALERTS,
-                  f"{rel}: alert {n} has NO unit test in {tests}. This file is already covered, "
-                  f"so a new alert in it must be tested too — otherwise a 'tested' alert file "
-                  f"quietly grows untested rules")
-    INVENTORY.append(f"  {rel:<46} alerts={len(names):<3} tested={len(covered)}/{len(names):<3} "
-                     f"{','.join(tests)}")
-
-# stale register entries
-for rel in sorted(UNTESTED_FILES - set(alert_files)):
-    check(False, f"UNTESTED_FILES lists {rel}, which carries no PrometheusRule in this repo any "
-                 f"more — a stale exemption hides the next real gap; delete the line")
-for rel in sorted({f for f, _ in UNTESTED_ALERTS} - set(alert_files)):
-    check(False, f"UNTESTED_ALERTS lists alerts in {rel}, which carries no PrometheusRule in this "
-                 f"repo any more — delete those lines")
-
-# --------------------------------------------------------------------------------- report
-print("Alert-rule unit-test inventory (PrometheusRule -> covering test file(s)):")
-for line in INVENTORY:
-    print(line)
-print(f"\nharness: {HARNESS.relative_to(ROOT)}   promtool: {version_line or 'NOT FOUND'}")
-print(f"pinned:  PROMTOOL_VERSION={DEV_PIN} (devcontainer) / {CI_PIN} (ci) — prod runs Prometheus "
-      f"v{DEV_PIN}")
-print(f"alert files: {len(alert_files)}   covered by a test file: "
-      f"{len(alert_files) - len(GAP_FILES)}   with NO test file: {len(GAP_FILES)}")
-print(f"alert rules: {alerts_total}   unit-tested: {covered_total}   "
-      f"NOT unit-tested: {alerts_total - covered_total}")
-
-if GAP_FILES:
-    print("\nDECLARED GAPS — alert files nothing tests at all (shrink-only register; each is a")
-    print("rule nothing has ever evaluated, NOT a rule known to work):")
-    for g in GAP_FILES:
-        print(f"  - {g}  ({len(alert_files[g])} alert(s))")
-if UNTESTED_ALERTS:
-    print("\nDECLARED GAPS — individual alerts inside a covered file that nothing asserts on:")
-    for f, n in sorted(UNTESTED_ALERTS):
-        print(f"  - {f}  {n}")
-
-print()
-if not PROMTOOL and not FAILURES:
-    print("check SKIPPED (exit 3, NOT a pass): promtool is not on PATH, so NO alert rule was "
-          "parsed and NO unit test was evaluated.", file=sys.stderr)
-    print(f"  promtool is pinned at v{DEV_PIN} in infra/devcontainer/versions.env + "
-          f"ci/versions.env.", file=sys.stderr)
-    print("  Run the gate in the pinned toolbox (./infra/devcontainer/dev.sh build, then "
-          "make -C ci gate), or set PROMTOOL=/path/to/promtool.", file=sys.stderr)
-    print(f"  The {CHECKS} offline assertions above (pins, lockstep, one harness, no run-time "
-          f"engine fetch, the gap register) DID run and passed.", file=sys.stderr)
-    print("  This test declares scope: whole-repo, so the runner reports rc 3 — UNWIRED, never "
-          "green.", file=sys.stderr)
-    raise SystemExit(3)
-
-if not FAILURES:
-    print(f"check OK: alert-rules-unit-tested — {CHECKS} assertions "
-          f"({covered_total}/{alerts_total} alert rules unit-tested, "
-          f"{len(GAP_FILES)} files with no test file, all declared) (R-OBS-04, R-TEST-02)")
-    raise SystemExit(0)
-print(f"CHECK FAIL (alert-rules-unit-tested): {len(FAILURES)} failed, "
-      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
-for f in FAILURES:
-    print(f"  FAIL {f}", file=sys.stderr)
-raise SystemExit(1)
-PY

conformance/tests/l0-static/alert-runbooks-resolvable/chainsaw-test.yaml

@@ -1,89 +0,0 @@
-# =============================================================================
-# L0 STATIC — every alert's runbook_url resolves to a runbook that is actually served
-# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
-#
-# Every platform-health alert rule in this repo carries
-# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations across four files, over 34
-# distinct names. For the whole of phases 4-6 there was NO runbooks/ directory and NO HTTPRoute serving
-# that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404.
-# So an alert reaching a human handed them a dead link at the one moment documentation is read, while the
-# annotation made the problem look solved — the same green-surface-over-absent-mechanism class as the
-# import Job that reported success while storing placeholder secrets, and the guard whose glob silently
-# stopped matching.
-#
-# The link is served now (runbooks/, clusters/base/runbooks.yaml). This test is the guard that keeps it
-# true, and it asserts BOTH halves of the claim because the regression is silent in both directions: an
-# alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its
-# configMapGenerator entry is in git, reviewed, merged — and still 404s in the cluster. "The file exists"
-# and "the URL resolves" are different claims; only the second one helps an operator.
-#
-# GIVEN every `runbook_url` annotation on every PrometheusRule in the repo,
-# WHEN  each is parsed out of git and resolved against runbooks/,
-# THEN  (1) each URL has EXACTLY the shape the server resolves — https://${BASE_DOMAIN}/runbooks/<name>,
-#           the ${BASE_DOMAIN} token (never a literal hostname), no `.md`, no trailing slash, no query;
-#       (2) every referenced <name> has a runbooks/<name>.md in git;
-#       (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
-#           dev-container kubectl invocation with --context k3s-prod — so a stub cannot satisfy (2) and
-#           make the link look fixed while still helping nobody;
-#       (4) every runbook is PROJECTED INTO THE SERVER by runbooks/kustomization.yaml's
-#           configMapGenerator under the extensionless key the URL asks for;
-#       (5) route, listener and server agree with the annotations (PathPrefix == the annotations' prefix,
-#           a sectionName that platform/gateway/gateway.yaml actually defines, the namespace opted in to
-#           the Gateway, and an nginx location for that same prefix with a text/plain default_type);
-#       (6) the Flux tier is wired into clusters/base AND declares NO postBuild.substituteFrom —
-#           substitution would run envsubst over the runbook prose and silently empty every `${...}`
-#           shell example in it (it fails open: DECISIONS-LOG 2026-09-01, the emptied Cilium
-#           k8sServiceHost), and the flux-postbuild-substitution check cannot see it because that check
-#           scans files reachable through `resources:`, not configMapGenerator inputs;
-#   AND (7) the check is LOAD-BEARING: this Test DECLARES
-#           `conformance.home.k3s/scope: whole-repo`, so ci/l0.sh Stage 3b's
-#           `--include-scope whole-repo` unions it into EVERY gate invocation rather than only the runs
-#           whose touched requirements happen to include R-OBS-04; and ci/gate.sh sets the merge block
-#           when the l0 stage fails. A SKIP of a scope-declaring check is UNWIRED, never green.
-#
-# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library only);
-# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test wraps the SAME
-# script.
-#
-# NOT proven (named in check.sh's header): this does not FETCH the URL. That it returns HTTP 200 with
-# real content was proven by an in-cluster probe recorded in the commit message; a live probe needs a
-# cluster, which L0 by definition does not have. What is asserted here is every static precondition for
-# that 200. It also does not judge whether a runbook is correct or current — (3) is a floor, not a
-# review — and it does not prove Alertmanager delivers the annotation to a human (that is the L3
-# `alerting` test). R-OBS-04 is tagged for the narrow slice only: the platform's own health alerts carry
-# runbook links that resolve to real, served procedure rather than to a 404.
-# =============================================================================
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: alert-runbooks-resolvable
-  labels:
-    conformance.home.k3s/R-OBS-04: "true"
-  annotations:
-    conformance.home.k3s/requirements: "R-OBS-04"
-    conformance.home.k3s/layer: "L0"
-    # WHOLE-REPO SCOPE — this is what makes the check unconditional. Stage 3b of ci/l0.sh runs
-    # `conformance/run.sh --layer l0 --include-scope whole-repo`, which UNIONS every test carrying this
-    # annotation into the selection regardless of which requirements the change touched. That union is
-    # load-bearing here: ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" —
-    # NOT R-OBS-04 — and maps platform/kubevirt/* and runbooks/* to nothing at all, so the single most
-    # likely breaking change (adding an alert with a runbook_url, or adding a runbook without its
-    # configMapGenerator entry) attributes a requirement set that would otherwise FILTER THIS CHECK OUT
-    # of exactly the change that introduces the bug. Declaring the scope replaces the hand-written
-    # per-test ci/l0.sh stage that this repo used to need (and that nobody remembered to add).
-    conformance.home.k3s/scope: "whole-repo"
-    conformance.home.k3s/safety-critical: "false"
-    conformance.home.k3s/executable: "true"
-    conformance.home.k3s/intent: >-
-      Every runbook_url on every alert rule resolves to a runbook that exists in git, clears a quality
-      floor, and is actually projected into the server that answers that path — so an alert can never
-      hand an operator a dead link.
-spec:
-  steps:
-    - name: audit-every-alert-runbook-link
-      description: >-
-        Parse every runbook_url out of every PrometheusRule and assert the runbook exists, is real, is
-        served at that exact path, and that the serving tier is wired and unsubstituted.
-      try:
-        - script:
-            content: bash ./check.sh

conformance/tests/l0-static/alert-runbooks-resolvable/check.sh

@@ -1,410 +0,0 @@
-#!/usr/bin/env bash
-# =============================================================================
-# L0 static check (offline, no cluster) — alert-runbooks-resolvable
-# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
-#         VPN)." — spec docs/10-observability-and-alerting.md)
-#
-# WHY THIS EXISTS — A DEAD LINK IN AN ALERT IS WORSE THAN NO LINK
-# --------------------------------------------------------------
-# Every platform-health alert rule in this repo carries
-#   runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
-# 42 such annotations across four files, over 34 distinct names. For the whole of phases 4-6
-# there was NO runbooks/ directory and NO HTTPRoute serving that path: a live in-cluster probe
-# of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. So an alert reaching a
-# human at 3am handed them a link that fails, at the one moment the documentation is read. The
-# annotation LOOKED like the problem was solved, which is the specific failure this repo keeps
-# finding (an import Job that reports success while storing placeholder secrets; a guard whose
-# glob silently stopped matching) — a green-looking surface over an absent mechanism.
-#
-# The link is now served (runbooks/, clusters/base/runbooks.yaml). This check is the guard that
-# keeps it true, because the regression is SILENT IN BOTH DIRECTIONS and nothing else catches it:
-#   * add an alert with a new runbook_url and forget the runbook  -> that alert links to a 404,
-#     and no test, lint, render or policy notices;
-#   * write runbooks/<name>.md but forget to add it to the ConfigMap generator -> the file is in
-#     git, reviewable, apparently done, and the URL still 404s in the cluster. "The file exists"
-#     and "the URL resolves" are different claims and only the second one helps an operator.
-# Both halves are asserted below, which is why this check reaches past the markdown into the
-# kustomization, the HTTPRoute, the nginx config and the Flux tier.
-#
-# GIVEN the repository as authored,
-# WHEN  every `runbook_url` annotation on every PrometheusRule is parsed out of git,
-# THEN  (1) each URL has EXACTLY the shape the server resolves —
-#           https://${BASE_DOMAIN}/runbooks/<name>, `${BASE_DOMAIN}` as the token (never a literal
-#           hostname: a literal is how prod was authored to serve dev's hostnames, see the
-#           ServiceDefinition schema's note), no `.md` suffix, no trailing slash, no query;
-#       (2) every referenced <name> has a runbooks/<name>.md in git;
-#       (3) each runbook clears a QUALITY FLOOR — the four operator sections, a real body, and the
-#           dev-container kubectl invocation — so a one-line stub cannot satisfy (2) and make the
-#           link look fixed while still helping nobody (the failure mode this check is most likely
-#           to be defeated by is a placeholder file, not a missing one);
-#       (4) every runbook markdown file is actually PROJECTED INTO THE SERVER by
-#           runbooks/kustomization.yaml's configMapGenerator, under the EXTENSIONLESS key the URL
-#           asks for — the "exists in git but 404s in the cluster" half;
-#       (5) the route and the server agree with the annotations: the HTTPRoute's PathPrefix is the
-#           prefix the URLs use, it attaches to a Gateway listener that actually exists, its
-#           namespace is opted in to that Gateway, and the nginx config answers that same prefix;
-#       (6) the Flux tier that reconciles all of it is wired into clusters/base AND declares NO
-#           postBuild.substituteFrom — substitution here would run envsubst over the runbook prose
-#           and silently empty every `${...}` shell example in it (it fails open: DECISIONS-LOG
-#           2026-09-01, the emptied Cilium k8sServiceHost);
-#   AND (7) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and ci/gate.sh sets the
-#           merge block when the l0 stage fails.
-#
-# WHY (7) IS AN UNCONDITIONAL ci/l0.sh STAGE and not left to the requirement-scoped run: Stage 3b
-# scopes `conformance/run.sh --layer l0` to the TOUCHED requirements (R-TEST-03), and
-# ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02" — NOT R-OBS-04 — and
-# maps platform/kubevirt/* and runbooks/* to nothing at all. So the single most likely breaking
-# change (adding an alert with a runbook_url to platform/monitoring/alerts.yaml, or adding a
-# runbook and forgetting the generator entry) attributes a requirement set that would FILTER THIS
-# CHECK OUT. Declaring `conformance.home.k3s/scope: whole-repo` makes ci/l0.sh Stage 3b union it into
-# EVERY gate invocation instead; a SKIP of a scope-declaring check is UNWIRED, never green.
-#
-# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
-#   - It does not fetch the URL. That it returns HTTP 200 with real content was proven by an
-#     in-cluster probe (recorded in the commit message and DECISIONS-LOG); a live probe needs a
-#     cluster, which L0 by definition does not have. What is asserted here is every static
-#     precondition for that 200: the file, its projection, the key name, the route, the prefix.
-#   - It does not judge whether a runbook is CORRECT or current. (3) is a floor, not a review: it
-#     can tell a stub from a runbook, not a good runbook from a stale one. Nothing automatable can.
-#   - It does not prove Alertmanager actually delivers the annotation to a human. That is the L3
-#     `alerting` test.
-#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and is not consulted.
-# =============================================================================
-set -euo pipefail
-DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
-PYTHON="${PYTHON:-python3}"
-
-# Honest SKIP (never green) if there are no alert rules at all — nothing to audit yet.
-if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" >/dev/null 2>&1; then
-  echo "check SKIPPED: no PrometheusRule found under platform/ — no alert rules to audit" >&2
-  exit 3
-fi
-
-cd "${ROOT}"
-"${PYTHON}" - "${ROOT}" <<'PY'
-"""Static audit: every alert runbook_url resolves to a runbook that is actually served.
-
-Pure standard library, no PyYAML — the same constraint and the same style as the neighbouring
-L0 checks (lib/coverage.py, lib/status.py). Every parse contract is documented at its parser;
-all of them are line-constrained shapes in hand-authored manifests.
-"""
-from __future__ import annotations
-
-import re
-import sys
-from pathlib import Path
-
-ROOT = Path(sys.argv[1])
-sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
-
-FAILURES: list[str] = []
-ADVISORIES: list[str] = []
-CHECKS = 0
-
-
-def check(cond, msg):
-    global CHECKS
-    CHECKS += 1
-    if not cond:
-        FAILURES.append(msg)
-
-
-def read(p: Path) -> str:
-    try:
-        return p.read_text()
-    except (OSError, UnicodeDecodeError):
-        return ""
-
-
-# =============================================================== (1) parse the annotations
-# Parse contract: `runbook_url:` is an annotation on a Prometheus alerting rule, authored one per
-# line as `runbook_url: "<url>"` (single or double quoted, or bare). Only files that declare
-# `kind: PrometheusRule` are scanned, so a runbook_url quoted inside this check or inside prose
-# elsewhere in the repo is not mistaken for an alert annotation. Commented-out lines are skipped:
-# platform/monitoring/alerts.yaml carries a `# runbook_url points at a docs base` header comment.
-SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
-RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
-
-rule_files = sorted(
-    p for p in ROOT.rglob("*.yaml")
-    if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
-)
-check(len(rule_files) > 0,
-      "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
-      "this check would silently audit nothing")
-
-# url -> [ "<relpath>:<lineno>", ... ]
-refs: dict[str, list[str]] = {}
-for f in rule_files:
-    rel = f.relative_to(ROOT)
-    for lineno, line in enumerate(read(f).splitlines(), 1):
-        if line.lstrip().startswith("#"):
-            continue
-        m = RUNBOOK_URL_RE.match(line)
-        if m:
-            refs.setdefault(m.group(1), []).append(f"{rel}:{lineno}")
-
-n_annotations = sum(len(v) for v in refs.values())
-check(n_annotations > 0,
-      "no runbook_url annotation parsed out of any PrometheusRule — the parser contract broke; "
-      "this check would pass vacuously while every alert link rots")
-
-# ---- URL SHAPE. The server resolves exactly one shape. Anything else is a link that 404s even
-# though a file with a plausible name exists, so shape is a failure, not a style note.
-URL_RE = re.compile(r"^https://\$\{BASE_DOMAIN\}/(runbooks)/([a-z0-9][a-z0-9-]*)$")
-names: dict[str, list[str]] = {}        # runbook name -> annotation sites
-prefixes: set[str] = set()
-for url, sites in sorted(refs.items()):
-    m = URL_RE.match(url)
-    check(m is not None,
-          f"runbook_url '{url}' ({', '.join(sites)}) is not the shape this platform serves. "
-          f"Required: https://${{BASE_DOMAIN}}/runbooks/<name> — the ${{BASE_DOMAIN}} TOKEN (never a "
-          f"literal hostname: a literal is how prod got authored to serve dev's public names), a "
-          f"lowercase-hyphen <name>, and NO '.md' suffix, trailing slash or query. The runbook "
-          f"server maps the URL path straight onto an extensionless ConfigMap key, so any other "
-          f"shape is a 404 no matter which files exist")
-    if m:
-        prefixes.add(m.group(1))
-        names.setdefault(m.group(2), []).extend(sites)
-
-check(len(prefixes) <= 1,
-      f"alert annotations use more than one runbook path prefix ({sorted(prefixes)}); the single "
-      f"HTTPRoute + nginx location can only serve one, so the others are dead links")
-PREFIX = (sorted(prefixes) or ["runbooks"])[0]
-
-# =================================================== (2)(3) the runbooks exist and are real
-RUNBOOK_DIR = ROOT / PREFIX
-check(RUNBOOK_DIR.is_dir(),
-      f"{PREFIX}/ is not a directory in this repo, so every one of the {n_annotations} "
-      f"runbook_url annotations is a dead link")
-
-# The quality floor. (2) alone is satisfiable by `touch runbooks/node-down.md`, which makes the
-# link resolve and still abandons the operator — a worse outcome than the 404, because the 404 at
-# least told the truth. These four sections are the operator's path through an incident: what the
-# alert means HERE, what to type first, what is probably wrong, and when to stop. `dev.sh` +
-# `--context k3s-prod` are required because kubectl does not work on the workstation at all
-# (the toolchain lives only in the pinned dev container) and because dev.sh defaults its
-# current-context to k3s-DEV — a runbook that omits the context teaches the reader to answer
-# questions about the wrong cluster during an incident on this one.
-REQUIRED_SECTIONS = (
-    "## What this means here",
-    "## First commands",
-    "## Likely causes, most likely first",
-    "## When to escalate or stop",
-)
-MIN_BYTES = 1500
-
-for name, sites in sorted(names.items()):
-    md = RUNBOOK_DIR / f"{name}.md"
-    if not md.is_file():
-        check(False,
-              f"runbook '{name}' does not exist: {md.relative_to(ROOT)} is missing, but "
-              f"{len(sites)} alert annotation(s) link to it ({', '.join(sites[:3])}"
-              f"{', ...' if len(sites) > 3 else ''}). An alert firing now hands the operator a 404. "
-              f"Write {PREFIX}/{name}.md (see {PREFIX}/README.md for the required sections)")
-        continue
-    body = read(md)
-    check(len(body) >= MIN_BYTES,
-          f"{md.relative_to(ROOT)} is {len(body)} bytes, under the {MIN_BYTES}-byte floor — a stub "
-          f"that makes the link resolve while still helping nobody is worse than the 404 it "
-          f"replaced, because it looks fixed")
-    for sec in REQUIRED_SECTIONS:
-        check(sec in body,
-              f"{md.relative_to(ROOT)} has no '{sec}' section — the four operator sections are what "
-              f"make this a runbook rather than a restatement of the alert name")
-    check("dev.sh" in body,
-          f"{md.relative_to(ROOT)} never mentions dev.sh — kubectl does not exist on the "
-          f"workstation; every command in a runbook here must be runnable as "
-          f"`infra/devcontainer/dev.sh run bash -c '...'` and the runbook must say so")
-    check("--context k3s-prod" in body,
-          f"{md.relative_to(ROOT)} never passes --context k3s-prod — dev.sh merges both clusters "
-          f"and picks a current-context PER INVOCATION — a default that has already changed once "
-          f"(k3s-dev until dev was decommissioned on 2026-09-13, k3s-prod now). A context-less command "
-          f"silently answers about whichever cluster the default happens to be, in the middle of an "
-          f"incident")
-
-# ADVISORY, not a failure: a runbook nothing links to yet. Writing the runbook BEFORE the alert is
-# the right order, and the repo's precedent for "authored but unreferenced" is advisory (see the
-# dead-knob advisory in flux-postbuild-substitution/check.sh).
-served_extra = {"README"}
-if RUNBOOK_DIR.is_dir():
-    for md in sorted(RUNBOOK_DIR.glob("*.md")):
-        stem = md.stem
-        if stem in served_extra or stem in names:
-            continue
-        ADVISORIES.append(f"{md.relative_to(ROOT)} is a runbook no alert links to (written ahead of "
-                          f"its alert, or the alert was removed — advisory, not a failure)")
-
-# ================================= (4) every runbook is PROJECTED INTO THE SERVER, not just in git
-# THIS IS THE HALF A FILE-EXISTENCE CHECK WOULD MISS. The pod serves a ConfigMap, not the git tree.
-# Parse contract: runbooks/kustomization.yaml declares one configMapGenerator whose `files:` entries
-# are `- <key>=<file>.md`, one per line. The key is EXTENSIONLESS on purpose (the URL carries no
-# `.md`, and the ConfigMap volume mount turns each key into a file the nginx root resolves directly).
-KUST = RUNBOOK_DIR / "kustomization.yaml"
-check(KUST.is_file(), f"{PREFIX}/kustomization.yaml is missing — the Flux tier renders nothing and "
-                      f"every runbook 404s no matter what is in git")
-GEN_FILE_RE = re.compile(r"^\s*-\s*([A-Za-z0-9._-]+)=([A-Za-z0-9._/-]+\.md)\s*(?:#.*)?$")
-generated: dict[str, str] = {}
-kust_text = read(KUST)
-if kust_text:
-    check("configMapGenerator:" in kust_text,
-          f"{PREFIX}/kustomization.yaml has no configMapGenerator — the runbook markdown is not "
-          f"rendered into the ConfigMap the server mounts")
-    for line in kust_text.splitlines():
-        if line.lstrip().startswith("#"):
-            continue
-        m = GEN_FILE_RE.match(line)
-        if m:
-            generated[m.group(1)] = m.group(2)
-    check(len(generated) > 0,
-          f"{PREFIX}/kustomization.yaml declares no `<key>=<file>.md` generator entries — the "
-          f"parser contract broke, or the markdown is being added with implicit keys (which would "
-          f"carry the '.md' suffix the alert URLs do NOT have, so every link would 404)")
-
-for name in sorted(names):
-    check(generated.get(name) == f"{name}.md",
-          f"runbook '{name}' is not projected into the server: {PREFIX}/kustomization.yaml has no "
-          f"configMapGenerator entry `- {name}={name}.md`. The file can be in git, reviewed and "
-          f"merged and the URL still returns 404 — 'the file exists' and 'the URL resolves' are "
-          f"different claims")
-
-# And the reverse, so a generator entry cannot point at a file that is not there.
-for key, fname in sorted(generated.items()):
-    check((RUNBOOK_DIR / fname).is_file(),
-          f"{PREFIX}/kustomization.yaml generates key '{key}' from {PREFIX}/{fname}, which does not "
-          f"exist — `kustomize build` fails, so the WHOLE tier stops reconciling and every runbook "
-          f"goes stale at once")
-
-# ============================== (5) the route, the listener and the server agree with the URLs
-ROUTE = RUNBOOK_DIR / "serving" / "httproute.yaml"
-NGINX = RUNBOOK_DIR / "serving" / "nginx-config.yaml"
-NS = RUNBOOK_DIR / "serving" / "namespace.yaml"
-GATEWAY = ROOT / "platform" / "gateway" / "gateway.yaml"
-
-route_text = read(ROUTE)
-check(bool(route_text), f"{PREFIX}/serving/httproute.yaml is missing — nothing routes the "
-                        f"/{PREFIX} prefix to the server")
-# Parse contract: a single HTTPRoute with one PathPrefix match and one parentRefs entry naming a
-# listener via sectionName; hand-authored, one key per line.
-m_path = re.search(r"^\s*type:\s*PathPrefix\s*$\n\s*value:\s*(\S+)\s*$", route_text, re.M)
-check(m_path is not None and m_path.group(1) == f"/{PREFIX}",
-      f"{PREFIX}/serving/httproute.yaml does not match PathPrefix /{PREFIX} "
-      f"(found {m_path.group(1) if m_path else 'no PathPrefix match'}). The alert annotations point "
-      f"at /{PREFIX}/<name>; if the route's prefix drifts from the annotations the links 404 while "
-      f"both files look individually correct")
-m_sec = re.search(r"^\s*sectionName:\s*(\S+)", route_text, re.M)
-check(m_sec is not None,
-      f"{PREFIX}/serving/httproute.yaml declares no parentRefs sectionName. The apex host needs the "
-      f"dedicated apex listener: the wildcard `*.${{BASE_DOMAIN}}` listener does NOT match the bare "
-      f"apex, so a route without sectionName binds nothing and serves nothing")
-if m_sec:
-    listener = m_sec.group(1)
-    gw = read(GATEWAY)
-    check(re.search(rf"^\s*-\s*name:\s*{re.escape(listener)}\s*$", gw, re.M) is not None,
-          f"{PREFIX}/serving/httproute.yaml attaches to Gateway listener '{listener}', which "
-          f"platform/gateway/gateway.yaml does not define — the route would report NotAllowed and "
-          f"never serve")
-# An HTTPRoute with no `hostnames:` inherits the listener's hostname, which is how this tier avoids
-# needing Flux substitution (see (6)). Either form is routable; a LITERAL hostname is not acceptable,
-# for the same reason the schema forbids it in a ServiceDefinition.
-m_host = re.search(r"^\s*hostnames:\s*$\n\s*-\s*(\S+)", route_text, re.M)
-check(m_host is None or "${BASE_DOMAIN}" in m_host.group(1),
-      f"{PREFIX}/serving/httproute.yaml hardcodes hostname '{m_host.group(1) if m_host else ''}' — "
-      f"use the ${{BASE_DOMAIN}} token, or omit `hostnames:` entirely and inherit the listener's")
-
-ns_text = read(NS)
-check('gateway-access: "shared"' in ns_text or "gateway-access: shared" in ns_text,
-      f"{PREFIX}/serving/namespace.yaml does not carry gateway-access=shared — the shared Gateway "
-      f"selects namespaces by that label, so without it the HTTPRoute is refused attachment")
-
-nginx_text = read(NGINX)
-check(f"location /{PREFIX}/" in nginx_text,
-      f"{PREFIX}/serving/nginx-config.yaml has no `location /{PREFIX}/` block — the route would "
-      f"deliver the request and the server would answer 404 for every runbook")
-check("default_type" in nginx_text and "text/plain" in nginx_text,
-      f"{PREFIX}/serving/nginx-config.yaml does not force a text/plain default_type. The ConfigMap "
-      f"keys are EXTENSIONLESS, so with the stock MIME table nginx serves them as "
-      f"application/octet-stream and a browser DOWNLOADS the runbook instead of showing it")
-
-# ====================================== (6) the Flux tier is wired, and is NOT substituted
-FLUX = ROOT / "clusters" / "base" / f"{PREFIX}.yaml"
-BASE_KUST = ROOT / "clusters" / "base" / "kustomization.yaml"
-flux_text = read(FLUX)
-check(bool(flux_text),
-      f"clusters/base/{PREFIX}.yaml is missing — nothing reconciles {PREFIX}/ into the cluster, so "
-      f"the runbooks exist in git and the URL still 404s")
-check(re.search(rf"^\s*path:\s*\./{re.escape(PREFIX)}\s*$", flux_text, re.M) is not None,
-      f"clusters/base/{PREFIX}.yaml does not set `path: ./{PREFIX}`")
-check(re.search(r"^\s*wait:\s*false\s*$", flux_text, re.M) is not None,
-      f"clusters/base/{PREFIX}.yaml is not wait:false — a leaf web server that can be legitimately "
-      f"unhealthy must never gate the reconcile chain (DECISIONS-LOG: 'Headlamp gating stall')")
-# Strip comments before testing: this file's own comment block EXPLAINS at length why
-# substituteFrom must never be added here, and a naive substring test would fire on the warning.
-flux_code = "\n".join(l for l in flux_text.splitlines() if not l.lstrip().startswith("#"))
-check("substituteFrom" not in flux_code,
-      f"clusters/base/{PREFIX}.yaml declares postBuild.substituteFrom. It MUST NOT: this tier "
-      f"renders documentation into a ConfigMap, and Flux's envsubst would run over the prose and "
-      f"silently EMPTY every `${{...}}` shell example in every runbook — it fails open, so the "
-      f"Kustomization would still report Ready (DECISIONS-LOG 2026-09-01, the emptied Cilium "
-      f"k8sServiceHost). The flux-postbuild-substitution check cannot see this: it scans files "
-      f"reachable through `resources:`, not configMapGenerator inputs")
-check(re.search(rf"^\s*-\s*{re.escape(PREFIX)}\.yaml\b", read(BASE_KUST), re.M) is not None,
-      f"clusters/base/kustomization.yaml does not list {PREFIX}.yaml — the Flux Kustomization is "
-      f"authored and never applied, which is the 'looks done, is absent' class this repo has been "
-      f"bitten by repeatedly")
-
-# ============================================ (7) this check is load-bearing in the merge gate
-L0 = read(ROOT / "ci" / "l0.sh")
-GATE = read(ROOT / "ci" / "gate.sh")
-# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism, in two halves.
-# It is not enough that check.sh exists: the merge gate must SELECT it on every run. The repo used to
-# get that with a hand-written "UNCONDITIONAL" stage per whole-repo check in ci/l0.sh, hardcoding the
-# path; those stages are gone, because the next whole-repo invariant only got one if somebody
-# REMEMBERED to write it. The property is now DECLARED by the test and HONOURED by the gate, so both
-# halves are asserted — neither alone is sufficient. The mechanism itself (that the union really
-# selects, and that a scope-declared SKIP exits 3 rather than passing green) is proven end-to-end and
-# falsifiably by conformance/tests/l0-static/whole-repo-invariants-wired.
-SELF_YAML = (ROOT / "conformance" / "tests" / "l0-static" / "alert-runbooks-resolvable"
-             / "chainsaw-test.yaml")
-SELF = read(SELF_YAML)
-check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
-      "alert-runbooks-resolvable/chainsaw-test.yaml does not declare "
-      "conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
-      "change happens to touch a requirement it is labelled with. ci/touched-requirements.sh maps "
-      "platform/monitoring/* to 'R-OBS-01 R-OBS-02' (NOT R-OBS-04) and maps platform/kubevirt/* and "
-      "runbooks/* to nothing at all — so adding an alert with a new runbook_url, or adding a runbook "
-      "without its configMapGenerator entry, would be filtered out of exactly the change that "
-      "introduces the dead link")
-check("--include-scope" in L0,
-      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
-      "purely requirement-scoped run again and the scope annotation above is inert decoration — this "
-      "check would then run only by coincidence")
-check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
-      "ci/gate.sh does not set the merge block when the l0 stage fails")
-
-# ======================================================================== report
-print(f"Alert runbook inventory: {n_annotations} runbook_url annotation(s) across "
-      f"{len(rule_files)} PrometheusRule file(s) -> {len(names)} distinct runbook(s), "
-      f"prefix /{PREFIX}")
-missing = sorted(n for n in names if not (RUNBOOK_DIR / f"{n}.md").is_file())
-print(f"  runbooks present: {len(names) - len(missing)}/{len(names)}   "
-      f"projected into the ConfigMap: {sum(1 for n in names if generated.get(n) == f'{n}.md')}/{len(names)}")
-if missing:
-    print(f"  MISSING: {', '.join(missing)}", file=sys.stderr)
-
-if ADVISORIES:
-    print("\nADVISORIES (not failures):")
-    for a in ADVISORIES:
-        print(f"  - {a}")
-
-print()
-if not FAILURES:
-    print(f"check OK: alert-runbooks-resolvable — {CHECKS} assertions (R-OBS-04)")
-    raise SystemExit(0)
-print(f"CHECK FAIL (alert-runbooks-resolvable): {len(FAILURES)} failed, "
-      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
-for f in FAILURES:
-    print(f"  FAIL {f}", file=sys.stderr)
-raise SystemExit(1)
-PY

conformance/tests/l0-static/auditability-proofs-producible/chainsaw-test.yaml

@@ -1,114 +0,0 @@
-# =============================================================================
-# L0 STATIC — the five auditability proofs can be produced on demand  (R-SEC-03)  [EXECUTABLE]
-# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
-# Spec:    docs/00 R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an
-#          operator can prove isolation, egress path, cert validity, and backup integrity)"),
-#          made concrete by docs/07 §"Auditability requirements (R-SEC-03) — what an operator
-#          must be able to prove": Isolation (denied-probe test), Egress path (egress IP
-#          check), Kill-switch (fault-injection test), TLS (cert status + external check),
-#          Backups (DR drill) — "These proofs are part of the platform acceptance suite, not
-#          one-time checks", with the acceptance gate "The five auditability proofs above can
-#          be produced on demand."
-#
-# HOW THIS AVOIDS BEING A TAUTOLOGY (the whole design question for a meta-requirement)
-# -----------------------------------------------------------------------------------
-# "A test exists for R-NET-01, therefore isolation is auditable" would pass in any suite that
-# compiles and would keep passing while every proof rotted into a scaffold. Four distinct
-# failure modes are asserted instead, each by DRIVING the real machinery:
-#   (a) each of the five proofs has an EXECUTABLE test — not a placeholder scaffold — at the layer
-#       the spec's own words require (fault injection / a DR drill => L3), taken from
-#       lib/status.py's own verdict so this check cannot disagree with the status report;
-#   (b) the proofs are PRODUCIBLE by the documented command: `run.sh --requirements <the five>`
-#       must resolve to real test dirs AND pass the merge gate's coverage assertion;
-#   (c) each Chainsaw-layer proof declares a PASS SENTINEL that lib/record_run.py can
-#       attribute — a proof that passes but records nothing leaves the operator with a green
-#       terminal and no evidence;
-#   (d) the evidence chain REFUSES FORGERIES: this check mounts two of its own against the
-#       real tools — a green JUnit for a real proof test whose log holds only that test's SKIP
-#       token, and a ledger row claiming `proven` at the repository's ROOT commit — and
-#       requires record_run.py to report not-proven and status.py to report `stale-proof`. It
-#       also drives `record_run.py --selftest` and `status.py --selftest`, which NOTHING in
-#       ci/ or run.sh ran before this test existed.
-# The five proofs are not hand-listed where the spec can be read: when the read-only spec repo
-# is mounted, docs/07's bullet labels are parsed and must match this check's mapping exactly,
-# so a sixth proof added to the spec fails here instead of being silently unverified.
-#
-# WHY L0: every clause is a property of the repository and its own tooling, needs no cluster,
-# and belongs in the merge gate — an audit chain is worth having only if it is intact BEFORE
-# anyone needs it.
-#
-# GIVEN the repository as git tracks it (plus the read-only spec repo when mounted),
-# WHEN  lib/status.py, lib/common.sh's requirement resolver, run.sh's coverage gate,
-#       lib/record_run.py's sentinel parser and both evidence tools' self-tests are DRIVEN,
-#       and two synthetic forgeries are presented to them,
-# THEN  each of docs/07's five proofs maps to an executable test at the required layer, the
-#       documented one-command invocation selects and asserts them, each is capturable as a
-#       ledger row, and neither a self-skip nor a stale row can be recorded as proof.
-#
-# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + the harness's
-# own stdlib tooling; no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run
-# it directly, and this Chainsaw Test wraps the SAME script.
-#
-# WHAT THIS TEST FIXED WHILE BEING WRITTEN (recorded, not hidden): three of the tests that ARE
-# the automated half of docs/07's isolation and TLS proofs — l2-behavioral/default-deny,
-# l2-behavioral/trusted-tls and l2-behavioral/gateway-ingress-reachable — declared NO pass
-# sentinel, so lib/record_run.py could never record them as proven and those two audit proofs
-# were structurally unrecordable. Each now ends with a proof-of-pass sentinel step. The
-# assertion here was NOT relaxed to accommodate them (R-TEST-06).
-#
-# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
-#   * That the five proofs PASS. This asserts they are present, strong, runnable by one
-#     command, capturable, and that the chain refuses forgeries — not that isolation holds
-#     today (R-NET-01's test) or that the restore works (R-STATE-03's).
-#   * HUBBLE. docs/07 names Hubble flow logs as part of the isolation proof; no test in this
-#     suite queries Hubble. Only the denied-probe half is automated — a real, named gap.
-#   * LEDGER COMPLETENESS OVER TIME. Nothing offline can prove a row was never removed; that
-#     would need signed history.
-#   * R-SEC-03's wording is broader than docs/07's five bullets. Suite-wide coverage of every
-#     requirement is R-TEST-01/04's claim (tests/meta/harness-selfcheck) and is deliberately
-#     not re-asserted here.
-# =============================================================================
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: auditability-proofs-producible
-  labels:
-    conformance.home.k3s/R-SEC-03: "true"
-  annotations:
-    conformance.home.k3s/requirements: "R-SEC-03"
-    conformance.home.k3s/layer: "L0"
-    # WHOLE-REPO SCOPE. This is a META-invariant over the ENTIRE conformance suite: the five
-    # spec-named proofs must each still have an EXECUTABLE test at the right layer, with a
-    # pass sentinel, resolvable by the documented command. Every one of its inputs lives under
-    # conformance/, which ci/touched-requirements.sh keys to R-TEST-01..04 — never to R-SEC-03.
-    # So downgrading conformance/tests/l3-fault-injection/dr-restore to a scaffold attributes
-    # R-STATE-03/R-TEST-*, and requirement-scoped selection drops the one check that notices
-    # the audit evidence just stopped being producible.
-    conformance.home.k3s/scope: "whole-repo"
-    conformance.home.k3s/safety-critical: "false"
-    conformance.home.k3s/executable: "true"
-    conformance.home.k3s/intent: >-
-      Given the tracked tree and the harness's own tooling, when status.py, the requirement
-      resolver, run.sh's coverage gate, record_run.py's sentinel parser and both evidence
-      tools' self-tests are driven and two synthetic forgeries are presented to them, then each
-      of docs/07's five auditability proofs has an executable test at the layer the spec
-      requires, is selectable and asserted by the documented one-command invocation, is
-      capturable as a ledger row, and neither a self-skip nor a stale proof can be recorded as
-      evidence.
-spec:
-  steps:
-    - name: then-each-of-the-five-proofs-is-producible-and-the-evidence-chain-refuses-forgeries
-      description: >-
-        Run the offline assertion script over the tracked tree (same script ci/l0.sh and
-        conformance/run.sh --layer l0 execute).
-      try:
-        - script:
-            content: bash ./check.sh
-# ---------------------------------------------------------------------------------------------
-# NOTE ON WORDING (deliberate, do not "improve"): this file and check.sh say "placeholder
-# scaffold" rather than the four-letter marker lib/status.py looks for, because that detector is
-# a bare substring test over the whole file. Writing the marker in PROSE — even to say a proof
-# must NOT be one — makes status.py classify THIS test as a scaffold, so the R-SEC-03 test would
-# report itself as unexecutable. Observed and worked around here; the detector's crudeness is
-# reported to the PM rather than loosened, since loosening a fake-green guard to suit one file is
-# the wrong trade (R-TEST-06).

conformance/tests/l0-static/auditability-proofs-producible/check.sh

@@ -1,454 +0,0 @@
-#!/usr/bin/env bash
-# =============================================================================
-# L0 static check (offline, no cluster) — auditability-proofs-producible
-# Serves: R-SEC-03 ("All the above properties MUST be demonstrable/auditable (an operator can
-#         prove isolation, egress path, cert validity, and backup integrity)."
-#         — spec docs/00-vision-and-scope.md).
-#
-# The spec makes this meta-requirement concrete. docs/07-security-and-secrets.md
-# §"Auditability requirements (R-SEC-03) — what an operator must be able to prove":
-#   - **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a
-#     denied-probe test.
-#   - **Egress path**: "service X's public IP is the VPN's" — via an egress IP check.
-#   - **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test.
-#   - **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert
-#     status + external check.
-#   - **Backups**: "irreplaceable data is restorable" — via a DR drill.
-#   "These proofs are part of the platform acceptance suite, not one-time checks."
-# and its acceptance gate: "The five auditability proofs above can be produced on demand."
-#
-# HOW A META-REQUIREMENT IS TESTED WITHOUT BECOMING A TAUTOLOGY
-# ------------------------------------------------------------
-# The trap is obvious and this check is built to avoid it: "a test exists for R-NET-01,
-# therefore isolation is auditable" is true of any suite that compiles, proves nothing, and
-# would keep passing while every proof rotted into a scaffold. Four separate failure modes
-# have to be caught for "an operator can PROVE it" to mean anything, and each one is asserted
-# by DRIVING the real machinery rather than by reading files:
-#
-#   (a) THE PROOF EXISTS, AT THE RIGHT STRENGTH. Each of the five named proofs maps to a
-#       requirement that must have an EXECUTABLE verifying test — not a placeholder scaffold — and,
-#       where the spec's own words name fault injection ("with VPN down"; "a DR drill"), an
-#       executable L3 test specifically. Executable-vs-scaffold is not re-derived here: this
-#       check RUNS lib/status.py and reads its verdict, so it cannot disagree with the
-#       repository's own status report.
-#   (b) THE PROOF IS PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND. `run.sh
-#       --requirements <the five IDs>` must resolve to a non-empty test set and must ASSERT
-#       their coverage (that flag is the merge gate's own path: it passes --require to
-#       coverage.py). This check drives run.sh itself, so if the resolver, the labels or the
-#       gate wiring break, R-SEC-03 fails.
-#   (c) THE PROOF IS CAPTURABLE. A test that passes but records nothing leaves an operator
-#       with a green terminal and no evidence. For each Chainsaw-layer proof, the test must
-#       declare a PASS SENTINEL that lib/record_run.py can actually attribute — computed with
-#       record_run.py's OWN parser, so deleting a test's sentinel makes this check fail
-#       instead of silently making the requirement unprovable.
-#   (d) THE EVIDENCE CANNOT BE FORGED, AND IS NOT HONOURED WHEN STALE. This is the clause
-#       that makes the check bite. Two INDEPENDENT forgery attempts are mounted against the
-#       real tools, in a throwaway directory:
-#         * a Chainsaw run in which the JUnit says a real proof test PASSED while the run log
-#           carries only that test's SKIP token — the exact shape of a self-skipping L2/L3
-#           test. record_run.py must refuse to call it `proven`.
-#         * a ledger row claiming `proven` at a commit that PREDATES the requirement's
-#           implementing code — status.py must degrade it to `stale-proof`, never green.
-#       Both tools' own built-in self-tests (`--selftest`) are also driven, because nothing
-#       in ci/ or run.sh runs them today: the guards that keep a skip from becoming a proof
-#       were themselves unexercised in the merge gate until this check.
-#
-#   The FIVE PROOFS ARE NOT HAND-LISTED where the spec can be read: when the read-only spec
-#   repo is mounted, the bullet labels are parsed out of docs/07 and must match this check's
-#   mapping EXACTLY. Add a sixth proof to the spec and this check fails as unmapped rather
-#   than quietly ignoring it.
-#
-# WHY L0: every clause above is a property of the repository and its own tooling, provable
-# offline, and it belongs in the merge gate — the point of an audit chain is that it is intact
-# BEFORE anyone needs it. Nothing here needs a cluster, and nothing here would be more true
-# with one.
-#
-# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
-#   - THAT THE FIVE PROOFS PASS. This asserts the proofs are present, strong, runnable by one
-#     documented command, capturable as evidence, and that the evidence chain refuses
-#     forgeries. Whether isolation actually holds today is R-NET-01's test; whether the DR
-#     restore works is R-STATE-03's. R-SEC-03 is about the operator's ABILITY to produce the
-#     proof, which is what is asserted.
-#   - HUBBLE. docs/07 names "Hubble flow logs" as part of the isolation proof. No test in this
-#     suite queries Hubble, and this check does not pretend otherwise: it asserts the
-#     denied-probe half, which is the half that is automated. The Hubble half remains an
-#     operator-interactive capability — a real, named gap in R-SEC-03's coverage.
-#   - THE OPERATOR-FACING NARRATIVE. That a human can find and read the evidence (the status
-#     report's prose, the ledger's provenance fields) is a usability property, not a testable
-#     one.
-#   - LEDGER COMPLETENESS OVER TIME. The ledger is append-only by convention and by review;
-#     nothing here can prove a row was never removed from it (that would need signed history).
-#   - THAT THE PROOFS COVER "ALL THE ABOVE PROPERTIES". R-SEC-03's own wording is broader than
-#     the five bullets docs/07 enumerates. The five are what the spec made concrete and are
-#     what is asserted; suite-wide coverage of every requirement is R-TEST-01/04's claim,
-#     tested by tests/meta/harness-selfcheck, and is deliberately NOT re-asserted here.
-# =============================================================================
-set -euo pipefail
-DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
-PYTHON="${PYTHON:-python3}"
-: "${SPEC_REPO:=${ROOT}/../home-network-k3s}"
-export SPEC_REPO
-
-cd "${ROOT}"
-"${PYTHON}" - "${ROOT}" <<'PY'
-import json
-import os
-import re
-import shutil
-import subprocess
-import sys
-import tempfile
-from pathlib import Path
-
-ROOT = Path(sys.argv[1])
-CONF = ROOT / "conformance"
-LIB = CONF / "lib"
-FAILURES: list[str] = []
-NOTES: list[str] = []
-CHECKS = 0
-
-
-def check(cond, msg):
-    global CHECKS
-    CHECKS += 1
-    if not cond:
-        FAILURES.append(msg)
-
-
-def note(msg):
-    NOTES.append(msg)
-
-
-def die():
-    print(f"CHECK FAIL (auditability-proofs-producible): {len(FAILURES)} of {CHECKS} "
-          "assertions failed", file=sys.stderr)
-    for f in FAILURES:
-        print(f"  - {f}", file=sys.stderr)
-    raise SystemExit(1)
-
-
-def run(cmd, **kw) -> subprocess.CompletedProcess:
-    return subprocess.run(cmd, capture_output=True, text=True, cwd=kw.pop("cwd", CONF), **kw)
-
-
-# ================================================= THE FIVE PROOFS docs/07 ENUMERATES
-# key  = the bullet label docs/07 uses (verified against the spec below when it is mounted)
-# reqs = the requirement(s) whose verifying test IS that proof
-# layer= the WEAKEST layer the spec's own words allow. Where docs/07 says "a fault-injection
-#        test" / "a DR drill", an L2 happy-path test does not discharge the proof.
-PROOFS = {
-    "Isolation":   {"reqs": ["R-NET-01"], "layers": {"L2", "L3"},
-                    "why": '"service X cannot reach service Y" — via Hubble flow logs + a '
-                           "denied-probe test"},
-    "Egress path": {"reqs": ["R-NET-05"], "layers": {"L2", "L3"},
-                    "why": '"service X\'s public IP is the VPN\'s" — via an egress IP check'},
-    "Kill-switch": {"reqs": ["R-NET-06"], "layers": {"L3"},
-                    "why": '"with VPN down, X leaks nothing" — via a fault-injection test'},
-    "TLS":         {"reqs": ["R-TLS-01"], "layers": {"L2", "L3"},
-                    "why": '"every endpoint serves a valid, non-self-signed, unexpired cert" '
-                           "— via cert status + external check"},
-    "Backups":     {"reqs": ["R-STATE-03"], "layers": {"L3"},
-                    "why": '"irreplaceable data is restorable" — via a DR drill'},
-}
-PROOF_REQS = sorted({r for p in PROOFS.values() for r in p["reqs"]})
-
-# ---- SELF-POLICING: the proof list must still be the SPEC's proof list ---------------
-spec = Path(os.environ.get("SPEC_REPO", ""))
-sec_doc = spec / "docs" / "07-security-and-secrets.md"
-if sec_doc.is_file():
-    text = sec_doc.read_text()
-    m = re.search(r"^##\s+Auditability requirements \(R-SEC-03\).*?$(.*?)^##\s", text,
-                  re.M | re.S)
-    check(m is not None,
-          "docs/07 no longer carries an '## Auditability requirements (R-SEC-03)' section — "
-          "the proof list this check is coupled to has moved or been removed, so the mapping "
-          "below can no longer be trusted (R-TEST-06: follow the spec, never the other way)")
-    if m:
-        labels = re.findall(r"^\s*-\s+\*\*([^*]+)\*\*:", m.group(1), re.M)
-        check(set(labels) == set(PROOFS),
-              f"docs/07 enumerates the auditability proofs {sorted(labels)} but this check "
-              f"maps {sorted(PROOFS)} — every proof the spec names must map to a verifying "
-              "test, and a proof this check does not know about would be silently unverified")
-        check(len(labels) == 5,
-              f"docs/07 enumerates {len(labels)} auditability proof(s); its own acceptance "
-              "gate says 'The five auditability proofs above can be produced on demand' — the "
-              "prose and the list have drifted apart")
-else:
-    note(f"the read-only spec repo is not mounted at {spec} — the proof list could not be "
-         "re-derived from docs/07, so the mapping below was used as written. Every other "
-         "assertion still ran. (ci/l0.sh mounts the spec; a bare local run may not.)")
-
-# ============================================ (a) EACH PROOF EXISTS, AT THE RIGHT STRENGTH
-# Driven through lib/status.py so this check and the repository's own status report cannot
-# disagree about what is executable and what is a scaffold.
-tmp = Path(tempfile.mkdtemp(prefix="r-sec-03-"))
-try:
-    out = tmp / "status"
-    st = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
-              "--registry", str(CONF / "requirements" / "registry.yaml"),
-              "--tests", str(CONF / "tests"), "--out", str(out)])
-    check(st.returncode == 0,
-          f"lib/status.py exited {st.returncode} — the report an operator would use to find "
-          f"the audit evidence does not generate: {(st.stderr or st.stdout).strip()[:400]}")
-    status = {}
-    sj = out / "status.json"
-    if sj.is_file():
-        status = json.loads(sj.read_text())
-    check(bool(status.get("by_requirement")),
-          "lib/status.py produced no per-requirement report — the audit posture of each "
-          "requirement is not derivable, so R-SEC-03 cannot be assessed at all")
-    by_req = status.get("by_requirement", {})
-
-    for label, spec_proof in sorted(PROOFS.items()):
-        for rid in spec_proof["reqs"]:
-            rec = by_req.get(rid)
-            check(rec is not None,
-                  f"proof '{label}' maps to {rid}, which the status report does not know — "
-                  "either the registry lost the requirement or this mapping is stale")
-            if not rec:
-                continue
-            tests = (rec.get("verification") or {}).get("tests") or []
-            executable = [t for t in tests if t.get("state", "").startswith("executable")]
-            check(bool(executable),
-                  f"proof '{label}' ({rid}) has no EXECUTABLE verifying test — its tests are "
-                  f"{[t.get('name') + ':' + t.get('state', '?') for t in tests]}. docs/07: "
-                  f"{spec_proof['why']}. A placeholder scaffold cannot be produced on demand, so the "
-                  "operator cannot prove this property")
-            at_layer = [t for t in executable if t.get("layer") in spec_proof["layers"]]
-            check(bool(at_layer),
-                  f"proof '{label}' ({rid}) has no executable test at layer(s) "
-                  f"{sorted(spec_proof['layers'])} — it has "
-                  f"{[t.get('name') + '@' + str(t.get('layer')) for t in executable]}. "
-                  f"docs/07 requires this proof to be produced {spec_proof['why']}, and a "
-                  "weaker layer does not discharge it (a rendering check is not a drill)")
-
-    # ==================================== (b) PRODUCIBLE ON DEMAND, BY THE DOCUMENTED COMMAND
-    # The resolver: --requirements must translate the five R-IDs into real test dirs.
-    resolver = run(["bash", "-c",
-                    f'export CONF_ROOT="{CONF}"; . "{LIB}/common.sh"; '
-                    f'conf_dirs_for_requirements "{" ".join(PROOF_REQS)}"'])
-    dirs = [l for l in resolver.stdout.splitlines() if l.strip()]
-    check(resolver.returncode == 0 and len(dirs) >= len(PROOF_REQS),
-          f"`run.sh --requirements {','.join(PROOF_REQS)}` resolves to {len(dirs)} test dir(s) "
-          f"for {len(PROOF_REQS)} proof requirement(s) — the single command docs/07's "
-          "acceptance gate depends on ('produced on demand') cannot select the proofs")
-
-    # The gate: the same flag ASSERTS coverage (run.sh passes --require to coverage.py), so a
-    # zero exit here is the repository's own statement that all five proofs are covered.
-    gate = run(["./run.sh", "--coverage-only", "--requirements", ",".join(PROOF_REQS)])
-    check(gate.returncode == 0,
-          f"`./run.sh --coverage-only --requirements {','.join(PROOF_REQS)}` exited "
-          f"{gate.returncode} — the merge gate's own coverage assertion rejects the audit "
-          f"proof set: {(gate.stderr or gate.stdout).strip()[:400]}")
-
-    # ================================================ (c) THE PROOF IS CAPTURABLE AS EVIDENCE
-    # Use record_run.py's OWN sentinel parser: a Chainsaw-layer proof with no declared pass
-    # token can pass forever and never be recorded as proof of anything.
-    sys.path.insert(0, str(LIB))
-    os.environ.setdefault("PYTHONDONTWRITEBYTECODE", "1")
-    sys.dont_write_bytecode = True
-    import record_run as rr                                   # noqa: E402
-
-    # status.py owns the latest-wins rule; import it rather than reimplementing it, so the two
-    # cannot disagree about which rows are live evidence.
-    import status as _st                                        # noqa: E402
-    rr_latest_by_test = _st.latest_by_test
-
-    smap = rr.sentinel_map(CONF / "tests")
-    check(len(smap) >= 10,
-          f"record_run.py's sentinel map contains {len(smap)} test(s) — the parser lost its "
-          "grip on the test tree, so the assertions below would assert nothing")
-    for label, spec_proof in sorted(PROOFS.items()):
-        for rid in spec_proof["reqs"]:
-            rec = by_req.get(rid) or {}
-            tests = (rec.get("verification") or {}).get("tests") or []
-            names = [t.get("name") for t in tests
-                     if t.get("state", "").startswith("executable")
-                     and t.get("layer") in spec_proof["layers"]]
-            for name in names:
-                ent = smap.get(name)
-                check(ent is not None,
-                      f"proof '{label}': record_run.py does not know the test {name!r} — its "
-                      "result could never be attributed to a ledger row")
-                if not ent:
-                    continue
-                if ent.get("layer") in rr.SENTINEL_EXEMPT_LAYERS:
-                    continue          # offline layers are truthful by JUnit alone
-                check(bool(ent.get("pass")),
-                      f"proof '{label}': the Chainsaw test {name!r} declares NO pass sentinel, "
-                      "so lib/record_run.py can never record it as proven. It would go green "
-                      "in the terminal and leave the operator with no evidence — exactly the "
-                      "gap the ledger exists to close (docs/07: the proofs are 'part of the "
-                      "platform acceptance suite, not one-time checks')")
-                check(bool(ent.get("terminal_pass")),
-                      f"proof '{label}': the test {name!r} has no TERMINAL pass token, so "
-                      "record_run.py cannot decide its verdict")
-
-    # =============================== (d) THE EVIDENCE CANNOT BE FORGED, NOR HONOURED WHEN STALE
-    # Both tools' own guards, which NOTHING in ci/ or run.sh drives today.
-    for tool in ("record_run.py", "status.py"):
-        r = run([sys.executable, str(LIB / tool), "--selftest"])
-        check(r.returncode == 0,
-              f"`lib/{tool} --selftest` exited {r.returncode} — the guards that keep a "
-              "self-skipped or stale run from being recorded as proof are broken, so nothing "
-              f"in the evidence chain can be trusted: {(r.stderr or r.stdout).strip()[:500]}")
-
-    # ---- FORGERY 1: a green JUnit for a REAL proof test whose log holds only a SKIP token.
-    # This is the precise shape of a self-skipping L2/L3 test, and it must NOT become proof.
-    forge_target, forge_skip = None, None
-    for label, spec_proof in sorted(PROOFS.items()):
-        for rid in spec_proof["reqs"]:
-            for name, ent in smap.items():
-                if rid in (ent.get("requirements") or []) and ent.get("skip") \
-                        and ent.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS:
-                    forge_target, forge_skip = name, ent["skip"][0]
-                    break
-            if forge_target:
-                break
-        if forge_target:
-            break
-    check(forge_target is not None,
-          "no Chainsaw-layer proof test declares a skip token, so the forgery attempt below "
-          "could not be mounted. That is not a pass: it means this check could not verify "
-          "that a self-skip is refused as proof, which is the whole point of the ledger")
-    if forge_target:
-        run_dir = tmp / "20260101T000000Z-forged"
-        run_dir.mkdir(parents=True)
-        (run_dir / "report.junit.xml").write_text(
-            '<?xml version="1.0" encoding="UTF-8"?>\n'
-            '<testsuites><testsuite name="chainsaw" tests="1" failures="0" errors="0">'
-            f'<testcase name="{forge_target}" classname="forged"/>'
-            "</testsuite></testsuites>\n")
-        (run_dir / "run.log").write_text(
-            f"=== RUN   chainsaw/{forge_target}\n"
-            f"    === STDOUT\n        {forge_skip}\n")
-        forged = run([sys.executable, str(LIB / "record_run.py"), "--run-dir", str(run_dir),
-                      "--dry-run", "--ledger", str(tmp / "never-written.jsonl"),
-                      "--tests", str(CONF / "tests"), "--repo", str(ROOT)])
-        verdicts = re.findall(rf"^\[record\]\s+\S+\s+(\S+)\s+{re.escape(forge_target)}\b",
-                              forged.stdout, re.M)
-        check(verdicts and all(v != "proven" for v in verdicts),
-              f"lib/record_run.py recorded a FORGED run as {verdicts!r} for {forge_target!r}: "
-              "a JUnit saying `passed` with nothing in the log but that test's own SKIP token "
-              f"({forge_skip}). A self-skip must never become proof — with this hole every "
-              "audit proof in docs/07 could be produced by a cluster that skipped it")
-        check(not (tmp / "never-written.jsonl").exists(),
-              "record_run.py --dry-run WROTE to the ledger — a tool that records while "
-              "claiming not to cannot be trusted to record only real runs")
-
-    # ---- FORGERY 2: a ledger row claiming `proven` at a commit that predates the code.
-    # status.py must degrade it to stale-proof (not green), or an old green run would keep
-    # vouching for code it never executed.
-    root_sha = run(["git", "-C", str(ROOT), "rev-list", "--max-parents=0", "HEAD"])
-    old_sha = (root_sha.stdout.strip().splitlines() or [""])[0]
-    check(len(old_sha) == 40,
-          "the repository's root commit could not be resolved, so the stale-proof guard could "
-          "not be exercised (that is a gap in this check, not a pass)")
-    if len(old_sha) == 40 and forge_target:
-        stale_ledger = tmp / "stale.jsonl"
-        rid = sorted(smap[forge_target].get("requirements") or ["R-NET-06"])[0]
-        stale_ledger.write_text(json.dumps({
-            "ts_utc": "2026-01-01T00:00:00Z", "git_sha": old_sha, "env": "dev",
-            "chainsaw_version": "v0.0.0", "kube_version": "unknown",
-            "layer": smap[forge_target].get("layer", "L3"), "test": forge_target,
-            "requirements": [rid], "junit_result": "passed",
-            "sentinel": (smap[forge_target].get("terminal_pass") or "X"),
-            "proof": "proven", "run_id": "20260101T000000Z-forged",
-            "note": "synthetic row written by the R-SEC-03 conformance check",
-        }) + "\n")
-        out2 = tmp / "status-stale"
-        st2 = run([sys.executable, str(LIB / "status.py"), "--repo", str(ROOT),
-                   "--registry", str(CONF / "requirements" / "registry.yaml"),
-                   "--tests", str(CONF / "tests"), "--out", str(out2),
-                   "--ledger", str(stale_ledger)])
-        check(st2.returncode == 0,
-              f"lib/status.py failed against a synthetic ledger: "
-              f"{(st2.stderr or st2.stdout).strip()[:300]}")
-        s2 = json.loads((out2 / "status.json").read_text()) if (out2 / "status.json").is_file() else {}
-        rec2 = (s2.get("by_requirement") or {}).get(rid) or {}
-        per = {p.get("test"): p.get("state") for p in ((rec2.get("proof") or {}).get("per_test") or [])}
-        check(per.get(forge_target) == "stale-proof",
-              f"a ledger row claiming `proven` for {forge_target!r} at the repository's ROOT "
-              f"commit was reported as {per.get(forge_target)!r} instead of 'stale-proof'. A "
-              "proof from a run that predates the code it vouches for is not evidence, and "
-              "honouring it would let one old green run certify every later change")
-        check((rec2.get("proof") or {}).get("state") != "proven",
-              f"{rid} is reported PROVEN on the strength of a row from the root commit — the "
-              "freshness guard is not holding")
-
-    # ---- the ledger itself is machine-readable, and no row proves a test that is gone ----
-    ledger = CONF / "runs" / "ledger.jsonl"
-    if ledger.is_file():
-        rows, bad = [], []
-        for i, line in enumerate(ledger.read_text().splitlines(), 1):
-            if not line.strip():
-                continue
-            try:
-                rows.append(json.loads(line))
-            except json.JSONDecodeError:
-                bad.append(i)
-        check(not bad,
-              f"conformance/runs/ledger.jsonl has unparsable line(s) {bad} — the audit record "
-              "is not machine-readable, so no proof can be derived from it")
-        # Which rows are HONOURED: status.py resolves duplicates latest-wins (latest_by_test),
-        # so a superseded row is not evidence of anything. The dirty-tree assertion below is
-        # therefore scoped to honoured rows. Without that scoping, ONE historical dirty row reds
-        # this gate permanently even after a clean re-run of the same test — and a dirty-then-
-        # clean sequence is the normal shape of "noticed the tree was dirty, fixed it, re-ran".
-        # The row is still kept in the ledger and its run.log stays committed: nothing is erased,
-        # it simply stops being treated as a live claim. Structural assertions (provenance keys,
-        # sentinel, test-still-exists) still apply to EVERY row.
-        honoured_ids = {id(v) for v in rr_latest_by_test(rows).values()}
-        for r in rows:
-            for key in ("ts_utc", "git_sha", "test", "requirements", "junit_result", "proof",
-                        "run_id"):
-                check(key in r,
-                      f"a ledger row for {r.get('test', '?')!r} lacks {key!r} — a row without "
-                      "provenance cannot be audited")
-            if r.get("proof") == "proven":
-                # L0/L1 rows are recorded proven WITHOUT a sentinel, deliberately, by
-                # record_run.py's own SENTINEL_EXEMPT_LAYERS: run_l1_offline.py renders skips as
-                # JUnit <skipped/>, so those layers have no pass token to attribute. This
-                # assertion had NO layer exemption, so the two halves of the harness contradicted
-                # each other — ANY `--record` run touching L0 or L1 wrote a row this gate then
-                # rejected, which meant a requirement whose verifier set includes an L1 check
-                # (R-NET-06 is one) could never be both PROVEN and L0-green. Honour the same
-                # constant rather than restating the policy here, so the two cannot drift apart.
-                if r.get("layer") not in rr.SENTINEL_EXEMPT_LAYERS:
-                    check(bool(r.get("sentinel")),
-                          f"a ledger row claims `proven` for {r.get('test')!r} with NO sentinel — "
-                          "the sentinel is the only thing separating a real pass from a self-skip")
-                check(r.get("test") in smap,
-                      f"a ledger row claims `proven` for {r.get('test')!r}, which is not a "
-                      "test in the suite any more — a proof of something that no longer exists")
-                check(id(r) not in honoured_ids or "-dirty" not in str(r.get("git_sha", "")),
-                      f"a ledger row claims `proven` for {r.get('test')!r} from a DIRTY tree "
-                      f"({r.get('git_sha')}) — what ran is not what is committed")
-    else:
-        note("conformance/runs/ledger.jsonl does not exist yet — the ledger-shape assertions "
-             "were skipped; every other assertion ran, and the forgery probes above use "
-             "synthetic ledgers, not this file")
-finally:
-    shutil.rmtree(tmp, ignore_errors=True)
-
-# --------------------------------------------------------------------------- verdict
-# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
-check(CHECKS >= 30,
-      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
-      "would be meaningless")
-
-if FAILURES:
-    die()
-for n in NOTES:
-    print(f"  note: {n}")
-print(f"check OK: auditability-proofs-producible — {CHECKS} assertions; the five docs/07 "
-      f"proofs ({', '.join(sorted(PROOFS))}) each map to an executable test at the layer the "
-      f"spec's own wording requires, `run.sh --requirements {','.join(PROOF_REQS)}` resolves "
-      "and asserts them, each Chainsaw-layer proof declares a sentinel record_run.py can "
-      "attribute, both evidence tools' self-tests pass, and two independent forgeries — a "
-      "self-skip presented as a pass, and a `proven` row from the root commit — were REFUSED. "
-      "R-SEC-03's producibility half only: whether each proof PASSES is its own "
-      "requirement's test, and the Hubble half of the isolation proof is automated nowhere "
-      "(see the header's gap list).")
-PY

conformance/tests/l0-static/bulk-tier-distinct-from-replicated/chainsaw-test.yaml

@@ -1,114 +0,0 @@
-# =============================================================================
-# L0 STATIC — the bulk tier's capacity strategy is DISTINCT from the replicated tier's
-# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
-#
-# THE BUG CLASS THIS CLOSES. The `bulk` StorageClass existed from Phase 0 as a stub:
-# `provisioner: nfs.csi.k8s.io`, `server: REPLACE-ME`, `share: REPLACE-ME`, TODO(OQ-2).
-# Four ServiceDefinitions carried a comment promising to attach their media once OQ-2 was
-# resolved, and the ONLY thing keeping a multi-terabyte library off the 3-replica
-# `longhorn` class was those four comments. A bulk volume that lands on the replicated
-# tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
-# three mini-PCs cannot hold. R-STATE-04 is precisely that the two tiers are distinct, so
-# the distinctness is asserted rather than reviewed.
-#
-# It also pins the four decisions that make the STATIC form of this tier safe, each with a
-# plausible "simplification" that breaks something silently: `Retain` (bulk data pre-exists
-# the cluster and has consumers outside it, so anything else makes `delete pvc` a data-loss
-# operation); NO dynamic provisioning (a dynamic bulk class hands out a FRESH EMPTY
-# directory that mounts and Binds perfectly while the real 8 TiB is never attached — the
-# most deceptive failure available here); `hard` not `soft` (a soft mount hands a server
-# hiccup to the app as a short read, which for a media library is silent corruption); and
-# NFSv4 (mount and lock in-protocol ⇒ ONE reachable port, tcp/2049, which is what the
-# operator must open in pfSense for prod to reach this server at all).
-#
-# AND the EXDEV trap, which is not about storage. Sonarr/Radarr import by hardlinking or
-# atomically renaming, both same-filesystem operations, and to the kernel "same filesystem"
-# means SAME MOUNT. Splitting one export into two subPath mounts looks TIGHTER and breaks
-# link() with EXDEV; the arrs then degrade to copy-then-delete, rewriting every import over
-# NFS and destroying the seeding copy. It surfaces weeks later as "imports are slow and my
-# ratio is gone". A reviewer cannot see that; this check can.
-#
-# GIVEN the repository as authored,
-# WHEN  every `class: bulk` storage entry in catalog/ is compared against the objects the
-#       COMMITTED apps/ goldens carry for it (what Flux reconciles — deliberately not the
-#       renderer, which would only prove it agrees with itself),
-# THEN  (1) no bulk volume is on a Longhorn-provisioned class, and its class is explicitly
-#           EMPTY so nothing can dynamically provision an empty stand-in;
-#       (2) its capacity is declared and is at least an order of magnitude above the same
-#           service's replicated volume — the requirement stated as arithmetic;
-#       (3) PV and PVC are pre-bound from BOTH ends (claimRef + volumeName) and agree;
-#       (4) the PV reclaim policy is Retain;
-#       (5) mountOptions carry `hard` and exactly one `nfsvers=4.x`, and never `soft`;
-#       (6) no workload mounts one export more than once (the EXDEV trap);
-#       (7) a readOnly entry renders nfs.readOnly AND a readOnly volumeMount AND
-#           ReadOnlyMany; a writable one renders neither and is ReadWriteMany; every bulk
-#           access mode is a `...Many`, since one export serves several namespaces and
-#           several nodes;
-#       (8) the PV's nfs.server is a BARE cluster-vars token (never a literal — dev and
-#           prod reach this host by different addresses), that knob has a base default, and
-#           EVERY overlay supplies it;
-#   AND (9) the check is LOAD-BEARING (ci/l0.sh runs it unconditionally, ci/gate.sh blocks
-#           on the l0 stage) and NON-VACUOUS (an empty bulk corpus FAILS rather than
-#           passing trivially, and a floor on the assertion count keeps a pass from
-#           degenerating into a no-op).
-#
-# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
-# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
-# SAME script.
-#
-# NOT proven — the runtime half, which needs a cluster: that the server is reachable, that
-# the export admits these clients, that the PVC binds, that a write succeeds, or that a
-# hardlink between the library and the download path actually works. The uid analysis is
-# reasoning (docs/BULK-NFS-STORAGE.md §6) and the in-cluster probes that would settle it
-# are written down in §9 of that document, unrun. Nor can this prove prod's address is
-# CORRECT — only that it is a supplied knob; while it is a placeholder, prod's pods will
-# not mount, and catching that statically needs the placeholder guard, which is absent from
-# main (§7). Read-only on the media tree is a CLIENT-side guarantee only: the export serves
-# one reader and two writers from the same node addresses, so the server cannot tell them
-# apart (§5).
-# =============================================================================
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: bulk-tier-distinct-from-replicated
-  labels:
-    conformance.home.k3s/R-STATE-04: "true"
-  annotations:
-    conformance.home.k3s/requirements: "R-STATE-04"
-    conformance.home.k3s/layer: "L0"
-    # WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3e
-    # (UNCONDITIONAL)", which this annotation replaces:
-    #
-    #   The `bulk` StorageClass sat as a REPLACE-ME stub for six roadmap phases, and the only
-    #   thing keeping a multi-terabyte media library off the 3-replica `longhorn` class was four
-    #   hand-written comments in ServiceDefinitions. A bulk volume that lands on the replicated
-    #   tier does not fail loudly — Longhorn accepts the PVC and starts replicating 3x of data
-    #   three mini-PCs cannot hold.
-    #
-    #   WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's path map keys
-    #   NEITHER apps/* nor platform/longhorn/* to the bulk requirement. So the change most
-    #   likely to break this — moving a bulk volume onto the replicated tier, or narrowing one
-    #   export into two mounts — could attribute no requirement at all and be gated by nothing.
-    #   The invariant spans catalog/, apps/ and clusters/.
-    conformance.home.k3s/scope: "whole-repo"
-    conformance.home.k3s/safety-critical: "false"
-    conformance.home.k3s/executable: "true"
-    conformance.home.k3s/intent: >-
-      Bulk media storage is provably NOT on the 3-replica Longhorn tier, is sized for a
-      different order of magnitude, cannot be silently replaced by a dynamically
-      provisioned empty volume, is never reclaimable by the cluster, mounts hard over
-      NFSv4, and is never split into two mounts of one export — so hardlinked imports keep
-      working. Its server is a per-environment knob, not a literal, because the two
-      clusters reach the same storage host by different addresses.
-spec:
-  steps:
-    - name: assert-bulk-tier-is-distinct-and-statically-bound
-      description: >-
-        Read every `class: bulk` storage entry from catalog/, pair it with the
-        PersistentVolume, PersistentVolumeClaim and volumeMount the committed apps/ goldens
-        carry for it, and assert the tier separation, the static pre-binding, the reclaim
-        and mount-option decisions, the read-only coupling, the one-export-one-mount
-        invariant, and that the server address is supplied by every environment overlay.
-      try:
-        - script:
-            content: bash ./check.sh

conformance/tests/l0-static/bulk-tier-distinct-from-replicated/check.sh

@@ -1,496 +0,0 @@
-#!/usr/bin/env bash
-# =============================================================================
-# L0 static check (offline, no cluster) — bulk-tier-distinct-from-replicated
-# Serves: R-STATE-04 ("Bulk media storage has a capacity strategy distinct from the
-#                      replicated tier." — spec docs/04-storage-and-state.md)
-#
-# WHY THIS EXISTS — THE BULK TIER WAS A STUB FOR SIX ROADMAP PHASES, AND STUBS ROT QUIETLY
-# ----------------------------------------------------------------------------------------
-# `platform/longhorn/storageclasses.yaml` declared a `bulk` StorageClass from Phase 0 with
-# `provisioner: nfs.csi.k8s.io`, `server: REPLACE-ME`, `share: REPLACE-ME` and a TODO(OQ-2).
-# Four ServiceDefinitions carried a comment promising to attach their media "once OQ-2 is
-# resolved", and each of them said, correctly, that adding `class: bulk` before then would
-# leave the PVC unbound and the pod Pending. So the tier existed as a NAME with nothing
-# behind it, and the only thing preventing a multi-terabyte media library from being
-# authored onto the 3-replica `longhorn` class was four hand-written comments.
-#
-# That is the failure this check makes impossible to reintroduce. R-STATE-04 is not "bulk
-# storage exists" — it is that bulk storage's capacity strategy is DISTINCT from the
-# replicated tier's. Three mini-PCs cannot hold 3x of 8.33 TiB. A bulk volume that lands on
-# `longhorn` does not fail loudly: Longhorn accepts the PVC and starts replicating, and the
-# tier fills up. Distinctness is therefore a property worth asserting statically, on every
-# gate run, rather than trusting to review.
-#
-# It also pins the four decisions that make the STATIC form of this tier safe, each of
-# which has a plausible-looking "simplification" that silently breaks something:
-#
-#   * RECLAIM POLICY. Bulk data pre-exists the cluster, is larger than the cluster, and has
-#     consumers OUTSIDE it. `Retain` is the whole reason a PV is used here instead of an
-#     inline volume. `Delete` on a dynamically-provisioned bulk class would make `kubectl
-#     delete pvc` a data-loss operation.
-#   * NO DYNAMIC PROVISIONING. A dynamic bulk class hands out FRESH EMPTY directories. The
-#     pod mounts successfully, the PVC is Bound, everything is green — and the 8 TiB of
-#     existing data was never attached. This is the single most deceptive failure available
-#     in this tier, so `storageClassName` must be explicitly empty and binding must be
-#     pinned from BOTH ends (claimRef + volumeName).
-#   * `hard`, NOT `soft`. A soft NFS mount converts a server hiccup into a short read or
-#     EIO handed to the application. For a media library that is silent corruption. `hard`
-#     hangs instead — recoverable, where truncated is not.
-#   * NFSv4. v4 carries mount and lock in-protocol, so the tier needs ONE reachable port
-#     (tcp/2049) instead of v3's rpcbind/111 + mountd + statd on ephemeral numbers. That is
-#     a FIREWALL fact as much as a storage one: prod reaches this server through pfSense,
-#     and the number of ports the operator must open is a consequence of this field.
-#
-# AND ONE INVARIANT THAT IS NOT ABOUT STORAGE AT ALL — the EXDEV trap (assertion 6).
-# Sonarr and Radarr import by hardlinking or atomically renaming a finished download into
-# the library. Both are same-filesystem operations, and to the kernel "same filesystem"
-# means SAME MOUNT. Mounting one export twice (say `subPath: tv` and `subPath: downloads`)
-# looks TIGHTER — less of the tree exposed per mount — and breaks `link()` with EXDEV no
-# matter that the server knows they are one dataset. The arrs do not fail loudly on that:
-# they degrade to copy-then-delete, rewriting every imported file over NFS and destroying
-# the seeding copy. The symptom appears weeks later as "imports are slow and my ratio is
-# gone". A reviewer cannot see it, so this check does: one export, at most one mount, per
-# workload.
-#
-# GIVEN the repository as authored,
-# WHEN  every `class: bulk` storage entry in catalog/ is compared against the manifests
-#       render.py produces for it (read from the committed apps/ goldens, i.e. what Flux
-#       actually reconciles — not from the renderer, which would only agree with itself),
-# THEN  (1) DISTINCTNESS: no bulk volume is on the replicated tier. Its PV/PVC
-#           storageClassName is explicitly empty — never `longhorn`, never any class the
-#           repo declares with the Longhorn provisioner — and its PV's source is an
-#           EXTERNAL one (nfs), not a Longhorn volume;
-#       (2) CAPACITY STRATEGY: every bulk PV declares a capacity, and for each service the
-#           bulk volume is at least an ORDER OF MAGNITUDE larger than that service's own
-#           replicated volume. This is the requirement stated as arithmetic: the two tiers
-#           exist because the data has two magnitudes, so a "bulk" volume sized like a
-#           config PVC means the strategy was named and not actually applied;
-#       (3) NOT DYNAMICALLY PROVISIONED, AND PRE-BOUND BOTH WAYS: every bulk PVC names its
-#           PV by volumeName, every bulk PV names its PVC by claimRef (namespace+name), and
-#           the two agree — so no other claim can win a race, and nothing can provision an
-#           empty replacement;
-#       (4) RECLAIM: every bulk PV is `Retain`;
-#       (5) MOUNT SAFETY: every bulk PV declares mountOptions carrying `hard` and an
-#           `nfsvers=4.x`, and carrying NEITHER `soft` NOR an `nfsvers=3`;
-#       (6) ONE EXPORT, ONE MOUNT PER WORKLOAD (the EXDEV trap above);
-#       (7) READ-ONLY IS REAL AND CONSISTENT: a bulk entry declared readOnly renders BOTH
-#           `spec.nfs.readOnly: true` on the PV AND `readOnly: true` on the volumeMount,
-#           and its access mode is ReadOnlyMany; a writable one renders NEITHER and is
-#           ReadWriteMany. Every bulk access mode is a `...Many` — one export is consumed
-#           by several namespaces and several nodes, which ReadWriteOnce cannot express;
-#       (8) AUTHORED ONCE, PROMOTED (the R-ENV-02 seam this tier depends on): every bulk
-#           PV's `nfs.server` is the BARE cluster-vars token, never a literal address; that
-#           knob has a base default; and EVERY environment overlay supplies it — so the two
-#           clusters, which reach this storage by different addresses, cannot be served by
-#           one hardcoded value;
-#   AND (9) LOAD-BEARING + NON-VACUOUS: this check DECLARES `conformance.home.k3s/scope:
-#           whole-repo`, so ci/l0.sh runs it unconditionally via --include-scope, and
-#           ci/gate.sh blocks the merge when the l0 stage fails; and the corpus is NOT
-#           empty — a repo with zero bulk volumes fails here rather than passing trivially,
-#           because a check that is satisfiable by deleting the feature proves nothing.
-#
-# WHAT THIS DOES *NOT* PROVE (honest gaps)
-#   - Nothing about RUNTIME. Not that the server is reachable, not that the export permits
-#     these clients, not that the PVC binds, not that a write succeeds. The uid analysis in
-#     docs/BULK-NFS-STORAGE.md §6 is reasoning, and the write/hardlink probes in §9 are the
-#     in-cluster proof — they need a cluster and are not run here.
-#   - It cannot prove the prod address is CORRECT, only that it is a supplied knob. While
-#     prod's value is a placeholder, prod's pods will not mount (docs/BULK-NFS-STORAGE.md
-#     §7); catching that needs the placeholder guard, which is absent from main.
-#   - Read-only for the media tree is a CLIENT-side guarantee, and this check can only
-#     assert the client side. The export serves a reader and two writers from the same node
-#     addresses, so the server cannot distinguish them (docs/BULK-NFS-STORAGE.md §5).
-#   - Assertion (2)'s magnitude ratio is a heuristic with a deliberately generous floor. It
-#     catches "bulk volume sized like a config volume", not a subtly wrong capacity — and a
-#     static NFS PV's capacity is advisory anyway, since the ZFS pool is the real limit.
-#   - Assertion (6) reasons over ONE workload's mounts. It cannot see two different
-#     services hardlinking between each other's mounts, which is not a thing that happens.
-# =============================================================================
-set -euo pipefail
-DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
-PYTHON="${PYTHON:-python3}"
-
-cd "${ROOT}"
-
-if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ]; then
-  echo "check SKIPPED (bulk-tier-distinct-from-replicated): catalog/ or apps/ absent — nothing to audit" >&2
-  exit 3
-fi
-
-"${PYTHON}" - "${ROOT}" <<'PY'
-import re
-import sys
-from pathlib import Path
-
-try:
-    import yaml
-except ImportError:                                  # pragma: no cover
-    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
-          file=sys.stderr)
-    raise SystemExit(1)
-
-ROOT = Path(sys.argv[1])
-FAILURES: list[str] = []
-CHECKS = 0
-
-
-def check(cond, msg):
-    global CHECKS
-    CHECKS += 1
-    if not cond:
-        FAILURES.append(msg)
-
-
-def bail(msg):
-    """A failure that makes the rest of the check meaningless: report and stop."""
-    print(f"CHECK FAIL (bulk-tier-distinct-from-replicated): {msg}", file=sys.stderr)
-    raise SystemExit(1)
-
-
-def qty_to_bytes(q):
-    """Kubernetes quantity -> bytes. Only the binary suffixes the schema permits."""
-    m = re.fullmatch(r"([0-9]+)(Mi|Gi|Ti)", str(q).strip())
-    if not m:
-        return None
-    return int(m.group(1)) * {"Mi": 2 ** 20, "Gi": 2 ** 30, "Ti": 2 ** 40}[m.group(2)]
-
-
-# =====================================================================================
-# The replicated tier, named by the repo rather than by this file. Any StorageClass the
-# repo declares with the Longhorn provisioner is "the replicated tier" for assertion (1);
-# deriving it means adding a new Longhorn class cannot create a blind spot here.
-# =====================================================================================
-REPLICATED_CLASSES: set[str] = set()
-LONGHORN_DIR = ROOT / "platform" / "longhorn"
-if LONGHORN_DIR.is_dir():
-    for f in sorted(LONGHORN_DIR.rglob("*.yaml")):
-        for doc in yaml.safe_load_all(f.read_text()):
-            if isinstance(doc, dict) and doc.get("kind") == "StorageClass":
-                if "longhorn" in str(doc.get("provisioner", "")):
-                    REPLICATED_CLASSES.add(doc.get("metadata", {}).get("name", ""))
-# `longhorn` is the name every ServiceDefinition uses for the replicated tier, so it is in
-# the set even if platform/longhorn/ is not present in this checkout.
-REPLICATED_CLASSES.add("longhorn")
-REPLICATED_CLASSES.discard("")
-
-# =====================================================================================
-# The bulk corpus: every `class: bulk` storage entry in every ServiceDefinition, paired
-# with the objects the COMMITTED goldens carry for it. Reading the goldens (not calling
-# the renderer) is deliberate: the goldens are what Flux reconciles, and render==golden is
-# a separate gate stage. Asking the renderer would only prove it agrees with itself.
-# =====================================================================================
-sds: dict[str, dict] = {}
-for f in sorted((ROOT / "catalog").glob("*.yaml")):
-    doc = yaml.safe_load(f.read_text())
-    if isinstance(doc, dict) and doc.get("name"):
-        sds[doc["name"]] = doc
-
-if not sds:
-    bail("no ServiceDefinitions under catalog/ — cannot audit the bulk tier")
-
-# Every rendered app-tier document, indexed by (kind, name).
-rendered: dict[tuple, dict] = {}
-for f in sorted((ROOT / "apps").rglob("*.yaml")):
-    if f.name == "kustomization.yaml" or ".sops." in f.name:
-        continue
-    try:
-        docs = list(yaml.safe_load_all(f.read_text()))
-    except yaml.YAMLError:
-        continue
-    for doc in docs:
-        if isinstance(doc, dict) and doc.get("kind") and doc.get("metadata", {}).get("name"):
-            rendered[(doc["kind"], doc["metadata"]["name"])] = doc
-
-# (svc, vol_entry) for every bulk entry, and the replicated sizes alongside for (2).
-corpus: list[tuple[str, dict]] = []
-replicated_bytes: dict[str, list[int]] = {}
-for svc, sd in sds.items():
-    for vol in sd.get("storage") or []:
-        cls = vol.get("class", "longhorn")
-        if cls == "bulk":
-            corpus.append((svc, vol))
-        else:
-            b = qty_to_bytes(vol.get("size"))
-            if b:
-                replicated_bytes.setdefault(svc, []).append(b)
-
-# ------------------------------------------------------------------- (9) NON-VACUOUS
-# A corpus of zero would make every loop below trivially satisfied. The bulk tier is
-# DELIVERED; a change that removes the last bulk volume must fail here and be argued for,
-# not sail through on an empty iteration.
-check(len(corpus) > 0,
-      "NO `class: bulk` storage entry exists in catalog/ — this check would then pass "
-      "vacuously, which is exactly the posture that let the bulk tier sit as a REPLACE-ME "
-      "stub for six roadmap phases. If the tier is genuinely being retired, retire this "
-      "check in the same change and say so; do not let it pass on an empty corpus")
-if not corpus:
-    bail("empty bulk corpus — refusing to report a vacuous pass (see assertion 9)")
-
-inventory: list[str] = []
-
-for svc, vol in corpus:
-    vname = vol.get("name", "<unnamed>")
-    label = f"{svc}/{vname}"
-    obj = f"{svc}-{vname}"
-    ro = bool(vol.get("readOnly"))
-
-    # `export` is what the schema requires of a bulk entry; without it there is nothing to
-    # attach and the rest of the assertions have no subject.
-    export = vol.get("export")
-    check(bool(export) and str(export).startswith("/"),
-          f"{label}: bulk entry has no absolute `export` path — a bulk volume attaches an "
-          f"EXISTING dataset, so it must say which one (schema requires it when class is bulk)")
-
-    pv = rendered.get(("PersistentVolume", obj))
-    pvc = rendered.get(("PersistentVolumeClaim", obj))
-    check(pv is not None,
-          f"{label}: no PersistentVolume '{obj}' in the committed apps/ goldens. A bulk "
-          f"entry that renders no PV has NO external storage behind it — the pod would get "
-          f"an unbound PVC, or (worse, with a dynamic class) a fresh EMPTY directory")
-    check(pvc is not None,
-          f"{label}: no PersistentVolumeClaim '{obj}' in the committed apps/ goldens")
-    if pv is None or pvc is None:
-        continue
-
-    pvs, pvcs = pv.get("spec", {}) or {}, pvc.get("spec", {}) or {}
-
-    # ------------------------------------------------------ (1) DISTINCT FROM REPLICATED
-    for what, spec in (("PV", pvs), ("PVC", pvcs)):
-        sc = spec.get("storageClassName")
-        check(sc not in REPLICATED_CLASSES,
-              f"{label}: bulk {what} storageClassName is '{sc}', which the repo declares "
-              f"with the Longhorn provisioner. That puts BULK data on the REPLICATED tier "
-              f"— 3x of a multi-terabyte library across three mini-PCs. Longhorn will not "
-              f"refuse it; it will accept the PVC and fill the tier (R-STATE-04)")
-        check(sc == "",
-              f"{label}: bulk {what} storageClassName is {sc!r}, not the explicitly EMPTY "
-              f"string. Empty is what disables dynamic provisioning AND the default-class "
-              f"annotation. Any named class risks a provisioner handing out a fresh EMPTY "
-              f"directory that mounts and binds cleanly while the real data is never attached")
-    check("nfs" in pvs,
-          f"{label}: bulk PV declares no external `nfs` source — the bulk tier's whole "
-          f"premise is that durability lives OUTSIDE the cluster (docs/04)")
-
-    # ------------------------------------------------------------ (4) RECLAIM POLICY
-    check(pvs.get("persistentVolumeReclaimPolicy") == "Retain",
-          f"{label}: bulk PV reclaim policy is "
-          f"{pvs.get('persistentVolumeReclaimPolicy')!r}, not Retain. This data pre-exists "
-          f"the cluster and has consumers outside it; anything but Retain makes `kubectl "
-          f"delete pvc` a data-loss operation")
-
-    # ------------------------------------------------- (3) PRE-BOUND FROM BOTH ENDS
-    check(pvcs.get("volumeName") == obj,
-          f"{label}: bulk PVC does not name its PV via volumeName (got "
-          f"{pvcs.get('volumeName')!r}, expected {obj!r}) — without it the claim can be "
-          f"satisfied by some OTHER volume of matching size and mode")
-    cref = pvs.get("claimRef") or {}
-    check(cref.get("name") == obj and cref.get("namespace") == svc,
-          f"{label}: bulk PV claimRef is {cref!r}, expected namespace={svc!r} name={obj!r} "
-          f"— pre-binding must be pinned from BOTH ends or another PVC can win the race")
-
-    # ------------------------------------------------------------- (5) MOUNT SAFETY
-    mopts = [str(o) for o in (pvs.get("mountOptions") or [])]
-    check(bool(mopts),
-          f"{label}: bulk PV declares no mountOptions — then `hard` vs `soft` and the NFS "
-          f"version are whatever the client happens to default to, and both are decisions "
-          f"with consequences (silent short reads; how many firewall ports must open)")
-    check("hard" in mopts,
-          f"{label}: bulk PV mountOptions do not include `hard` ({mopts}). A soft mount "
-          f"hands a server hiccup to the application as a short read or EIO — silent "
-          f"corruption for a media library. `hard` hangs instead, which is recoverable")
-    check("soft" not in mopts,
-          f"{label}: bulk PV mountOptions include `soft` ({mopts}) — see above; this is the "
-          f"one option that turns an outage into damaged data")
-    vers = [o for o in mopts if o.startswith("nfsvers=")]
-    check(len(vers) == 1 and re.fullmatch(r"nfsvers=4(\.[0-9]+)?", vers[0]),
-          f"{label}: bulk PV must pin exactly one NFSv4 `nfsvers=` option (got {vers}). "
-          f"NFSv4 carries mount and lock in-protocol, so the tier needs ONE reachable port "
-          f"(tcp/2049); v3 also needs rpcbind/111, mountd and statd, which changes what the "
-          f"operator has to open in pfSense for prod to reach this server at all")
-
-    # ---------------------------------------------- (7) READ-ONLY IS REAL + CONSISTENT
-    want_mode = "ReadOnlyMany" if ro else "ReadWriteMany"
-    for what, spec in (("PV", pvs), ("PVC", pvcs)):
-        modes = spec.get("accessModes") or []
-        check(modes == [want_mode],
-              f"{label}: bulk {what} accessModes {modes} != ['{want_mode}']. Every bulk "
-              f"volume is a `...Many` (one export, several namespaces, several nodes — "
-              f"ReadWriteOnce cannot express that), and WHICH `Many` is how the read/write "
-              f"posture stays legible in `kubectl get pv`")
-    nfs_ro = bool((pvs.get("nfs") or {}).get("readOnly"))
-    check(nfs_ro == ro,
-          f"{label}: ServiceDefinition readOnly={ro} but the PV's nfs.readOnly={nfs_ro}. "
-          f"The PV field is what makes the kernel mount `ro`; a mismatch means the declared "
-          f"posture is not the one the kernel enforces")
-
-    # The volumeMount half, read out of the rendered Deployment.
-    dep = rendered.get(("Deployment", svc))
-    check(dep is not None, f"{label}: no rendered Deployment '{svc}' to check the mount against")
-    if dep is not None:
-        containers = (((dep.get("spec") or {}).get("template") or {}).get("spec") or {}).get("containers") or []
-        # The app container is containers[0] by the renderer's contract (a forward-auth
-        # sidecar is appended AFTER it).
-        mounts = [m for m in (containers[0].get("volumeMounts") or []) if m.get("name") == vname] if containers else []
-        check(len(mounts) == 1,
-              f"{label}: the app container has {len(mounts)} volumeMount(s) named {vname!r}, "
-              f"expected exactly 1 — a bulk volume that is declared and never mounted is a "
-              f"PV/PVC pair doing nothing, and two mounts of one volume is assertion (6)")
-        if len(mounts) == 1:
-            m = mounts[0]
-            check(bool(m.get("readOnly")) == ro,
-                  f"{label}: volumeMount readOnly={bool(m.get('readOnly'))} but the "
-                  f"ServiceDefinition says readOnly={ro} — the PV and the mount must agree, "
-                  f"so that dropping one still leaves the other")
-            check(m.get("subPath", vol.get("subPath")) == vol.get("subPath"),
-                  f"{label}: volumeMount subPath {m.get('subPath')!r} does not match the "
-                  f"declared subPath {vol.get('subPath')!r}")
-
-    # ------------------------------------------ (8) AUTHORED ONCE, SUPPLIED PER ENV
-    server = (pvs.get("nfs") or {}).get("server")
-    m = re.fullmatch(r"\$\{([A-Z0-9_]+)\}", str(server or ""))
-    check(m is not None,
-          f"{label}: bulk PV nfs.server is {server!r} — it must be a BARE cluster-vars "
-          f"token, not a literal address. dev and prod reach this storage host by DIFFERENT "
-          f"addresses (dev is on its LAN; prod is across pfSense), so a literal is correct "
-          f"in at most one environment and silently wrong in the other (R-ENV-02)")
-
-    cap = (pvs.get("capacity") or {}).get("storage")
-    req = ((pvcs.get("resources") or {}).get("requests") or {}).get("storage")
-    check(cap is not None and qty_to_bytes(cap) is not None,
-          f"{label}: bulk PV declares no parseable capacity ({cap!r}) — the API requires it, "
-          f"and it is what makes `kubectl get pv` report something true about this tier")
-    check(cap == req,
-          f"{label}: PV capacity {cap!r} != PVC request {req!r}; a request above the "
-          f"capacity never binds")
-
-    inventory.append(
-        f"  {label:<28} export={export}  {'ro' if ro else 'rw'}  {want_mode:<14} "
-        f"cap={cap}  server={server}  opts={','.join(mopts)}"
-    )
-
-    # ---------------------------------- (2) CAPACITY STRATEGY: two tiers, two magnitudes
-    cap_b = qty_to_bytes(cap)
-    reps = replicated_bytes.get(svc) or []
-    if cap_b and reps:
-        biggest = max(reps)
-        # ONE ORDER OF MAGNITUDE. Calibrated against the real corpus rather than guessed:
-        # the media library is ~460x its service's config volume, and the Nextcloud archive
-        # ~13x its 20Gi Longhorn volume — so a floor above ~13x would fire on a legitimate
-        # bulk volume, and this check would be teaching people to weaken it. 10x cannot be
-        # met by accident and still catches the regression that matters: a "bulk" volume
-        # sized like a config PVC, i.e. the strategy named but never applied.
-        check(cap_b >= 10 * biggest,
-              f"{label}: bulk capacity {cap} is less than 10x this service's largest "
-              f"replicated volume ({biggest // 2 ** 30}Gi). R-STATE-04 exists because the "
-              f"two tiers hold data of two different MAGNITUDES — the replicated tier keeps "
-              f"the small config/DB and the bulk tier keeps what cannot be replicated 3x. A "
-              f"bulk volume sized like a config volume means the tier was named, not applied")
-
-# ------------------------------------------------- (6) ONE EXPORT, AT MOST ONE MOUNT
-# The EXDEV trap. Grouped per SERVICE, because the hazard is two mounts inside ONE pod.
-by_service: dict[str, list[dict]] = {}
-for svc, vol in corpus:
-    by_service.setdefault(svc, []).append(vol)
-for svc, vols in sorted(by_service.items()):
-    seen: dict[str, str] = {}
-    for vol in vols:
-        exp = vol.get("export")
-        prev = seen.get(exp)
-        check(prev is None,
-              f"{svc}: mounts the export {exp} TWICE (volumes {prev!r} and "
-              f"{vol.get('name')!r}). Two mounts of one export are two FILESYSTEMS to the "
-              f"kernel, so link() between them fails EXDEV even though the server knows "
-              f"they are one dataset. Sonarr/Radarr do not fail loudly on that — they "
-              f"degrade to copy-then-delete, rewriting every imported file over NFS and "
-              f"destroying the seeding copy. Mount the common ancestor ONCE instead "
-              f"(docs/BULK-NFS-STORAGE.md §4)")
-        seen[exp] = vol.get("name")
-
-# ------------------------------------- (8b) the knob has a base default AND every overlay
-tokens = set()
-for _svc, vol in corpus:
-    pv = rendered.get(("PersistentVolume", f"{_svc}-{vol.get('name')}"))
-    if pv:
-        mm = re.fullmatch(r"\$\{([A-Z0-9_]+)\}", str(((pv.get("spec") or {}).get("nfs") or {}).get("server") or ""))
-        if mm:
-            tokens.add(mm.group(1))
-
-cv = ROOT / "clusters" / "base" / "cluster-vars.yaml"
-base_keys: set[str] = set()
-if cv.is_file():
-    doc = yaml.safe_load(cv.read_text()) or {}
-    base_keys = set((doc.get("data") or {}).keys())
-
-overlays = sorted(
-    d for d in (ROOT / "clusters").iterdir()
-    if cv.is_file() and d.is_dir() and d.name != "base" and (d / "kustomization.yaml").is_file()
-) if (ROOT / "clusters").is_dir() else []
-
-for tok in sorted(tokens):
-    check(tok in base_keys,
-          f"bulk PVs reference ${{{tok}}} but clusters/base/cluster-vars.yaml declares no "
-          f"such knob. Under StrictPostBuildSubstitutions that fails the whole tier; without "
-          f"it the token ships into the cluster as a literal server name")
-    for ov in overlays:
-        text = (ov / "kustomization.yaml").read_text()
-        check(f"/data/{tok}" in text,
-              f"clusters/{ov.name} does not supply ${{{tok}}}. The base default is the DEV "
-              f"value by this repo's convention, so an overlay that omits it silently "
-              f"inherits an address on the WRONG SIDE of pfSense — a mount that either "
-              f"never connects or connects to something unintended (R-ENV-02)")
-
-# --------------------------------------------------------------- (9) LOAD-BEARING
-L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
-GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
-# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
-# It used to assert `"bulk-tier-distinct-from-replicated/check.sh" in L0`: ci/l0.sh carried a hand-written
-# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
-# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
-# to write it). The same property is now asserted two ways: this test DECLARES
-# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
-# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
-# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
-# passing) is proven end-to-end and falsifiably by
-# conformance/tests/l0-static/whole-repo-invariants-wired.
-SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "bulk-tier-distinct-from-replicated" / "chainsaw-test.yaml"
-SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
-check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
-      "bulk-tier-distinct-from-replicated/chainsaw-test.yaml does not declare "
-      'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
-      "when a change happens to touch one of the requirements it is labelled with. "
-      "The path map keys NEITHER apps/* nor platform/longhorn/* to the bulk requirement, so moving a multi-terabyte bulk volume onto the 3-replica tier could attribute no requirement at all and be gated by nothing")
-check("--include-scope" in L0,
-      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
-      "a purely requirement-scoped run again and the scope annotation is inert decoration — "
-      "this check would then run only by coincidence")
-check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
-      "ci/gate.sh does not set the merge block when the l0 stage fails")
-
-# ------------------------------------------------------------------------- report
-print("Bulk tier inventory (from catalog/ + the committed apps/ goldens):")
-for line in inventory:
-    print(line)
-print(f"\nreplicated tier StorageClasses (derived from platform/longhorn/): "
-      f"{', '.join(sorted(REPLICATED_CLASSES))}")
-print(f"bulk volumes: {len(corpus)}   distinct exports: "
-      f"{len({v.get('export') for _s, v in corpus})}   "
-      f"server knob(s): {', '.join(sorted(tokens)) or 'NONE'}")
-
-# A floor on the assertion count, for the same reason the corpus may not be empty: a pass
-# must not be able to degenerate into "nothing was actually asked".
-MIN_CHECKS = 40
-check(CHECKS >= MIN_CHECKS,
-      f"only {CHECKS} assertions ran (floor {MIN_CHECKS}) — a pass this thin is not "
-      f"evidence; the corpus or the extraction has degraded")
-
-print()
-if FAILURES:
-    print(f"CHECK FAIL (bulk-tier-distinct-from-replicated): {len(FAILURES)} finding(s)",
-          file=sys.stderr)
-    for f in FAILURES:
-        print(f"  - {f}", file=sys.stderr)
-    raise SystemExit(1)
-
-print(f"check OK: bulk-tier-distinct-from-replicated — {CHECKS} assertions (R-STATE-04)")
-PY

conformance/tests/l0-static/c2-root-surface/chainsaw-test.yaml

@@ -1,116 +0,0 @@
-# =============================================================================
-# L0 STATIC — the C2 root surfaces every service the cluster runs, plus platform
-# health and controlled actions  (R-C2-01, R-C2-02)   [EXECUTABLE]
-# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
-#
-# THE REQUIREMENTS, VERBATIM (spec docs/00-vision-and-scope.md):
-#   R-C2-01  "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services
-#             running in the cluster and links to them."
-#   R-C2-02  "The C2 UI MUST surface platform health (cluster, storage, GitOps, alerts) and
-#             provide controlled operational actions where safe."
-# Detail: spec docs/12-command-center-ui.md — Homepage at the root is "the primary R-C2-01
-# surface"; Headlamp is "the R-C2-02 'controlled operational actions where safe' surface";
-# Grafana is the dashboards/alerts surface; "never a raw cluster-admin kubeconfig in a browser".
-#
-# WHY L0, AND NOT L2
-# ------------------
-# The REACHABILITY half of R-C2-01 is already exercised against a live cluster:
-# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls
-# verifies the certificate it serves. What no live probe can decide is the half the
-# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING
-# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts:
-# what git declares the cluster serves (every in-repo HTTPRoute hostname, every L4 Service
-# that publishes a hostname, every catalog ServiceDefinition) against what the dashboard's
-# ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides half the
-# estate and cannot tell the difference.
-#
-# It is also the assertion that ROTS, in the direction nothing else notices: onboard a
-# service, forget the dashboard, and every cluster test stays green while the operator's
-# front door hides a running service. That is not a styling preference — it is exactly the
-# R-C2-01 failure mode, and this check found a live instance of it when it was written (see
-# the commit message: platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root
-# linked to it).
-#
-# GIVEN the repository as git tracks it,
-# WHEN  the apex ingress, the dashboard's declared link set, every in-repo HTTPRoute, every
-#       hostname-publishing L4 Service, the versioned catalog and the platform-health
-#       surfaces are read and compared,
-# THEN — R-C2-01 —
-#       the bare apex is claimed by exactly one route, that route binds the ONE Gateway
-#       listener whose hostname matches a bare apex, the apex is a SAN on the certificate
-#       that listener terminates with, the dashboard is RECONCILED (not merely authored),
-#       FORWARD every HTTP service the repo routes is linked from the root, FORWARD every
-#       catalog `kind: container` service with a route is linked, and REVERSE every
-#       home-zone link the root offers resolves to something this repo actually serves —
-#       an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4);
-#     — R-C2-02 —
-#       both health surfaces docs/12 names are linked AND reconciled, the health DATA exists
-#       as code (PrometheusRule groups for cluster/nodes, storage, GitOps and cert expiry,
-#       reconciled by the platform-config tier), and actions are CONTROLLED: the console
-#       binds NO ClusterRole to its own ServiceAccount, authenticates per-user by native
-#       OIDC from a named SOPS Secret, and cluster rights arrive through OIDC GROUPS at more
-#       than one privilege level.
-#
-# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files + PyYAML —
-# the same dependency catalog/render.py already requires). conformance/run.sh --layer l0 and
-# ci/l0.sh run it directly; this Chainsaw Test wraps the SAME script.
-#
-# WHAT IT DOES NOT PROVE — honest gaps, restated in check.sh's own header:
-#   - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2;
-#     the Gateway/TLS half is covered under R-DNS-01 / R-TLS-01.
-#   - That the C2 is access-CONTROLLED in fact. The auth assertions read a DECLARED posture,
-#     because the Gateway ext-authz attachment that would enforce forward-auth is not built
-#     anywhere in this repo (platform/homepage/httproute.yaml says so in its own header, and
-#     every forward-auth app shares the gap). Declared, not enforced — and labelled as such
-#     in the assertion text.
-#   - That an operational action in the console succeeds, or that it is audited. The
-#     assertions prove there is no shared cluster-admin identity to act with and that rights
-#     arrive per-user; the audit trail is a runtime property.
-#   - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up
-#     (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for.
-# =============================================================================
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: c2-root-surface
-  labels:
-    conformance.home.k3s/R-C2-01: "true"
-    conformance.home.k3s/R-C2-02: "true"
-  annotations:
-    conformance.home.k3s/requirements: "R-C2-01, R-C2-02"
-    conformance.home.k3s/layer: "L0"
-    # WHOLE-REPO SCOPE, and it is the whole point of the forward assertion.
-    #
-    # The invariant is "every service the repo routes is linked from the C2 root". Its input
-    # surface is therefore EVERY tracked manifest that can declare an HTTPRoute — apps/wave-*,
-    # platform/*, runbooks/ — not just the two components that own the C2. ci/touched-
-    # requirements.sh keys `platform/homepage/*` and `platform/headlamp/*` to nothing at all
-    # today (there is no path-map arm for either), and even with one, the change that BREAKS
-    # this invariant is the change that adds a route somewhere else: `apps/wave-1/newthing.yaml`
-    # attributes R-GITOPS-02 / R-NET-* / nothing, never R-C2-01. Requirement-scoped selection
-    # would filter this check out of exactly the change that introduces the bug — the R-TEST-03
-    # blind spot recorded in conformance/lib/common.sh.
-    #
-    # Declaring the scope is the whole fix: conformance/run.sh --include-scope whole-repo
-    # (ci/l0.sh Stage 3b) unions this test into EVERY gate invocation. Do NOT hand-add a stage
-    # to ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
-    # A SKIP of a scope-declaring check is rc 3 (UNWIRED), never green.
-    conformance.home.k3s/scope: "whole-repo"
-    conformance.home.k3s/safety-critical: "false"
-    conformance.home.k3s/executable: "true"
-    conformance.home.k3s/intent: >-
-      The bare apex is bound to the landing dashboard over trusted TLS and is reconciled; the
-      dashboard links every HTTP service this repo routes and every link it offers resolves to
-      something the repo serves; and the C2 surfaces platform health backed by real alert rules
-      while granting operational rights per-user through OIDC groups rather than through a
-      shared cluster-admin identity.
-spec:
-  steps:
-    - name: compare-the-c2-root-against-what-the-repo-actually-serves
-      description: >-
-        Read the apex ingress chain, the dashboard's declared links, every in-repo HTTPRoute and
-        hostname-publishing Service, the catalog, the platform-health alert rules and the
-        console's RBAC posture, and assert the two sets agree in both directions.
-      try:
-        - script:
-            content: bash ./check.sh

conformance/tests/l0-static/c2-root-surface/check.sh

@@ -1,656 +0,0 @@
-#!/usr/bin/env bash
-# =============================================================================
-# L0 static check (offline, no cluster) — c2-root-surface
-#
-# Serves (spec docs/00-vision-and-scope.md; detail in docs/12-command-center-ui.md):
-#
-#   R-C2-01  "The root `home.keithjackson.dev` MUST serve a UI that surfaces the
-#             services running in the cluster and links to them."
-#   R-C2-02  "The C2 UI MUST surface platform health (cluster, storage, GitOps,
-#             alerts) and provide controlled operational actions where safe."
-#
-# WHY THIS IS AN L0 STATIC CHECK AND NOT AN L2 PROBE
-# --------------------------------------------------
-# The REACHABILITY half of R-C2-01 is already exercised against a live cluster:
-# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls
-# verifies the certificate it serves. What no live probe can decide is the half the
-# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING
-# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts:
-# what git declares the cluster serves (every in-repo HTTPRoute hostname, every
-# hostname-publishing L4 Service, every catalog ServiceDefinition) against what the
-# dashboard's ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides
-# half the estate and cannot tell the difference.
-#
-# It is also the assertion that ROTS, in the direction nothing else notices: onboard a
-# service, forget the dashboard, and every cluster test stays green while the operator's
-# front door hides a running service. That is not a styling preference — it is exactly the
-# R-C2-01 failure mode, and it was a LIVE defect on main when this check was written
-# (platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root linked to it).
-#
-# THE RULE THIS IMPLEMENTS, in one sentence: every home-zone hostname the repository routes
-# over HTTP must be linked from the apex dashboard's configuration, every home-zone link the
-# apex dashboard offers must resolve to something the repository actually serves, and the two
-# R-C2-02 surfaces (health data + a per-user-authorized console) must be linked, reconciled
-# and backed by real alert rules.
-#
-# GIVEN the repository as git tracks it,
-# WHEN  the apex ingress chain, the dashboard's declared link set, every in-repo HTTPRoute,
-#       every hostname-publishing L4 Service, the versioned catalog, the platform-health
-#       alert rules and the console's RBAC posture are read and compared,
-# THEN — R-C2-01 —
-#       (0)  the INPUT SURFACE is intact: every Flux-reconciled path lies inside the roots
-#            this check scans, and every manifest in them parses. A root this check does not
-#            scan, or a file it cannot parse, is a service it cannot see — which would
-#            silently weaken every assertion below;
-#       (1)  EXACTLY ONE HTTPRoute claims the bare apex (an ambiguous root is not a root),
-#            and it belongs to the landing-dashboard component;
-#       (2)  that route pins by sectionName the ONE Gateway listener whose hostname is the
-#            bare apex, and that listener is HTTPS — the wildcard `*.${BASE_DOMAIN}` listener
-#            does NOT intersect the bare apex, so without the sectionName the root binds
-#            nothing and serves nothing (asserted against the wildcard listener's existence,
-#            so the claim is proven rather than asserted from prose);
-#       (3)  the apex is a SAN on the Certificate whose Secret that listener terminates with;
-#       (4)  the dashboard is RECONCILED, not merely authored: one Flux Kustomization points
-#            at its path and is listed in the base entrypoint set;
-#       (5)  that Kustomization carries postBuild.substituteFrom, because the dashboard's
-#            hrefs and its route's hostname are written as `${BASE_DOMAIN}` tokens — without
-#            substitution the apex route binds a literal `${BASE_DOMAIN}` and EVERY tile on
-#            the root links to a nonexistent host, while Flux still reports Ready;
-#       (6)  the component's own kustomization ships config + workload + Service + route;
-#       (7)  the dashboard is a service CATALOG: grouped entries carrying hrefs;
-#       (8)  FORWARD — every home-zone hostname an in-repo HTTPRoute serves, other than the
-#            apex itself, is LINKED from the root. This is the rot-catcher;
-#       (9)  FORWARD — every catalog `kind: container` ServiceDefinition whose declared
-#            ingress host is routed is likewise linked, tying the C2 to the VERSIONED
-#            onboarding contract and not only to whatever is rendered today;
-#       (10) REVERSE — every home-zone link the root offers resolves to something this repo
-#            actually serves: an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service
-#            (L4). A dead tile is a link the root offers to nothing;
-#       (11) the off-cluster escape hatch is not a blanket one: every name declared in the
-#            ConfigMap's `off-cluster-links` annotation is a specific home-zone hostname and
-#            none of them shadows a hostname the repo DOES serve;
-#       (12) no href names a LITERAL base-domain host. Every hostname is the `${BASE_DOMAIN}`
-#            token, so prod's root cannot link dev's services — the 2026-09-04 incident class
-#            (two clusters authored to describe the same names), applied to the front door;
-#     — R-C2-02 —
-#       (13) both health surfaces docs/12 names are linked: the Kubernetes console
-#            (cluster/storage/GitOps state) and the dashboards/alerts surface;
-#       (14) both are RECONCILED by Flux, so the links lead to something running;
-#       (15) the health DATA those surfaces render exists as code — PrometheusRule groups for
-#            each axis R-C2-02 enumerates (cluster/nodes, storage, GitOps reconciliation)
-#            plus certificate expiry — and those rule files are reconciled;
-#       (16) actions are CONTROLLED, not cluster-admin-in-a-browser (docs/12: "never a raw
-#            cluster-admin kubeconfig in a browser"): the console chart is explicitly told NOT
-#            to bind its ServiceAccount to a ClusterRole, so there is no shared privileged
-#            identity to act as;
-#       (17) actions are instead PER-USER: the console is configured for native OIDC with
-#            credentials from a named (SOPS) Secret, not inline;
-#       (18) cluster rights arrive through OIDC GROUPS at MORE THAN ONE privilege level (an
-#            admin group and a lesser one) — least-privilege, not one tier for everybody;
-#       (19) the C2 root DECLARES the authenticated-group posture docs/12 requires, on its own
-#            route (declared, NOT enforced — see the gap note below).
-#
-# WHAT THIS DOES *NOT* PROVE — honest gaps, the same ones the Test declaration names:
-#   - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2; the
-#     Gateway/TLS half is covered under R-DNS-01 / R-TLS-01.
-#   - That the C2 is access-CONTROLLED in fact. (19) reads a DECLARED posture, because the
-#     Gateway ext-authz attachment that would ENFORCE forward-auth is not built anywhere in
-#     this repo (platform/homepage/httproute.yaml says so in its own header, and every
-#     forward-auth app shares the gap). Declared, not enforced, and labelled as such in the
-#     assertion text itself.
-#   - That an operational action in the console succeeds, or that it is audited. (16)-(18)
-#     prove there is no shared cluster-admin identity to act with and that rights arrive
-#     per-user; the audit trail is a runtime property.
-#   - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up
-#     (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for.
-#   - That an L4-ONLY endpoint is surfaced. A raw TCP/UDP endpoint (a Minecraft game port) has
-#     no URL a browser can follow, so the FORWARD direction quantifies over HTTP routes;
-#     L4-published hostnames are listed as ADVISORIES and are honoured in the REVERSE
-#     direction (that is how the UniFi tile, an L4 LoadBalancer on :11443, resolves).
-#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and the
-#     requirement text above is quoted, not parsed.
-#
-# STATUS: EXECUTABLE. Pure offline: `git ls-files` + PyYAML (the dependency ten neighbouring
-# l0-static checks already take). No cluster, no network, no spec repo.
-# =============================================================================
-set -euo pipefail
-DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
-PYTHON="${PYTHON:-python3}"
-
-# Honest SKIP (never green, and rc 3 is UNWIRED for a scope-declaring test) if the landing
-# dashboard component is absent entirely — there is then no C2 root to audit, which is a
-# report about the tree, not a pass.
-if [ ! -f "${ROOT}/platform/homepage/configmap.yaml" ]; then
-  echo "check SKIPPED: platform/homepage/configmap.yaml absent — no C2 landing dashboard to audit" >&2
-  exit 3
-fi
-
-cd "${ROOT}"
-"${PYTHON}" - "${ROOT}" <<'PY'
-"""Static audit: the C2 root surfaces every service the repository serves, plus health.
-
-Every parse contract is documented at its parser. PyYAML rather than regex because the
-subjects are genuinely structured: multi-document apps/ bundles, and a Homepage config that
-is nested YAML *inside* a ConfigMap string value.
-"""
-from __future__ import annotations
-
-import re
-import subprocess
-import sys
-from pathlib import Path
-
-try:
-    import yaml
-except ImportError:                                  # pragma: no cover
-    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
-          file=sys.stderr)
-    raise SystemExit(1)
-
-ROOT = Path(sys.argv[1])
-sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
-
-FAILURES: list[str] = []
-ADVISORIES: list[str] = []
-CHECKS = 0
-
-
-def check(cond, msg):
-    global CHECKS
-    CHECKS += 1
-    if cond:
-        print(f"  ok   {msg}")
-    else:
-        print(f"  FAIL {msg}", file=sys.stderr)
-        FAILURES.append(msg)
-
-
-def read(p: Path) -> str:
-    try:
-        return p.read_text()
-    except (OSError, UnicodeDecodeError):
-        return ""
-
-
-# ============================================================ the hostname vocabulary
-# BASE_DOMAIN is a cluster-vars knob (R-ENV-02): every public hostname the cluster serves is
-# `<svc>.${BASE_DOMAIN}`, and dev/prod differ ONLY in that value. So a hostname in this repo
-# is canonically the TOKEN form, and the literal value below exists only to be FORBIDDEN in
-# (12): a literal is how prod came to be authored to serve dev's hostnames (2026-09-04,
-# clusters/base/cluster-vars.yaml's own header).
-TOKEN = "${BASE_DOMAIN}"
-_vars = yaml.safe_load(read(ROOT / "clusters" / "base" / "cluster-vars.yaml")) or {}
-LITERAL_BASE = ((_vars.get("data") or {}).get("BASE_DOMAIN") or "").strip()
-
-
-def strip_port(host: str) -> str:
-    return host.split(":", 1)[0]
-
-
-def zone_label(host: str):
-    """The sub-label of a home-zone hostname, "" for the bare apex, or None if not in zone.
-
-    Accepts only the TOKEN form; a literal-base hostname returns None here and is caught
-    separately by (12), so a literal can never silently satisfy a set comparison.
-    """
-    h = strip_port(host)
-    if h == TOKEN:
-        return ""
-    if h.endswith("." + TOKEN):
-        return h[: -len("." + TOKEN)]
-    return None
-
-
-def is_literal_base(host: str) -> bool:
-    h = strip_port(host)
-    return bool(LITERAL_BASE) and (h == LITERAL_BASE or h.endswith("." + LITERAL_BASE))
-
-
-# ============================================================ (0) the input surface
-# DELIVERY ROOTS — the subtrees Flux reconciles, plus catalog/ (the versioned source the
-# apps/ manifests are rendered FROM). conformance/ is deliberately EXCLUDED: its fixtures and
-# goldens declare synthetic HTTPRoutes (`paas-demo.…`, `gw-declared-conformance.…`) that are
-# test data, not services the cluster serves, and counting them would make (8) assert that
-# the operator's dashboard links this harness's fixtures.
-DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
-                  "runbooks", "clusters", "bootstrap", "catalog")
-
-
-def tracked_yaml() -> list[Path]:
-    out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
-                         cwd=ROOT, capture_output=True, text=True, check=True)
-    return [ROOT / p for p in out.stdout.split("\0") if p]
-
-
-ALL_TRACKED = tracked_yaml()
-DELIVERY = [p for p in ALL_TRACKED
-            if p.relative_to(ROOT).parts[0] in DELIVERY_ROOTS]
-
-parse_failed: list[str] = []
-DOCS: dict[Path, list[dict]] = {}
-for p in DELIVERY:
-    try:
-        DOCS[p] = [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
-    except yaml.YAMLError:
-        DOCS[p] = []
-        parse_failed.append(str(p.relative_to(ROOT)))
-
-
-def docs_all():
-    for p, ds in DOCS.items():
-        for d in ds:
-            yield p, d
-
-
-check(len(DELIVERY) >= 100,
-      f"(0) the delivery tree is present and scanned ({len(DELIVERY)} tracked manifest(s) "
-      f"under {'/, '.join(DELIVERY_ROOTS)}/)")
-check(not parse_failed,
-      f"(0) every manifest in the scanned roots PARSES — an unparseable one is a service "
-      f"this check cannot see, so it would weaken every set comparison below "
-      f"{parse_failed if parse_failed else ''}")
-
-# Every Flux-reconciled path must lie inside a root this check scans. This is the guard on
-# the guard: if a future tier reconciles, say, ./services, the forward direction would stop
-# seeing its routes and would go quietly green on a dashboard that hides all of them.
-flux_ks = [(p, d) for p, d in docs_all()
-           if d.get("kind") == "Kustomization"
-           and str(d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io")]
-# A RECONCILING Kustomization is one carrying a sourceRef. The pathless documents under
-# bootstrap/flux/patches/ are strategic-merge PATCH FRAGMENTS (they add a decryption stanza to
-# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
-# a path of them would fail on a file that is correct.
-reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
-unscanned = sorted({
-    f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
-    for p, d in reconcilers
-    if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
-    not in DELIVERY_ROOTS
-})
-check(len(reconcilers) >= 10 and not unscanned,
-      f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
-      f"reconciled tree this check does not read is a service it cannot require a link for "
-      f"{unscanned if unscanned else ''}")
-
-# ============================================================ what the repo SERVES
-# L7: every hostname an in-repo HTTPRoute claims, and the component that owns it.
-routes: dict[str, list[tuple[Path, dict]]] = {}
-apex_inheriting: list[Path] = []
-for p, d in docs_all():
-    if d.get("kind") != "HTTPRoute":
-        continue
-    hosts = (d.get("spec", {}) or {}).get("hostnames") or []
-    if not hosts:
-        # An HTTPRoute with NO hostnames inherits the hostname of every listener it attaches
-        # to (runbooks/serving/httproute.yaml binds `https-apex` only, so it serves a PATH on
-        # the apex). It introduces no new hostname, so it owes no tile.
-        apex_inheriting.append(p)
-        continue
-    for h in hosts:
-        routes.setdefault(str(h), []).append((p, d))
-
-routed_labels = {}          # sub-label -> owning component dir
-for h, owners in routes.items():
-    lbl = zone_label(h)
-    if lbl:                 # "" is the apex; None is out of zone
-        routed_labels[lbl] = owners[0][0].parent
-
-# L4: a Service that publishes a home-zone hostname to DNS is also "served by this repo" —
-# it is how UniFi (a KubeVirt VM behind a pinned Cilium LoadBalancer VIP on :11443) reaches
-# the LAN. external-dns's annotation IS the declaration; there is no HTTPRoute to find.
-DNS_ANN = "external-dns.alpha.kubernetes.io/hostname"
-l4_labels: dict[str, Path] = {}
-for p, d in docs_all():
-    if d.get("kind") != "Service":
-        continue
-    ann = (d.get("metadata", {}) or {}).get("annotations", {}) or {}
-    for h in re.split(r"[,\s]+", str(ann.get(DNS_ANN, ""))):
-        lbl = zone_label(h) if h else None
-        if lbl:
-            l4_labels[lbl] = p
-
-check(len(routed_labels) >= 10,
-      f"(0) the repo's north-south HTTP surface is discovered "
-      f"({len(routed_labels)} routed home-zone hostname(s), "
-      f"{len(apex_inheriting)} apex-path route(s), {len(l4_labels)} L4-published hostname(s))")
-
-# ============================================================ (1) the apex is unambiguous
-apex_claims = routes.get(TOKEN, [])
-check(len(apex_claims) == 1,
-      f"(1) exactly ONE HTTPRoute claims the bare apex {TOKEN} — an ambiguous root is not a "
-      f"root (found {len(apex_claims)}: {[str(p.relative_to(ROOT)) for p, _ in apex_claims]})")
-apex_path, apex_route = (apex_claims[0] if apex_claims else (None, {}))
-DASH = apex_path.parent if apex_path is not None else None
-check(DASH is not None and DASH.name == "homepage",
-      f"(1) the apex route belongs to the landing-dashboard component "
-      f"(found {DASH.name if DASH else None!r})")
-
-# ============================================================ (2) it binds the apex listener
-parents = (apex_route.get("spec", {}) or {}).get("parentRefs") or []
-section = next((pr.get("sectionName") for pr in parents if pr.get("sectionName")), None)
-gw_name = next((pr.get("name") for pr in parents), None)
-gw_ns = next((pr.get("namespace") for pr in parents), None)
-check(section is not None,
-      "(2) the apex route pins a Gateway listener by sectionName — the wildcard listener's "
-      "hostname cannot match a bare apex, so an unpinned apex route binds NOTHING")
-
-gw = next((d for _, d in docs_all()
-           if d.get("kind") == "Gateway"
-           and (d.get("metadata", {}) or {}).get("name") == gw_name
-           and (gw_ns is None or (d.get("metadata", {}) or {}).get("namespace") == gw_ns)), None)
-check(gw is not None, f"(2) the parent Gateway '{gw_name}' is declared in the repo")
-listeners = ((gw or {}).get("spec", {}) or {}).get("listeners") or []
-apex_listener = next((l for l in listeners if l.get("name") == section), None)
-check(apex_listener is not None,
-      f"(2) the Gateway declares the '{section}' listener the apex route binds")
-check(apex_listener is not None and apex_listener.get("hostname") == TOKEN,
-      f"(2) that listener's hostname is EXACTLY the bare apex "
-      f"(found {(apex_listener or {}).get('hostname')!r})")
-check(apex_listener is not None and apex_listener.get("protocol") == "HTTPS",
-      "(2) the apex listener terminates HTTPS (docs/12 acceptance gate: trusted TLS on the root)")
-# Prove the sectionName is LOAD-BEARING rather than decorative: a wildcard listener exists,
-# it is a DIFFERENT listener, and its hostname does not equal the apex.
-wildcard = [l for l in listeners if str(l.get("hostname", "")) == "*." + TOKEN]
-check(bool(wildcard) and all(l.get("name") != section for l in wildcard),
-      f"(2) the wildcard listener(s) {[l.get('name') for l in wildcard]} are DISTINCT from "
-      f"'{section}' — which is why the apex needs its own listener, and why dropping the "
-      f"sectionName would leave the root unserved")
-
-# ============================================================ (3) trusted TLS on the root
-cert_secret = next((ref.get("name")
-                    for ref in ((apex_listener or {}).get("tls", {}) or {}).get("certificateRefs") or []),
-                   None)
-cert = next((d for _, d in docs_all()
-             if d.get("kind") == "Certificate"
-             and (d.get("spec", {}) or {}).get("secretName") == cert_secret), None)
-check(cert is not None,
-      f"(3) a cert-manager Certificate produces the apex listener's Secret ({cert_secret!r})")
-check(cert is not None and TOKEN in ((cert.get("spec", {}) or {}).get("dnsNames") or []),
-      "(3) the bare apex is a SAN on that Certificate — the wildcard alone does NOT cover it, "
-      "so without this SAN the root serves an untrusted certificate")
-
-# ============================================================ (4)(5) it is RECONCILED
-dash_rel = f"./{DASH.relative_to(ROOT)}" if DASH else None
-dash_ks = [(p, d) for p, d in flux_ks
-           if (d.get("spec", {}) or {}).get("path", "").rstrip("/") == dash_rel]
-check(len(dash_ks) == 1,
-      f"(4) exactly one Flux Kustomization reconciles the dashboard ({dash_rel}) — authored "
-      f"but unreconciled is the 'looks done, is absent' class this repo keeps hitting "
-      f"(found {len(dash_ks)})")
-if dash_ks:
-    base_txt = read(ROOT / "clusters" / "base" / "kustomization.yaml")
-    entry = dash_ks[0][0].name
-    check(re.search(rf"^\s*-\s*{re.escape(entry)}\s*(#.*)?$", base_txt, re.M) is not None,
-          f"(4) that Kustomization is listed in the base entrypoint set ({entry})")
-    # (5) The dashboard's hrefs and its route's hostname are ${BASE_DOMAIN} TOKENS. Without
-    # postBuild.substituteFrom the token reaches the cluster LITERALLY: the apex route binds
-    # the host `${BASE_DOMAIN}` (matching no listener) and every tile links to a host that
-    # does not exist — and Flux still reports the Kustomization Ready. It fails OPEN.
-    subs = (((dash_ks[0][1].get("spec", {}) or {}).get("postBuild") or {}).get("substituteFrom") or [])
-    check(any((s.get("name") == "cluster-vars") for s in subs),
-          f"(5) {dash_ks[0][0].name} carries postBuild.substituteFrom cluster-vars. It MUST: "
-          f"this component's route hostname and EVERY tile href are ${{BASE_DOMAIN}} tokens, "
-          f"and an unsubstituted token is applied literally while Flux still reports Ready — "
-          f"a root bound to no listener and a dashboard of dead links, with a green tier")
-
-# ============================================================ (6) it ships the UI
-dash_kust = yaml.safe_load(read(DASH / "kustomization.yaml")) if DASH else None
-dash_res = set((dash_kust or {}).get("resources") or [])
-for want in ("configmap.yaml", "deployment.yaml", "service.yaml", "httproute.yaml"):
-    check(want in dash_res, f"(6) the dashboard kustomization ships {want}")
-
-# ================================================== the dashboard's DECLARED link set
-# Homepage's config is nested YAML inside ConfigMap string values (services.yaml,
-# bookmarks.yaml). Both are read: a bookmark is a link the root offers just as a tile is.
-cm = next((d for d in (DOCS.get(DASH / "configmap.yaml") or []) if d.get("kind") == "ConfigMap"),
-          {}) if DASH else {}
-cm_data = cm.get("data") or {}
-
-
-def embedded(key):
-    try:
-        return yaml.safe_load(cm_data.get(key) or "") or []
-    except yaml.YAMLError:
-        return None
-
-
-svc_cfg = embedded("services.yaml")
-bm_cfg = embedded("bookmarks.yaml")
-check(svc_cfg is not None and bm_cfg is not None,
-      "(7) the dashboard's embedded services.yaml / bookmarks.yaml parse as YAML — the app "
-      "mounts these verbatim, so a malformed one is a dashboard that renders nothing")
-
-
-def hrefs(node):
-    """Every href anywhere in Homepage's nested group/entry structure."""
-    if isinstance(node, dict):
-        for k, v in node.items():
-            if k == "href" and isinstance(v, str):
-                yield v
-            else:
-                yield from hrefs(v)
-    elif isinstance(node, list):
-        for v in node:
-            yield from hrefs(v)
-
-
-def host_of(url):
-    m = re.match(r"^https?://([^/]+)", url or "")
-    return m.group(1) if m else None
-
-
-tile_urls = list(hrefs(svc_cfg or []))
-all_urls = tile_urls + list(hrefs(bm_cfg or []))
-all_hosts = [h for h in (host_of(u) for u in all_urls) if h]
-linked = {lbl for lbl in (zone_label(h) for h in all_hosts) if lbl}
-tile_labels = {lbl for lbl in (zone_label(h) for h in (host_of(u) for u in tile_urls) if h) if lbl}
-
-groups = [g for g in (svc_cfg or []) if isinstance(g, dict)]
-check(len(groups) >= 1 and len(tile_labels) >= 5,
-      f"(7) the root is a grouped service catalog with links "
-      f"({len(groups)} group(s), {len(tile_labels)} linked home-zone service(s))")
-
-# ============================================ (8) FORWARD: nothing routed is hidden
-unsurfaced = sorted(set(routed_labels) - linked)
-check(not unsurfaced,
-      "(8) every service the repo ROUTES over HTTP is surfaced on the C2 root. Unsurfaced: "
-      + ", ".join(f"{n}.{TOKEN} ({routed_labels[n].relative_to(ROOT)})" for n in unsurfaced)
-      if unsurfaced else
-      f"(8) every service the repo ROUTES over HTTP is surfaced on the C2 root "
-      f"({len(routed_labels)} hostname(s) compared)")
-
-# L4-only endpoints are reported, never required: a raw TCP/UDP port has no browsable URL.
-for lbl in sorted(set(l4_labels) - linked):
-    ADVISORIES.append(
-        f"{lbl}.{TOKEN} is published to DNS by an L4 Service "
-        f"({l4_labels[lbl].relative_to(ROOT)}) and is not linked from the root. Not a "
-        f"failure: a raw TCP/UDP endpoint has no URL a browser can follow. Verify the "
-        f"service it belongs to is reachable through a tile that IS linked")
-
-# ================================= (9) FORWARD: the VERSIONED onboarding contract
-cat_dir = ROOT / "catalog"
-cat_seen, cat_missing, cat_unrouted = 0, [], []
-for p in sorted(cat_dir.glob("*.yaml")):
-    sd = next((d for d in (DOCS.get(p) or [])), None)
-    if not isinstance(sd, dict) or sd.get("kind") != "container":
-        continue                      # kind: vm has no north-south HTTP surface (docs/09)
-    for r in sd.get("ingressRoutes") or []:
-        lbl = zone_label(str(r.get("host", "")))
-        if not lbl:
-            continue
-        if lbl not in routed_labels:
-            # Declared in the catalog but not rendered into a route: an apps/ render drift
-            # problem owned by R-GITOPS-02, reported here rather than silently skipped.
-            cat_unrouted.append(f"{sd.get('name')}:{lbl}")
-            continue
-        cat_seen += 1
-        if lbl not in linked:
-            cat_missing.append(f"{sd.get('name')}:{lbl}")
-check(cat_seen >= 5,
-      f"(9) the versioned catalog contributes routed HTTP services ({cat_seen} found) — the "
-      f"C2 is tied to the onboarding contract, not only to today's rendered output")
-check(not cat_missing,
-      f"(9) every catalog `kind: container` service with a route is linked from the C2 root "
-      f"{cat_missing if cat_missing else ''}")
-for u in cat_unrouted:
-    ADVISORIES.append(f"catalog declares an ingress host with no rendered HTTPRoute: {u} "
-                      f"(render drift — R-GITOPS-02 owns it, not R-C2-01)")
-
-# ============================ (10)(11) REVERSE: no link points at nothing
-ann = (cm.get("metadata", {}) or {}).get("annotations", {}) or {}
-OFF_ANN = "homelab.keithjackson.dev/off-cluster-links"
-declared_off_raw = [t for t in re.split(r"[,\s]+", str(ann.get(OFF_ANN, ""))) if t]
-declared_off = {zone_label(t) for t in declared_off_raw}
-served = set(routed_labels) | set(l4_labels)
-dead = sorted(linked - served - declared_off)
-check(not dead,
-      f"(10) every home-zone link the root offers resolves to something this repo serves — "
-      f"an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4). Dead: "
-      f"{[f'{d}.{TOKEN}' for d in dead] if dead else ''}")
-# The escape hatch is a DECLARATION, not a hole: same doctrine as an explicit `NONE` in
-# ci/touched-requirements.sh's path map. Empty today, and these two hold vacuously when it is
-# — they exist so it cannot be widened into a blanket without failing.
-check(all(zone_label(t) for t in declared_off_raw),
-      f"(11) every declared off-cluster link is a specific home-zone hostname in TOKEN form "
-      f"({declared_off_raw})")
-shadowed = sorted(declared_off & served)
-check(not shadowed,
-      f"(11) no off-cluster declaration shadows a hostname the repo DOES serve "
-      f"{shadowed if shadowed else ''}")
-
-# ============================ (12) tokens, never literals
-literal_hosts = sorted({h for h in all_hosts if is_literal_base(h)})
-check(not literal_hosts,
-      f"(12) no link on the root names a LITERAL '{LITERAL_BASE}' host — every hostname is the "
-      f"${{BASE_DOMAIN}} token, so prod's front door cannot link dev's services (the "
-      f"2026-09-04 class: two clusters authored to describe the same names). Literals: "
-      f"{literal_hosts if literal_hosts else ''}")
-
-# =====================================================================
-# R-C2-02 — platform health, and controlled operational actions
-# =====================================================================
-# docs/12 names the surfaces the C2 composes: Headlamp is "the R-C2-02 'controlled
-# operational actions where safe' surface"; Grafana is the dashboards/alerts surface. Each is
-# identified by the component that OWNS ITS ROUTE, not by a magic string in the config.
-CONSOLE, DASHBOARDS = "headlamp", "grafana"
-check(CONSOLE in linked,
-      f"(13) the C2 root links the Kubernetes console ({CONSOLE}.{TOKEN}) — cluster, storage "
-      f"and GitOps state, and the R-C2-02 operational-actions surface (docs/12)")
-check(DASHBOARDS in linked,
-      f"(13) the C2 root links the dashboards/alerts surface ({DASHBOARDS}.{TOKEN})")
-
-AGG = ("\n".join(read(ROOT / t / "kustomization.yaml")
-                 for t in ("platform", "platform-config", "platform-config-cdi")))
-for label, name in (("console", CONSOLE), ("dashboards", DASHBOARDS)):
-    comp = routed_labels.get(name)
-    check(comp is not None, f"(14) the {label} surface has an in-repo HTTPRoute")
-    if comp is None:
-        continue
-    rel = f"./{comp.relative_to(ROOT)}"
-    reconciled = any((d.get("spec", {}) or {}).get("path", "").rstrip("/") == rel
-                     for _, d in flux_ks)
-    if not reconciled:
-        # A component may reconcile as a MEMBER of an aggregate tier's kustomize tree
-        # (platform / platform-config) instead of owning a Flux Kustomization. Both count;
-        # being in NEITHER is "authored, never applied".
-        reconciled = (f"\n  - {comp.name}" in AGG
-                      or f"../{comp.relative_to(ROOT)}/" in AGG)
-    check(reconciled,
-          f"(14) the {label} surface ({comp.name}) is reconciled by Flux, not merely authored "
-          f"— a link to an unreconciled component is a link to nothing")
-
-# (15) the health DATA exists as code, for every axis R-C2-02 enumerates.
-rule_files = sorted({p for p, d in docs_all() if d.get("kind") == "PrometheusRule"})
-rule_groups = {g.get("name", "")
-               for p, d in docs_all() if d.get("kind") == "PrometheusRule"
-               for g in ((d.get("spec", {}) or {}).get("groups") or [])}
-for axis, needle in (("cluster/nodes", "nodes"),
-                     ("storage", "longhorn"),
-                     ("GitOps reconciliation", "flux"),
-                     ("certificate expiry", "certificates")):
-    check(any(needle in g for g in rule_groups),
-          f"(15) platform-health alert rules exist for {axis} (a '*{needle}*' rule group) — "
-          f"the C2 cannot surface health that is not measured")
-pc_txt = read(ROOT / "platform-config" / "kustomization.yaml")
-reconciled_rules = [p for p in rule_files if f"../{p.relative_to(ROOT)}" in pc_txt]
-check(len(reconciled_rules) >= 2,
-      f"(15) those alert rules are RECONCILED — PrometheusRule is a CR of a CRD the platform "
-      f"tier installs, so the rules are applied by platform-config "
-      f"({len(reconciled_rules)}/{len(rule_files)} rule file(s) referenced there)")
-
-# (16)(17) controlled actions: no shared cluster-admin identity; per-user OIDC instead.
-console_dir = routed_labels.get(CONSOLE)
-hr = next((d for p in (sorted(console_dir.glob("*.yaml")) if console_dir else [])
-           for d in (DOCS.get(p) or []) if d.get("kind") == "HelmRelease"), None)
-check(hr is not None, "(16) the Kubernetes console is deployed by a HelmRelease")
-hv = ((hr or {}).get("spec", {}) or {}).get("values", {}) or {}
-check((hv.get("clusterRoleBinding") or {}).get("create") is False,
-      "(16) the console chart is told NOT to bind its ServiceAccount to a ClusterRole. The "
-      "chart's DEFAULT binds it to cluster-admin, which is exactly the 'raw cluster-admin "
-      "kubeconfig in a browser' docs/12 forbids — every visitor would act as one shared "
-      "omnipotent identity")
-oidc = ((hv.get("config") or {}).get("oidc") or {})
-check(bool(oidc),
-      "(17) the console is configured for native OIDC login, so an action carries the "
-      "operator's OWN identity (docs/12: Headlamp/Grafana use native OIDC)")
-check(bool((oidc.get("externalSecret") or {}).get("name")),
-      "(17) its OIDC client credentials come from a NAMED (SOPS) Secret, not inline in the "
-      "HelmRelease values — R-SEC-01")
-
-# (18) rights arrive through GROUPS, at more than one privilege level.
-group_bindings = []
-for p, d in docs_all():
-    if d.get("kind") not in ("ClusterRoleBinding", "RoleBinding"):
-        continue
-    for s in d.get("subjects") or []:
-        if s.get("kind") == "Group" and str(s.get("name", "")).startswith("oidc:"):
-            group_bindings.append(((d.get("metadata", {}) or {}).get("name"),
-                                   s.get("name"),
-                                   (d.get("roleRef") or {}).get("name")))
-check(len(group_bindings) >= 2,
-      f"(18) cluster rights are bound to OIDC GROUPS, so access follows Keycloak group "
-      f"membership rather than a per-user grant somebody must remember to revoke "
-      f"({[(g, r) for _, g, r in group_bindings]})")
-check(any(r == "cluster-admin" for _, _, r in group_bindings)
-      and any(r != "cluster-admin" for _, _, r in group_bindings),
-      "(18) MORE THAN ONE privilege level exists (an admin group AND a lesser one) — "
-      "least-privilege, not one tier for everybody (docs/12 access control, R-IDP-06)")
-
-# (19) the root DECLARES the authenticated-group posture. DECLARED, NOT ENFORCED.
-route_ann = (apex_route.get("metadata", {}) or {}).get("annotations", {}) or {}
-check(route_ann.get("homelab.keithjackson.dev/auth-mode") in ("forward-auth", "oidc", "oauth2-proxy"),
-      f"(19) the C2 root's route DECLARES an authentication mode "
-      f"(found {route_ann.get('homelab.keithjackson.dev/auth-mode')!r}) "
-      f"[DECLARED, NOT ENFORCED — no Gateway ext-authz attachment exists in this repo]")
-check(bool(route_ann.get("homelab.keithjackson.dev/auth-groups")),
-      "(19) it DECLARES the authorized group(s) — docs/12 requires an authenticated user IN "
-      "AN AUTHORIZED GROUP, not merely logged in "
-      "[DECLARED, NOT ENFORCED — see the header's gap note]")
-
-# ======================================================================== report
-print()
-print(f"C2 surface inventory (apex {TOKEN}, literal base default '{LITERAL_BASE}'):")
-print(f"  routed over HTTP : {len(routed_labels)}  -> {', '.join(sorted(routed_labels))}")
-print(f"  L4-published     : {len(l4_labels)}  -> {', '.join(sorted(l4_labels)) or '(none)'}")
-print(f"  linked from root : {len(linked)}  -> {', '.join(sorted(linked))}")
-print(f"  apex-path routes : {len(apex_inheriting)}  -> "
-      f"{', '.join(str(p.relative_to(ROOT)) for p in apex_inheriting) or '(none)'}")
-print(f"  catalog services compared: {cat_seen}   off-cluster declarations: "
-      f"{sorted(declared_off_raw) or '(none)'}")
-
-if ADVISORIES:
-    print("\nADVISORIES (not failures):")
-    for a in ADVISORIES:
-        print(f"  - {a}")
-
-print()
-if not FAILURES:
-    print(f"check OK: c2-root-surface — {CHECKS} assertions (R-C2-01, R-C2-02)")
-    raise SystemExit(0)
-print(f"CHECK FAIL (c2-root-surface): {len(FAILURES)} failed, "
-      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
-for f in FAILURES:
-    print(f"  FAIL {f}", file=sys.stderr)
-raise SystemExit(1)
-PY

conformance/tests/l0-static/dev-nodes-provisioned-as-code/chainsaw-test.yaml

@@ -1,83 +0,0 @@
-# =============================================================================
-# L0 STATIC — dev/test nodes are provisioned as code, end to end  (R-ENV-01)  [EXECUTABLE]
-# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
-# Spec:    docs/00 R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC:
-#          created, baselined, cluster-joined, and managed as code"); the four stages
-#          are enumerated in docs/02 §"Node lifecycle stages".
-#
-# WHY L0: R-ENV-01 is a requirement about HOW the dev nodes came to exist, and a live
-# Node object cannot answer it — a node built by OpenTofu and one built by hand in a
-# hypervisor UI are indistinguishable through the Kubernetes API. What is provable, with
-# no cluster, is that a COMPLETE and EXECUTABLE provisioning path exists for all four
-# stages and covers EVERY dev node. For the Ansible half that is proven by DRIVING the
-# real toolchain offline (`--syntax-check`, `--list-hosts` — neither opens a connection
-# to any node), not by reading files: a chain that does not resolve fails the check.
-#
-# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count appears in the test.
-# The dev group names are read from the playbooks' own default host patterns, the node
-# names are generated from the IaC module's prefix/count variables, and the two sets are
-# compared against each other — so the assertion survives a rename or a resize.
-#
-# GIVEN the repository as git tracks it,
-# WHEN  the provisioning chain is parsed and (where offline execution is possible) driven,
-# THEN  the IaC module creates the dev nodes on a LOCAL KVM hypervisor (allowlisted
-#       provider — a cloud provider fails), from a for_each map rather than a count index,
-#       each with CPU/RAM/OS disk/dedicated Longhorn data disk/NIC/cloud-init seed; the
-#       node set it creates is EXACTLY the set Ansible manages; the baseline stage applies
-#       the package floor, kernel modules, sysctls, data-disk mount, swap-off and iscsid
-#       declaratively; one k3s role founds the cluster with --cluster-init and joins the
-#       rest with --server while disabling flannel/kube-proxy/traefik/servicelb at
-#       install; and site.yml's chain (baseline -> k3s -> Cilium -> verify -> Flux)
-#       resolves and targets every dev node at every stage.
-#
-# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git, PyYAML, and —
-# where present — ansible-playbook in no-connection modes; no cluster, no network);
-# conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw Test
-# wraps the SAME script.
-#
-# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
-#   * That the RUNNING dev VMs were created by this module — needs the OpenTofu state or
-#     hypervisor access, neither available offline.
-#   * `tofu validate` — needs `tofu init` (provider registry), i.e. network. The module is
-#     parsed, not evaluated; a network-allowed CI stage would close this.
-#   * "Never hand-edited on the box" (docs/02 stage 4's second half) — a live drift check.
-#   * REBUILD/re-converge (docs/02's own proof line) — exercised at L3 by
-#     tests/l3-fault-injection/node-loss (R-STATE-02).
-#   * docs/02 prose puts the baseline in cloud-init; the implementation moved it into the
-#     Ansible node_baseline role. R-ENV-01 requires the node to be "baselined ... as code",
-#     not that cloud-init be the tool, so the check asserts the STAGE and RECORDS the
-#     divergence rather than hiding it.
-# =============================================================================
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: dev-nodes-provisioned-as-code
-  labels:
-    conformance.home.k3s/R-ENV-01: "true"
-  annotations:
-    conformance.home.k3s/requirements: "R-ENV-01"
-    conformance.home.k3s/layer: "L0"
-    # NO `scope` ANNOTATION, DELIBERATELY — this invariant IS path-scoped, and was audited as
-    # such (2026-09-12). Its entire input surface is infra/ (OpenTofu + the Ansible roles that
-    # provision the dev nodes); it reads nothing else. The gap was that
-    # ci/touched-requirements.sh keyed infra/* to NOTHING, so an infra/ edit attributed no
-    # requirement and this check was filtered out. The right fix for a path-scoped invariant is
-    # a path-map entry, not an unconditional run: `infra/*) echo "R-ENV-01"` was added there in
-    # the same change. Keep the two in sync — if this check ever grows an assertion outside
-    # infra/, either widen that map entry or give this test a whole-repo scope.
-    conformance.home.k3s/safety-critical: "false"
-    conformance.home.k3s/executable: "true"
-    conformance.home.k3s/intent: >-
-      Given the tracked tree, when the dev provisioning chain is parsed and driven
-      offline, then every dev node is created by local-KVM IaC, baselined, cluster-joined
-      and handed to Flux by one code path that covers all of them — with no node outside
-      it and no stage missing.
-spec:
-  steps:
-    - name: then-every-dev-node-is-created-baselined-joined-and-managed-as-code
-      description: >-
-        Run the offline assertion script over the tracked tree (same script ci/l0.sh and
-        conformance/run.sh --layer l0 execute).
-      try:
-        - script:
-            content: bash ./check.sh

conformance/tests/l0-static/dev-nodes-provisioned-as-code/check.sh

@@ -1,442 +0,0 @@
-#!/usr/bin/env bash
-# =============================================================================
-# L0 static check (offline, no cluster) — dev-nodes-provisioned-as-code
-# Serves: R-ENV-01 ("dev/test MUST be provisioned as local KVMs via IaC: created,
-#         baselined, cluster-joined, and managed as code." — spec
-#         docs/00-vision-and-scope.md; the four stages are enumerated in
-#         docs/02-environments-and-provisioning.md §"Node lifecycle stages
-#         (R-ENV-01 — 'created, baselined, cluster-joined, managed as code')":
-#           1. Create   — OpenTofu instantiates the VM
-#           2. Baseline — OS baseline and prerequisites
-#           3. Cluster-join — Ansible installs/joins k3s with Cilium as CNI; disables
-#              k3s bundled flannel/traefik/servicelb/kube-proxy
-#           4. Manage   — GitOps-managed by Flux; node config changes are made via IaC
-#              and re-applied, never hand-edited on the box.)
-#
-# WHY L0 IS THE RIGHT LAYER, AND WHAT IT MEANS TO TEST THIS BEHAVIOURALLY
-# ----------------------------------------------------------------------
-# R-ENV-01 is a requirement about HOW the dev nodes come to exist. The live cluster
-# cannot answer it: a Kubernetes node object looks identical whether it was built by
-# OpenTofu or by hand in a hypervisor UI (there is no API field for "was I created by
-# IaC"). What CAN be proven, with no cluster, is that a COMPLETE, EXECUTABLE
-# provisioning path exists for all four stages and covers every dev node — and, for the
-# Ansible half, that is proven by RUNNING the real toolchain in a no-connection mode
-# (`--syntax-check`, `--list-hosts`) rather than by reading the files, so a chain that
-# does not actually resolve fails here. The IaC half is asserted by parsing the module:
-# `tofu init/validate` needs the provider registry, and L0 is offline by contract.
-#
-# GIVEN the repository as git tracks it,
-# WHEN  the provisioning chain is parsed and (where offline execution is possible)
-#       actually driven,
-# THEN  CREATE:  the IaC module targets a LOCAL KVM hypervisor (every declared provider
-#                is a KVM-family hypervisor provider — no cloud provider may appear),
-#                instantiates its nodes from a for_each map (never a count index, so one
-#                node can be replaced without disturbing its siblings), and gives each
-#                node CPU, RAM, an OS disk, a DEDICATED Longhorn data disk, a NIC and a
-#                cloud-init seed carrying the login user + SSH key;
-#       AND      the node set it creates is EXACTLY the dev node set Ansible manages —
-#                names derived from the module's own prefix/count, so no node can be
-#                hand-added to the inventory or hand-created in the hypervisor;
-#       BASELINE: a baseline stage exists as declarative Ansible (package floor, kernel
-#                modules persisted AND loaded, sysctls, the Longhorn data disk formatted
-#                and mounted, swap off, iscsid enabled), driven by variables with
-#                defaults rather than per-host special cases;
-#       JOIN:    one role installs k3s for both roles of every node — `--cluster-init`
-#                on the primary, `--server` join on the secondaries — with flannel,
-#                kube-proxy, traefik and servicelb disabled AT INSTALL, and Cilium
-#                installed as the CNI afterwards;
-#       MANAGE:  site.yml chains baseline -> k3s -> Cilium -> verify -> Flux bootstrap;
-#                `ansible-playbook --syntax-check` on that chain SUCCEEDS; and
-#                `--list-hosts` shows every one of those stages targeting EVERY dev
-#                node, so no node is outside the code path.
-#
-# ENVIRONMENT-AGNOSTIC: no hostname, address, prefix or node count is written down here.
-# The dev group names come from the playbooks' own default host patterns, the node names
-# from the IaC module's prefix/count variables, and the two sets are compared to each
-# other. This check therefore keeps working when the estate is renamed or resized.
-#
-# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this test as claiming them):
-#   - That the RUNNING dev VMs were in fact created by this module. Nothing offline can:
-#     it would need the OpenTofu state (a remote R2 backend + credentials) or hypervisor
-#     access. `tofu plan` against the live estate is the operator's own check.
-#   - That the module APPLIES cleanly. `tofu validate` needs `tofu init`, which downloads
-#     the provider from the registry; L0 is offline by contract, so the module is parsed,
-#     not evaluated. ci/lint.sh does not cover .tf either — a gap worth closing in CI
-#     with a network-allowed stage, not here.
-#   - "Never hand-edited on the box" (stage 4's second half) — a drift check against the
-#     live nodes, which is an operator/Ansible-check-mode exercise, not a repo property.
-#   - REBUILD/re-converge ("destroy a dev node VM and rebuild it", docs/02's own proof
-#     line) — that is exercised at L3 by tests/l3-fault-injection/node-loss (R-STATE-02).
-#   - That the baseline actually ran on a node (cluster/host state), and that the
-#     hypervisor is on the operator's own LAN rather than remote — "local" is asserted as
-#     "a KVM hypervisor provider driving a self-hosted endpoint variable", not as a
-#     network topology fact.
-#   - DEVIATION FROM docs/02 PROSE, ASSERTED DELIBERATELY: docs/02 describes stage 2 as
-#     cloud-init applying the OS baseline. The implementation moved the rich baseline into
-#     the Ansible `node_baseline` role (cloud-init keeps hostname/user/SSH/network),
-#     because Proxmox provisioning is API-only and a custom cloud-init snippet needs
-#     SSH/SFTP to the hypervisor. R-ENV-01's own wording requires the node to be
-#     "baselined ... as code", not that cloud-init be the tool, so this check asserts the
-#     STAGE, wherever it lives, and records the divergence rather than hiding it.
-# =============================================================================
-set -euo pipefail
-DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
-PYTHON="${PYTHON:-python3}"
-
-cd "${ROOT}"
-"${PYTHON}" - "${ROOT}" <<'PY'
-import re
-import shutil
-import subprocess
-import sys
-from pathlib import Path
-
-try:
-    import yaml
-except ImportError:                                  # pragma: no cover
-    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
-          file=sys.stderr)
-    raise SystemExit(1)
-
-ROOT = Path(sys.argv[1])
-FAILURES: list[str] = []
-NOTES: list[str] = []
-CHECKS = 0
-
-
-def check(cond, msg):
-    global CHECKS
-    CHECKS += 1
-    if not cond:
-        FAILURES.append(msg)
-
-
-def note(msg):
-    NOTES.append(msg)
-
-
-# A LOCAL DEPENDENCY the repo deliberately does not carry — the vault passphrase (kept
-# outside the repo, R-SEC-01) or the Galaxy collections (gitignored; installed from
-# collections/requirements.yml). Their absence is an ENVIRONMENT gap, not a repo defect,
-# and is the ONLY tolerated reason the driven sub-assertions may not run. Deliberately
-# narrow: every other non-zero exit is a real failure. The parse assertions always run.
-ENV_ONLY_FAILURE = re.compile(
-    r"vault password|vault-id|Attempting to decrypt"
-    r"|No module named 'ansible_collections|couldn't resolve module/action", re.I)
-
-TOFU = ROOT / "infra" / "tofu"
-ANS = ROOT / "infra" / "ansible"
-check(TOFU.is_dir(), "infra/tofu/ is missing — the CREATE stage has no IaC at all (R-ENV-01)")
-check(ANS.is_dir(), "infra/ansible/ is missing — the BASELINE/JOIN/MANAGE stages have no code")
-if not TOFU.is_dir() or not ANS.is_dir():
-    print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} failed",
-          file=sys.stderr)
-    for f in FAILURES:
-        print(f"  - {f}", file=sys.stderr)
-    raise SystemExit(1)
-
-TF = "\n".join((p.read_text(errors="replace") for p in sorted(TOFU.glob("*.tf"))))
-check(len(TF) > 500, "infra/tofu/*.tf is empty or a stub — nothing creates the dev nodes")
-
-# ============================================================ STAGE 1 — CREATE (IaC)
-# A LOCAL KVM hypervisor, by allowlist: any provider outside this set fails, so moving
-# dev onto a cloud (or anything that is not a KVM hypervisor the operator runs) is caught
-# here rather than being noticed later. Each entry is justified, not just listed:
-#   bpg/proxmox        Proxmox VE — a KVM/QEMU hypervisor the operator hosts.
-#   dmacvicar/libvirt  libvirt/KVM directly (the pre-Proxmox substrate).
-KVM_PROVIDERS = {"bpg/proxmox", "dmacvicar/libvirt"}
-providers = dict(re.findall(r'source\s*=\s*"([^"]+)"', TF) and
-                 [(m.group(1), m.group(2)) for m in
-                  re.finditer(r'(\w[\w-]*)\s*=\s*\{[^}]*?source\s*=\s*"([^"]+)"', TF, re.S)])
-check(len(providers) > 0,
-      "infra/tofu declares no required_providers — the substrate is unpinned, so 'via IaC' "
-      "cannot be checked at all")
-for alias, source in sorted(providers.items()):
-    check(source.lower() in KVM_PROVIDERS,
-          f"infra/tofu declares provider {alias}={source!r}, which is not a local KVM hypervisor "
-          f"provider (allowed: {sorted(KVM_PROVIDERS)}) — R-ENV-01 requires dev/test to be LOCAL KVMs")
-
-# The VM resource: for_each-keyed, and carrying the hardware + seed docs/02 enumerates.
-vm_blocks = re.findall(r'resource\s+"([\w]*(?:_vm|_domain))"\s+"(\w+)"\s*\{(.*?)\n\}',
-                       TF, re.S)
-check(len(vm_blocks) >= 1,
-      "no VM/domain resource found in infra/tofu — nothing in IaC creates a dev node")
-for rtype, rname, body in vm_blocks:
-    check("for_each" in body,
-          f'resource "{rtype}" "{rname}" is not for_each-keyed. A count index makes single-node '
-          "replace unsafe (a -target'ed destroy cascaded to sibling nodes — DECISIONS-LOG F1), so "
-          "'created ... as code' would not survive a rebuild")
-    check(not re.search(r"^\s*count\s*=", body, re.M),
-          f'resource "{rtype}" "{rname}" uses a positional count index')
-    for want, why in (
-        (r"\bcpu\b", "CPU"),
-        (r"\bmemory\b", "RAM"),
-        (r"network_device|network_interface", "a NIC"),
-        (r"initialization|cloud_init|cloudinit", "a cloud-init seed"),
-    ):
-        check(re.search(want, body) is not None,
-              f'resource "{rtype}" "{rname}" declares no {why} — docs/02 stage 1 requires CPU, RAM, '
-              "disks, NICs and a cloud-init seed to come from IaC")
-    disks = len(re.findall(r"^\s*disk\s*\{", body, re.M))
-    check(disks >= 2,
-          f'resource "{rtype}" "{rname}" declares {disks} disk block(s); docs/02 requires the OS '
-          "disk PLUS a dedicated data disk for Longhorn")
-    check(re.search(r"user_account|ssh_authorized_key|ssh_keys", body) is not None,
-          f'resource "{rtype}" "{rname}" seeds no login user / SSH key — the node could not be '
-          "reached by the BASELINE stage without a hand step")
-
-# Node identity comes from variables, and those variables define the whole node set.
-def tf_default(var: str):
-    m = re.search(r'variable\s+"' + re.escape(var) + r'"\s*\{(.*?)\n\}', TF, re.S)
-    if not m:
-        return None
-    d = re.search(r'default\s*=\s*(.+)', m.group(1))
-    return d.group(1).strip() if d else None
-
-
-prefix_raw = tf_default("node_name_prefix")
-count_raw = tf_default("node_count")
-ips_raw = tf_default("node_ips")
-check(prefix_raw is not None and count_raw is not None,
-      "infra/tofu has no node_name_prefix/node_count variables — the node set is not derived "
-      "from code, so the created nodes cannot be compared with the managed ones")
-prefix = (prefix_raw or "").strip('"')
-try:
-    node_count = int((count_raw or "0").strip())
-except ValueError:
-    node_count = 0
-check(node_count >= 1, f"node_count default is {count_raw!r} — no node would be created")
-if ips_raw is not None:
-    n_ips = len(re.findall(r'"[^"]+"', ips_raw))
-    check(n_ips >= node_count,
-          f"node_ips default has {n_ips} address(es) for node_count={node_count} — the module "
-          "cannot address every node it creates")
-check((TOFU / "outputs.tf").is_file() and
-      re.search(r'output\s+"(ansible_inventory|nodes)"', (TOFU / "outputs.tf").read_text()) is not None,
-      "infra/tofu/outputs.tf surfaces no node inventory output — the JOIN stage would have to be "
-      "told the node facts by hand instead of consuming them from IaC")
-
-# ------------------------------------------------- the dev node set, derived twice over
-# Group names are taken from the PLAYBOOKS' own default host patterns, so nothing here
-# restates the estate's naming.
-PLAYBOOKS = ANS / "playbooks"
-
-
-def default_patterns(playbook: str) -> list[str]:
-    p = PLAYBOOKS / playbook
-    if not p.is_file():
-        return []
-    return re.findall(r"default\('([^']+)'\)", p.read_text())
-
-
-dev_groups = set(default_patterns("node-baseline.yml") + default_patterns("k3s.yml"))
-check(len(dev_groups) > 0,
-      "no playbook declares a default host pattern — the dev target is not expressed in code")
-
-INV = ANS / "inventory" / "hosts.ini"
-check(INV.is_file(), "infra/ansible/inventory/hosts.ini is missing — no node is managed as code")
-ini_groups: dict[str, list[str]] = {}
-children: dict[str, list[str]] = {}
-cur = None
-for line in (INV.read_text().splitlines() if INV.is_file() else []):
-    s = line.strip()
-    if not s or s.startswith("#"):
-        continue
-    m = re.match(r"^\[([^\]]+)\]$", s)
-    if m:
-        cur = m.group(1)
-        (children if cur.endswith(":children") else ini_groups).setdefault(
-            cur.split(":")[0] if cur.endswith(":children") else cur, [])
-        continue
-    if cur is None or cur.endswith(":vars"):
-        continue
-    host = s.split()[0]
-    if cur.endswith(":children"):
-        children.setdefault(cur.split(":")[0], []).append(host)
-    else:
-        ini_groups.setdefault(cur, []).append(host)
-
-
-def hosts_in(group: str, seen=None) -> set[str]:
-    seen = seen or set()
-    if group in seen:
-        return set()
-    seen.add(group)
-    out = set(ini_groups.get(group, []))
-    for child in children.get(group, []):
-        out |= hosts_in(child, seen)
-    return out
-
-
-dev_hosts: set[str] = set()
-for g in sorted(dev_groups):
-    dev_hosts |= hosts_in(g)
-check(len(dev_hosts) > 0,
-      f"the dev groups {sorted(dev_groups)} resolve to NO hosts in hosts.ini — nothing is managed")
-expected = {f"{prefix}{i + 1}" for i in range(node_count)}
-check(dev_hosts == expected,
-      f"the dev nodes Ansible manages ({sorted(dev_hosts)}) are not exactly the nodes the IaC "
-      f"module creates ({sorted(expected)}) — a node outside IaC is a hand-built node (R-ENV-01)")
-
-# ============================================================ STAGE 2 — BASELINE (code)
-BASE_ROLE = ANS / "roles" / "node_baseline"
-check(BASE_ROLE.is_dir(),
-      "no node_baseline role — the BASELINE stage has no code (docs/02 stage 2)")
-base_tasks = "\n".join(p.read_text(errors="replace")
-                       for p in sorted((BASE_ROLE / "tasks").glob("*.yml"))) if BASE_ROLE.is_dir() else ""
-base_defaults = "\n".join(p.read_text(errors="replace")
-                          for p in sorted((BASE_ROLE / "defaults").glob("*.yml"))) if BASE_ROLE.is_dir() else ""
-# Each item docs/02 names for stage 2, matched on the ANSIBLE MODULE that would do it —
-# a declarative, idempotent task, not a shell one-liner.
-for pattern, what in (
-    (r"ansible\.builtin\.apt:", "a package baseline (apt module)"),
-    (r"modprobe:", "kernel modules loaded (modprobe module)"),
-    (r"modules-load|/etc/modules", "kernel modules PERSISTED across reboot"),
-    (r"sysctl", "sysctl tuning"),
-    (r"filesystem:|mkfs", "a filesystem on the Longhorn data disk"),
-    (r"mount:|ansible\.posix\.mount", "the Longhorn data disk MOUNTED"),
-    (r"swap", "swap disabled"),
-    (r"iscsid", "iscsid enabled (Longhorn prerequisite)"),
-):
-    check(re.search(pattern, base_tasks) is not None,
-          f"node_baseline does not declare {what} — docs/02 stage 2 requires the OS baseline and "
-          "k3s/Longhorn/KubeVirt prerequisites to be applied as code")
-check(re.search(r"^node_baseline_packages:", base_defaults, re.M) is not None and
-      re.search(r"^node_baseline_kernel_modules:", base_defaults, re.M) is not None,
-      "node_baseline's package/module sets are not variables with defaults — a per-host special "
-      "case is not 'managed as code' (and could not be scoped per environment, R-ENV-02)")
-
-# ========================================================== STAGE 3 — CLUSTER-JOIN (k3s)
-K3S_ROLE = ANS / "roles" / "k3s_server"
-k3s_text = "\n".join(p.read_text(errors="replace")
-                     for p in sorted((K3S_ROLE).rglob("*.yml"))) if K3S_ROLE.is_dir() else ""
-check(K3S_ROLE.is_dir() and len(k3s_text) > 200,
-      "no k3s_server role — nothing joins the nodes into a cluster (docs/02 stage 3)")
-check("--cluster-init" in k3s_text,
-      "the k3s role never passes --cluster-init — the HA embedded-etcd cluster is never founded")
-check(re.search(r"--server\s+https", k3s_text) is not None,
-      "the k3s role never joins secondaries with --server — nodes would not form one cluster")
-check(re.search(r"K3S_TOKEN|--token", k3s_text) is not None,
-      "the k3s role never supplies a join token — the join could only be completed by hand")
-# The bundled components docs/02 requires to be disabled AT INSTALL, each checked by the
-# flag that actually disables it.
-for pattern, what in (
-    (r"--flannel-backend=none", "flannel (Cilium is the CNI)"),
-    (r"--disable-kube-proxy", "kube-proxy (Cilium kube-proxy replacement)"),
-    (r"traefik", "traefik (Gateway API via Cilium)"),
-    (r"servicelb", "servicelb"),
-):
-    check(re.search(pattern, k3s_text) is not None,
-          f"the k3s install does not disable {what} — docs/02 stage 3 requires it disabled at "
-          "install, or the bundled component fights the platform's own")
-check((ANS / "roles" / "cilium").is_dir(),
-      "no cilium role — stage 3 requires Cilium to be installed as the CNI at bootstrap")
-check((ANS / "roles" / "flux_bootstrap").is_dir(),
-      "no flux_bootstrap role — stage 4's hand-off to GitOps has no code")
-
-# ============================================== STAGE 4 — MANAGE + the chain is REAL
-SITE = PLAYBOOKS / "site.yml"
-check(SITE.is_file(), "no playbooks/site.yml — the provisioning chain is not expressed as one path")
-site_txt = SITE.read_text() if SITE.is_file() else ""
-order = [m for m in re.findall(r"import_playbook:\s*(\S+)", site_txt)]
-check(order[:1] == ["node-baseline.yml"],
-      f"site.yml imports {order} — the BASELINE stage must come first (docs/02 sequencing)")
-for want in ("k3s.yml", "cilium.yml", "flux-bootstrap.yml"):
-    check(want in order, f"site.yml does not import {want} — a lifecycle stage is missing from the chain")
-if "k3s.yml" in order and "cilium.yml" in order:
-    check(order.index("k3s.yml") < order.index("cilium.yml"),
-          "site.yml installs Cilium before k3s — the CNI cannot precede the cluster")
-
-# --- DRIVE THE REAL TOOLCHAIN, OFFLINE ------------------------------------------------
-# --syntax-check and --list-hosts open NO connection to any node: they resolve the
-# playbook, roles, collections and inventory only. Running them is the difference between
-# "the chain is written down" and "the chain resolves".
-if shutil.which("ansible-playbook") is None:
-    note("ansible-playbook is not installed here (e.g. the in-cluster chainsaw image), so the "
-         "chain was PARSED but not DRIVEN; the merge gate's container has it and does drive it")
-else:
-    # cwd=infra/ansible so the tracked ansible.cfg (roles_path, collections_path, inventory)
-    # is the one in force — the same resolution a real run gets.
-    def ansible(*args) -> subprocess.CompletedProcess:
-        return subprocess.run(["ansible-playbook", *args], cwd=ANS,
-                              capture_output=True, text=True)
-
-    syn = ansible("--syntax-check", "playbooks/site.yml")
-    syn_out = (syn.stderr or "") + (syn.stdout or "")
-    lh = None
-    if syn.returncode != 0 and ENV_ONLY_FAILURE.search(syn_out):
-        # NARROW, SELF-POLICING SKIP — see ENV_ONLY_FAILURE above.
-        note("ansible-playbook lacks a LOCAL dependency kept outside the repo by design (the vault "
-             "passphrase and/or the gitignored Galaxy collections), so --syntax-check/--list-hosts "
-             "were not driven; every parse assertion above still ran")
-    else:
-        check(syn.returncode == 0,
-              "`ansible-playbook --syntax-check playbooks/site.yml` FAILED — the provisioning chain "
-              f"does not even load: {syn_out.strip()[:400]}")
-        lh = ansible("--list-hosts", "playbooks/site.yml")
-        check(lh.returncode == 0,
-              "`ansible-playbook --list-hosts playbooks/site.yml` FAILED — the chain does not "
-              f"resolve against the tracked inventory: "
-              f"{((lh.stderr or '') + (lh.stdout or '')).strip()[:400]}")
-    if lh is not None and lh.returncode == 0:
-        # Parse plays -> hosts. NEVER `ansible-inventory --list`: it prints DECRYPTED vault values.
-        plays: list[tuple[str, set[str]]] = []
-        name, hosts, in_hosts = None, set(), False
-        for line in lh.stdout.splitlines():
-            m = re.match(r"\s*play #\d+ \([^)]*\):\s*(.*?)\s*TAGS:", line)
-            if m:
-                if name is not None:
-                    plays.append((name, hosts))
-                name, hosts, in_hosts = m.group(1), set(), False
-                continue
-            if re.match(r"\s*hosts \(\d+\):", line):
-                in_hosts = True
-                continue
-            if in_hosts and line.strip() and not line.strip().startswith("pattern:"):
-                hosts.add(line.strip())
-        if name is not None:
-            plays.append((name, hosts))
-        check(len(plays) >= 4,
-              f"--list-hosts resolved {len(plays)} play(s) — the four-stage chain did not expand")
-        # Every lifecycle stage must cover EVERY dev node. A stage that reaches two of three
-        # nodes means the third was finished by hand.
-        STAGES = {
-            "baseline": r"baseline",
-            "cluster-join": r"^k3s (primary|secondar)",
-            "cni": r"cilium",
-            "manage (flux hand-off)": r"flux",
-        }
-        for stage, rx in STAGES.items():
-            covered: set[str] = set()
-            matched = 0
-            for pname, phosts in plays:
-                if re.search(rx, pname, re.I):
-                    matched += 1
-                    covered |= phosts
-            check(matched >= 1, f"no play in site.yml implements the {stage} stage")
-            check(covered == dev_hosts,
-                  f"the {stage} stage targets {sorted(covered)}, not every dev node "
-                  f"{sorted(dev_hosts)} — a node outside the code path is a hand-provisioned "
-                  "node (R-ENV-01)")
-
-# --------------------------------------------------------------------------- verdict
-# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
-check(CHECKS >= 30,
-      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
-      "would be meaningless")
-
-if FAILURES:
-    print(f"CHECK FAIL (dev-nodes-provisioned-as-code): {len(FAILURES)} of {CHECKS} assertions failed",
-          file=sys.stderr)
-    for f in FAILURES:
-        print(f"  - {f}", file=sys.stderr)
-    raise SystemExit(1)
-for n in NOTES:
-    print(f"  note: {n}")
-print(f"check OK: dev-nodes-provisioned-as-code — {CHECKS} assertions; "
-      f"{len(dev_hosts)} dev node(s) {sorted(dev_hosts)} created by IaC ({sorted(providers.values())}), "
-      f"baselined, cluster-joined and Flux-managed by the same chain. R-ENV-01's authored-path "
-      "clause only — the live VMs' provenance, `tofu validate`, and rebuild/re-converge are NOT "
-      "proven here (see the header's gap list; rebuild is L3 node-loss).")
-PY

conformance/tests/l0-static/egress-paths-selected-by-rules/chainsaw-test.yaml

@@ -1,99 +0,0 @@
-# =============================================================================
-# L0 STATIC — two egress paths, and the rule set that selects between them  (R-NET-04)  [EXECUTABLE]
-# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
-# Spec:    docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
-#          routing rules ... realized as two paths — the local `internet` path and the
-#          in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress
-#          routing: two paths, one physical WAN": "the `internet` path is simply the default
-#          route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and
-#          the nodes themselves — egress the local/private WAN by default and **never** the
-#          VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the
-#          kill-switch blackholes **only** that class."
-#
-# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here)
-# ------------------------------------------------------------------------
-# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class
-# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and
-# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES
-# that select among them, which nothing else asserts:
-#   * exactly TWO paths, the same closed set in the schema, the admission policy and the
-#     renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope);
-#   * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one
-#     Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one
-#     `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because
-#     absence of a matching rule IS the default path;
-#   * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a
-#     vpn-class pod's cluster/LAN traffic still takes the local path;
-#   * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing
-#     table, `route-noexec` is set and a pushed `redirect-gateway` is ignored;
-#   * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every
-#     DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN.
-#     (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.)
-#
-# WHY L0: every clause above is a statement about ALL rules — including rules that must be
-# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default
-# route in the main table). A cluster can show which path two pods took; it cannot show that
-# no third rule exists or that the vocabulary is closed.
-#
-# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are
-# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod
-# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked
-# against its own values.
-#
-# GIVEN the repository as git tracks it,
-# WHEN  the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy
-#       routing / OpenVPN / nftables templates, and every environment's overlay addresses are
-#       parsed and cross-checked,
-# THEN  there are exactly two paths with `internet` as the default; exactly one steering rule
-#       selects exactly the vpn class by label; the gateway is selected by node label with a
-#       named egress interface; all RFC1918 space and every address any environment assigns
-#       itself is excluded from the tunnel; the tunnel's default route and its fail-closed
-#       blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN
-#       cannot take the main table; and no nftables DROP is unscoped.
-#
-# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
-# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
-# Chainsaw Test wraps the SAME script.
-#
-# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
-#   * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by
-#     tests/l2-behavioral/egress-default-path (same requirement label); the two labelled
-#     classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test.
-#   * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates
-#     that generate them; a hand-added rule on the box is invisible here.
-#   * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no
-#     honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default".
-#   * IPv6 egress.
-# =============================================================================
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: egress-paths-selected-by-rules
-  labels:
-    conformance.home.k3s/R-NET-04: "true"
-  annotations:
-    conformance.home.k3s/requirements: "R-NET-04"
-    conformance.home.k3s/layer: "L0"
-    # WHOLE-REPO SCOPE. Universally quantified over tracked YAML: every manifest that carries
-    # an egress-class label/annotation must use a value the policy set actually steers. The
-    # file most likely to break it is a NEW workload under apps/ — which attributes no
-    # requirement — not platform/vpn-egress/ (the only path the map keys to R-NET-04).
-    conformance.home.k3s/scope: "whole-repo"
-    conformance.home.k3s/safety-critical: "false"
-    conformance.home.k3s/executable: "true"
-    conformance.home.k3s/intent: >-
-      Given the tracked tree, when the egress vocabulary, every steering rule and the gateway
-      node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers
-      exactly two egress paths with the WAN as the default, exactly one rule selects exactly
-      the vpn class, local destinations stay on the local path in every environment, the
-      tunnel never becomes the default route, and nothing unscoped can blackhole the default
-      path.
-spec:
-  steps:
-    - name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly
-      description: >-
-        Run the offline assertion script over the tracked tree (same script ci/l0.sh and
-        conformance/run.sh --layer l0 execute).
-      try:
-        - script:
-            content: bash ./check.sh

conformance/tests/l0-static/egress-paths-selected-by-rules/check.sh

@@ -1,521 +0,0 @@
-#!/usr/bin/env bash
-# =============================================================================
-# L0 static check (offline, no cluster) — egress-paths-selected-by-rules
-# Serves: R-NET-04 ("The platform MUST support **multiple egress paths** selected by routing
-#         rules. Per OQ-1 the environment has **one physical commercial WAN and no
-#         multi-ISP plans**, so this is realized as two paths — the local `internet` path and
-#         the in-cluster `vpn` path — not multi-ISP routing. Physical multi-WAN (multiple ISP
-#         uplinks) is **out of scope** unless a future ADR reopens it."
-#         — spec docs/00-vision-and-scope.md).
-#
-# The elaborating spec text this check quotes and asserts against — docs/03
-# §"Egress routing: two paths, one physical WAN (R-NET-04/05 — resolved by OQ-1)":
-#     | `internet` | Out the single commercial WAN | Normal pod egress via node -> SOHO edge
-#                                                    router -> internet |
-#     | `vpn`      | Routed through the in-cluster VPN tunnel | Steered to the shared VPN
-#                                                    egress gateway (ADR-0013) |
-#     "**Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing
-#      to policy-route across; the `internet` path is simply the default route out."
-#     "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes
-#      themselves — egress the local/private WAN by default and **never** the VPN. Only
-#      services labeled `egress-class=vpn` are steered to the tunnel, and the kill-switch
-#      blackholes **only** that class."
-#     "It is **split-tunnel**: it creates `tun0` but does **not** set the node's default
-#      route (`route-noexec` + policy routing), so the node's own traffic stays on the WAN —
-#      only what Cilium points at `tun0` uses the VPN."
-#
-# WHAT R-NET-04 ADDS OVER R-NET-05/07 (which are ALREADY covered) — READ THIS FIRST
-# ---------------------------------------------------------------------------------
-# R-NET-05/07 are about a SERVICE: it can declare its class, enforcement happens outside it,
-# and the class can be flipped via GitOps. Those are proven by
-# tests/l1-rendering/egress-class-render (the render) and tests/l2-behavioral/egress-class
-# (two labelled pods observe two different public IPs, and a relabel re-routes the same pod).
-# Re-tagging either of those with R-NET-04 would be a duplicate wearing a new label.
-#
-# R-NET-04 is about the PLATFORM's PATH SET and the RULES THAT SELECT AMONG THEM. Its
-# distinctive content — none of which any existing test asserts — is:
-#
-#   (1) THERE ARE EXACTLY TWO PATHS, and the same two everywhere. The class vocabulary is
-#       one closed set of two values agreed by the schema, the admission policy and the
-#       renderer. A third value appearing in one of them but not the others would be a path
-#       nothing routes (or a path nothing validates); "multi-ISP" reappearing as a third
-#       class would contradict OQ-1's resolution.
-#   (2) SELECTION IS BY A ROUTING RULE, and the rule set is TOTAL AND DISJOINT. The `vpn`
-#       path is entered only via rules that select the vpn class — a Cilium
-#       CiliumEgressGatewayPolicy pod-selector, and on the gateway node an `ip rule` scoped
-#       to the tunnel source address. Nothing selects the `internet` class, because
-#       "the `internet` path is simply the default route out": ABSENCE of a matching rule IS
-#       the default path. So an unlabelled pod, and an `internet`-class pod, take the WAN.
-#   (3) THE PATHS COEXIST — the choice is per-destination, not per-pod-lifetime. The vpn
-#       rule tunnels public destinations while EXCLUDING all private space, so a vpn-class
-#       pod's cluster/LAN traffic still takes the local path at the same time as its
-#       internet traffic takes the tunnel.
-#   (4) THE VPN PATH NEVER BECOMES THE DEFAULT. Split tunnel: the tunnel's default route
-#       lives in a DEDICATED routing table reached only through the scoped rule, the
-#       provider's pushed `redirect-gateway` is ignored, and `route-noexec` keeps OpenVPN
-#       from touching the main table. Otherwise there would be one path (the VPN), not two.
-#   (5) THE KILL-SWITCH BLACKHOLES ONLY THAT CLASS. The blackhole default is inside the
-#       dedicated table; the nftables chain's policy is ACCEPT and every DROP is qualified
-#       by the vpn class's own source set / fwmark. The `internet` path therefore fails
-#       OPEN and can never be blackholed. (R-NET-06's opposite claim — that `vpn` fails
-#       CLOSED — is proven at L3 by tests/l3-fault-injection/vpn-killswitch. This check
-#       asserts the OTHER direction, which that test does not.)
-#
-# WHY L0. Every clause above is a property of the RULE SET, i.e. of authored configuration:
-# totality ("nothing selects the default path"), disjointness ("exactly one rule, selecting
-# exactly one class"), and scoping ("no unqualified drop, no default route in the main
-# table") are statements about ALL rules, including the ones that would have to be absent.
-# A running cluster can show you which path two pods took; it cannot show you that no third
-# rule exists, that no rule is unscoped, or that the vocabulary is closed. The behavioural
-# complement — that an UNLABELLED pod really does take the WAN path — is at L2 in
-# tests/l2-behavioral/egress-default-path (same requirement label).
-#
-# ENVIRONMENT-AGNOSTIC: no address is written down. The private ranges the vpn rule must
-# exclude are checked as CONTAINMENT of the addresses each environment's own Flux overlay
-# declares (pod CIDR, Gateway VIP, LB pool, API host) — so a new environment on a different
-# LAN is checked against its own addresses, and the check keeps working when they change.
-#
-# WHAT THIS DOES *NOT* PROVE (honest gaps — do not read this check as claiming them):
-#   - Behaviour. That a vpn-class pod's packets really leave via tun0, that an unlabelled
-#     pod's really do not, and that the tunnel is up at all. L2 covers the two labelled
-#     classes (tests/l2-behavioral/egress-class, R-NET-05) and the unlabelled default
-#     (tests/l2-behavioral/egress-default-path, R-NET-04); the kill-switch firing is L3.
-#   - THE RUNNING NODE's routing tables and nftables ruleset. This reads the templates that
-#     generate them, driven by no node. A hand-added `ip rule` on the gateway node is
-#     invisible here (and to every other test); drift on the box is an Ansible check-mode
-#     exercise for the operator.
-#   - PHYSICAL MULTI-WAN. Out of scope per OQ-1, so its absence is not asserted: there is no
-#     honest repository signal that would distinguish "one WAN, correctly" from "one WAN,
-#     because nobody tried" — and inventing one would be a hollow assertion.
-#   - IPv6. The policy's CIDRs and the routing rules are IPv4-only, matching the platform.
-#     If IPv6 egress is ever enabled, the `internet` path would need the same analysis and
-#     this check would NOT catch its absence.
-# =============================================================================
-set -euo pipefail
-DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
-PYTHON="${PYTHON:-python3}"
-
-cd "${ROOT}"
-"${PYTHON}" - "${ROOT}" <<'PY'
-import ipaddress
-import re
-import sys
-from pathlib import Path
-
-try:
-    import yaml
-except ImportError:                                  # pragma: no cover
-    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
-          file=sys.stderr)
-    raise SystemExit(1)
-
-ROOT = Path(sys.argv[1])
-FAILURES: list[str] = []
-CHECKS = 0
-LBL = "homelab.keithjackson.dev"
-CLASS_LABEL = f"{LBL}/egress-class"
-
-
-def check(cond, msg):
-    global CHECKS
-    CHECKS += 1
-    if not cond:
-        FAILURES.append(msg)
-
-
-def die():
-    print(f"CHECK FAIL (egress-paths-selected-by-rules): {len(FAILURES)} of {CHECKS} "
-          "assertions failed", file=sys.stderr)
-    for f in FAILURES:
-        print(f"  - {f}", file=sys.stderr)
-    raise SystemExit(1)
-
-
-def docs(path: Path):
-    """Every YAML document in a file, skipping empties."""
-    try:
-        return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
-    except yaml.YAMLError:
-        return []
-
-
-# Skip dot-DIRECTORIES (.git, .claude, .cache) while still matching dot-FILES such as
-# .sops.yaml. A bare rglob from ROOT also descends into NESTED WORKTREES —
-# .claude/worktrees/agent-*/ created for isolated agent runs — which made this check see
-# 5 CiliumEgressGatewayPolicy objects instead of the 1 authored, failing R-NET-04 for a
-# reason that has nothing to do with egress. .claude/ is gitignored (.gitignore:43), so
-# nothing under it is part of the authored surface by definition.
-TRACKED_YAML = sorted(p for p in ROOT.rglob("*.yaml")
-                      if "_artifacts" not in p.parts
-                      and not any(part.startswith(".")
-                                  for part in p.relative_to(ROOT).parent.parts))
-SCHEMA = ROOT / "schemas" / "servicedefinition.schema.json"
-RENDER = ROOT / "catalog" / "render.py"
-VPN_ROLE = ROOT / "infra" / "ansible" / "roles" / "vpn_egress_gateway"
-CLUSTERS = ROOT / "clusters"
-
-for p in (SCHEMA, RENDER, VPN_ROLE, CLUSTERS):
-    check(p.exists(), f"{p.relative_to(ROOT)} is missing — the egress path model cannot be "
-                      "read at all (R-NET-04)")
-if FAILURES:
-    die()
-
-
-# ======================================================= (1) EXACTLY TWO PATHS, AGREED
-import json  # noqa: E402  — only needed for the schema
-
-schema = json.loads(SCHEMA.read_text())
-egress_prop = ((schema.get("properties") or {}).get("egress") or {})
-enum = egress_prop.get("enum")
-check(isinstance(enum, list) and len(enum) == 2,
-      f"schemas/servicedefinition.schema.json declares egress enum {enum!r} — R-NET-04 is "
-      "realized as EXACTLY TWO paths (OQ-1: one WAN, no multi-ISP). A third value is a path "
-      "nothing routes; one value is not 'multiple egress paths'")
-PATHS = set(enum or [])
-check(PATHS == {"internet", "vpn"},
-      f"the declared egress paths are {sorted(PATHS)} — docs/03's table fixes them as "
-      "`internet` (out the single commercial WAN) and `vpn` (the in-cluster tunnel)")
-check(egress_prop.get("default") == "internet",
-      f"the schema's default egress class is {egress_prop.get('default')!r} — docs/03: "
-      "'Default is the WAN; the VPN is strictly opt-in', and 'the `internet` path is simply "
-      "the default route out'. A service that says nothing must get the WAN")
-
-# The admission policy must constrain the pod label to the SAME closed set, or a workload
-# could carry a class the schema never allowed (and that no rule routes).
-pol_files = sorted((ROOT / "platform" / "kyverno").rglob("*egress*.yaml"))
-check(len(pol_files) >= 1,
-      "no Kyverno policy file mentions egress under platform/kyverno — nothing stops a pod "
-      "carrying an egress class outside the declared path set")
-pol_text = "\n".join(p.read_text(errors="replace") for p in pol_files)
-for cls in sorted(PATHS):
-    check(re.search(rf"\b{re.escape(cls)}\b", pol_text) is not None,
-          f"the egress admission policy never names the {cls!r} path — a class the policy "
-          "does not know cannot be validated against the path set")
-# No tracked object may carry a class outside the closed set — with ONE self-policing
-# exemption. A Kyverno policy-test fixture deliberately carries an INVALID class so the
-# admission test can assert the policy REJECTS it; that pod is exempt only while a
-# kyverno-test.yaml in the same tests tree still expects `result: fail` for it. Delete the
-# expectation (or flip it to pass) and the exemption evaporates — never a bare path allowlist.
-def rejected_by_a_policy_test(resource_name: str, fixture: Path) -> bool:
-    tests_root = fixture.parent
-    while tests_root != ROOT and tests_root.name != "tests":
-        tests_root = tests_root.parent
-    if tests_root == ROOT:
-        return False
-    for tf in sorted(tests_root.rglob("kyverno-test.yaml")):
-        for d in docs(tf):
-            for case in (d.get("results") or []):
-                if case.get("result") == "fail" and resource_name in (case.get("resources") or []):
-                    return True
-    return False
-
-
-stray: list[str] = []
-_STRAY_RE = re.compile(rf"{re.escape(CLASS_LABEL)}:\s*[\"']?([A-Za-z][A-Za-z0-9-]*)")
-for f in TRACKED_YAML:
-    for d in docs(f):
-        labels = ((d.get("metadata") or {}).get("labels") or {})
-        cls = labels.get(CLASS_LABEL)
-        # Also catch pod-template labels (Deployments/DaemonSets/Jobs/VMs).
-        tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {}).get("labels") or {}
-        for value, owner in ((cls, d), (tmpl.get(CLASS_LABEL), d)):
-            if value is None or value in PATHS or value == "CLASS":
-                continue
-            name = ((owner.get("metadata") or {}).get("name") or "?")
-            if rejected_by_a_policy_test(name, f):
-                continue
-            stray.append(f"{f.relative_to(ROOT)}:{name}={value}")
-    # Values written as raw text (templates, docs-as-code) are caught too.
-    for value in set(_STRAY_RE.findall(f.read_text(errors="replace"))):
-        if value in PATHS or value == "CLASS":
-            continue
-        if any(value in s for s in stray):
-            continue
-        # Raw-text hit with no parsed owner: exempt only if EVERY object in the file that
-        # carries this value is a policy-test fixture proven to be rejected.
-        owners = [((d.get("metadata") or {}).get("name") or "?") for d in docs(f)
-                  if ((d.get("metadata") or {}).get("labels") or {}).get(CLASS_LABEL) == value]
-        if owners and all(rejected_by_a_policy_test(o, f) for o in owners):
-            continue
-        stray.append(f"{f.relative_to(ROOT)}:<text>={value}")
-check(not stray,
-      f"tracked YAML assigns egress-class value(s) outside the declared path set "
-      f"{sorted(PATHS)}: {sorted(stray)} — a value no routing rule selects and no admission "
-      "policy allows is a workload on no path at all (the only exemption is a Kyverno "
-      "policy-test fixture that a kyverno-test.yaml still expects to FAIL)")
-
-# The renderer must project the DECLARED class onto the pod label, unmodified: that label is
-# what the routing rule selects on, so a renderer that rewrote it would break the coupling
-# between the declaration and the path.
-rtext = RENDER.read_text()
-check(re.search(rf'f?"\{{LBL\}}/egress-class":\s*s\["egress"\]', rtext) is not None
-      or re.search(r'egress-class\"\]?\s*:\s*s\[\"egress\"\]', rtext) is not None,
-      "catalog/render.py does not project the ServiceDefinition's own `egress` value onto the "
-      f"{CLASS_LABEL} pod label — the declaration and the routing rule's selector would be "
-      "two independent facts that could disagree")
-
-
-# ============================== (2) SELECTION IS BY RULE; THE RULE SET IS TOTAL & DISJOINT
-policies: list[tuple[Path, dict]] = []
-for p in TRACKED_YAML:
-    for d in docs(p):
-        if d.get("kind") == "CiliumEgressGatewayPolicy":
-            policies.append((p, d))
-check(len(policies) == 1,
-      f"{len(policies)} CiliumEgressGatewayPolicy object(s) are authored "
-      f"({[str(p.relative_to(ROOT)) for p, _ in policies]}) — R-NET-04's steering must be ONE "
-      "rule. Two rules can overlap, and which one wins for an overlapping pod is not "
-      "something the repository states")
-
-for path, pol in policies:
-    rel = str(path.relative_to(ROOT))
-    spec = pol.get("spec") or {}
-    selectors = spec.get("selectors") or []
-    check(len(selectors) == 1,
-          f"{rel}: the steering rule has {len(selectors)} selector(s) — every additional "
-          "selector widens the set of pods steered off the default path, and R-NET-04's "
-          "selection must be exactly 'the vpn class'")
-    for sel in selectors:
-        pod_sel = sel.get("podSelector")
-        check(isinstance(pod_sel, dict) and pod_sel,
-              f"{rel}: a selector has no podSelector — an empty selector in Cilium matches "
-              "EVERY pod, which would put the whole cluster on the VPN path and leave no "
-              "default path at all")
-        ml = (pod_sel or {}).get("matchLabels") or {}
-        me = (pod_sel or {}).get("matchExpressions") or []
-        check(not me,
-              f"{rel}: the podSelector uses matchExpressions {me!r}. The class contract is a "
-              "single label equality; expressions (In/NotIn/Exists) can select pods that "
-              "never declared a class, so the rule set would no longer be disjoint from the "
-              "default path")
-        check(set(ml) == {CLASS_LABEL},
-              f"{rel}: the podSelector matches on {sorted(ml)} — it must select on exactly "
-              f"{CLASS_LABEL}. Any other key makes path selection depend on something other "
-              "than the declared class (and a namespace/app key would silently exempt pods)")
-        check(ml.get(CLASS_LABEL) == "vpn",
-              f"{rel}: the steering rule selects {CLASS_LABEL}={ml.get(CLASS_LABEL)!r}. Only "
-              "the `vpn` class may be steered: docs/03 — 'Only services labeled "
-              "egress-class=vpn are steered to the tunnel'. A rule selecting `internet` would "
-              "route the default path through the tunnel, i.e. leave one path, not two")
-
-    # The gateway is chosen by a NODE LABEL, not a hostname: the path's endpoint is itself a
-    # rule, so the gateway can move (or gain a standby) with no rule rewrite.
-    gw = spec.get("egressGateway") or {}
-    node_sel = (gw.get("nodeSelector") or {}).get("matchLabels") or {}
-    check(bool(node_sel),
-          f"{rel}: the egressGateway selects no node by label — pinning the path's exit to a "
-          "named host means the vpn path cannot fail over (docs/03 requires two gateway "
-          "nodes + Egress Gateway HA)")
-    check("nodeName" not in gw,
-          f"{rel}: the egressGateway pins a nodeName — the exit node must be selected by "
-          "label so the path survives losing that node")
-    check(isinstance(gw.get("interface"), str) and gw.get("interface"),
-          f"{rel}: the egressGateway names no egress interface — without it the steered class "
-          "is not actually sent down a different path from the default")
-
-    # ------------------- (3) THE PATHS COEXIST: the choice is per DESTINATION -------------
-    dests = spec.get("destinationCIDRs") or []
-    excl = spec.get("excludedCIDRs") or []
-    check([str(c) for c in dests] == ["0.0.0.0/0"],
-          f"{rel}: destinationCIDRs is {dests!r}. The vpn path is for PUBLIC destinations "
-          "('Routed through the in-cluster VPN tunnel'), expressed as the whole internet "
-          "minus the exclusions below — narrowing it here would silently leave public "
-          "destinations on the WAN for a vpn-class service")
-    try:
-        excluded = [ipaddress.ip_network(str(c)) for c in excl]
-    except ValueError as e:
-        excluded = []
-        check(False, f"{rel}: excludedCIDRs contains an unparsable network ({e})")
-    check(len(excluded) >= 3,
-          f"{rel}: only {len(excluded)} excluded CIDR(s) — docs/03 requires intra-cluster, "
-          "pod->service and LAN traffic to stay on the LOCAL path, which is all of RFC1918")
-
-    def covered(net: ipaddress.IPv4Network | ipaddress.IPv6Network) -> bool:
-        return any(net.subnet_of(e) for e in excluded if e.version == net.version)
-
-    # RFC1918 in full — computed, not string-matched, so a typo'd mask is caught.
-    for rfc in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"):
-        net = ipaddress.ip_network(rfc)
-        check(covered(net),
-              f"{rel}: the private range {rfc} is NOT excluded from the tunnel — a vpn-class "
-              "pod's cluster/LAN traffic would be SNAT'd out tun0, so the two paths would not "
-              "coexist for the same pod and east-west traffic would break")
-
-    # Every address each ENVIRONMENT declares for itself must fall in the excluded set: that
-    # is what makes 'local stays local' true per environment rather than by coincidence.
-    def overlay_addresses() -> list[tuple[str, str]]:
-        found: list[tuple[str, str]] = []
-        base = CLUSTERS / "base" / "cluster-vars.yaml"
-        data = {}
-        if base.is_file():
-            for d in docs(base):
-                if d.get("kind") == "ConfigMap":
-                    data = {k: str(v) for k, v in (d.get("data") or {}).items()}
-        for f in sorted(CLUSTERS.glob("*/kustomization.yaml")):
-            env = {**data}
-            for m in re.finditer(
-                    r"-\s+op:\s*replace\s*\n\s*path:\s*/data/(?P<k>[A-Z0-9_]+)\s*\n"
-                    r"\s*value:\s*(?P<v>\S.*?)\s*$", f.read_text(), re.M):
-                env[m.group("k")] = m.group("v").strip().strip('"').strip("'")
-            label = f.parent.name
-            for key in ("CLUSTER_POD_CIDR", "GATEWAY_VIP", "CILIUM_LB_POOL_START",
-                        "CILIUM_LB_POOL_STOP", "K3S_API_HOST"):
-                v = env.get(key, "")
-                if v and not v.startswith("TBD"):
-                    found.append((f"{label}/{key}", v))
-        return found
-
-    addrs = overlay_addresses()
-    check(len(addrs) >= 5,
-          f"only {len(addrs)} environment address(es) could be read from the Flux overlays — "
-          "the 'local traffic stays local' assertion would be checking almost nothing")
-    for label, value in addrs:
-        try:
-            net = ipaddress.ip_network(value, strict=False)
-        except ValueError:
-            continue
-        check(covered(net),
-              f"{rel}: {label}={value}, an address THIS repository assigns to a cluster, is "
-              "NOT inside the tunnel's excludedCIDRs — a vpn-class pod in that environment "
-              "would try to reach its own cluster/LAN through the VPN")
-
-
-# ============ (4) THE VPN PATH IS A SECOND TABLE, NEVER THE DEFAULT (split tunnel) ========
-TPL = VPN_ROLE / "templates"
-up = (TPL / "vpn-egress-up.sh.j2")
-down = (TPL / "vpn-egress-down.sh.j2")
-conf = (TPL / "vpn-egress.conf.j2")
-nft = (TPL / "vpn-egress-killswitch.nft.j2")
-for f in (up, down, conf, nft):
-    check(f.is_file(), f"{f.relative_to(ROOT)} is missing — the second egress path's routing "
-                       "rules are not authored")
-if FAILURES:
-    die()
-
-defaults = yaml.safe_load((VPN_ROLE / "defaults" / "main.yml").read_text()) or {}
-
-
-def render(text: str) -> str:
-    """Substitute `{{ var }}` with the role's own default value, for every default.
-
-    The templates reference their knobs by NAME, so an analysis that looked for the VALUE
-    (e.g. the nft set name) would silently match nothing and pass vacuously — which is
-    exactly what an earlier draft of this check did. Rendering first means the assertions
-    below examine the text the node actually gets. Only simple `{{ name }}` expressions are
-    substituted; anything else is left as-is (and the assertions are written so an
-    unsubstituted expression fails rather than passes).
-    """
-    out = text
-    for k, v in defaults.items():
-        if isinstance(v, (str, int)):
-            out = re.sub(r"\{\{\s*" + re.escape(str(k)) + r"\s*\}\}", str(v), out)
-    return out
-
-
-up_t, conf_t, nft_t = render(up.read_text()), render(conf.read_text()), render(nft.read_text())
-table_id = str(defaults.get("vpn_egress_gateway_route_table_id", ""))
-check(table_id.isdigit() and table_id not in ("0", "253", "254", "255"),
-      f"the tunnel's routing table id is {table_id!r} — the second path must live in a "
-      "DEDICATED table. 254 (main), 255 (local) and 253 (default) are the kernel's own, and "
-      "putting the tunnel's default route in any of them makes the VPN the node's default "
-      "route instead of a selectable second path")
-check(str(defaults.get("vpn_egress_gateway_rule_priority", "")).isdigit(),
-      "no ip-rule priority is declared for the tunnel table — 'selected by routing rules' "
-      "needs a rule, at a defined precedence")
-
-# Every `ip route ... default` the up-script installs must name the dedicated table.
-route_lines = [l.strip() for l in up_t.splitlines()
-               if re.search(r"^\s*ip\s+route\s+(replace|add)\b", l)]
-check(len(route_lines) >= 2,
-      f"the up-script installs {len(route_lines)} route(s) — the vpn path needs its own "
-      "default route via the tunnel AND the fail-closed blackhole behind it")
-for l in route_lines:
-    check(re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
-          f"the up-script installs a route WITHOUT naming the dedicated table: {l!r}. A route "
-          "landing in the main table changes the path every other pod and the node itself "
-          "uses — the VPN would stop being opt-in")
-    check("blackhole" not in l or re.search(r'table\s+"?\$\{?TABLE_ID', l) is not None,
-          f"a blackhole route is installed outside the tunnel table: {l!r}. docs/03: the "
-          "kill-switch 'blackholes **only** that class'; `internet` fails OPEN and must never "
-          "be blackholed")
-
-# Every `ip rule` must be SCOPED. An unscoped rule (`ip rule add lookup <table>`) sends
-# everything into the tunnel table, which would leave one path instead of two.
-rule_lines = [l.strip() for l in up_t.splitlines() if re.search(r"^\s*ip\s+rule\s+add\b", l)]
-check(len(rule_lines) >= 1,
-      "the up-script adds no `ip rule` — nothing SELECTS the tunnel path, so the second "
-      "routing table would never be consulted (R-NET-04's 'selected by routing rules')")
-for l in rule_lines:
-    check(re.search(r"\bfrom\s+\S|\bfwmark\s+\S|\biif\s+\S", l) is not None,
-          f"an unscoped ip rule is installed: {l!r}. Without a `from`/`fwmark`/`iif` selector "
-          "EVERY packet is looked up in the tunnel table, so the WAN path disappears and the "
-          "VPN is no longer opt-in")
-    check(re.search(r'lookup\s+"?\$\{?TABLE_ID', l) is not None,
-          f"an ip rule looks up a table other than the tunnel's: {l!r}")
-check(re.search(r'ip rule add from\s+"\$\{SRC\}"', up_t) is not None,
-      "the steering rule is not scoped to ${SRC}, the tunnel's own local address — that "
-      "address is what Cilium SNATs vpn-class pods to, and it is the ONLY thing that "
-      "distinguishes the steered class from every other flow on the gateway node")
-check(re.search(r"^\s*if \[ -z \"\$\{SRC\}\" \]", up_t, re.M) is not None,
-      "the up-script does not refuse to run when the tunnel address is unknown — installing "
-      "a partial rule set is how a path ends up half-selected")
-
-# The provider must not be able to make the tunnel the default route either.
-check(re.search(r"^\s*route-noexec\s*$", conf_t, re.M) is not None,
-      "the OpenVPN wrapper config lacks `route-noexec` — OpenVPN would install the pushed "
-      "routes itself, in the MAIN table, and the VPN would become the node's default route "
-      "(docs/03: 'does **not** set the node's default route (`route-noexec` + policy "
-      "routing)')")
-check(re.search(r'pull-filter ignore "redirect-gateway"', conf_t) is not None,
-      "the OpenVPN wrapper config does not ignore a pushed `redirect-gateway` — the VPN "
-      "provider could redirect the node's whole default route into the tunnel, collapsing "
-      "the two paths into one")
-check(re.search(r"^\s*redirect-gateway", conf_t, re.M) is None,
-      "the OpenVPN wrapper config itself sets `redirect-gateway` — that IS making the VPN the "
-      "default route")
-
-
-# ================= (5) THE KILL-SWITCH BLACKHOLES ONLY THE VPN CLASS (internet fails OPEN)
-src_set = str(defaults.get("vpn_egress_gateway_nft_src_set", ""))
-fwmark = str(defaults.get("vpn_egress_gateway_fwmark", ""))
-check(bool(src_set),
-      "the role declares no nftables source set — the kill-switch would have no way to name "
-      "'the steered class' and could only match broadly")
-policies_lines = [l.strip() for l in nft_t.splitlines() if re.search(r"^\s*type filter hook", l)]
-check(len(policies_lines) >= 1,
-      "the kill-switch nft template declares no base chain — nothing would be enforced")
-for l in policies_lines:
-    check(re.search(r"policy\s+accept\s*;", l) is not None,
-          f"an nft base chain is not `policy accept`: {l!r}. A default-drop chain here "
-          "blackholes the `internet` path too, and docs/03 requires the two classes to fail "
-          "in OPPOSITE directions — 'internet fails **open** (WAN, never blackholed)'")
-drop_lines = [l.strip() for l in nft_t.splitlines()
-              if re.search(r"\bdrop\b", l) and not l.lstrip().startswith("#")]
-check(len(drop_lines) >= 1,
-      "the kill-switch nft template contains no drop rule at all — there is no kill-switch, "
-      "and R-NET-04's 'the kill-switch blackholes only that class' has nothing to scope")
-for l in drop_lines:
-    scoped = (src_set and src_set in l) or (fwmark and "mark" in l)
-    check(scoped,
-          f"an nft DROP rule is not scoped to the vpn class: {l!r}. It must match the tunnel "
-          f"source set (@{src_set}) or the class's fwmark; an unqualified drop would blackhole "
-          "the default `internet` path, which must never be blackholed")
-
-
-# --------------------------------------------------------------------------- verdict
-# NO-OP GUARD: a check that quietly asserts nothing is worse than no check.
-check(CHECKS >= 35,
-      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op and its pass "
-      "would be meaningless")
-
-if FAILURES:
-    die()
-print(f"check OK: egress-paths-selected-by-rules — {CHECKS} assertions; "
-      f"paths={sorted(PATHS)} (default={egress_prop.get('default')}), "
-      f"{len(policies)} steering rule selecting {CLASS_LABEL}=vpn only, tunnel default route "
-      f"in dedicated table {table_id} behind a scoped ip rule, blackhole and every nft DROP "
-      f"scoped to the vpn class. R-NET-04's RULE-SET half only: the paths' actual BEHAVIOUR "
-      "is L2 (egress-class for the two labelled classes, egress-default-path for the "
-      "unlabelled default) and the kill-switch firing is L3 (vpn-killswitch).")
-PY

conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml

@@ -8,18 +8,6 @@
 # the empty string while the Kustomization still reports Ready — which emptied Cilium's
 # k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01).
 #
-# It is also LOSSY IN THE OTHER DIRECTION, and that is the second incident on this record.
-# envsubst consumes dollars in PAIRS and emits one per pair, unconditionally and everywhere —
-# so no substituted path can carry a literal `$$` at all. On 2026-09-13 a ConfigMap under
-# apps-wave-0 shipped a dollar-quoted PL/pgSQL `DO $$ ... $$;` self-check; Postgres received
-# a half-quoted `DO $` and every Flux-reconciled run of the nextcloud-db grant reconciler died
-# with a syntax error, nine Error pods deep before anyone looked (commit 16b7fcd). It passed
-# review because it was hand-applied with `kubectl apply`, which runs no envsubst: the defect
-# existed ONLY on the rendered path, and it landed on the assertion whose whole job was to
-# make a silent regression visible. Nothing here failed — the manifest rendered
-# "successfully" into something broken, because the check treated any doubled dollar as a
-# deliberate escape and stopped looking.
-#
 # GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom,
 # WHEN  each `spec.path` is resolved to the exact file set it renders (following
 #       kustomization.yaml `resources:` transitively, not globbing the repo),
@@ -27,12 +15,7 @@
 #       or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default
 #       operator on an undefined name (the silent-empty class strict mode does NOT catch);
 #       no unescaped expansion is unparseable by envsubst (which fails the WHOLE
-#       Kustomization); NO RUN OF TWO OR MORE DOLLARS IS SILENTLY REWRITTEN — the only shape
-#       that passes unremarked is exactly `$${`, the documented total escape, and every other
-#       run (a bare `$$` that cannot survive verbatim, an ODD `$$${...}` that only LOOKS
-#       escaped and in fact expands, an over-counted `$$$${...}`) is named and justified in
-#       `dollar-collapse.allow`, which is itself held to being exact and non-stale;
-#       every path rendering a knob actually substitutes cluster-vars;
+#       Kustomization); every path rendering a knob actually substitutes cluster-vars;
 #       every overlay-patched knob has a base default; and the check is wired into the
 #       merge gate as a blocking, unconditionally-run stage.
 #
@@ -43,19 +26,6 @@
 # only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
 # Test wraps the SAME script.
 #
-# NOT proven, and deliberately NOT proxied: the wider lesson of 2026-09-13 is that the bug
-# survived review because verification used a path (`kubectl apply`) that runs no
-# substitution. That property is NOT statically decidable from this repo — which command
-# someone ran is a fact about a shell session, and git stores the manifest, never the
-# invocation. Grepping commit prose or runbooks for `kubectl apply` would be trivially
-# satisfiable by rewording and would fire on the break-glass runbooks whose whole content is
-# that command, so no such proxy is attempted. The dollar-run assertion is the honest answer
-# instead: it moves the proof off the verification PATH and onto the SOURCE, where it IS
-# decidable, so the mangling is caught by reading the manifest however a human chose to try
-# it. An end-to-end `kustomize build | flux envsubst --strict` per substituted path would be
-# the real thing, and it is how the arithmetic in check.sh's header was measured — but the L0
-# CI toolchain installs neither tool, so as a gate stage it would SKIP permanently.
-#
 # NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests
 # a path includes — a deliberate superset of the post-`kustomize build` stream. It does
 # not render, does not prove the feature gate is enabled on the live kustomize-controller,
@@ -73,36 +43,18 @@ metadata:
   annotations:
     conformance.home.k3s/requirements: "R-ENV-02, R-GITOPS-01"
     conformance.home.k3s/layer: "L0"
-    # WHOLE-REPO SCOPE — relocated verbatim from ci/l0.sh's hand-written "Stage 3c
-    # (UNCONDITIONAL)", which this annotation replaces:
-    #
-    #   R-ENV-02's promote-once mechanism is `postBuild.substituteFrom cluster-vars`, and it
-    #   FAILS OPEN: an undefined ${TOKEN} becomes the empty string and the Kustomization still
-    #   reports Ready. That emptied Cilium's k8sServiceHost and broke the CNI on 2/3 dev nodes
-    #   (DECISIONS-LOG 2026-09-01, "substitution race that fails OPEN").
-    #
-    #   WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh's coarse path map
-    #   keys only clusters/* to R-ENV-02 — an edit to a SUBSTITUTED manifest under apps/ or
-    #   platform/ attributes no requirement, so the requirement-scoped selection would filter
-    #   this check out of exactly the change that introduces the bug. The invariant spans every
-    #   path any Flux Kustomization substitutes (apps/, platform/, platform-config/), so it runs
-    #   on EVERY gate invocation.
-    conformance.home.k3s/scope: "whole-repo"
     conformance.home.k3s/safety-critical: "false"
     conformance.home.k3s/executable: "true"
     conformance.home.k3s/intent: >-
       Every variable a substituted Flux path renders is defined in cluster-vars or
       escaped, so post-build substitution cannot silently empty a value and enabling
-      StrictPostBuildSubstitutions cannot fail a legitimate manifest; and no dollar run
-      envsubst rewrites reaches the cluster unreviewed, so content that must arrive
-      verbatim cannot be silently mangled on the Flux-rendered path (16b7fcd).
+      StrictPostBuildSubstitutions cannot fail a legitimate manifest.
 spec:
   steps:
     - name: audit-every-substituted-flux-path
       description: >-
         Resolve each postBuild.substituteFrom Kustomization to the files it renders and
-        assert every ${...} is defined or escaped and no dollar run is silently collapsed,
-        then assert the gate wires it blocking.
+        assert every ${...} is defined or escaped, then assert the gate wires it blocking.
       try:
         - script:
             content: bash ./check.sh

conformance/tests/l0-static/flux-postbuild-substitution/check.sh

@@ -18,32 +18,6 @@
 # substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in
 # the repo asserted that. This check does.
 #
-# SECOND INCIDENT, 2026-09-13 — THE INVERSE HAZARD (commit 16b7fcd)
-# ------------------------------------------------------------------
-# The first incident was a variable the author wanted EXPANDED that rendered EMPTY. This one
-# is its mirror: content the author wanted to reach the workload UNTOUCHED that substitution
-# silently REWROTE. A CronJob reconciler in apps/wave-0/nextcloud-db.yaml asserted its own
-# convergence in a dollar-quoted PL/pgSQL block —
-#
-#     DO <dq> BEGIN ... END <dq>;     where <dq> is two literal dollars: standard Postgres
-#
-# — inside a ConfigMap under apps-wave-0, which declares postBuild.substituteFrom. envsubst
-# collapses a doubled dollar into ONE dollar, ALWAYS, so Postgres received a half-quoted
-# `DO <one dollar>` and died with `syntax error at or near "$"`. Three scheduled runs left
-# nine Error pods in nextcloud-db before anyone looked.
-#
-# It PASSED verification because it was hand-applied with `kubectl apply`, which runs NO
-# envsubst. The defect existed ONLY on the Flux-rendered path. And the damage landed
-# precisely on the self-check: the GRANT on line 1 still ran, so the privilege stayed
-# correct and the only thing broken was the assertion meant to make a silent regression
-# visible — the part whose whole job is to stop a blind spot being quiet.
-#
-# THE ASSERTION GAP THIS CLOSES: assertions (1)-(3) below ask whether every `${...}` is
-# defined, escaped, or parseable. They said NOTHING about a doubled dollar that is not an
-# escape, because the old classifier treated a `$` run of length >= 2 as "escaped, therefore
-# fine" and stopped looking. Nothing failed the gate; the manifest rendered "successfully"
-# into something broken. Assertion (7) is that hole, closed.
-#
 # ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain,
 # `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption):
 #   ${DEFINED}        -> substituted with the cluster-vars value.
@@ -59,66 +33,6 @@
 #                        shell/cloud-init consumer. Escaping is total: it neutralises
 #                        even the indexed form.
 #   $VAR, $(cmd)      -> left alone by envsubst; they need no escaping.
-#
-# DOLLAR-RUN ARITHMETIC — the exact rule, MEASURED, not inferred (see assertion (7))
-# ---------------------------------------------------------------------------------
-# envsubst here is `github.com/fluxcd/pkg/envsubst` v1.1.0 — Flux's own maintained FORK of
-# `github.com/drone/envsubst` ("This package is a fork of drone/envsubst", its README) — and it
-# is exposed by the `flux envsubst` CLI ("replicate the behavior of the Flux Kustomization
-# post-build substitutions", its own --help). Named from the binary, not from memory: the
-# pinned flux 2.4.0 in the dev image embeds the Go module path
-# `/home/runner/go/pkg/mod/github.com/fluxcd/pkg/[email protected]/parse`
-# (`grep -ao 'github.com/fluxcd/pkg/envsubst[^ ]*' "$(command -v flux)"`). Its scanner walks
-# the input left to right; on a `$` it looks at the NEXT byte:
-#   * another `$`   -> BOTH are consumed and ONE `$` is emitted. This is UNCONDITIONAL. What
-#                      follows the pair is irrelevant — `{`, a letter, a digit, a semicolon,
-#                      a newline, end of input. `$$` is not "the escape for `${`"; it is the
-#                      escape for `$` itself, everywhere.
-#   * `{`           -> the braced expression is parsed and EXPANDED.
-#   * anything else -> the `$` is emitted literally (hence `$VAR` and `$(cmd)` survive).
-# So a run of N consecutive dollars emits floor(N/2) literal dollars, and when N is ODD the
-# final dollar is still LIVE. Measured with flux 2.4.0 `flux envsubst`, DEFINED=VALUE — the
-# left column is authored bytes, the right is what reaches the cluster:
-#
-#   $                 ->  $                  ${DEFINED}       ->  VALUE
-#   $$                ->  $                  $${DEFINED}      ->  ${DEFINED}   <- the escape
-#   $$$               ->  $$                 $$${DEFINED}     ->  $VALUE   <- ODD: brace LIVE
-#   $$$$              ->  $$                 $$$${DEFINED}    ->  $${DEFINED}
-#   $$$$$             ->  $$$                $$$$${DEFINED}   ->  $$VALUE
-#   DO $$ .. $$;      ->  DO $ .. $;         $$${NOPE:-d}     ->  $d
-#   awk '{print $$2}' ->  awk '{print $2}'   $${ARR[0]}       ->  ${ARR[0]}    (safe)
-#   $$@ $$< $$^       ->  $@ $< $^           $$${ARR[0]}      ->  HARD PARSE ERROR, whole
-#                                                                 Kustomization dies
-#
-# TWO CONSEQUENCES THIS CHECK IS BUILT ON:
-#   (a) THERE IS NO WAY TO SHIP A LITERAL `$$` THROUGH A SUBSTITUTED PATH. To make two
-#       dollars arrive you must author FOUR. So any authored `$$` that is meant to reach the
-#       workload verbatim is a BUG, full stop — that is the 2026-09-13 incident, and Postgres
-#       dollar-quoting is its commonest shape (Makefile `$$@`, awk `$$2` and docker-compose
-#       `$${}` are the same hazard arriving from the other direction: there the collapse is
-#       what the author wanted).
-#   (b) `$$$` + brace is NOT "more escaped" than `$$` + brace — it is NOT escaped at all. An
-#       odd run leaves one live dollar, so the brace expands, or, for an inner envsubst
-#       cannot parse, fails the ENTIRE Kustomization (proven above with $$${ARR[0]}, which
-#       aborted the whole stream while $${ARR[0]} on the line before it was fine).
-# `--strict` reports NONE of this: every row of that table except the parse error was
-# accepted SILENTLY by `flux envsubst --strict`, exit 0, and its output was BYTE-IDENTICAL to
-# non-strict (`cmp`). Strict mode cannot see this class, so a source-level assertion is the
-# only available guard.
-#
-# TO REPRODUCE THE TABLE (30 seconds, no cluster) — it is a measurement, so re-measure it
-# after any FLUX_VERSION bump in infra/devcontainer/versions.env. Open the toolbox with
-# `./infra/devcontainer/dev.sh` and run these INSIDE it (do NOT pipe into `dev.sh run` from
-# the host: it starts the container with `-it`, so a piped stdin hangs):
-#   printf '[$$] [$$$] [$${D}] [$$${D}] [DO $$ x $$;]\n' | D=VALUE flux envsubst
-#   -> [$] [$$] [${D}] [$VALUE] [DO $ x $;]
-# And end-to-end, on the actual delivery path, with the pre-16b7fcd manifest checked out
-# (`git show 16b7fcd^:apps/wave-0/nextcloud-db.yaml > apps/wave-0/nextcloud-db.yaml`):
-#   kubectl kustomize apps/wave-0 | grep 'DO \$'            -> "DO $$"  (the kubectl path)
-#   kubectl kustomize apps/wave-0 | flux envsubst --strict | grep 'DO \$'
-#                                                           -> "DO $"   (the FLUX path), exit 0
-# (`kubectl kustomize` IS the kustomize build here; the image ships no standalone binary.)
-# That two-command contrast IS the 2026-09-13 incident, and is why review missed it.
 #   ${...}, ${a b}, ${ARR[0]}  (UNESCAPED) -> HARD PARSE ERROR ("unable to parse
 #                        variable name" / "missing closing brace") which fails the
 #                        WHOLE Kustomization — every resource in the path, not just
@@ -141,14 +55,7 @@
 #       (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
 #           the "add the knob with a base default BEFORE referencing it" rule that the
 #           incident's interim discipline depends on;
-#       (7) NO DOLLAR RUN IS SILENTLY REWRITTEN: the only run of length >= 2 that passes
-#           unremarked is exactly `$$` immediately followed by `{` — the documented, total,
-#           already-inventoried escape. EVERY other run of >= 2 dollars (a bare `$$`, an odd
-#           `$$$`, an over-counted `$$$$`) is rewritten by envsubst between git and the
-#           container, so it must be named and justified in `dollar-collapse.allow`, a file
-#           this check also holds to being exact and non-stale;
-#   AND (6) this check is LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`,
-#           so ci/l0.sh's `--include-scope whole-repo` runs it on every gate run, and
+#   AND (6) this check is LOAD-BEARING: ci/l0.sh runs it on every gate run and
 #           ci/gate.sh sets the merge block when the l0 stage fails.
 # Assertion (1)-(3) are applied to COMMENTS as well as content: `kustomize build`
 # preserves body comments into the stream envsubst reads (verified: a `$`-carrying
@@ -171,21 +78,6 @@
 #     kustomize-controller. Both are cluster facts (L2), untested here. Tagging
 #     R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders
 #     is fully determined, with no variable silently emptied on the way in.
-#   - It CANNOT prove the thing that actually let 16b7fcd's bug reach a cluster: that a
-#     change was VERIFIED through a substituting path. "It was hand-applied with kubectl" is
-#     a fact about a shell session, not about any tracked file — git records the manifest,
-#     never the command someone ran to try it, so the property is not statically decidable
-#     here and no honest formulation of it exists. Nor is a proxy worth having: grepping
-#     commit prose or runbooks for `kubectl apply` would be trivially satisfiable by
-#     rewording, and would fire on the many legitimate break-glass runbooks whose whole
-#     content is that command. So this check does not attempt it, and does not pretend to.
-#     Assertion (7) answers that incident a different way: it moves the proof off the
-#     verification PATH, which is unknowable, and onto the SOURCE, which is not — the
-#     mangling is now caught by reading the manifest, whichever way a human chose to try it.
-#     (The one decidable half-measure, `kustomize build <path> | flux envsubst --strict` per
-#     substituted path in CI, is already ruled out above: the L0 toolchain installs neither
-#     tool, so the stage would SKIP permanently. The dev image has both, and it is how the
-#     arithmetic table above was measured.)
 #   - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool,
 #     that an overlay's LAN block is self-consistent). Presence, not correctness.
 #   - A cluster-vars key nothing references is reported as an ADVISORY, never a failure:
@@ -215,7 +107,6 @@ definition of "what a Flux path actually renders" in the harness.
 """
 from __future__ import annotations
 
-import bisect
 import os
 import re
 import sys
@@ -339,18 +230,9 @@ LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$")
 
 
 def classify(dollars: int, inner: str) -> tuple[str, str] | None:
-    """Return (severity-tag, remediation) for a hazardous expansion, else None.
-
-    PARITY, not ">= 2" (the 2026-09-13 correction). envsubst consumes dollars in PAIRS and
-    emits one per pair, so a run of N dollars in front of a brace leaves the brace ESCAPED
-    only when N is EVEN. An ODD run emits floor(N/2) literal dollars and its last dollar is
-    still LIVE, so `$$${X}` expands exactly as `${X}` does — and `$$${ARR[0]}` fails the
-    whole Kustomization where `$${ARR[0]}` is harmless. The old `dollars >= 2 -> return None`
-    called every such run "escaped" and stopped looking. Assertion (7) flags the run itself;
-    this function additionally classifies the still-live expansion an odd run carries.
-    """
-    if dollars % 2 == 0:
-        return None                                   # fully escaped: emitted as a literal
+    """Return (severity-tag, remediation) for a hazardous expansion, else None."""
+    if dollars >= 2:
+        return None                                   # escaped: emitted as a literal
     if NAME_RE.match(inner):
         if inner in KEYS:
             return None                               # defined knob — the happy path
@@ -385,9 +267,6 @@ def rendered_files(path_spec: str) -> tuple[list[str], bool]:
 findings: list[str] = []
 referenced: dict[str, set[str]] = {}
 inventory: list[str] = []
-# rel path -> the Kustomization name(s) that render it, for assertion (7)'s second pass.
-# A file can be rendered by more than one Kustomization; all of them substitute.
-subst_files: dict[str, set[str]] = {}
 
 for k in kustomizations:
     files, exists = rendered_files(k["path"])
@@ -401,14 +280,13 @@ for k in kustomizations:
             text = f.read_text()
         except (OSError, UnicodeDecodeError):
             continue
-        subst_files.setdefault(rel, set()).add(k["name"])
         for lineno, line in enumerate(text.splitlines(), 1):
             for m in TOKEN_RE.finditer(line):
                 dollars, inner = len(m.group(1)), m.group(2)
                 verdict = classify(dollars, inner)
                 if verdict is None:
-                    if dollars % 2 == 0:              # PARITY, see classify(): an odd run is
-                        n_esc += 1                    # live, not escaped
+                    if dollars >= 2:
+                        n_esc += 1
                     else:
                         n_def += 1
                         referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner,
@@ -434,156 +312,6 @@ for k in kustomizations:
 check(not findings,
       f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths")
 
-# ----------------------------------- (7) no dollar run is silently rewritten on the way out
-# THE 2026-09-13 CLASS. envsubst consumes dollars in PAIRS and emits one per pair,
-# unconditionally and everywhere (header: DOLLAR-RUN ARITHMETIC, measured against flux 2.4.0).
-# So every run of >= 2 dollars in a substituted manifest is REWRITTEN between git and the
-# container, and from the text alone the two intents are indistinguishable:
-#
-#   (a) a deliberate escape the author WANTS collapsed  — `$${VAR}` -> literal `${VAR}`;
-#   (b) content that must arrive VERBATIM               — `DO $$ ... $$;` -> `DO $ ... $;`
-#                                                          i.e. the incident. A BUG.
-#
-# THE RULE CHOSEN, AND WHY IT IS NEITHER NOISY NOR VACUOUS.
-# Exactly ONE shape passes unremarked: a run of EXACTLY TWO dollars immediately followed by
-# `{`. That is not a guess about intent, it is the escape contract this repo already
-# documents in clusters/base/apps-vms.yaml, that assertions (1)-(3) already validate the
-# inside of, and that 18 reviewed occurrences already rely on (unifi cloud-init, the gitea
-# and nextcloud OIDC bootstrap scripts, two longhorn comments). Failing those would be a
-# permanently-firing alert on documented, correct intent, and this repo has already been
-# bitten by one of those.
-#
-# EVERY OTHER run of >= 2 dollars FAILS unless it is named in `dollar-collapse.allow`:
-#   * DOLLAR-COLLAPSE   — a run not followed by `{`. Cannot survive verbatim: `$$` -> `$`.
-#                         This is the incident's exact shape and a blanket bug by default.
-#   * DOLLAR-HALF-ESCAPED — an ODD run >= 3 before `{`. Looks escaped, is not: the last
-#                         dollar is live. Occasionally legitimate (`$$${PRICE}` to render a
-#                         literal `$` before a value), so it is allowlistable, not banned.
-#   * DOLLAR-OVER-ESCAPED — an EVEN run >= 4 before `{`. Emits N/2 dollars then a literal
-#                         brace; nearly always a miscount, sometimes deliberate.
-#
-# WHY AN ALLOWLIST AND NOT A BLANKET BAN: a blanket ban was the first choice and was
-# rejected on evidence, not taste — main carries two real non-braced `$$` occurrences today
-# (prose inside embedded shell comments), so a blanket ban would have shipped RED. Nor is
-# the allowlist a hole: an entry is a hand-written edit to a file inside the conformance
-# test, which is loud in review; it must quote a >= 24-character VERBATIM context, so it can
-# only ever cover the occurrence it was written for and never a future one that drifts into
-# the same file; it must carry a >= 40-character reason; and every entry must still MATCH
-# something, so the allowlist cannot rot into a blanket exemption after the code it excused
-# is deleted. A wildcard, a bare `$$` context and a stale entry each FAIL.
-ALLOW_FILE = (ROOT / "conformance" / "tests" / "l0-static"
-              / "flux-postbuild-substitution" / "dollar-collapse.allow")
-MIN_CONTEXT = 24
-MIN_REASON = 40
-DOLLAR_RUN_RE = re.compile(r"\$+")
-
-allow_entries: list[dict] = []
-check(ALLOW_FILE.is_file(),
-      f"{ALLOW_FILE.relative_to(ROOT)} is missing — assertion (7) needs its allowlist to "
-      f"exist even when empty, because an absent file and an empty file are the same green "
-      f"and the next author would not know the mechanism is there")
-if ALLOW_FILE.is_file():
-    for lineno, raw in enumerate(ALLOW_FILE.read_text().splitlines(), 1):
-        if not raw.strip() or raw.lstrip().startswith("#"):
-            continue
-        parts = [x.strip() for x in raw.split("|")]
-        if len(parts) != 3:
-            check(False, f"dollar-collapse.allow:{lineno}: expected exactly 3 '|'-separated "
-                         f"fields (path | verbatim-context | reason), got {len(parts)}")
-            continue
-        allow_entries.append({"line": lineno, "path": parts[0], "ctx": parts[1],
-                              "reason": parts[2], "used": 0})
-
-for e in allow_entries:
-    tag = f"dollar-collapse.allow:{e['line']}"
-    check((ROOT / e["path"]).is_file(),
-          f"{tag}: names '{e['path']}', which is not a file in this repo — a stale entry is "
-          f"a standing exemption for a path someone could recreate")
-    check(len(e["ctx"]) >= MIN_CONTEXT,
-          f"{tag}: context {e['ctx']!r} is {len(e['ctx'])} chars; at least {MIN_CONTEXT} are "
-          f"required so an entry cannot be a near-wildcard that swallows future occurrences")
-    check(DOLLAR_RUN_RE.search(e["ctx"]) is not None
-          and max((len(m.group(0)) for m in DOLLAR_RUN_RE.finditer(e["ctx"])), default=0) >= 2,
-          f"{tag}: context {e['ctx']!r} contains no run of two or more dollars, so it excuses "
-          f"nothing and is either a typo or a decoy")
-    check(len(e["reason"]) >= MIN_REASON,
-          f"{tag}: reason is {len(e['reason'])} chars; at least {MIN_REASON} are required. "
-          f"State what envsubst will actually emit and why that is acceptable — an entry "
-          f"nobody had to explain is an accident wearing an approval")
-
-RUN_TAGS = {
-    "collapse": ("DOLLAR-COLLAPSE",
-                 "envsubst collapses each PAIR of dollars into one, everywhere and "
-                 "unconditionally, so this CANNOT reach the workload verbatim (a literal $$ "
-                 "needs $$$$ in source). This is the 2026-09-13 nextcloud-db shape: a "
-                 "dollar-quoted PL/pgSQL DO block arrived half-quoted and every Flux-"
-                 "reconciled run died. Rewrite without dollars, double them again if the "
-                 "collapse is what you want, or justify it in dollar-collapse.allow"),
-    "half": ("DOLLAR-HALF-ESCAPED",
-             "an ODD run of dollars before a brace is NOT escaped: floor(N/2) literal "
-             "dollars are emitted and the LAST dollar is still live, so this expansion "
-             "EVALUATES at reconcile time (and if its inner is unparseable it fails the "
-             "entire Kustomization). Use an EVEN run to escape, or justify it in "
-             "dollar-collapse.allow if a literal dollar before a substituted value is "
-             "really what you meant"),
-    "over": ("DOLLAR-OVER-ESCAPED",
-             "an EVEN run of four or more dollars emits N/2 literal dollars followed by a "
-             "LITERAL brace — the value is never substituted and extra dollars appear in "
-             "the rendered manifest. Almost always a miscount of the escape; justify it in "
-             "dollar-collapse.allow if it is not"),
-}
-
-run_findings: list[str] = []
-n_runs_ok = 0
-for rel in sorted(subst_files):
-    text = (ROOT / rel).read_text(errors="replace")
-    # Absolute offsets, not per-line: the gitea/nextcloud bootstrap scripts are single
-    # 4000-character folded scalars, so "the line" is not a reviewable unit of context here.
-    nl = [i for i, ch in enumerate(text) if ch == "\n"]
-    spans = []                                       # allowlisted [start, end) regions
-    for e in allow_entries:
-        if e["path"] != rel:
-            continue
-        start = text.find(e["ctx"])
-        while start >= 0:
-            spans.append((start, start + len(e["ctx"]), e))
-            start = text.find(e["ctx"], start + 1)
-    for m in DOLLAR_RUN_RE.finditer(text):
-        n = len(m.group(0))
-        if n < 2:
-            continue
-        braced = text[m.end():m.end() + 1] == "{"
-        if n == 2 and braced:
-            n_runs_ok += 1                           # the documented escape; (1)-(3) own it
-            continue
-        covering = [e for (a, b, e) in spans if a <= m.start() and m.end() <= b]
-        if covering:
-            for e in covering:
-                e["used"] += 1
-            continue
-        kind = "collapse" if not braced else ("half" if n % 2 else "over")
-        tag, fix = RUN_TAGS[kind]
-        lineno = bisect.bisect_right(nl, m.start()) + 1
-        ctx = text[max(0, m.start() - 60):m.end() + 60].replace("\n", "\\n")
-        run_findings.append(
-            f"  [{tag}] {rel}:{lineno}  a run of {n} dollar(s)"
-            f"{' before a brace' if braced else ''}\n"
-            f"        rendered by Flux Kustomization(s) {', '.join(sorted(subst_files[rel]))}"
-            f" — postBuild substitution REWRITES it\n"
-            f"        {fix}\n"
-            f"        | ...{ctx}...")
-
-check(not run_findings,
-      f"{len(run_findings)} dollar run(s) in substituted paths that envsubst rewrites and "
-      f"nothing justifies (the 2026-09-13 nextcloud-db class)")
-
-for e in allow_entries:
-    check(e["used"] > 0,
-          f"dollar-collapse.allow:{e['line']}: STALE — its context is not present in "
-          f"{e['path']} any more (or no longer contains a rewritten dollar run), so the entry "
-          f"now excuses nothing. Delete it. A stale allowlist is how an opt-in decays into a "
-          f"blanket exemption for whatever lands in that file next")
-
 # --------------------------------------------- (5) overlay knobs exist in the base
 # Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` /
 # `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml
@@ -600,27 +328,10 @@ for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
 # ------------------------------------------------------- (6) the check is load-bearing
 L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
 GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
-# THIS CHECK IS LOAD-BEARING — asserted against the DECLARATIVE mechanism.
-# It used to assert `"flux-postbuild-substitution/check.sh" in L0`: ci/l0.sh carried a hand-written
-# "UNCONDITIONAL" stage per whole-repo check, hardcoding this path. Those stages are gone
-# (they did not scale, and the next whole-repo invariant only got one if someone REMEMBERED
-# to write it). The same property is now asserted two ways: this test DECLARES
-# `conformance.home.k3s/scope: "whole-repo"`, and ci/l0.sh unions every declaring test into
-# the run via `--include-scope`. Neither half alone is enough, so both are checked here; the
-# mechanism ITSELF (the union really selects, and a scope-declared SKIP exits 3 rather than
-# passing) is proven end-to-end and falsifiably by
-# conformance/tests/l0-static/whole-repo-invariants-wired.
-SELF_YAML = ROOT / "conformance" / "tests" / "l0-static" / "flux-postbuild-substitution" / "chainsaw-test.yaml"
-SELF = SELF_YAML.read_text() if SELF_YAML.is_file() else ""
-check(re.search(r'^\s*conformance\.home\.k3s/scope:\s*["\']?whole-repo', SELF, re.M) is not None,
-      "flux-postbuild-substitution/chainsaw-test.yaml does not declare "
-      'conformance.home.k3s/scope: "whole-repo", so the merge gate selects this check only '
-      "when a change happens to touch one of the requirements it is labelled with. "
-      "The path map keys only clusters/* to R-ENV-02, so an edit to a SUBSTITUTED manifest under apps/ or platform/ attributes nothing and this check would be filtered out of exactly the change that empties a variable (the 2026-09-01 Cilium k8sServiceHost CNI outage)")
-check("--include-scope" in L0,
-      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is "
-      "a purely requirement-scoped run again and the scope annotation is inert decoration — "
-      "this check would then run only by coincidence")
+check("flux-postbuild-substitution/check.sh" in L0,
+      "ci/l0.sh does not run flux-postbuild-substitution/check.sh — the requirement-scoped "
+      "conformance selection can filter this check out of a gate run, so l0.sh must invoke "
+      "it directly and unconditionally or the gate is not actually enforcing it")
 check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) and re.search(r"fail\).*BLOCK=1", GATE),
       "ci/gate.sh does not set the merge block when the l0 stage fails")
 
@@ -634,10 +345,6 @@ print("Flux postBuild substitution inventory (per Kustomization):")
 for line in inventory:
     print(line)
 print(f"\ncluster-vars knobs defined: {len(KEYS)}   referenced: {len(referenced)}")
-print(f"dollar runs >=2 in substituted files: {n_runs_ok} documented $${{...}} escapes, "
-      f"{sum(e['used'] for e in allow_entries)} justified by dollar-collapse.allow "
-      f"({len(allow_entries)} entr{'y' if len(allow_entries) == 1 else 'ies'}), "
-      f"{len(run_findings)} unjustified")
 
 if ADVISORIES:
     print("\nADVISORIES (not failures):")
@@ -649,12 +356,6 @@ if findings:
     for f in findings:
         print(f, file=sys.stderr)
 
-if run_findings:
-    print("\nHAZARDS — dollar runs envsubst rewrites between git and the container:",
-          file=sys.stderr)
-    for f in run_findings:
-        print(f, file=sys.stderr)
-
 print()
 if not FAILURES:
     print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions "

conformance/tests/l0-static/flux-postbuild-substitution/dollar-collapse.allow

@@ -1,52 +0,0 @@
-# =============================================================================
-# dollar-collapse.allow — the reviewed exceptions to assertion (7) of
-# conformance/tests/l0-static/flux-postbuild-substitution/check.sh.
-#
-# WHY THIS FILE EXISTS
-# --------------------
-# Flux's postBuild substitution runs github.com/fluxcd/pkg/envsubst (Flux's fork of
-# github.com/drone/envsubst; the module path is embedded in the pinned flux 2.4.0
-# binary, and check.sh's header says how to read it back out) over everything a
-# substituted Kustomization renders, and that scanner consumes dollars in PAIRS,
-# emitting ONE dollar per pair — unconditionally, wherever they appear, not only in
-# front of a brace. A run of N dollars becomes floor(N/2) dollars, and an ODD run
-# leaves one live dollar behind (so `$$${X}` is NOT escaped; it expands).
-#
-# Consequence: there is NO way to ship a literal `$$` through a substituted path.
-# On 2026-09-13 that broke the nextcloud-db grant reconciler — a dollar-quoted
-# PL/pgSQL `DO $$ ... $$;` block reached Postgres as `DO $ ... $;` and every
-# Flux-reconciled run died with `syntax error at or near "$"`. It had passed review
-# because it was hand-applied with `kubectl apply`, which runs no envsubst at all.
-# See commit 16b7fcd and check.sh's DOLLAR-RUN ARITHMETIC table (measured against
-# flux 2.4.0, not inferred).
-#
-# From the source text alone, a doubled dollar the author WANTS collapsed and one
-# that must arrive verbatim are indistinguishable. check.sh therefore accepts exactly
-# one shape without comment — a run of EXACTLY TWO dollars immediately followed by
-# `{`, the escape contract clusters/base/apps-vms.yaml already documents and that 18
-# reviewed occurrences already depend on — and FAILS every other run of two or more
-# dollars unless it is named here.
-#
-# ADDING AN ENTRY IS A DECISION, NOT A MUTE BUTTON
-# -----------------------------------------------
-# One entry per line, exactly three `|`-separated fields:
-#
-#     <repo-relative path> | <verbatim context> | <reason>
-#
-#   * the context must be a VERBATIM substring of the file AS STORED IN GIT (for a
-#     folded/JSON-ish scalar that means the escaped bytes, e.g. `\n` and `—`
-#     spelled literally), at least 24 characters long, containing the dollar run it
-#     excuses, and containing no `|` — shift the window if a pipe is in the way;
-#   * the reason must be at least 40 characters and should say what envsubst will
-#     actually EMIT and why that is acceptable;
-#   * an entry covers only the dollar runs inside the region its context matches, so
-#     it can never silently cover a new occurrence that lands elsewhere in the file;
-#   * check.sh FAILS a stale entry — one whose context no longer matches, or which
-#     excuses nothing any more. Delete it rather than leaving it to rot into a
-#     blanket exemption for whatever lands in that file next.
-#
-# If the collapse would change BEHAVIOUR, this file is the wrong answer. Rewrite the
-# content without dollars (16b7fcd) or double them again (`$$$$` -> `$$`).
-# =============================================================================
-apps/wave-0/gitea.yaml | # NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild. | Prose, not code: this is a comment INSIDE the gitea OIDC bootstrap shell script, and the two dollars are the sentence quoting the escape idiom it goes on to explain. envsubst renders it as "the `$` escape below", which is cosmetically wrong and behaviourally inert — a shell comment. Cannot be written correctly at all: `$$$$` would render `$$` but would then be wrong in the source a reader sees in git, and the file has no non-dollar way to name the idiom. Reviewed and accepted as cosmetic.
-apps/wave-1/nextcloud.yaml | # NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild. | Identical case to the gitea entry above, in the nextcloud OIDC bootstrap script: a shell comment quoting the escape idiom, rendered by envsubst as "the `$` escape below". Cosmetic only — the collapse happens inside a `#` comment line that no interpreter reads, and the escaped expansion the comment is describing is on a later line and is unaffected. Reviewed and accepted as cosmetic.

conformance/tests/l0-static/ha-control-plane-as-code/chainsaw-test.yaml

@@ -1,98 +0,0 @@
-# =============================================================================
-# L0 STATIC — the HA control plane is authored, for every environment  (R-PLAT-01)  [EXECUTABLE]
-# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
-# Spec:    docs/00 R-PLAT-01 ("The platform MUST run on k3s across 3 nodes with
-#          production-grade HA control plane in prod; dev/test MAY run fewer nodes but MUST
-#          exercise the same multi-node topology"), elaborated by docs/01 ("k3s (HA,
-#          embedded etcd)") and docs/02 ("HA embedded etcd (k3s server on all 3)",
-#          "3 physical nodes", "no external datastore", "the *same* Ansible bootstrap runs").
-#
-# WHY L0: R-PLAT-01 has an authored half and an observed half. The authored half — that the
-# configuration in git BUILDS a 3-server embedded-etcd control plane in BOTH environments
-# from one code path, with exactly one --cluster-init founder, no external datastore, and a
-# floating API endpoint that is SAN'd into the serving cert and cannot collide with the
-# Service-LB allocator — is provable with no cluster and is ONLY provable here: a live
-# cluster cannot tell you what the NEXT environment brought up from this repo would look
-# like. The observed half is asserted separately at L2 by
-# tests/l2-behavioral/ha-control-plane, which carries the same requirement label.
-#
-# ENVIRONMENT-AGNOSTIC: no hostname, address, node count or group name is written down.
-# Environments are enumerated from the R-ENV-02 seam itself (k3s_primary_group /
-# k3s_servers_group in group_vars), and WHICH one is prod is derived from the Flux overlay
-# that environment bootstraps (flux_bootstrap_env_path) — never from a group's name.
-#
-# GIVEN the repository as git tracks it,
-# WHEN  the inventory, the k3s_server and kube_vip roles, their plays and each
-#       environment's Flux overlay are parsed and cross-checked against each other,
-# THEN  k3s is the pinned distro and nothing is installed as an agent; every environment
-#       resolves to an ODD, quorum-capable server count (prod exactly three) with EXACTLY
-#       ONE --cluster-init founder and every other server joining it with --server;
-#       the datastore is k3s's embedded etcd (no --datastore-endpoint, no --disable-etcd);
-#       the API's floating endpoint is a leader-elected, control-plane-ONLY kube-vip
-#       DaemonSet on every server whose address is the environment-scoped VIP, that VIP is
-#       in the apiserver serving cert's SANs together with every server's own address, and
-#       it is neither a node address nor the ingress Gateway VIP nor inside that
-#       environment's Cilium LoadBalancer pool; and dev/test and prod are built by the SAME
-#       roles and plays retargeted by variables only.
-#
-# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
-# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
-# Chainsaw Test wraps the SAME script.
-#
-# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
-#   * The LIVE clusters' shape (three Ready servers, all etcd members, VIP actually floating,
-#     API answering through it) — asserted at L2 by tests/l2-behavioral/ha-control-plane.
-#   * FAILOVER. That the control plane survives losing a server (quorum holds, the VIP moves)
-#     is a fault-injection claim and is proven NOWHERE yet — tests/l3-fault-injection/node-loss
-#     drains a node for R-STATE-02's data claim and asserts nothing about the control plane.
-#   * "Production-grade" as a hardware/performance judgement (operator decision, §2 of
-#     infra/PROD-PROVISIONING.md).
-#
-# TWO GAPS CLOSED 2026-09-06 (they used to be listed above; recorded so the change is legible):
-#   * `K3S_API_HOST` (Cilium's k8sServiceHost) was the PRIMARY NODE's address in both
-#     environments rather than the floating VIP, contradicting the prod overlay's own comment.
-#     It was ESCALATED, not asserted, because the workstream could not decide it (R-TEST-06).
-#     The operator decided the VIP, so it is now ASSERTED — on both sides of the [parity]
-#     contract and per environment.
-#   * kube-vip was NOT imported by playbooks/site.yml. It is now, BETWEEN k3s and cilium, and
-#     the order is asserted — not as a tidiness preference but because the CNI's endpoint is
-#     the VIP, so an unfloated VIP means a cold build brings up a CNI that cannot reach its
-#     API server.
-# =============================================================================
-apiVersion: chainsaw.kyverno.io/v1alpha1
-kind: Test
-metadata:
-  name: ha-control-plane-as-code
-  labels:
-    conformance.home.k3s/R-PLAT-01: "true"
-  annotations:
-    conformance.home.k3s/requirements: "R-PLAT-01"
-    conformance.home.k3s/layer: "L0"
-    # WHOLE-REPO SCOPE. The invariant spans THREE independent subtrees — infra/ (the Ansible
-    # roles + OpenTofu that create the control-plane nodes and the VIP), clusters/ (the
-    # CONTROL_PLANE_VIP knob and its per-overlay values) and platform/cilium/ (the HelmRelease
-    # whose k8sServiceHost IS that VIP). ci/touched-requirements.sh keys infra/* to NOTHING,
-    # clusters/* to R-GITOPS-01/R-ENV-02 and platform/cilium/* to the R-NET-* family: not one
-    # of them attributes R-PLAT-01, so no edit that can break HA control-plane-as-code selects
-    # this test. Note this is the same VIP whose empty substitution took the CNI down on 2/3
-    # dev nodes (2026-09-01) — the two halves of that incident are guarded here and in
-    # flux-postbuild-substitution.
-    conformance.home.k3s/scope: "whole-repo"
-    conformance.home.k3s/safety-critical: "false"
-    conformance.home.k3s/executable: "true"
-    conformance.home.k3s/intent: >-
-      Given the tracked tree, when the inventory, the k3s/kube-vip roles and plays and each
-      environment's Flux overlay are parsed and cross-checked, then every environment is
-      authored as an odd, quorum-capable set of k3s servers (prod exactly three) with one
-      embedded-etcd founder, no external datastore, and a leader-elected control-plane-only
-      VIP that is in the serving cert's SANs and collides with no other allocator — built by
-      one code path for both environments.
-spec:
-  steps:
-    - name: then-every-environment-is-authored-as-an-ha-embedded-etcd-control-plane
-      description: >-
-        Run the offline assertion script over the tracked tree (same script ci/l0.sh and
-        conformance/run.sh --layer l0 execute).
-      try:
-        - script:
-            content: bash ./check.sh