docs(gitea): a skipped job reports SUCCESS, not pending -- correct the deadlock claim

The runner came up on 2026-09-29 and the first CI jobs this repo has ever run went
green, which falsified a claim merged in #2.

MEASURED:
  gate / gate (pull_request)          success   1m58s (PR 1), 1m34s (PR 2)
  gate / gate (push)                  success   1m32s (PR 2 only)
  gate / conformance (pull_request)   success   "Has been skipped"

#2 said listing the conformance context as a required check would deadlock every PR,
because the job is `if: false` and never runs. That is wrong. Gitea reports a skipped
job as SUCCESS. Requiring it would not deadlock anything -- it would pass VACUOUSLY,
forever, while nothing ran.

The conclusion is unchanged (never require it) but the reason inverts, and the corrected
reason matters more: under HARD RULE 4 a permanently-green check that verifies nothing is
worse than a deadlock. A deadlock is loud. This is the exact fail-open shape the
2026-09-18 audit and this one both keep finding.

Also records that the (push) contexts only appear for branches matching gate.yml's push
trigger -- feature/**, fix/**, docs/**, chore/** -- and NOT feat/**, which is why PR 1's
feat/ branch produced only pull_request contexts. Require the pull_request context alone,
or PRs from non-matching prefixes never satisfy the rule.

CLAUDE.md's enforcement table is updated: the runner half is genuinely fixed and observed
green, the branch-protection half is not. Adds the warning that a green tick is not by
itself evidence that anything ran.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/gate.yml

@@ -1,25 +1,44 @@
 # .gitea/workflows/gate.yml — the ENFORCED PR merge gate (Phase 6, WS-GITOSS).
 #
 # This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
-# Docker executor + DinD). It invokes the AGGREGATE merge gate — `make -C ci gate` — so CI
+# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
-# runs BYTE-FOR-BYTE the same command a human runs in the devcontainer (R-TEST-02): the
+# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
-# runner just installs the pinned tools from ci/versions.env first. `make -C ci gate`
+# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
-# (ci/gate.sh) runs every stage in order — spec-immutability, lint, Kyverno L0,
+# below for why the script, not `make`, is invoked here). It runs every stage in order —
-# render==golden, L1 offline render — and applies the R-TEST-03 decision, printing
+# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
-# "MERGE GATE: PASS" / "MERGE GATE: BLOCKED". It is the single required status check on
+# R-TEST-03 decision, printing
-# `main` (see branch protection). A non-zero exit fails the job, so a BLOCKED gate blocks
+# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
-# the merge.
+# single required status check on `main` (see branch protection). A non-zero exit fails the job,
+# so a BLOCKED gate blocks the merge.
+#
+# WHY THIS STEP DECLARES GATE_STRICT_SKIPS AND TOLERATES EXIT 3 (2026-09-13). A SKIPPED stage is
+# never green, and since 2026-09-13 gate.sh refuses to print PASS when one did not run (it used to
+# print "— not green" and then "MERGE GATE: PASS", exit 0 — fail-open in the gate itself). Two
+# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
+# environment it is:
+#   * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
+#     installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
+#     "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
+#   * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
+#     not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
+#     is the ONLY skip that can still reach exit 3 under strict skips, which is what makes
+#     `|| [ "$?" -eq 3 ]` a narrow, reviewed tolerance rather than a blanket one. The log still
+#     says UNDETERMINED and names R-TEST-06 as unverified. Wire the spec checkout (or set
+#     REQUIRE_SPEC_VERIFICATION=true) and this tolerance should be deleted.
+# Called DIRECTLY, not through `make`: GNU make reports any recipe failure as its own exit 2, so
+# `make -C ci gate || [ "$?" -eq 3 ]` would see 2 and fail the job. Same reason the
+# spec-immutability pre-step below calls the script directly.
 #
 # The cluster-gated L2/L3 conformance stage SKIPS honestly here (exit 3, surfaced not faked
-# — HARD RULE 4) because this runner declares no cluster: `make -C ci gate` does not set
+# — HARD RULE 4) because this runner declares no cluster: nothing here sets CLUSTER_AVAILABLE,
-# CLUSTER_AVAILABLE, so gate.sh reports the touched requirements UNVERIFIED without blocking
+# so gate.sh reports the touched requirements UNVERIFIED — and NAMES that declared gap in its
-# Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
+# PASS headline — without blocking Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
 # no broad kubeconfig) driven by the separate `conformance` job below.
 #
 # NOTE (R-TEST-06 / gate-semantics): spec-immutability is ALSO surfaced as its own explicit
 # early step (below) so its result is a distinct, tolerant status in the Gitea UI; gate.sh
-# re-runs it as its first stage, which is the authoritative one. `make -C ci gate` here is
+# re-runs it as its first stage, which is the authoritative one. The gate step below is the
-# the byte-identical laptop gate; the extra pre-step is a read-only git-diff check and is
+# same script the laptop runs; the extra pre-step is a read-only git-diff check and is
 # purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
 # is retained dormant in merge-gate.yaml.
 ---
@@ -69,13 +88,17 @@ jobs:
       - name: Spec immutability (R-TEST-06, skips if spec repo absent)
         run: ./ci/spec-immutability.sh || [ "$?" -eq 3 ]
 
-      # Aggregate merge gate: byte-for-byte the command a human runs in the devcontainer
+      # Aggregate merge gate: the same script a human runs in the devcontainer via
-      # (`make -C ci gate` -> ci/gate.sh). Runs spec-immutability, lint, Kyverno L0,
+      # `make -C ci gate`. Runs spec-immutability, lint, Kyverno L0, render==golden and L1
-      # render==golden and L1 render, then the R-TEST-03 decision. Exits non-zero on a
+      # render, then the R-TEST-03 decision. Exits non-zero on a BLOCKED gate, which fails this
-      # BLOCKED gate, which fails this job. The cluster-gated L2/L3 stage SKIPS here (no
+      # job. The cluster-gated L2/L3 stage SKIPS here (no CLUSTER_AVAILABLE) — surfaced as
-      # CLUSTER_AVAILABLE) — surfaced as UNVERIFIED, never faked green (see header).
+      # UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
-      - name: Merge gate — make -C ci gate
+      # tolerated exit 3 are explained in the file header; the tolerance covers exactly one
-        run: make -C ci gate
+      # thing, the unverifiable spec repo, and nothing else can produce a 3 here.
+      - name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
+        env:
+          GATE_STRICT_SKIPS: "true"
+        run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
       # ("not supported on GHES" — Gitea does not implement the artifact-v2 backend), which

.gitea/workflows/scripts/install-tools.sh

@@ -39,6 +39,21 @@ install_chainsaw() {
   curl -fsSL "$url" | tar -xz -C "$bindir" chainsaw
 }
 
+# promtool — the alert-rule UNIT-TEST engine the L0 check
+# conformance/tests/l0-static/alert-rules-unit-tested runs. Installed for the l0 stage because
+# without it that check exits 3 (SKIPPED, never green) on the runner and the alert unit tests
+# would be enforced nowhere — the exact "coverage that looks present" failure it exists to end.
+# Pinned from ci/versions.env to the Prometheus version prod runs (promtool IS the rule engine);
+# the check asserts `promtool --version` equals that pin, so a drifted install FAILS, not skips.
+install_promtool() {
+  have promtool && return 0
+  log "promtool ${PROMTOOL_VERSION} (from the prometheus release tarball)"
+  local tarball="prometheus-${PROMTOOL_VERSION}.${os}-${arch}"
+  curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/${tarball}.tar.gz" \
+    | tar -xz -C "$bindir" --strip-components=1 "${tarball}/promtool"
+  chmod +x "${bindir}/promtool"
+}
+
 install_kubectl() {
   have kubectl && return 0
   log "kubectl ${KUBECTL_VERSION}"
@@ -50,10 +65,10 @@ have() { command -v "$1" >/dev/null 2>&1; }
 
 case "$stage" in
   lint)        install_yamllint ;;
-  l0)          install_yamllint; install_check_jsonschema; install_kyverno ;;  # schema (check-jsonschema) + Kyverno L0
+  l0)          install_yamllint; install_check_jsonschema; install_kyverno; install_promtool ;;  # schema (check-jsonschema) + Kyverno L0 + alert unit tests
   l1)          install_chainsaw ;;              # L1 render via conformance runner (Chainsaw)
   conformance) install_chainsaw; install_kubectl ;;   # trigger the in-cluster Job
-  all)         install_yamllint; install_check_jsonschema; install_kyverno; install_chainsaw; install_kubectl ;;
+  all)         install_yamllint; install_check_jsonschema; install_kyverno; install_promtool; install_chainsaw; install_kubectl ;;
   *) echo "unknown stage: $stage" >&2; exit 2 ;;
 esac
 log "stage '${stage}' tools ready"

.standup-dns/dns-20260914-pre-externaldns-scope.txt

@@ -0,0 +1,97 @@
+# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev`
+# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied.
+#
+# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers,
+# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not
+# merely left alone. That is a one-way, outward-facing operation against a live authoritative
+# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the
+# reconcile removes can be restored by hand from this file. It is the same discipline as
+# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable
+# change rather than to a migration.
+#
+# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones/<id>/dns_records, all pages) using the
+# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the
+# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the
+# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the
+# devcontainer, which has previously produced false "no records" readings — the API is the only
+# reading that is both complete and authoritative.
+#
+# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the
+# TTL and the proxy flag, not just the value.
+#
+#   <type>|<name>|<content>|<ttl>|<proxied>
+#
+# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here
+# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address.
+#
+# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry
+# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which
+# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy
+# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS
+# cannot touch them regardless of what this branch does:
+#
+#   *.keithjackson.dev            -> 136.51.115.252   legacy stack (the zone wildcard)
+#   brickbreak.keithjackson.dev   -> 20.236.242.14    unrelated, off-prem
+#   dev.brickbreak…               -> 20.236.242.14    unrelated, off-prem
+#
+# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in
+# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of
+# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does
+# NOT cover `<svc>.home.keithjackson.dev`: when a specific `<svc>.home` record is removed, that
+# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended
+# end state for `internal` services and it is also precisely why Pi-hole must already be
+# answering the LAN authoritatively before this branch merges.
+#
+# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog
+# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its
+# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed
+# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an
+# HTTPRoute that already had a published record produced
+# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile
+# interval (~60s) later, and nothing else in the zone moved.
+#
+# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its
+# own): recreate the A record with its value/ttl/proxied from this file, then recreate the
+# matching `a-<name>` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod,
+# external-dns/resource=…` string is what re-establishes ownership. An A record restored without
+# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value
+# points the registry at a source object that does not exist.
+A|*.keithjackson.dev|136.51.115.252|1|false
+A|auth.home.keithjackson.dev|192.168.2.20|1|false
+A|brickbreak.keithjackson.dev|20.236.242.14|1|false
+A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false
+A|crafty.home.keithjackson.dev|192.168.2.20|1|false
+A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false
+A|gitea.home.keithjackson.dev|192.168.2.20|1|false
+A|grafana.home.keithjackson.dev|192.168.2.20|1|false
+A|headlamp.home.keithjackson.dev|192.168.2.20|1|false
+A|home.keithjackson.dev|192.168.2.20|1|false
+A|jackett.home.keithjackson.dev|192.168.2.20|1|false
+A|keycloak.home.keithjackson.dev|192.168.2.20|1|false
+A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false
+A|pihole.home.keithjackson.dev|192.168.2.20|1|false
+A|plex.home.keithjackson.dev|192.168.2.20|1|false
+A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false
+A|radarr.home.keithjackson.dev|192.168.2.20|1|false
+A|seerr.home.keithjackson.dev|192.168.2.20|1|false
+A|semaphore.home.keithjackson.dev|192.168.2.20|1|false
+A|sonarr.home.keithjackson.dev|192.168.2.20|1|false
+A|unifi.home.keithjackson.dev|192.168.2.21|1|false
+TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false
+TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false
+TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false
+TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false
+TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false
+TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false
+TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false
+TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false
+TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false
+TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false
+TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false
+TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false
+TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false
+TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false
+TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false
+TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false
+TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false
+TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false

.standup-dns/dns-before.txt

@@ -0,0 +1,57 @@
+A|*.keithjackson.dev|136.51.115.252
+A|auth.home.keithjackson.dev|192.168.1.53
+A|brickbreak.keithjackson.dev|20.236.242.14
+A|crafty-mc.home.keithjackson.dev|192.168.1.55
+A|crafty.home.keithjackson.dev|192.168.1.53
+A|dev.brickbreak.keithjackson.dev|20.236.242.14
+A|gitea.home.keithjackson.dev|192.168.1.53
+A|grafana.home.keithjackson.dev|192.168.1.53
+A|headlamp.home.keithjackson.dev|192.168.1.53
+A|home.keithjackson.dev|192.168.1.53
+A|jackett.home.keithjackson.dev|192.168.1.53
+A|keycloak.home.keithjackson.dev|192.168.1.53
+A|nextcloud.home.keithjackson.dev|192.168.1.53
+A|pihole.home.keithjackson.dev|192.168.1.53
+A|plex.home.keithjackson.dev|192.168.1.53
+A|qbittorrent.home.keithjackson.dev|192.168.1.53
+A|radarr.home.keithjackson.dev|192.168.1.53
+A|seerr.home.keithjackson.dev|192.168.1.53
+A|semaphore.home.keithjackson.dev|192.168.1.53
+A|sonarr.home.keithjackson.dev|192.168.1.53
+A|unifi.home.keithjackson.dev|192.168.1.54
+TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
+TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
+TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
+TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
+TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
+TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
+TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
+TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
+TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
+TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
+TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
+TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
+TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
+TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
+TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
+TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
+TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
+TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
+TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
+TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
+TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
+TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
+TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
+TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp

.standup-dns/dns-post-l2.txt

@@ -0,0 +1,95 @@
+A|*.keithjackson.dev|136.51.115.252
+A|auth.home.keithjackson.dev|192.168.1.53
+A|auth.prod.home.keithjackson.dev|192.168.2.20
+A|brickbreak.keithjackson.dev|20.236.242.14
+A|conformance-dns.home.keithjackson.dev|192.168.1.56
+A|crafty-mc.home.keithjackson.dev|192.168.1.55
+A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
+A|crafty.home.keithjackson.dev|192.168.1.53
+A|crafty.prod.home.keithjackson.dev|192.168.2.20
+A|dev.brickbreak.keithjackson.dev|20.236.242.14
+A|gitea.home.keithjackson.dev|192.168.1.53
+A|gitea.prod.home.keithjackson.dev|192.168.2.20
+A|grafana.home.keithjackson.dev|192.168.1.53
+A|grafana.prod.home.keithjackson.dev|192.168.2.20
+A|headlamp.home.keithjackson.dev|192.168.1.53
+A|headlamp.prod.home.keithjackson.dev|192.168.2.20
+A|home.keithjackson.dev|192.168.1.53
+A|jackett.home.keithjackson.dev|192.168.1.53
+A|jackett.prod.home.keithjackson.dev|192.168.2.20
+A|keycloak.home.keithjackson.dev|192.168.1.53
+A|keycloak.prod.home.keithjackson.dev|192.168.2.20
+A|nextcloud.home.keithjackson.dev|192.168.1.53
+A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
+A|pihole.home.keithjackson.dev|192.168.1.53
+A|pihole.prod.home.keithjackson.dev|192.168.2.20
+A|plex.home.keithjackson.dev|192.168.1.53
+A|plex.prod.home.keithjackson.dev|192.168.2.20
+A|prod.home.keithjackson.dev|192.168.2.20
+A|qbittorrent.home.keithjackson.dev|192.168.1.53
+A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
+A|radarr.home.keithjackson.dev|192.168.1.53
+A|radarr.prod.home.keithjackson.dev|192.168.2.20
+A|seerr.home.keithjackson.dev|192.168.1.53
+A|seerr.prod.home.keithjackson.dev|192.168.2.20
+A|semaphore.home.keithjackson.dev|192.168.1.53
+A|semaphore.prod.home.keithjackson.dev|192.168.2.20
+A|sonarr.home.keithjackson.dev|192.168.1.53
+A|sonarr.prod.home.keithjackson.dev|192.168.2.20
+A|unifi.home.keithjackson.dev|192.168.1.54
+A|unifi.prod.home.keithjackson.dev|192.168.2.21
+TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|a-conformance-dns.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/chainsaw-growing-grouse/dns-probe-svc
+TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
+TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
+TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
+TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
+TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
+TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
+TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
+TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
+TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
+TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
+TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
+TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
+TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
+TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
+TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
+TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
+TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
+TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
+TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
+TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
+TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
+TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
+TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
+TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
+TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
+TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
+TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
+TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
+TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
+TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
+TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
+TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
+TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
+TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
+TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
+TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp

.standup-dns/dns-verify.txt

@@ -0,0 +1,93 @@
+A|*.keithjackson.dev|136.51.115.252
+A|auth.home.keithjackson.dev|192.168.1.53
+A|auth.prod.home.keithjackson.dev|192.168.2.20
+A|brickbreak.keithjackson.dev|20.236.242.14
+A|crafty-mc.home.keithjackson.dev|192.168.1.55
+A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
+A|crafty.home.keithjackson.dev|192.168.1.53
+A|crafty.prod.home.keithjackson.dev|192.168.2.20
+A|dev.brickbreak.keithjackson.dev|20.236.242.14
+A|gitea.home.keithjackson.dev|192.168.1.53
+A|gitea.prod.home.keithjackson.dev|192.168.2.20
+A|grafana.home.keithjackson.dev|192.168.1.53
+A|grafana.prod.home.keithjackson.dev|192.168.2.20
+A|headlamp.home.keithjackson.dev|192.168.1.53
+A|headlamp.prod.home.keithjackson.dev|192.168.2.20
+A|home.keithjackson.dev|192.168.1.53
+A|jackett.home.keithjackson.dev|192.168.1.53
+A|jackett.prod.home.keithjackson.dev|192.168.2.20
+A|keycloak.home.keithjackson.dev|192.168.1.53
+A|keycloak.prod.home.keithjackson.dev|192.168.2.20
+A|nextcloud.home.keithjackson.dev|192.168.1.53
+A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
+A|pihole.home.keithjackson.dev|192.168.1.53
+A|pihole.prod.home.keithjackson.dev|192.168.2.20
+A|plex.home.keithjackson.dev|192.168.1.53
+A|plex.prod.home.keithjackson.dev|192.168.2.20
+A|prod.home.keithjackson.dev|192.168.2.20
+A|qbittorrent.home.keithjackson.dev|192.168.1.53
+A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
+A|radarr.home.keithjackson.dev|192.168.1.53
+A|radarr.prod.home.keithjackson.dev|192.168.2.20
+A|seerr.home.keithjackson.dev|192.168.1.53
+A|seerr.prod.home.keithjackson.dev|192.168.2.20
+A|semaphore.home.keithjackson.dev|192.168.1.53
+A|semaphore.prod.home.keithjackson.dev|192.168.2.20
+A|sonarr.home.keithjackson.dev|192.168.1.53
+A|sonarr.prod.home.keithjackson.dev|192.168.2.20
+A|unifi.home.keithjackson.dev|192.168.1.54
+A|unifi.prod.home.keithjackson.dev|192.168.2.21
+TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
+TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
+TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
+TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
+TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
+TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
+TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
+TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
+TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
+TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
+TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
+TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
+TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
+TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
+TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
+TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
+TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
+TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
+TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
+TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
+TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
+TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
+TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
+TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
+TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
+TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
+TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
+TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
+TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
+TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
+TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
+TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
+TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
+TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
+TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
+TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
+TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
+TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
+TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
+TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
+TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
+TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp

CLAUDE.md

@@ -0,0 +1,117 @@
+# CLAUDE.md — how work lands in this repo
+
+Read this before doing anything durable. It is the delivery contract for AI coding
+sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
+takes precedence on *what* to work on; this file governs *how* it lands.
+
+## 1. Never push to `main`. Ever.
+
+Every durable change reaches `main` exactly one way:
+
+    branch  ->  push the branch  ->  open a PR  ->  CI gate green  ->  human approval  ->  merge
+
+There is no other path. Specifically:
+
+- **Do not `git push origin main`.**
+- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
+- **Do not merge your own PR.** A human approves and clicks merge.
+- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
+  anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
+
+> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
+> That wording is **stale** — it predates the PR flow. It describes what Flux does
+> (main is what the cluster follows), not what you are permitted to do. Honour this
+> file, not that line. Fixing those runbooks is tracked work.
+
+## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
+
+Verified against the live Gitea on 2026-09-29:
+
+| Expected | Actual |
+|---|---|
+| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
+| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
+| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m |
+| Approval required before merge | **`required_approvals: 0`** |
+
+So nothing mechanically stops you from pushing to `main` right now. **That is a gap
+being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a
+"required status check" with a runner online; neither was true, which is how this went
+unnoticed for a month. The runner half is now genuinely fixed and observed green. The
+protection half is not — see `infra/gitea/`.
+
+**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
+the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
+strength of a check you have not seen report — and note that a **skipped** Gitea job
+reports as `success`, so a green tick is not by itself evidence that anything ran.
+
+## 3. Gate before you push
+
+    ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+
+Exit codes are a contract — do not flatten them:
+
+| rc | Meaning |
+|---|---|
+| 0 | PASS |
+| 1 | BLOCKED — something failed |
+| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
+
+`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
+beside this one; that single skip is expected and tolerated. Any *other* skip is a
+real gap in your run.
+
+**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
+so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
+
+### Running the gate from a git worktree
+
+Two unrelated failures, both with known workarounds — put both in any agent brief:
+
+1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
+   and exists only in the main checkout. Do **not** copy it (it is a credential).
+   Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
+   normally.
+2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
+   gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
+   the `ENGINE` hook — a shim that intercepts `run` and re-execs
+   `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
+   `tracked_files()` is this bug, not a broken tree.
+
+## 4. Branch naming
+
+Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
+trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
+The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
+
+## 5. Cluster access
+
+`kubectl` does not work on the operator workstation; the toolchain lives only in the
+pinned container. Always name the context explicitly:
+
+    ./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+`dev.sh` merges both clusters and picks a current-context **per invocation**; that
+default has already changed once. Never rely on it. Do not run two `dev.sh`
+invocations concurrently.
+
+**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
+
+## 6. Secrets
+
+- **Never `sops -d` a file into a terminal, a log, or a transcript.**
+- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
+  list key names or object names only. A password has been leaked into a transcript
+  this way before.
+- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
+  the single creation-rules file; do not add a second one in a subtree.
+- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
+  flow. Do not design around committing one — read the expiry from the issuing page and
+  treat a burnt token as free to retry.
+
+## 7. Before you finish
+
+Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
+verified, and what you did **not** verify. State gaps plainly — an honest
+"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
+burned by the difference.

README.md

@@ -38,22 +38,56 @@ ci/             the gate as scripts (lint · schema · Kyverno L0 · L1 render
 
 ## Status
 
-Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
+Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in
-open** — `clusters/prod` is a design-only overlay.
+progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the
+apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration:
+dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
+hardware rather than on Proxmox VMs.
+
+**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring +
+alerts live*:
+
+| criterion | state |
+|---|---|
+| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
+| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
+| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only |
+
+**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 —
+the first requirement in this repo to reach that state. Both of its verifiers are recorded:
+`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`)
+and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn`
+class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy**
+(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
+*different* PersistentVolume — so an in-cluster survival cannot pass as a restore.
+
+**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned
+dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
+exercise rather than something to run in passing.
+
+Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a
+bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
 
 | Phase | Delivered |
 |---|---|
 | 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
-| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
+| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
-| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
+| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
 | 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
-| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
+| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
 | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
 | 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
+| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
 
-**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
+**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
-(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
+2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with
-the full report is `conformance/_artifacts/coverage/coverage.md`.
+`./conformance/run.sh --coverage-only`; the full report is
+`conformance/_artifacts/coverage/coverage.md`.
+
+Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and
+*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current
+SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the
+second.
 
 The running narrative — decisions, root causes, incidents, follow-ups — is
 [`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.

apps/vms/unifi/30-virtualmachine.yaml

@@ -41,10 +41,41 @@ spec:
       # disk KubeVirt cannot migrate and eviction would hard-stop the guest.
       evictionStrategy: LiveMigrate
       domain:
+        # =====================================================================================
+        # FIRMWARE UUID == THE CLOUD-INIT INSTANCE-ID. THIS IS THE RE-RUN LEVER. READ BEFORE EDIT.
+        # =====================================================================================
+        # MEASURED on prod 2026-09-14, not assumed:
+        #   * the NoCloud seed KubeVirt builds for this VM carries
+        #       {"instance-id":"de5c3338-6d32-4956-89d1-12c8272b2c07","local-hostname":"unifi"}
+        #     (read out of noCloud.iso inside virt-launcher-unifi-gp594), and that value was
+        #     EXACTLY `spec.template.spec.domain.firmware.uuid`, which KubeVirt's mutating webhook
+        #     had stamped onto the VM OBJECT — so it is PERSISTENT, not per-VMI (the VMI's own uid
+        #     was b532fd7c-…, a different value). It is not a hash of the VM name either: the four
+        #     standard uuid5 namespaces over "unifi" / "unifi.unifi" all produce something else.
+        #   * cloud-init's NoCloud datasource decides "is this the same instance?" by comparing the
+        #     cached instance-id against the guest's DMI product_uuid — which IS this firmware UUID.
+        #
+        # CONSEQUENCE, and it is the whole reason the previous unattended attempt could never have
+        # worked: EDITING `userData` BELOW AND RESTARTING THE VM CHANGES NOTHING. cloud-init sees an
+        # unchanged instance-id, restores its CACHED datasource, never re-reads the seed, and skips
+        # every per-instance module (write_files, runcmd). The new payload is silently ignored.
+        #
+        # So the UUID is pinned HERE, in git, and BUMPING IT IS THE SUPPORTED WAY to make the guest
+        # process a new cloud-init payload: change the trailing hex, commit, restart the VM. Pinning
+        # it also makes a rebuild reproducible — a fresh apply would otherwise take a random
+        # webhook-generated UUID and give the guest a different identity every time.
+        #
+        # The value below is DELIBERATELY NOT the de5c3338-…c07 the live guest booted with: that
+        # difference is what makes the next boot a NEW cloud-init instance and installs the
+        # `install-uos` helper + its systemd unit. `serial` is pinned to the value the webhook
+        # already stamped, so this change does not also churn the guest's DMI serial.
+        firmware:
+          uuid: "de5c3338-6d32-4956-89d1-12c8272b2c11"
+          serial: "24bb12ac-d311-45f6-a75b-90a9e1b63e18"
         # FLAG (verify-at-bringup): Ubuntu 24.04 amd64 cloud images are hybrid BIOS+UEFI bootable,
         # so KubeVirt's default SeaBIOS SHOULD boot cleanly (unlike the Fedora containerdisk, which
-        # is UEFI-only and needed an explicit efi block). If the guest drops to emergency/no-boot,
+        # is UEFI-only and needed an explicit efi block). RESOLVED at bring-up: boots as shipped;
-        # add:  firmware: { bootloader: { efi: { secureBoot: false } } }
+        # the guest reports Ubuntu 24.04.4 LTS.
         cpu:
           cores: 2
         memory:
@@ -58,9 +89,28 @@ spec:
             - name: cloudinit
               disk:
                 bus: virtio
+            # The operator's installer URL (+ optional SSH key), delivered as a tiny read-only
+            # iso9660 disk — see the `uosdata` volume near the bottom of this file.
+            - name: uosdata
+              disk:
+                bus: virtio
           interfaces:
+            # MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
+            # KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
+            # /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
+            # that file while the NoCloud instance-id is unchanged — and the instance-id here is
+            # domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
+            # handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
+            # administratively DOWN with no address and no DHCP attempt.
+            # That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
+            # device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
+            # (because the per-instance modules were skipped), yet UOS is installed and
+            # qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
+            # Pinning the MAC to the value the current VMI already holds makes the guest's netplan
+            # match permanently, so this cannot recur on the next recreation.
             - name: default
               masquerade: {}
+              macAddress: a6:d7:c0:b3:72:fa
               # Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
               # these and (b) surfaces them as containerPorts on the virt-launcher pod so the
               # LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
@@ -83,38 +133,112 @@ spec:
         - name: rootdisk
           dataVolume:
             name: unifi-rootdisk
+        # =====================================================================================
+        # THE OPERATOR'S ONE FIELD — SOPS Secret `unifi-uos-installer`, delivered as a DISK.
+        # =====================================================================================
+        # KubeVirt exposes a Secret as a small read-only iso9660 disk whose volume label we choose;
+        # each Secret key becomes a file on it. The guest mounts it by label (LABEL=UOSDATA) and
+        # reads `installer-url` plus an optional `console-password`. The Secret ships ENCRYPTED with
+        # REPLACE-ME placeholders, and the guest-side script treats anything that is not an https
+        # URL as "not provided yet" and exits 0 — so the VM boots and stays healthy until the
+        # operator fills it in. See unifi-uos-installer.sops.yaml.example next to this file.
+        #
+        # `console-password` exists because THERE IS CURRENTLY NO WAY INTO THIS GUEST. The stock
+        # Ubuntu cloud image ships a locked `ubuntu` account, this cloud-init never installed a key,
+        # and SSH could not help anyway: the masquerade interface above forwards ONLY the declared
+        # ports, and 22 is not one of them — probed from inside virt-launcher on 2026-09-14, the
+        # guest answers RST on 22 exactly as it does on the UniFi ports. The serial console is the
+        # only door, and a console needs a password. Set it and `virtctl -n unifi console unifi`
+        # works; `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` already gives
+        # read-only console output with no virtctl at all.
+        #
+        # WHY A SEPARATE DISK AND NOT `cloudInitNoCloud.secretRef` (which would also lift the
+        # 2048-byte inline cap): secretRef moves the ENTIRE cloud-config into ciphertext. Two
+        # concrete costs. (a) The shell below stops being reviewable in a diff and falls outside the
+        # L0 dollar-collapse guard, which can only read plaintext. (b) Flux runs its postBuild
+        # envsubst over the DECRYPTED resources, so the escape contract in
+        # clusters/base/apps-vms.yaml would still govern text nobody can see in git — the exact
+        # shape of the 2026-09-13 nextcloud-db incident. Keeping the script in plaintext and putting
+        # ONLY the operator's two values behind SOPS keeps both properties.
+        #
+        # A CHANGED SECRET REACHES THE GUEST ON THE NEXT VMI START with NO firmware-UUID bump: the
+        # disk is rebuilt from the Secret every time the VMI starts, and the systemd unit below
+        # re-reads it on EVERY boot instead of relying on cloud-init (which will not re-run — see
+        # the firmware block above). The UUID bump is a ONE-TIME step, to get that unit installed.
+        - name: uosdata
+          secret:
+            secretName: unifi-uos-installer
+            volumeLabel: UOSDATA
         - name: cloudinit
           cloudInitNoCloud:
-            # INLINE (kept < KubeVirt's 2048-byte userData limit — so NO SSH key here; add one back via
+            # INLINE, and it MUST stay <= 2048 bytes. That ceiling was MEASURED against the
-            # userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
+            # DEPLOYED KubeVirt v1.9.0 rather than quoted: a server-side dry-run apply of an
-            # signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
+            # otherwise-identical VM accepted 2048 bytes of userData and rejected 2049 with
-            # Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
+            # "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for
-            # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
+            # larger data." The payload below is ~1.5 KB; keep additions small.
-            # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
+            #
-            # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
+            # WHAT IT DOES: installs qemu-guest-agent (the KubeVirt AgentConnected signal, which is
-            # as a literal ${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
+            # also the proof cloud-init ran) and preps Podman for UOS, then writes `install-uos`
-            # by envsubst and need no escaping. NEVER write an unescaped ${...} here, and never write
+            # plus a systemd unit that runs it on EVERY boot. The unit — not cloud-init — is what
-            # ${ARR[0]} even escaped-adjacent: envsubst hard-fails on `[` ("missing closing brace")
+            # makes a later URL change take effect, because cloud-init will not re-run per-instance
-            # and would break the whole apps-vms apply.
+            # modules while the instance-id is unchanged.
+            #
+            # `install-uos` IS IDEMPOTENT AND WILL NOT REINSTALL OVER A WORKING CONTROLLER. It bails
+            # out the moment systemd knows a `uosserver.service` — which the live guest ALREADY has:
+            # the serial console log for the current boot shows both `uosserver.service` and
+            # `uosserver-updater.service` starting, and the root filesystem is 6.3 GB of 40 GB used.
+            # UOS is installed here; see the README for what is actually wrong (the guest lost its
+            # IP). A stamp file at /var/lib/uos-installed is the second, weaker guard for the case
+            # where the unit name changes in some future UOS release.
+            #
+            # ENVSUBST: apps-vms declares postBuild.substituteFrom (for ${UNIFI_L4_VIP} in
+            # 50-service-l4.yaml), so Flux's envsubst DOES run over this text. ${BASE_DOMAIN} below
+            # is a deliberate substitution. Everything else is bare `$name` or `$(cmd)`, both of
+            # which envsubst leaves untouched — verified against the live VMI, whose rendered
+            # userData still carries its shell variables verbatim. There is deliberately NO
+            # brace-form shell expansion left in this payload, so nothing here needs escaping; if
+            # you add one, escape it per the contract in clusters/base/apps-vms.yaml, and NEVER
+            # write an indexed expansion.
             userData: |-
               #cloud-config
               hostname: unifi
-              fqdn: unifi.home.keithjackson.dev
+              fqdn: unifi.${BASE_DOMAIN}
               package_update: true
               package_upgrade: false
-              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
+              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl]
               write_files:
                 - path: /usr/local/sbin/install-uos
                   permissions: '0755'
                   content: |
-                    #!/usr/bin/env bash
+                    #!/bin/bash
                     set -euo pipefail
-                    if [ -z "$${UOS_URL:-}" ]; then
+                    d=/run/uos; mkdir -p $d
-                      UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
+                    mountpoint -q $d || mount -o ro LABEL=UOSDATA $d
-                    fi
+                    p=$(tr -d '\r\n' < $d/console-password || true)
-                    [ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
+                    case "$p" in ""|REPLACE-ME) ;; *) echo "ubuntu:$p" | chpasswd ;; esac
-                    t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
+                    systemctl cat uosserver.service >/dev/null 2>&1 && exit 0
+                    s=/var/lib/uos-installed
+                    [ -e $s ] && exit 0
+                    u=$(tr -d '\r\n' < $d/installer-url || true)
+                    case "$u" in https://*) ;; *) echo "UOS installer-url not set yet"; exit 0;; esac
+                    t=$(mktemp)
+                    curl -fL --retry 5 --retry-delay 10 -o $t "$u"
+                    chmod +x $t
+                    yes y | $t install || yes y | $t
+                    rm -f $t; date -Is > $s
+                - path: /etc/systemd/system/uos-install.service
+                  content: |
+                    [Unit]
+                    Description=Install UniFi OS Server
+                    Wants=network-online.target
+                    After=network-online.target
+                    [Service]
+                    Type=oneshot
+                    ExecStart=/usr/local/sbin/install-uos
+                    StandardOutput=journal+console
+                    [Install]
+                    WantedBy=multi-user.target
               runcmd:
-                - [ systemctl, enable, --now, qemu-guest-agent ]
+                - [systemctl, enable, --now, qemu-guest-agent]
-                - [ systemctl, enable, --now, podman.socket ]
+                - [systemctl, enable, --now, podman.socket]
-                - [ bash, -lc, "/usr/local/sbin/install-uos 2>&1 | tee /var/log/uos-install.log > /dev/console" ]
+                - [systemctl, enable, --now, uos-install.service]

apps/vms/unifi/50-service-l4.yaml

@@ -21,13 +21,66 @@
 # k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
 #
 # selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
-# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
+# into the guest.
-# VIP pinned to the node running the VM (also what L2 announcement does).
+#
+# -------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
+# -------------------------------------------------------------------------------------------
+# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
+# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
+# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
+#
+# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
+#    per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
+#    matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
+#    MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
+#    `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
+#    announcing node that holds no backend has nowhere to send the packet and answers ICMP port
+#    unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
+#    (the leases were 8 days old); the operator's controller UI and device inform path were down.
+#    It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
+#    deleting the Lease forces a re-election — and that re-election is random, so it can pick a
+#    backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
+#    reference implementation this file now follows.
+#
+# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
+#    interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
+#    SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
+#    controller has always seen the pod-side masquerade gateway as the source, never the LAN
+#    client. There is a SECOND translation below the Service, and it already destroyed the very
+#    thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
+#    it is giving up an address the guest could not see in the first place.
+#
+#    Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
+#    MAC (and the device reports its own IP inside the inform payload), not the packet source,
+#    and guest-portal authorization runs on the gateway appliance, not on this controller.
+#
+# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
+# is what makes this survive a reschedule and a rebuild.
+#
+# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
+# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
+# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
+# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
+# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
+# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
+# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
+# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
+# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
+#
+# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
+# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
+# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
+# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
+# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
+# fix would be a NEW restriction that could silently break a remote site's device informing over
+# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
+# change, made deliberately and tested against the device fleet.
 #
 # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
 # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
 # Substituted by the `apps-vms` Flux Kustomization, which carries postBuild.substituteFrom for
-# exactly this token; every shell ${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
+# exactly this token; every shell $${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
 # accordingly. A new environment MUST set UNIFI_L4_VIP to a free address in ITS pool.
 #
 # REUSABLE PATTERN (the deferred `minecraft` :25565/L4 problem is the same shape): expose an
@@ -46,10 +99,10 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
     # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
     # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
-    external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
+    external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:
@@ -72,7 +125,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:

apps/vms/unifi/README.md

@@ -1,18 +1,48 @@
 # UniFi Network controller (UniFi OS Server) — VM onboarding
 
-> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
+> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
-> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
+> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
-> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
+> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
-> or install cannot stall the app tier.
+> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
+> cannot stall the app tier.
 >
-> **Status (verified 2026-08-26):**
+> **Status (re-verified against live prod 2026-09-14):**
-> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
+> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
-> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
+> - CDI import — **Succeeded** (100%).
-> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
+> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
-> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
+>   `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
-> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
+> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
->   (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
+> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
->   The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
+> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
+>   `:8080` refuse connections.
+
+## What is actually wrong (2026-09-14)
+
+The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
+That was wrong, and acting on it would have meant reinstalling over a working controller.
+
+Evidence, all read-only, from the live cluster:
+
+| Claim | How it was checked | Result |
+|---|---|---|
+| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
+| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
+| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
+| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
+| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
+
+So the controller is installed and its units start; nothing downstream of the guest's own network
+stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
+~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
+UOS taking over host networking, but **that has not been proven and cannot be, from outside the
+guest.** Diagnosing it needs a shell, which is the second problem below.
+
+**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
+no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
+nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
+ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
+guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
+new Secret is for.
 
 ## What this is
 
@@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
 | `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
 | `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
 | `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
-| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
+| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
 | `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
-| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
+| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
+| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
+| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
 | `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
 
-The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
+The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
-serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
+`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
-this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
+version. Browser access is via the L4 VIP. Re-add it if/when that lands.
 
-### Cloud-init delivery — two constraints learned the hard way
+### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)
 
-1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
+1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
-   prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
+   of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
-   shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
+   userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
-   Secret deliberately.
+   payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
-2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
+   is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
-   `$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
+   validate as an unknown field.
-   therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
+2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
+   nothing.** The NoCloud seed's `instance-id` is exactly
+   `spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
+   (so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
+   product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
+   `runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
+   effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
+   to force a re-run. See the long comment in `30-virtualmachine.yaml`.
+3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
+   been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
+   envsubst therefore runs over the cloud-init payload, and the escape contract in
+   `clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
+   against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
+   indexed expansion is a hard error that fails the whole Kustomization.
 
 ### Node prerequisites this VM forced onto the cluster
 
@@ -56,15 +101,31 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 - **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
   3-replica class made the import fail with insufficient storage.
 
-## UOS install method (researched)
+## UOS install method (researched 2026-09-14)
 
-- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
+- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
-  2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
+  Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
-- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
+  (the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
-  `ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
+  Podman must be at `/usr/bin/podman`.
-  **`https://<host>:11443`**.
+- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
-- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
+  in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
-  independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
+- **Is the URL a short-lived signed link?** *Probably not.* The links look like
+  `https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
+  **no signature query string**, and public community installers hard-code them. It is
+  account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
+  account-derived material out of plaintext config — but do **not** build logic around expiry.
+- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
+  Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
+  ecosystem actually does is pipe confirmations in: the widely used community installer
+  (`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
+  either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
+  `yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
+  fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
+  (no URL has been supplied, and UOS is already installed here), so treat it as "established by
+  convergent third-party practice", not as verified-by-us.
+- The previously shipped helper had two defects worth recording: it resolved the URL from
+  `fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
+  `unifi-os-server`, and it invoked the binary with **no `install` subcommand**.
 
 ### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
 
@@ -78,76 +139,143 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 | 3478 | UDP | **STUN** (device ⇄ controller) |
 | 10001 | UDP | **L2 device discovery** |
 
+**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
+the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.
+
 ## L4 exposure design (the hard part)
 
 The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
 or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
 
 - **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
-  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
+  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
-  The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
+  (`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
-  flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
+  `default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
+  the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
+  existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
+  / `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
+  decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
 - `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
-  declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
+  declared ports into the guest.
-- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
+- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
-  — via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
+  used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
-  Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
+  Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
-  reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
+  leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
-  masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
+  and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
-  explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
+  it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
-- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
+  why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
-  LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
+- **Device-adoption story:** point devices' **inform host** at the VIP
-  `minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
+  (`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
+  `set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
+  subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
+  broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
+  zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
+  on other VLANs must route to the VIP and be inform-pointed anyway.
+- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
+  per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.
 
-**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
+**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
-`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
+the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
-(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
+`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
+zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
+apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
+flip, so this hostname is current, not a dev leftover.
+
+## The operator's one field — installing UOS without a shell
+
+`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
+the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
+`uos-install.service` reads it **on every boot**:
+
+```bash
+sops apps/vms/unifi/unifi-uos-installer.sops.yaml    # fill installer-url and/or console-password
+git commit && git push                               # merge to the branch prod's Flux tracks
+kubectl -n unifi delete vmi unifi                    # `running: true` recreates it; no virtctl needed
+kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
+  -c guest-console-log                               # watch the boot + the install, read-only
+```
+
+Design notes that matter if you change this:
+
+- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
+  cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
+  L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
+  resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
+  operator values in SOPS.
+- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
+  the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
+  firmware-UUID bump.
+- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
+  a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
+  guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
+  controller on next boot".
+- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
+  `apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
+  `clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
+  guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.
 
 ## Open questions / risks
 
-1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
+1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
-   account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
+   requires a shell, which requires `console-password` above plus one restart. First things to run
-   **interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
+   on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
-   the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
+   `ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
-   **Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
+   own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
-   invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
+   capture the state before rebooting a second time if you can.
-2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
+2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
+   install-method section: no official silent flag exists, `yes y | <installer> install` is the
+   convergent community practice, and it has not been executed on our guest. Treat the first real
+   install as something to watch on the console log, not as a fire-and-forget.
+3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
    import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
    Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
    fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
-3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
+4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
-4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
+5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
-   `192.168.1.53–.60` pool (`.53` is the shared Gateway).
+   prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
-5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
+   (`.23`) or the node addresses (`.10/.11/.12`).
+6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
    migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
-6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
+7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
    installer and for optional remote access.
-7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
+8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
    masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
+9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
+   `kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
+   `virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
+   the cluster's **v1.9.0**.
 
 ## Remaining bring-up
 
-Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
+Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
+is written for a from-scratch cluster; the prod addresses are the live ones.
 
-1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
+1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
 2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
-   Kustomization (not an app wave).
+   Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
 3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
 4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
-   `192.168.1.54`.
+   `192.168.2.21`.
-5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
+5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
-   and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
+   `uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
-   ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
+   fill `installer-url`, commit, and the guest installs itself on first boot.
-6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
+6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
+   console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
+7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
    complete the first-run wizard.
-7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
+8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
-8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
+   split-horizon) and from in-cluster (coredns-custom).
+9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
    connects (green). Test guest portal / throughput if used.
-9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
+10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
-10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
+11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
 
 **Quick health check today:**
 ```bash
 kubectl -n unifi get dv,vmi,svc
 kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
+# the two that would have caught the current failure:
+kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}'   # expect an ipAddress
+kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
+  -c compute -- ip neigh                                                 # expect 10.0.2.2 REACHABLE
 ```

apps/vms/unifi/kustomization.yaml

@@ -11,3 +11,11 @@ resources:
   - 30-virtualmachine.yaml
   - 40-networkpolicy.yaml
   - 50-service-l4.yaml
+  # SOPS-encrypted. Carries the operator's UOS installer URL (+ optional SSH key) and is mounted
+  # into the guest as the UOSDATA disk by 30-virtualmachine.yaml. Decrypted by the `apps-vms` Flux
+  # Kustomization, which clusters/dev and clusters/prod patch with the `decryption` stanza — the
+  # FIRST encrypted content under apps/vms, so that patch was added with this file. The plaintext
+  # template + the operator hand-off steps live in the `.example` sibling of the file below, which
+  # is deliberately NOT named here: the L0 no-plaintext-secrets check fails any kustomization whose
+  # text contains an example template's filename, on the assumption it is being applied.
+  - unifi-uos-installer.sops.yaml

apps/vms/unifi/unifi-uos-installer.sops.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: unifi-uos-installer
+    namespace: unifi
+type: Opaque
+stringData:
+    installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
+    console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
+            WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
+            TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
+            cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
+            kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-19T02:47:13Z"
+    mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/vms/unifi/unifi-uos-installer.sops.yaml.example

@@ -0,0 +1,71 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the two values that (a) let the UniFi VM install UniFi OS Server by itself, with
+# nobody typing commands inside the guest, and (b) give the operator a way INTO that guest at all.
+# KubeVirt mounts this Secret into the VM as a tiny read-only iso9660 disk labelled `UOSDATA` (see
+# the `uosdata` volume in 30-virtualmachine.yaml); each key below becomes a file of the same name on
+# that disk, and the in-guest `uos-install.service` reads them on EVERY boot.
+#
+# READ THIS FIRST — UOS IS ALREADY INSTALLED ON THE LIVE PROD GUEST (verified 2026-09-14, see the
+# README "What is actually wrong"). `installer-url` is therefore NOT the fix for today's outage; it
+# is the rebuild path and the "the install never happened" path. `install-uos` refuses to run the
+# installer at all while systemd knows a `uosserver.service`, so filling this in cannot clobber a
+# working controller. `console-password` IS useful today: it is the only way to get a shell.
+#
+# WHY THE URL IS BEHIND SOPS: it is obtained through the operator's own (free) Ubiquiti account
+# session at ui.com/download/software/unifi-os-server, it identifies a specific build, and it is not
+# ours to redistribute. Treat it as account-derived material, not as public configuration.
+# HONEST CAVEAT, checked 2026-09-14: the resulting `https://fw-download.ubnt.com/data/
+# unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` URL carries NO signature query string,
+# and public community install scripts hard-code these URLs — so it does not behave like a
+# short-lived signed link. It does change every release. Encrypting it costs nothing and keeps the
+# repo's "account-derived material is not config" line clean, but do NOT design around "it expires".
+#
+# HAND-OFF STEPS (operator) — this is the whole job:
+#  (a) Sign in at https://ui.com/download/software/unifi-os-server, pick the Linux (x64) build, and
+#      copy the download link (right-click -> Copy link address). Do NOT download it here; the guest
+#      fetches it itself.
+#  (b) Open the encrypted sibling in your editor:
+#         sops apps/vms/unifi/unifi-uos-installer.sops.yaml
+#  (c) Replace the two REPLACE-ME values. Either may be left as the placeholder — the guest skips a
+#      placeholder rather than failing.
+#  (d) Save/quit (sops re-encrypts in place), commit, push, merge to the branch prod tracks.
+#  (e) Restart the VM once so the new cloud-init instance and the new disk are picked up:
+#         kubectl -n unifi delete vmi unifi        # `running: true` recreates it; no virtctl needed
+#      Watch it come up WITHOUT virtctl:
+#         kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher \
+#           -o name | head -1) -c guest-console-log
+#  (f) UI: https://192.168.2.21:11443 (prod) or https://unifi.home.keithjackson.dev:11443.
+#
+# IF YOU LATER NEED A NEW URL (new release, dead link): repeat (b)-(e). No firmware-UUID bump is
+# needed for that — the disk is rebuilt from this Secret on every VMI start and the systemd unit
+# re-reads it every boot. The UUID bump in 30-virtualmachine.yaml is only needed when the cloud-init
+# payload ITSELF changes.
+#
+# The file is already listed under `resources:` in apps/vms/unifi/kustomization.yaml, and `apps-vms`
+# is patched with the SOPS `decryption` stanza in clusters/dev + clusters/prod.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: unifi-uos-installer
+  namespace: unifi
+type: Opaque
+stringData:
+  # KEY NAMES ARE LOAD-BEARING: each key becomes a FILE of that exact name on the UOSDATA disk, and
+  # /usr/local/sbin/install-uos reads them by path. Renaming a key breaks the install silently.
+  #
+  # The signed-in download link for UniFi OS Server, Linux x64. Anything that is not an https URL
+  # (this placeholder included) is treated by the guest as "not provided yet": the unit logs
+  # "UOS installer-url not set yet" and exits 0, so the VM stays healthy and simply does not install.
+  installer-url: REPLACE-ME
+  # Password set on the guest's `ubuntu` account on every boot, for `virtctl -n unifi console unifi`.
+  # THIS IS THE ONLY DOOR INTO THE GUEST. The stock Ubuntu 24.04 cloud image ships `ubuntu` locked,
+  # no key was ever installed, and an SSH key would not help: the VM's masquerade interface forwards
+  # only the ports declared in 30-virtualmachine.yaml and 22 is not among them (probed from inside
+  # virt-launcher, 2026-09-14 — the guest RSTs on 22 just as it does on 11443). Generate one with
+  #     openssl rand -base64 24
+  # Leave it as REPLACE-ME to keep the account locked; the script skips the placeholder by name.
+  console-password: REPLACE-ME

apps/wave-0/gitea.yaml

@@ -90,9 +90,9 @@ spec:
               name: gitea-bootstrap
               key: secret-key
         - name: GITEA__server__DOMAIN
-          value: gitea.home.keithjackson.dev
+          value: gitea.${BASE_DOMAIN}
         - name: GITEA__server__ROOT_URL
-          value: https://gitea.home.keithjackson.dev/
+          value: https://gitea.${BASE_DOMAIN}/
         - name: GITEA__actions__ENABLED
           value: 'true'
         - name: GITEA__actions__DEFAULT_ACTIONS_URL
@@ -127,8 +127,8 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1; break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name\
+              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
-                \ groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n"
+                \ break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:
@@ -170,6 +170,27 @@ spec:
       - port: '3000'
         protocol: TCP
 ---
+# CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): gitea-allow-flux-ingress
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: gitea-allow-flux-ingress
+  namespace: gitea
+  labels:
+    app.kubernetes.io/name: gitea
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: gitea
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: flux-system
+    toPorts:
+    - ports:
+      - port: '3000'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): gitea-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -207,7 +228,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - gitea.home.keithjackson.dev
+  - gitea.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -217,26 +238,6 @@ spec:
     - name: gitea
       port: 3000
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: gitea
-  namespace: gitea
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: gitea
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: gitea
-  namespaceSelector:
-    matchNames:
-    - gitea
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -251,11 +252,11 @@ spec:
   - name: gitea.slis
     rules:
     - alert: gitea-down
-      expr: up{job="gitea"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"} == 0 or absent(kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"})
       for: 5m
       labels:
         severity: critical
         service: gitea
       annotations:
         summary: gitea is down
-        description: 'gitea has been unreachable for 5m (SLI: up).'
+        description: 'gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/jackett.yaml

@@ -74,7 +74,7 @@ spec:
         homelab.keithjackson.dev/auth-mode: forward-auth
       annotations:
         homelab.keithjackson.dev/auth-mode: forward-auth
-        homelab.keithjackson.dev/auth-groups: media-users
+        homelab.keithjackson.dev/auth-groups: media-users,platform-admins
     spec:
       containers:
       - name: jackett
@@ -83,7 +83,7 @@ spec:
         - containerPort: 9117
         volumeMounts:
         - name: config
-          mountPath: /data/config
+          mountPath: /config
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -91,16 +91,17 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:9117
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
+        - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://jackett.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -266,14 +267,15 @@ metadata:
     app.kubernetes.io/name: jackett
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
-    homelab.keithjackson.dev/auth-groups: media-users
+    homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - jackett.home.keithjackson.dev
+  - jackett.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -283,26 +285,6 @@ spec:
     - name: jackett
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: jackett
-  namespace: jackett
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: jackett
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: jackett
-  namespaceSelector:
-    matchNames:
-    - jackett
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -317,11 +299,11 @@ spec:
   - name: jackett.slis
     rules:
     - alert: jackett-down
-      expr: up{job="jackett"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"} == 0 or absent(kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"})
       for: 5m
       labels:
         severity: critical
         service: jackett
       annotations:
         summary: jackett is down
-        description: 'jackett has been unreachable for 5m (SLI: up).'
+        description: 'jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/kustomization.yaml

@@ -11,13 +11,17 @@ resources:
 - jackett-oauth2-proxy-credentials.sops.yaml
 - jackett.yaml
 - nextcloud-db-app.sops.yaml
+- nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
+- pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex.yaml
+- qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
 - qbittorrent.yaml
 - semaphore-db-app.sops.yaml
+- semaphore-db-backup-credentials-prod.sops.yaml
 - semaphore-db-backup-credentials.sops.yaml
 - semaphore-db.yaml

apps/wave-0/nextcloud-db-backup-credentials-prod.sops.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: nextcloud-db-backup-credentials-prod
+    namespace: nextcloud-db
+type: Opaque
+stringData:
+    ACCESS_KEY_ID: ENC[AES256_GCM,data:4lx6et4HcXDegulyIoa7jZR/lVgapGPEkukajfCKjvk=,iv:p6V7tizbikk55tOvpPOsPHTe/p52QLX5A5xrwuB14rA=,tag:oNJuipT/Lb6zynJ2K2Q71A==,type:str]
+    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:V5bHPoPeBOVyLp1n1Jilyso4i5P2aPSH0hrryfweg2q/SctWZxRbLBQ/FLzOigbJsAhlwOGu069mQPma6xjfdw==,iv:8u4UOvI8dC/cm6mEWDlyN5E3SfRZmwKY9uZY/DjPEsQ=,tag:tw7tGs29TmUvMPBCtCszAA==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiSUt5N2k2N2lDU2ZsMy9m
+            M1NFS0M2QjFsNFhnR0hCMzFyaTdiQWFZRFJZCmhBTWRXNmxLMGtsc3Rxd2FSMDZp
+            aktjcTE2c1hmU1EzM3RWNksyb3Q1YVUKLS0tIDBFS1I5aDBER2cvcWtLZ3BXTEp5
+            ZGI5ZmZIcStheUNQamxIVVVpc0ZMOHMKlE5aHewtHJpeU24F5sodc9CEm9KTjHQQ
+            8ThEQ/8vyazL9V4ojlICxSHU/SHkHpD1eBaTKcAQd/LZzujIcJ+QfA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-07T06:36:33Z"
+    mac: ENC[AES256_GCM,data:8awz1Jh8t4zjDMyd93/v+6on9iiIaSi01293A7uYhxj9L1+g63zeO+DPmSn6+KbZvHdqU2A7up09H1NzfXAEdGuvVbB36cdGq0zKank3PUi8c7IT6qNuWydxRAcqLnbbD+c30Z6GB+gAuZHstNOe4Rgc4aJvMN6FuP6x6Z0870c=,iv:KGGKJq2QA5xztvoHs/SViXW72DYwT16i+W+dJBUeIJQ=,tag:3UpYqx2+RIK0QdMH/BSIpA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/nextcloud-db.yaml

@@ -37,16 +37,53 @@ metadata:
   name: nextcloud-db
   namespace: nextcloud-db
 spec:
-  # Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
+  # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db; see
-  # keycloak-db). The storage is already 3-way replicated by Longhorn; raise to 3 for Postgres-level
+  # platform/keycloak/postgres-cluster.yaml for the full rationale. Short version: Longhorn's 3-way
-  # HA as a future hardening.
+  # replication saves the DATA on node loss, not the SERVICE, and with `instances: 1` all three CNPG
-  instances: 1
+  # primaries had drifted onto k3s-prod-2, so one node took Nextcloud down alongside SSO and
+  # Semaphore. A streaming standby lets CNPG promote in seconds and repoint the -rw Service.
+  # Gated on the Barman archive below actually working (ContinuousArchiving=True with a real
+  # firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
+  #
+  # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
+  # pg_basebackup from the running primary, which is neither restarted nor disconnected.
+  instances: 3
   imageName: ghcr.io/cloudnative-pg/postgresql:16.4
   # Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
   # nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
   inheritedMetadata:
     labels:
       app.kubernetes.io/name: nextcloud-db
+  # HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
+  # primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
+  # recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
+  # version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
+  # after a node loss it may stack two instances on one node — which is how `instances: 3` decays
+  # back into the single-node failure domain this change removes. `required` makes it a constraint;
+  # the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
+  # serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
+  #
+  # APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
+  # switchover. Expect a single brief connection reset for Nextcloud.
+  affinity:
+    enablePodAntiAffinity: true
+    podAntiAffinityType: required
+  primaryUpdateMethod: switchover
+  # SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
+  # rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
+  # last few transactions when a primary dies before its standbys catch up, and `method: any` +
+  # `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
+  # never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
+  # are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
+  # standby into a total write outage. `preferred` relaxes to async instead.
+  #
+  # NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
+  # no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
+  postgresql:
+    synchronous:
+      method: any
+      number: 1
+      dataDurability: preferred
   storage:
     storageClass: longhorn
     size: 5Gi
@@ -68,25 +105,38 @@ spec:
       memory: 256Mi
     limits:
       memory: 512Mi
-  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
+  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
   # Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
-  # backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
+  # backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
-  # (defaults to the Cluster name), so objects land under s3://cnpg/nextcloud-db/. Path-style HTTP.
+  # destinationPath, so objects land under ${CNPG_BACKUP_PATH}/nextcloud-db/ — dev
-  # Credentials: the scoped key from the in-namespace SOPS Secret nextcloud-db-backup-credentials
+  # s3://cnpg/nextcloud-db/ on the on-Proxmox MinIO, prod
-  # (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
+  # s3://k3s-prod-backup/barman/nextcloud-db/ on Cloudflare R2. Both endpointURL and
-  # endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
+  # destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
-  # substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
+  # Credentials: PER-ENVIRONMENT (2026-09-07). dev reads nextcloud-db-backup-credentials (the
+  # scoped MinIO key), prod reads nextcloud-db-backup-credentials-prod (the Cloudflare R2
+  # token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
+  # generated golden — render.py places the -prod sidecar in this DB's wave); the
+  # cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
+  # one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
+  # ONE shared Secret until the split, so giving prod its R2 token would have left dev
+  # signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
+  # disk-full clock below rather than a merely missing backup.
+  #
+  # archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
+  # Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
+  # the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
+  # endpoint. Treat ContinuousArchiving=False here as an availability incident.
   backup:
     retentionPolicy: "30d"
     barmanObjectStore:
-      destinationPath: s3://cnpg
+      destinationPath: ${CNPG_BACKUP_PATH}
       endpointURL: ${S3_BACKUP_ENDPOINT}
       s3Credentials:
         accessKeyId:
-          name: nextcloud-db-backup-credentials
+          name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
           key: ACCESS_KEY_ID
         secretAccessKey:
-          name: nextcloud-db-backup-credentials
+          name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
           key: ACCESS_SECRET_KEY
       wal:
         compression: gzip
@@ -108,3 +158,243 @@ spec:
   cluster:
     name: nextcloud-db
   method: barmanObjectStore
+---
+# =============================================================================
+# CNPG METRICS-EXPORTER CONNECT RECONCILER (R-OBS-01)
+#
+# THE BUG THIS CLOSES (true positive, verified live on prod 2026-09-12, and the
+# condition platform/cnpg/alerts.yaml already recorded on BOTH clusters 2026-09-06):
+# the `nextcloud` database's ACL had been narrowed out-of-band so that PUBLIC held
+# nothing, and CNPG's metrics exporter role therefore could not even CONNECT:
+#
+#   nextcloud-db-1/-2/-3, every scrape:
+#     FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
+#     DETAIL: User does not have CONNECT privilege.
+#     msg="Error collecting user query" targetDatabase=nextcloud
+#
+# CONSEQUENCE: `cnpg_last_error` = 1 on all three instances and the ENTIRE
+# cnpg_pg_stat_* / pg_settings / pg_postmaster family MISSING for this database,
+# while the scrape target still reported up=1 — a blind spot that looks like health.
+# Measured before the fix: cnpg_pg_settings_setting had 765 series for keycloak and
+# 765 for semaphore-db and ZERO for nextcloud-db. Nextcloud's database had never
+# been monitored.
+#
+# ROOT CAUSE, AND WHY ONLY THIS DATABASE. All three Cluster specs are identical in
+# their bootstrap; the difference is entirely in the live ACL:
+#
+#   nextcloud : datacl = {nextcloud=CTc/nextcloud}          <-- explicit, no PUBLIC entry
+#   keycloak  : datacl = NULL
+#   semaphore : datacl = NULL
+#
+# A NULL datacl means "PostgreSQL defaults", which include CONNECT and TEMP for
+# PUBLIC — so keycloak and semaphore let the exporter in only by inheriting PUBLIC's
+# implicit CONNECT. On `nextcloud` something ran the equivalent of
+# `REVOKE ALL ON DATABASE nextcloud FROM PUBLIC`, which MATERIALISES datacl and
+# drops the implicit PUBLIC entry with it. That REVOKE is not in this repo and not
+# in DECISIONS-LOG.md — it was applied out of band, which is exactly why a
+# git-declared fix is needed rather than another hand-run psql.
+#
+# WHY THE GRANT IS EXPLICIT RATHER THAN "RE-GRANT CONNECT TO PUBLIC": restoring
+# PUBLIC's CONNECT would undo whatever hardening that REVOKE intended and would
+# leave the exporter depending on an implicit default again — the precise fragility
+# that broke here and that keycloak/semaphore are still relying on. Naming the
+# exporter role makes the dependency explicit and survives a future REVOKE FROM
+# PUBLIC on any of the three databases.
+#
+# WHY A CronJob AND NOT A CNPG FIELD — CNPG 1.30.0's API GENUINELY CANNOT EXPRESS
+# THIS. Checked against the live CRDs on prod:
+#   * `Database` (databases.postgresql.cnpg.io) has no database-ACL field at all.
+#     `allowConnections` is the pg_database.datallowconn FLAG (open/closed to
+#     everyone), not a per-role GRANT; its only grant/revoke verbs are the `usage`
+#     lists on `fdws` and `servers`.
+#   * `spec.managed.roles` and `DatabaseRole` manage ROLE ATTRIBUTES (login,
+#     superuser, connectionLimit, ...) plus `inRoles` MEMBERSHIP. Membership is the
+#     wrong tool: the only local role holding CONNECT here is the OWNER, so
+#     `inRoles: [nextcloud]` would hand the exporter the owner's CREATE and TEMP on
+#     the database and read/write on every application table — the opposite of
+#     least privilege, to buy one CONNECT.
+#   * `bootstrap.initdb.postInitApplicationSQL` runs ONCE, at bootstrap. It cannot
+#     repair these three already-bootstrapped instances (the actual bug), and on a
+#     FRESH cluster the grant is unnecessary anyway (datacl starts NULL). Adding it
+#     would be a second mechanism that fixes nothing this one does not.
+# So the SQL has to be applied by something, and this repo already has the pattern
+# for state Flux cannot reach: the level-triggered CronJob reconciler in
+# platform/keycloak/client-secret-reconciler.yaml, whose reasoning transfers
+# verbatim — the drifted state lives inside PostgreSQL, not in a Kubernetes object,
+# so no admission or mutation policy can see it. A one-shot Job is deliberately NOT
+# used: a completed Job is never re-run, so a DB rebuild or a PITR restore would
+# silently reopen the blind spot. A loop converges instead.
+#
+# PRIVILEGE GRANTED — EXACTLY ONE, AND NOTHING ELSE:
+#   GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
+# Resulting ACL entry is `cnpg_metrics_exporter=c/nextcloud` — the `c` is CONNECT;
+# there is no `C` (CREATE) and no `T` (TEMPORARY). No ownership, no superuser, no
+# write, no table or schema grants. Nothing further is needed because CNPG already
+# creates this role as a member of the built-in `pg_monitor` role (verified on all
+# three prod clusters), and pg_monitor already carries the read access every
+# collector query wants — they read pg_catalog views (pg_stat_database,
+# pg_stat_archiver, pg_settings, pg_postmaster) and no application tables. Proof
+# that CONNECT alone is sufficient: after this grant, `Error collecting user query`
+# went to ZERO on all three pods, i.e. every collector query now succeeds.
+#
+# IT RUNS AS THE DATABASE OWNER, NOT AS A SUPERUSER. `enableSuperuserAccess` is off
+# on this Cluster (there is no nextcloud-db-superuser Secret), and it stays off: the
+# owner of a database holds the grant option on that database's privileges, so
+# `nextcloud` can issue this GRANT itself. Verified live before committing, by
+# running the statement under `SET ROLE nextcloud`. Credentials come from the
+# in-namespace SOPS Secret nextcloud-db-app that already exists for the app.
+#
+# NETWORKING: nextcloud-db is intentionally NOT an app-namespace (see this file's
+# header), so Kyverno generates no default-deny baseline here and this same-namespace
+# pod -> nextcloud-db-rw:5432 connection needs no rendered allow.
+#
+# THE POD LABEL IS DELIBERATELY *NOT* app.kubernetes.io/name=nextcloud-db. That
+# label is what `inheritedMetadata` stamps on the CNPG instance pods and what the
+# rendered nextcloud egress CiliumNetworkPolicy selects; reusing it here would quietly
+# make this reconciler a permitted destination for the Nextcloud app. It gets its own
+# name so that selector keeps meaning "the database".
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: nextcloud-db-exporter-grant
+  namespace: nextcloud-db
+  labels:
+    app.kubernetes.io/name: nextcloud-db-exporter-grant
+    app.kubernetes.io/part-of: nextcloud-db
+spec:
+  # Every 15 minutes, matching the keycloak reconciler's cadence. A converged
+  # database makes this one idempotent GRANT and exits; the cost is negligible, and
+  # after a rebuild or PITR restore monitoring is blind for at most one interval
+  # (well inside the alert's `for: 6h`).
+  schedule: "*/15 * * * *"
+  # A run still waiting on the primary must not be lapped by the next tick.
+  concurrencyPolicy: Forbid
+  startingDeadlineSeconds: 300
+  successfulJobsHistoryLimit: 1
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      activeDeadlineSeconds: 300
+      ttlSecondsAfterFinished: 3600
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/name: nextcloud-db-exporter-grant
+            app.kubernetes.io/part-of: nextcloud-db
+        spec:
+          restartPolicy: Never
+          # Talks only to PostgreSQL; its one input is a mounted Secret, so it needs
+          # no Kubernetes API access at all (R-SEC-02, least privilege).
+          automountServiceAccountToken: false
+          securityContext:
+            runAsNonRoot: true
+            # uid/gid 26 is `postgres` in the CNPG PostgreSQL image.
+            runAsUser: 26
+            runAsGroup: 26
+            seccompProfile:
+              type: RuntimeDefault
+          containers:
+            - name: grant
+              # The SAME image the instances above run, pinned to the same tag. psql
+              # is already in it and it is already present on every node, so this
+              # adds no new image and no new supply-chain surface. Keep the tag in
+              # step with spec.imageName above when Renovate bumps it.
+              image: ghcr.io/cloudnative-pg/postgresql:16.4
+              imagePullPolicy: IfNotPresent
+              command:
+                - psql
+                # -w never prompts; -X skips ~/.psqlrc; ON_ERROR_STOP makes the Job
+                # FAIL LOUDLY rather than reporting success on a refused grant.
+                - "-w"
+                - "-X"
+                - "-v"
+                - "ON_ERROR_STOP=1"
+                - "-f"
+                - /etc/grant/grant.sql
+              env:
+                # libpq connection, assembled from the owner Secret. Target the -rw
+                # Service: GRANT is a write and must reach the primary.
+                - name: PGHOST
+                  value: nextcloud-db-rw.nextcloud-db.svc.cluster.local
+                - name: PGPORT
+                  value: "5432"
+                # Connecting to the application database itself is deliberate: the
+                # owner has CONNECT on it, and a successful connection is the same
+                # check the exporter was failing.
+                - name: PGDATABASE
+                  value: nextcloud
+                - name: PGUSER
+                  valueFrom:
+                    secretKeyRef:
+                      name: nextcloud-db-app
+                      key: username
+                # Read by libpq from the environment and never logged; psql echoes
+                # no connection parameters.
+                - name: PGPASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: nextcloud-db-app
+                      key: password
+                - name: PGCONNECT_TIMEOUT
+                  value: "10"
+              volumeMounts:
+                - name: sql
+                  mountPath: /etc/grant
+                  readOnly: true
+              securityContext:
+                allowPrivilegeEscalation: false
+                readOnlyRootFilesystem: true
+                capabilities:
+                  drop:
+                    - ALL
+              resources:
+                requests:
+                  cpu: 10m
+                  memory: 32Mi
+                limits:
+                  memory: 128Mi
+          volumes:
+            - name: sql
+              configMap:
+                name: nextcloud-db-exporter-grant
+                defaultMode: 0444
+---
+# The desired state, as SQL. Idempotent: re-granting an existing privilege is a
+# no-op, so a converged run changes nothing. The DO block is the ASSERTION — it
+# turns "the grant did not take" into a failed Job instead of a silent success.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nextcloud-db-exporter-grant
+  namespace: nextcloud-db
+  labels:
+    app.kubernetes.io/name: nextcloud-db-exporter-grant
+    app.kubernetes.io/part-of: nextcloud-db
+data:
+  grant.sql: |
+    -- The ONE privilege the CNPG metrics exporter is missing on this database.
+    -- CONNECT only: no CREATE, no TEMPORARY, no ownership, no table/schema grants.
+    -- Read access for the collector's pg_catalog queries already comes from the
+    -- built-in pg_monitor role, which CNPG grants this role when it creates it.
+    GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
+
+    -- Fail the Job if the privilege is somehow still absent, so a broken
+    -- reconciler surfaces as a failed Job rather than as a quiet blind spot.
+    -- NO PL/pgSQL, AND NOT ONE DOLLAR SIGN IN THIS FILE — DELIBERATE, including in these
+    -- comments. This assertion began life as a dollar-quoted DO block, which cannot survive
+    -- being shipped this way: the ConfigMap is rendered through Flux's postBuild
+    -- substitution, and envsubst collapses a doubled dollar into a single one. The block
+    -- reached Postgres with its opening quote halved and every Flux-reconciled run died
+    -- with a syntax error pointing at that character.
+    -- It PASSED when hand-applied with kubectl, because that path runs no envsubst — so
+    -- the defect was invisible until Flux owned the object, and three runs of Error pods
+    -- were the first sign. Escaping by doubling again would work and would also be the
+    -- next person's trap; plain SQL leaves substitution nothing to touch.
+    --
+    -- has_database_privilege() returns boolean; ::int is 1 when granted and 0 when not, so
+    -- the division raises division_by_zero and ON_ERROR_STOP=1 fails the Job. The divisor
+    -- is not a constant, so it cannot be folded away at plan time.
+    \echo 'asserting cnpg_metrics_exporter has CONNECT on database nextcloud'
+    SELECT 1 / has_database_privilege('cnpg_metrics_exporter', 'nextcloud', 'CONNECT')::int
+      AS exporter_can_connect;

apps/wave-0/pihole-dns.yaml

@@ -0,0 +1,196 @@
+---
+# =================================================================================================
+# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole`
+# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS
+# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition
+# schema genuinely cannot express it: schemas/servicedefinition.schema.json is
+# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned
+# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at
+# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and
+# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this
+# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and
+# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog
+# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the
+# schema has no word for.
+#
+# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is
+# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service
+# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09
+# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on
+# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that
+# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0).
+#
+# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the
+# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config
+# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is
+# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it
+# can never be allowed to move.
+#
+# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53
+# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same
+# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the
+# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a
+# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently
+# and confusingly.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC
+# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself
+# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS,
+# and publishing a resolver's address in a public zone invites exactly the traffic the source-range
+# filter below exists to refuse. Internal name, internal answer, no public record — which is also
+# why this change touches nothing under platform/externaldns/.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to
+# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured,
+# not assumed (Cilium 1.20.1, prod, 2026-09-14):
+#
+#   * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the
+#     CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does
+#     NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services
+#     landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3,
+#     `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1.
+#   * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the
+#     packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three
+#     nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol.
+#   * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod
+#     rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three
+#     minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a
+#     re-election — and that re-election is random, so it may pick a backend-less node again.
+#
+# A resolver that works one time in three and stays broken after any reschedule is not a resolver.
+# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is
+# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement
+# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all
+# answering.
+#
+# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in
+# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the
+# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client
+# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two
+# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole
+# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts
+# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity —
+# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode:
+# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract
+# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
+#
+# -------------------------------------------------------------------------------------------------
+# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
+# -------------------------------------------------------------------------------------------------
+# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
+# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
+# datapath — it never reaches a backend and never reaches a network policy. This is what makes
+# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
+# range to a /24 that excludes the test client, after which the same query that had been answering
+# stopped being answered at all, and restoring it, after which it answered again.
+#
+# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
+# restriction once the source has been rewritten: measured on this cluster, the forwarded query
+# arrives from the ingress node's internal router address and Cilium resolves that source to the
+# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the
+# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction
+# lives here, where it is actually enforceable, and the policy below restricts the PORTS.
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-udp
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+  annotations:
+    lbipam.cilium.io/sharing-key: "pihole-dns"
+    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  loadBalancerSourceRanges:
+    - ${LAN_CIDR}
+  selector:
+    app.kubernetes.io/name: pihole
+  ports:
+    - { name: dns-udp, port: 53, targetPort: 53, protocol: UDP }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-tcp
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+  annotations:
+    lbipam.cilium.io/sharing-key: "pihole-dns"   # same key + same IP => shares the VIP with -udp
+    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  loadBalancerSourceRanges:
+    - ${LAN_CIDR}
+  selector:
+    app.kubernetes.io/name: pihole
+  ports:
+    - { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP }
+---
+# =================================================================================================
+# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports.
+# =================================================================================================
+# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered
+# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the
+# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything
+# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the
+# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule:
+# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that
+# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for
+# it and no policy for it.
+#
+# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is
+# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's
+# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium
+# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was
+# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced
+# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
+# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
+# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
+# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
+#
+# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
+# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
+# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none
+# needs PiHole on :53) on fewer ports, behind a filter those two do not have.
+#
+# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the
+# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to
+# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same
+# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not
+# an internet-facing open resolver. Say so out loud rather than implying the filter is total.
+#
+# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved
+# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the
+# node-to-node forwarding that IS the Cluster-policy datapath both need them.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-lan-dns-ingress
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+    - fromEntities:
+        - world         # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
+        - host          # this node's kubelet probes
+        - remote-node   # the node-to-node hop that forwards a query to wherever the pod runs
+      toPorts:
+        - ports:
+            - { port: "53", protocol: UDP }
+            - { port: "53", protocol: TCP }

apps/wave-0/pihole.yaml

@@ -81,6 +81,15 @@ spec:
         image: pihole/pihole
         ports:
         - containerPort: 80
+        env:
+        - name: FTLCONF_dns_listeningMode
+          value: ALL
+        - name: FTLCONF_dns_upstreams
+          value: 8.8.8.8;8.8.4.4
+        - name: FTLCONF_dns_rateLimit_count
+          value: '0'
+        - name: FTLCONF_misc_dnsmasq_lines
+          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
         volumeMounts:
         - name: config
           mountPath: /data/config
@@ -91,16 +100,16 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:80
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://pihole.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -223,13 +232,14 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - pihole.home.keithjackson.dev
+  - pihole.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -239,26 +249,6 @@ spec:
     - name: pihole
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: pihole
-  namespace: pihole
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: pihole
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: pihole
-  namespaceSelector:
-    matchNames:
-    - pihole
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -273,11 +263,11 @@ spec:
   - name: pihole.slis
     rules:
     - alert: pihole-down
-      expr: up{job="pihole"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
       for: 5m
       labels:
         severity: critical
         service: pihole
       annotations:
         summary: pihole is down
-        description: 'pihole has been unreachable for 5m (SLI: up).'
+        description: 'pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/plex.yaml

@@ -35,6 +35,34 @@ spec:
     - protocol: TCP
       port: 53
 ---
+# PersistentVolume plex-media - static external NFS (bulk tier)
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: plex-media
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  capacity:
+    storage: 9Ti
+  accessModes:
+  - ReadOnlyMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ''
+  mountOptions:
+  - nfsvers=4.1
+  - hard
+  - noatime
+  - timeo=600
+  - retrans=2
+  nfs:
+    server: ${BULK_NFS_SERVER}
+    path: /mnt/Storage/plex
+    readOnly: true
+  claimRef:
+    namespace: plex
+    name: plex-media
+---
 # PVC plex-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -51,6 +79,23 @@ spec:
     requests:
       storage: 8Gi
 ---
+# PVC plex-media (R-STATE)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: plex-media
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  accessModes:
+  - ReadOnlyMany
+  storageClassName: ''
+  volumeName: plex-media
+  resources:
+    requests:
+      storage: 9Ti
+---
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -90,10 +135,16 @@ spec:
         volumeMounts:
         - name: config
           mountPath: /config
+        - name: media
+          mountPath: /media
+          readOnly: true
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: plex-config
+      - name: media
+        persistentVolumeClaim:
+          claimName: plex-media
 ---
 # Service
 apiVersion: v1
@@ -183,7 +234,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - plex.home.keithjackson.dev
+  - plex.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -193,26 +244,6 @@ spec:
     - name: plex
       port: 32400
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: plex
-  namespace: plex
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: plex
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: plex
-  namespaceSelector:
-    matchNames:
-    - plex
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -227,11 +258,11 @@ spec:
   - name: plex.slis
     rules:
     - alert: plex-down
-      expr: up{job="plex"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})
       for: 5m
       labels:
         severity: critical
         service: plex
       annotations:
         summary: plex is down
-        description: 'plex has been unreachable for 5m (SLI: up).'
+        description: 'plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml

@@ -0,0 +1,60 @@
+# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux.
+#
+# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13):
+#   qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR
+#   `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW
+#   random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
+#   password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's
+#   saved download-client credentials stop working at the next qbittorrent pod restart, silently
+#   breaking Completed Download Handling; (b) a human operator has no stable login at all and must
+#   read the pod log after every restart.
+#
+# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env
+#   (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as
+#   gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores
+#   and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then
+#   verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore
+#   from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME
+#   credential, because the hook re-asserts it on every start rather than once.
+#
+# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is
+#   the name its own temporary-password path uses, so keeping it avoids inventing a second identity.
+# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client
+#   entries store in their own databases (app state on their config PVCs, NOT in git — see
+#   DECISIONS-LOG), so rotating it here means re-pointing both arrs.
+#   Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON
+#   payloads, and a shell/URL-special character in any of those is a bug waiting to happen.
+#
+# NEVER put a value in a comment: SOPS does NOT encrypt comments.
+# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml
+apiVersion: v1
+kind: Secret
+metadata:
+    name: qbittorrent-bootstrap
+    namespace: qbittorrent
+    labels:
+        app.kubernetes.io/name: qbittorrent
+type: Opaque
+stringData:
+    webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str]
+    webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2
+            TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3
+            UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv
+            VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv
+            FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-13T05:23:23Z"
+    mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/qbittorrent.yaml

@@ -35,6 +35,33 @@ spec:
     - protocol: TCP
       port: 53
 ---
+# PersistentVolume qbittorrent-downloads - static external NFS (bulk tier)
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: qbittorrent-downloads
+  labels:
+    app.kubernetes.io/name: qbittorrent
+spec:
+  capacity:
+    storage: 9Ti
+  accessModes:
+  - ReadWriteMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ''
+  mountOptions:
+  - nfsvers=4.1
+  - hard
+  - noatime
+  - timeo=600
+  - retrans=2
+  nfs:
+    server: ${BULK_NFS_SERVER}
+    path: /mnt/Storage/plex
+  claimRef:
+    namespace: qbittorrent
+    name: qbittorrent-downloads
+---
 # PVC qbittorrent-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -51,6 +78,23 @@ spec:
     requests:
       storage: 5Gi
 ---
+# PVC qbittorrent-downloads (R-STATE)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: qbittorrent-downloads
+  namespace: qbittorrent
+  labels:
+    app.kubernetes.io/name: qbittorrent
+spec:
+  accessModes:
+  - ReadWriteMany
+  storageClassName: ''
+  volumeName: qbittorrent-downloads
+  resources:
+    requests:
+      storage: 9Ti
+---
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -74,16 +118,43 @@ spec:
         homelab.keithjackson.dev/auth-mode: forward-auth
       annotations:
         homelab.keithjackson.dev/auth-mode: forward-auth
-        homelab.keithjackson.dev/auth-groups: media-users
+        homelab.keithjackson.dev/auth-groups: media-users,platform-admins
     spec:
       containers:
       - name: qbittorrent
         image: lscr.io/linuxserver/qbittorrent:5.1.2
         ports:
         - containerPort: 8080
+        env:
+        - name: PUID
+          value: '1000'
+        - name: PGID
+          value: '3000'
+        - name: QBT_WEBUI_USERNAME
+          valueFrom:
+            secretKeyRef:
+              name: qbittorrent-bootstrap
+              key: webui-username
+        - name: QBT_WEBUI_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: qbittorrent-bootstrap
+              key: webui-password
         volumeMounts:
         - name: config
-          mountPath: /data/config
+          mountPath: /config
+        - name: downloads
+          mountPath: /downloads
+          subPath: downloads
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n  echo \"[qbt-webui-cred] $*\"\n  printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n    >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n#     before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n#     racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n  attempt=$((attempt + 1))\n\n  # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n  #     10 = rewritten, anything else = could not write.\n  python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n    sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n    raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n#   WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
+                , \"[Preferences]\"\n\ndef matches(raw):\n    \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n    v = raw.strip().strip('\"')\n    if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n        v = v[len(\"@ByteArray(\"):-1]\n    try:\n        salt_b64, digest_b64 = v.split(\":\", 1)\n        salt = base64.b64decode(salt_b64, validate=True)\n        digest = base64.b64decode(digest_b64, validate=True)\n    except Exception:\n        return False\n    if not salt or not digest:\n        return False\n    return hmac.compare_digest(\n        hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n    lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n    lines = []\n\ndef value_of(key):\n    for i, line in enumerate(lines):\n        if line.startswith(key + \"=\"):\n            return i, line.split(\"=\", 1)[1]\n    return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n    raise SystemExit(0)          # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n    at = out.index(SECTION) + 1\nelse:\n    if out and out[-1].strip():\n        out.append(\"\")\n    out.append(SECTION)\n    at = len(out)\nout[at:at] = desired\n\ntry:\n    os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n    tmp = conf + \".webui-cred.new\"\n    with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n        fh.write(\"\\n\".join(out) + \"\\n\")\n        fh.flush()\n        os.fsync(fh.fileno())\n    # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n    # cost the app the ability to save its own settings, so carry the ownership over.\n    try:\n        st = os.stat(conf)\n        os.chown(tmp, st.st_uid, st.st_gid)\n        os.chmod(tmp, stat.S_IMODE(st.st_mode))\n    except FileNotFoundError:\n        os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n                 int(os.environ.get(\"PGID\", \"1000\")))\n        os.chmod(tmp, 0o664)\n    os.replace(tmp, conf)        # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n    sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n    raise SystemExit(1)\nraise SystemExit(10)\nPY\n  rc=$?\n\n  if [ \"$rc\" = 0 ]; then\n    log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n  elif [ \"$rc\" = 10 ]; then\n    log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n    # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n    # have read the old, credential-less conf and be serving a random temporary password.\n    # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n    # Nothing to do when the service has not started yet - it will read the new file.\n    # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n    # 2026-09-13 failure spent the entire verification budget before the first login and\n    # then fired attempts 2 and 3 with no wait left at all.\n    sv=0\n    while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n      sv=$((sv + 1)); sleep 1\n    done\n    if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n      log \"qbittorrent-nox was already running - restarting it to load the credential\"\n      s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n      sleep 3\n    fi\n  else\n    log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n    log \"minting a random password\
+                \ every start and BOTH arrs will break. Fix and restart.\" >&2\n    exit 0\n  fi\n\n  # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n  #     (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n  #     at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n  #\n  # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n  # conflating them is what made this hook lie once already (see the PORT comment). An\n  # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n  # its remaining login attempts to produce two more identical wrong diagnoses.\n  api_ok=0\n  while [ \"$waited\" -lt 120 ]; do\n    code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n    if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n    waited=$((waited + 1)); sleep 1\n  done\n  if [ \"$api_ok\" != 1 ]; then\n    log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n    log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n    log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n    exit 0\n  fi\n  body=$(curl -sS -m 10 \\\n           --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n           --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n           \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n  if [ \"$body\" = \"Ok.\" ]; then\n    log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n    exit 0\n  fi\n  log \"WebUI rejected the durable credential (login said '$body')\" >&2\n  # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n  # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n  # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n  # make the process RE-READ the file, so restart the supervised service and let the next\n  # attempt re-verify. Without this the retry loop could never change its own outcome.\n  if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n    log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n    s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n    sleep 3\n  fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -91,16 +162,17 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8080
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
+        - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://qbittorrent.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -148,10 +220,15 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
+      securityContext:
+        fsGroup: 3000
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: qbittorrent-config
+      - name: downloads
+        persistentVolumeClaim:
+          claimName: qbittorrent-downloads
 ---
 # Service
 apiVersion: v1
@@ -266,14 +343,15 @@ metadata:
     app.kubernetes.io/name: qbittorrent
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
-    homelab.keithjackson.dev/auth-groups: media-users
+    homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - qbittorrent.home.keithjackson.dev
+  - qbittorrent.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -283,26 +361,6 @@ spec:
     - name: qbittorrent
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: qbittorrent
-  namespace: qbittorrent
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: qbittorrent
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: qbittorrent
-  namespaceSelector:
-    matchNames:
-    - qbittorrent
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -317,11 +375,11 @@ spec:
   - name: qbittorrent.slis
     rules:
     - alert: qbittorrent-down
-      expr: up{job="qbittorrent"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"} == 0 or absent(kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"})
       for: 5m
       labels:
         severity: critical
         service: qbittorrent
       annotations:
         summary: qbittorrent is down
-        description: 'qbittorrent has been unreachable for 5m (SLI: up).'
+        description: 'qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/semaphore-db-backup-credentials-prod.sops.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: semaphore-db-backup-credentials-prod
+    namespace: semaphore-db
+type: Opaque
+stringData:
+    ACCESS_KEY_ID: ENC[AES256_GCM,data:OhWEEYqrvmDB40NmiETgeUisBvaO9ocevCZAsurNijw=,iv:kEjOIg9NHbRYm+Ul1WrBAzb3gLbIv+ipiYCxFK0QoVc=,tag:KAB1JAIw0ATwbjDEKZVT9A==,type:str]
+    ACCESS_SECRET_KEY: ENC[AES256_GCM,data:C0PNKv07LCYpFONLqd55sz+ioyU/i2YJUkbZ8OX/dCErM/DilqchyUAT5EqnT/r3YJmdaA1pX85S3fxlVVHkbg==,iv:4ha28b+xIebvX2JPITmtTovIe4vdmj2XfI3cgkUHH4c=,tag:xbMyflcHojytPzgaoOYcXQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZVTJ3cXR6cks3Y0RNS2hx
+            K3pjZEduYUxmL0V3SmFUcU9XbGVHSVFITWh3Cjl1eHlTLytoOW5PaVNFVkJoeTBo
+            T1FnRHNDVzJ6dWpuR09WaElvbWF4YU0KLS0tIEJoQ1VsM1cxRlg0R1dtNGdOenNn
+            aVNjMjVBTytLdHlFdmZCS2tPelo1Y2MKYt1G5t+sDiHgcg+Q9ECBzJn2gS/paSEC
+            7aoX60SBrPgHvnGls5lzc3Tin9nH4jWYl6Nu8DAvIE0YPXDFr5a0UA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-07T06:36:33Z"
+    mac: ENC[AES256_GCM,data:mhVP7tjdZK2FR2imvpJbTyT7BigvcIKEpLSnBS1oSITUirB8xZC7aHBfyFuIoKOrpOJgEe40nHUdDYCHRJtb4dLeVja8+7meVaQWzxC8BWVHsksW+73hsqhjAzlHeobx6Irbk4PAMqIY0/jnf4MHryjnRn3dUuwu9U1CqL30JbE=,iv:57oCcp6FW9EAIVKnYGtkaTrnbijyv54T+vmZxMbMqnw=,tag:agrRP/JLypU60KOeN0s/5w==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/semaphore-db.yaml

@@ -38,16 +38,53 @@ metadata:
   name: semaphore-db
   namespace: semaphore-db
 spec:
-  # Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
+  # THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db /
-  # keycloak-db / nextcloud-db). The storage is already 3-way replicated by Longhorn; raise to 3 for
+  # nextcloud-db; see platform/keycloak/postgres-cluster.yaml for the full rationale. Short version:
-  # Postgres-level HA as a future hardening.
+  # Longhorn's 3-way replication saves the DATA on node loss, not the SERVICE, and with
-  instances: 1
+  # `instances: 1` all three CNPG primaries had drifted onto k3s-prod-2, so one node took Semaphore
+  # down alongside SSO and Nextcloud. A streaming standby lets CNPG promote in seconds and repoint
+  # the -rw Service. Gated on the Barman archive below actually working (ContinuousArchiving=True
+  # with a real firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
+  #
+  # Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
+  # pg_basebackup from the running primary, which is neither restarted nor disconnected.
+  instances: 3
   imageName: ghcr.io/cloudnative-pg/postgresql:16.4
   # Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
   # semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
   inheritedMetadata:
     labels:
       app.kubernetes.io/name: semaphore-db
+  # HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
+  # primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
+  # recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
+  # version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
+  # after a node loss it may stack two instances on one node — which is how `instances: 3` decays
+  # back into the single-node failure domain this change removes. `required` makes it a constraint;
+  # the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
+  # serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
+  #
+  # APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
+  # switchover. Expect a single brief connection reset for Semaphore.
+  affinity:
+    enablePodAntiAffinity: true
+    podAntiAffinityType: required
+  primaryUpdateMethod: switchover
+  # SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
+  # rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
+  # last few transactions when a primary dies before its standbys catch up, and `method: any` +
+  # `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
+  # never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
+  # are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
+  # standby into a total write outage. `preferred` relaxes to async instead.
+  #
+  # NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
+  # no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
+  postgresql:
+    synchronous:
+      method: any
+      number: 1
+      dataDurability: preferred
   storage:
     storageClass: longhorn
     size: 5Gi
@@ -69,25 +106,38 @@ spec:
       memory: 256Mi
     limits:
       memory: 512Mi
-  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
+  # --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
   # Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
-  # backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
+  # backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
-  # (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP.
+  # destinationPath, so objects land under ${CNPG_BACKUP_PATH}/semaphore-db/ — dev
-  # Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
+  # s3://cnpg/semaphore-db/ on the on-Proxmox MinIO, prod
-  # (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
+  # s3://k3s-prod-backup/barman/semaphore-db/ on Cloudflare R2. Both endpointURL and
-  # endpointURL is ${S3_BACKUP_ENDPOINT} (cluster-vars, R-ENV-02; dev http://192.168.1.9:9000),
+  # destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
-  # substituted by the `apps-wave-0` Flux Kustomization. Prod supplies its own off-cluster target.
+  # Credentials: PER-ENVIRONMENT (2026-09-07). dev reads semaphore-db-backup-credentials (the
+  # scoped MinIO key), prod reads semaphore-db-backup-credentials-prod (the Cloudflare R2
+  # token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
+  # generated golden — render.py places the -prod sidecar in this DB's wave); the
+  # cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
+  # one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
+  # ONE shared Secret until the split, so giving prod its R2 token would have left dev
+  # signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
+  # disk-full clock below rather than a merely missing backup.
+  #
+  # archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
+  # Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
+  # the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
+  # endpoint. Treat ContinuousArchiving=False here as an availability incident.
   backup:
     retentionPolicy: "30d"
     barmanObjectStore:
-      destinationPath: s3://cnpg
+      destinationPath: ${CNPG_BACKUP_PATH}
       endpointURL: ${S3_BACKUP_ENDPOINT}
       s3Credentials:
         accessKeyId:
-          name: semaphore-db-backup-credentials
+          name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
           key: ACCESS_KEY_ID
         secretAccessKey:
-          name: semaphore-db-backup-credentials
+          name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
           key: ACCESS_SECRET_KEY
       wal:
         compression: gzip

apps/wave-0/tests/app-slis.test.yaml

@@ -0,0 +1,267 @@
+# promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
+# Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
+# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
+# the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
+# tests run against the exact expressions Flux applies, never a copy.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHY THESE EXIST. Until 2026-09-13 every one of these rules read `up{job="<app>"} == 0` while the
+# rendered ServiceMonitor named a Service port (`metrics`) that no rendered Service defined. The
+# monitor discovered ZERO targets, so no `up{job="<app>"}` series existed and the rule could not
+# fire — on prod, `count by (job) (up)` listed 24 jobs and not one of the twelve appeared.
+# `PrometheusTargetsMissing` could not see it either: a target that was never discovered has no `up`
+# series to be 0. The rules now read kube-state-metrics / kubevirt series that DO exist for these
+# workloads (verified on prod), and every rule below is pinned in BOTH directions:
+#   * QUIET on the healthy state -> it is not permanently firing;
+#   * FIRING when the workload has no ready replica -> it is not inert;
+#   * FIRING when the signal is ABSENT -> the failure mode that produced this whole defect class
+#     (a rule that goes quiet because its series vanished) is itself alerted on.
+#
+# CLOCK/`for` CONVENTION: sample interval 1m, series 21 samples long, evaluated at 10m — comfortably
+# past every rule's `for: 5m` and well inside Prometheus' 5m lookback.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # ===========================================================================================
+  # 1. HEALTHY WAVE-0 (what prod looks like right now) -> every rule SILENT.
+  #    This is the "not permanently firing" half. It is also the test that catches a mis-spelled
+  #    label: get `deployment=` wrong and the `== 0` half matches nothing while the `absent()` half
+  #    matches everything, so this test goes red instead of the rule going quietly wrong.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 healthy - every app has a ready replica, the demo VM runs - all quiet
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running", node="k3s-prod-1"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 2. EACH APP DOWN IN TURN (0 available replicas) -> ONLY that app's rule fires.
+  #    The per-app isolation matters: a rule that forgot its `namespace=`/`deployment=` matchers
+  #    would light up every app at once and be useless for routing.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 plex has no available replica - only plex-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: plex
+              namespace: plex
+              deployment: plex
+            exp_annotations:
+              summary: "plex is down"
+              description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+    # Before `for: 5m` has elapsed the rule must be PENDING, not firing — proof the 5m is real and
+    # a single scrape blip does not page anyone.
+    promql_expr_test:
+      - expr: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})'
+        eval_time: 2m
+        exp_samples:
+          - labels: 'kube_deployment_status_replicas_available{deployment="plex", namespace="plex"}'
+            value: 0
+
+  - interval: 1m
+    name: wave-0 gitea/jackett/pihole/qbittorrent each down - each own rule fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '0+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: gitea
+              namespace: gitea
+              deployment: gitea
+            exp_annotations:
+              summary: "gitea is down"
+              description: "gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: jackett
+              namespace: jackett
+              deployment: jackett
+            exp_annotations:
+              summary: "jackett is down"
+              description: "jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: pihole
+              namespace: pihole
+              deployment: pihole
+            exp_annotations:
+              summary: "pihole is down"
+              description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: qbittorrent
+              namespace: qbittorrent
+              deployment: qbittorrent
+            exp_annotations:
+              summary: "qbittorrent is down"
+              description: "qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 3. THE SIGNAL VANISHES -> the rule FIRES. This is the defect class itself, asserted on.
+  #    The old `up{job=...} == 0` rule had exactly this shape of hole and was silent for it. Here
+  #    the Deployment series for plex is simply not in the input at all, and plex-down must still
+  #    fire (via the `absent()` half), with the labels the selector's equality matchers carry.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 plex Deployment series ABSENT (never scraped / workload gone) - plex-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: plex
+              namespace: plex
+              deployment: plex
+            exp_annotations:
+              summary: "plex is down"
+              description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 4. THE VM: a VMI carries no replica count, so liveness is the EXISTENCE of a running-phase
+  #    series. Two ways to be down, both asserted: the VMI is gone entirely, and the VMI exists in
+  #    some OTHER phase. The second is the one a naive `kubevirt_vmi_info == 1` would miss, and it
+  #    is why the phase value is matched — LOWERCASE `running`, the value KubeVirt actually writes
+  #    (a capital-R matcher selects nothing and would be just as inert).
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 demo VM has no running-phase VMI - vm-demo-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: vm-demo
+              namespace: vm-demo
+              name: vm-demo
+              phase: running
+            exp_annotations:
+              summary: "vm-demo is down"
+              description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."
+
+  - interval: 1m
+    name: wave-0 demo VM exists but is Failed, not running - vm-demo-down still fires
+    input_series:
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="failed"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: vm-demo
+              namespace: vm-demo
+              name: vm-demo
+              phase: running
+            exp_annotations:
+              summary: "vm-demo is down"
+              description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."

apps/wave-0/vm-demo.yaml

@@ -0,0 +1,139 @@
+---
+# Namespace (trust zone; default-deny + policies apply)
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: vm-demo
+  labels:
+    homelab.keithjackson.dev/app-namespace: 'true'
+  annotations:
+    homelab.keithjackson.dev/backup-policy: daily-offcluster
+---
+# Default-deny baseline (R-NET-01): deny all, allow only DNS
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: default-deny
+  namespace: vm-demo
+  labels:
+    homelab.keithjackson.dev/baseline: default-deny
+spec:
+  podSelector: {}
+  policyTypes:
+  - Ingress
+  - Egress
+  ingress: []
+  egress:
+  - to:
+    - namespaceSelector:
+        matchLabels:
+          kubernetes.io/metadata.name: kube-system
+    ports:
+    - protocol: UDP
+      port: 53
+    - protocol: TCP
+      port: 53
+---
+# CDI DataVolume (standalone; survives rebuild, R-STATE-01)
+apiVersion: cdi.kubevirt.io/v1beta1
+kind: DataVolume
+metadata:
+  name: vm-demo-rootdisk
+  namespace: vm-demo
+spec:
+  source:
+    registry:
+      url: docker://quay.io/containerdisks/fedora:41
+  storage:
+    accessModes:
+    - ReadWriteOnce
+    storageClassName: longhorn
+    resources:
+      requests:
+        storage: 6Gi
+---
+# KubeVirt VirtualMachine (R-PLAT-03)
+apiVersion: kubevirt.io/v1
+kind: VirtualMachine
+metadata:
+  name: vm-demo
+  namespace: vm-demo
+  labels:
+    app.kubernetes.io/name: vm-demo
+spec:
+  running: false
+  template:
+    metadata:
+      labels:
+        kubevirt.io/vm: vm-demo
+        homelab.keithjackson.dev/egress-class: internet
+        homelab.keithjackson.dev/auth-mode: none
+    spec:
+      domain:
+        cpu:
+          cores: 1
+        memory:
+          guest: 1536Mi
+        devices:
+          disks:
+          - name: rootdisk
+            disk:
+              bus: virtio
+          - name: cloudinit
+            disk:
+              bus: virtio
+          interfaces:
+          - name: default
+            masquerade: {}
+        resources:
+          requests:
+            memory: 1536Mi
+      networks:
+      - name: default
+        pod: {}
+      volumes:
+      - name: rootdisk
+        dataVolume:
+          name: vm-demo-rootdisk
+      - name: cloudinit
+        cloudInitNoCloud:
+          userData: "#cloud-config\npackage_update: true\npackages:\n  - dnf-automatic\nwrite_files:\n  - path: /etc/dnf/automatic.conf\n    content: |\n      [commands]\n      upgrade_type = default\n      random_sleep = 0\n      download_updates = yes\n      apply_updates = yes\nruncmd:\n  - [ systemctl, enable, --now, dnf-automatic.timer ]\n"
+---
+# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): vm-demo-allow-internet-egress
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: vm-demo-allow-internet-egress
+  namespace: vm-demo
+  labels:
+    app.kubernetes.io/name: vm-demo
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: vm-demo
+  egress:
+  - toEntities:
+    - world
+---
+# PrometheusRule SLI alerts (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: vm-demo-slis
+  namespace: vm-demo
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: vm-demo
+spec:
+  groups:
+  - name: vm-demo.slis
+    rules:
+    - alert: vm-demo-down
+      expr: absent(kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"})
+      for: 5m
+      labels:
+        severity: critical
+        service: vm-demo
+      annotations:
+        summary: vm-demo is down
+        description: 'vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info).'

apps/wave-1/gitea-actions-runner-secret.sops.yaml

@@ -14,7 +14,7 @@ metadata:
         app.kubernetes.io/name: act-runner
 type: Opaque
 stringData:
-    token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
+    token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -30,8 +30,8 @@ sops:
             MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
             UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-08-26T04:48:14Z"
+    lastmodified: "2026-09-29T05:48:35Z"
-    mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
+    mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

apps/wave-1/gitea-actions.yaml

@@ -165,7 +165,7 @@ spec:
               exec /sbin/tini -- /opt/act/run.sh
           env:
             - name: GITEA_INSTANCE_URL
-              value: https://gitea.home.keithjackson.dev
+              value: https://gitea.${BASE_DOMAIN}
             - name: GITEA_RUNNER_REGISTRATION_TOKEN
               valueFrom:
                 secretKeyRef:

apps/wave-1/keycloak.yaml

@@ -0,0 +1,208 @@
+---
+# Namespace (trust zone; default-deny + policies apply)
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: keycloak
+  labels:
+    homelab.keithjackson.dev/app-namespace: 'true'
+    gateway-access: shared
+  annotations:
+    homelab.keithjackson.dev/backup-policy: daily-offcluster
+---
+# Default-deny baseline (R-NET-01): deny all, allow only DNS
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: default-deny
+  namespace: keycloak
+  labels:
+    homelab.keithjackson.dev/baseline: default-deny
+spec:
+  podSelector: {}
+  policyTypes:
+  - Ingress
+  - Egress
+  ingress: []
+  egress:
+  - to:
+    - namespaceSelector:
+        matchLabels:
+          kubernetes.io/metadata.name: kube-system
+    ports:
+    - protocol: UDP
+      port: 53
+    - protocol: TCP
+      port: 53
+---
+# PVC keycloak-db (R-STATE)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: keycloak-db
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  accessModes:
+  - ReadWriteOnce
+  storageClassName: longhorn
+  resources:
+    requests:
+      storage: 10Gi
+---
+# Deployment (R-PLAT-02)
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: keycloak
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: keycloak
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: keycloak
+        homelab.keithjackson.dev/egress-class: internet
+        homelab.keithjackson.dev/auth-mode: native
+    spec:
+      containers:
+      - name: keycloak
+        image: quay.io/keycloak/keycloak
+        ports:
+        - containerPort: 8080
+        volumeMounts:
+        - name: db
+          mountPath: /data/db
+      volumes:
+      - name: db
+        persistentVolumeClaim:
+          claimName: keycloak-db
+---
+# Service
+apiVersion: v1
+kind: Service
+metadata:
+  name: keycloak
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  selector:
+    app.kubernetes.io/name: keycloak
+  ports:
+  - name: port-8080
+    port: 8080
+    targetPort: 8080
+---
+# CiliumNetworkPolicy egress allow (R-NET-02): keycloak-allow-keycloak-db-5432
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: keycloak-allow-keycloak-db-5432
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: keycloak
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: keycloak-db
+        k8s:io.kubernetes.pod.namespace: keycloak-db
+    toPorts:
+    - ports:
+      - port: '5432'
+        protocol: TCP
+---
+# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): keycloak-allow-gateway-ingress
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: keycloak-allow-gateway-ingress
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: keycloak
+  ingress:
+  - fromEntities:
+    - ingress
+    toPorts:
+    - ports:
+      - port: '8080'
+        protocol: TCP
+---
+# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): keycloak-allow-internet-egress
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: keycloak-allow-internet-egress
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: keycloak
+  egress:
+  - toEntities:
+    - world
+---
+# HTTPRoute keycloak-8080 (R-TLS/R-DNS)
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: keycloak-8080
+  namespace: keycloak
+  labels:
+    app.kubernetes.io/name: keycloak
+spec:
+  parentRefs:
+  - name: shared
+    namespace: gateway
+    sectionName: https
+  hostnames:
+  - keycloak.${BASE_DOMAIN}
+  rules:
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /
+    backendRefs:
+    - name: keycloak
+      port: 8080
+---
+# PrometheusRule SLI alerts (R-OBS-01)
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: keycloak-slis
+  namespace: keycloak
+  labels:
+    release: kube-prometheus-stack
+    app.kubernetes.io/name: keycloak
+spec:
+  groups:
+  - name: keycloak.slis
+    rules:
+    - alert: keycloak-down
+      expr: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"})
+      for: 5m
+      labels:
+        severity: critical
+        service: keycloak
+      annotations:
+        summary: keycloak is down
+        description: 'keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'

apps/wave-1/nextcloud.yaml

@@ -35,6 +35,34 @@ spec:
     - protocol: TCP
       port: 53
 ---
+# PersistentVolume nextcloud-legacy-archive - static external NFS (bulk tier)
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: nextcloud-legacy-archive
+  labels:
+    app.kubernetes.io/name: nextcloud
+spec:
+  capacity:
+    storage: 270Gi
+  accessModes:
+  - ReadOnlyMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ''
+  mountOptions:
+  - nfsvers=4.1
+  - hard
+  - noatime
+  - timeo=600
+  - retrans=2
+  nfs:
+    server: ${BULK_NFS_SERVER}
+    path: /mnt/Storage/nextcloud
+    readOnly: true
+  claimRef:
+    namespace: nextcloud
+    name: nextcloud-legacy-archive
+---
 # PVC nextcloud-data (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -51,6 +79,23 @@ spec:
     requests:
       storage: 20Gi
 ---
+# PVC nextcloud-legacy-archive (R-STATE)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: nextcloud-legacy-archive
+  namespace: nextcloud
+  labels:
+    app.kubernetes.io/name: nextcloud
+spec:
+  accessModes:
+  - ReadOnlyMany
+  storageClassName: ''
+  volumeName: nextcloud-legacy-archive
+  resources:
+    requests:
+      storage: 270Gi
+---
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -104,13 +149,13 @@ spec:
               name: nextcloud-bootstrap
               key: admin-password
         - name: NEXTCLOUD_TRUSTED_DOMAINS
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
         - name: OVERWRITEHOST
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
         - name: OVERWRITEPROTOCOL
           value: https
         - name: OVERWRITECLIURL
-          value: https://nextcloud.home.keithjackson.dev
+          value: https://nextcloud.${BASE_DOMAIN}
         - name: NEXTCLOUD_OIDC_CLIENT_SECRET
           valueFrom:
             secretKeyRef:
@@ -119,18 +164,26 @@ spec:
         volumeMounts:
         - name: data
           mountPath: /var/www/html
+        - name: legacy-archive
+          mountPath: /mnt/legacy-nextcloud
+          subPath: nextcloud
+          readOnly: true
         lifecycle:
           postStart:
             exec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra\
+              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc]\
-                \ positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"$ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n"
+                \ client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
+                ); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
       volumes:
       - name: data
         persistentVolumeClaim:
           claimName: nextcloud-data
+      - name: legacy-archive
+        persistentVolumeClaim:
+          claimName: nextcloud-legacy-archive
 ---
 # Service
 apiVersion: v1
@@ -227,7 +280,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - nextcloud.home.keithjackson.dev
+  - nextcloud.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -237,26 +290,6 @@ spec:
     - name: nextcloud
       port: 80
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: nextcloud
-  namespace: nextcloud
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: nextcloud
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: nextcloud
-  namespaceSelector:
-    matchNames:
-    - nextcloud
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -271,11 +304,11 @@ spec:
   - name: nextcloud.slis
     rules:
     - alert: nextcloud-down
-      expr: up{job="nextcloud"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"} == 0 or absent(kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"})
       for: 5m
       labels:
         severity: critical
         service: nextcloud
       annotations:
         summary: nextcloud is down
-        description: 'nextcloud has been unreachable for 5m (SLI: up).'
+        description: 'nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/radarr.yaml

@@ -35,6 +35,33 @@ spec:
     - protocol: TCP
       port: 53
 ---
+# PersistentVolume radarr-media - static external NFS (bulk tier)
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: radarr-media
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  capacity:
+    storage: 9Ti
+  accessModes:
+  - ReadWriteMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ''
+  mountOptions:
+  - nfsvers=4.1
+  - hard
+  - noatime
+  - timeo=600
+  - retrans=2
+  nfs:
+    server: ${BULK_NFS_SERVER}
+    path: /mnt/Storage/plex
+  claimRef:
+    namespace: radarr
+    name: radarr-media
+---
 # PVC radarr-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -51,6 +78,23 @@ spec:
     requests:
       storage: 5Gi
 ---
+# PVC radarr-media (R-STATE)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: radarr-media
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  accessModes:
+  - ReadWriteMany
+  storageClassName: ''
+  volumeName: radarr-media
+  resources:
+    requests:
+      storage: 9Ti
+---
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -74,16 +118,25 @@ spec:
         homelab.keithjackson.dev/auth-mode: forward-auth
       annotations:
         homelab.keithjackson.dev/auth-mode: forward-auth
-        homelab.keithjackson.dev/auth-groups: media-users
+        homelab.keithjackson.dev/auth-groups: media-users,platform-admins
     spec:
       containers:
       - name: radarr
         image: lscr.io/linuxserver/radarr:5.21.1
         ports:
         - containerPort: 7878
+        env:
+        - name: PUID
+          value: '1000'
+        - name: PGID
+          value: '3000'
+        - name: TZ
+          value: Etc/UTC
         volumeMounts:
         - name: config
-          mountPath: /data/config
+          mountPath: /config
+        - name: media
+          mountPath: /media
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -91,16 +144,17 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:7878
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
+        - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://radarr.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -148,10 +202,15 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
+      securityContext:
+        fsGroup: 3000
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: radarr-config
+      - name: media
+        persistentVolumeClaim:
+          claimName: radarr-media
 ---
 # Service
 apiVersion: v1
@@ -288,14 +347,15 @@ metadata:
     app.kubernetes.io/name: radarr
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
-    homelab.keithjackson.dev/auth-groups: media-users
+    homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - radarr.home.keithjackson.dev
+  - radarr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -305,26 +365,6 @@ spec:
     - name: radarr
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: radarr
-  namespace: radarr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: radarr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: radarr
-  namespaceSelector:
-    matchNames:
-    - radarr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -339,11 +379,11 @@ spec:
   - name: radarr.slis
     rules:
     - alert: radarr-down
-      expr: up{job="radarr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"})
       for: 5m
       labels:
         severity: critical
         service: radarr
       annotations:
         summary: radarr is down
-        description: 'radarr has been unreachable for 5m (SLI: up).'
+        description: 'radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/semaphore-bootstrap-secret.sops.yaml

@@ -5,6 +5,15 @@
 #   credential from its own namespace (a Pod cannot read a Secret in another namespace).
 # access-key-encryption: FIXED base64 32-byte AES key Semaphore uses to encrypt stored SSH keys /
 #   credentials; MUST stay stable across restarts or stored secrets become undecryptable.
+# oidc-client-secret: the Keycloak `semaphore` CONFIDENTIAL client secret, and ONLY that. It MIRRORS
+#   `semaphore-client-secret` in platform/keycloak/keycloak-realm-clients.sops.yaml (the realm import
+#   injects that copy into the client), same cross-namespace mirror pattern as db-password above.
+#   It replaced the former `oidc-providers-json` key (2026-09-07): that key held the WHOLE
+#   SEMAPHORE_OIDC_PROVIDERS blob, which buried the environment-specific realm-issuer and redirect
+#   URLs inside ciphertext where Flux's ${BASE_DOMAIN} postBuild substitution could not reach them,
+#   so prod ran with dev's URLs. Those URLs are PUBLIC and now live in plaintext in
+#   catalog/semaphore.yaml (rendered to apps/wave-1/semaphore.yaml); the kubelet splices THIS value
+#   back into the blob via $(SEMAPHORE_OIDC_CLIENT_SECRET). Only the secret half is encrypted.
 # Recover: infra/devcontainer/dev.sh run sops -d apps/semaphore-bootstrap-secret.sops.yaml
 apiVersion: v1
 kind: Secret
@@ -15,11 +24,11 @@ metadata:
         app.kubernetes.io/name: semaphore
 type: Opaque
 stringData:
-    admin-username: ENC[AES256_GCM,data:uD/vO9A9MjerFg==,iv:i/NAqVD6rWg9oWWSGBrRkxzwcPp2IW1HLIKY0pdcPPg=,tag:r27tKLAJ+ydE7Ty3ccMNUw==,type:str]
+    admin-username: ENC[AES256_GCM,data:ju3wVdjNzRUP/w==,iv:d/5ZxgQZ5bympu3gT2ck3DJy9iUPPBAtZYsJvD3To7I=,tag:0UtBaDVqNSK4RCJrQicchw==,type:str]
-    admin-password: ENC[AES256_GCM,data:544nWY1UmBnlsdPJAQZJMXRumu4W9X7UXZvZ6vHN/ArjRgLXKTArOe5JkT/nNrNw,iv:AC1DDbZj2HfpU19EjzA7T+WpIaR4nZc3PNN61y8bAzU=,tag:r28LFi5w6ZeCCaRaX2jRdg==,type:str]
+    admin-password: ENC[AES256_GCM,data:pxIwpgh3BoND2TI0mPTixOTJoictzT77dqtuRdMKgY/2WaW4wM1Ra6H6n/dxnyR2,iv:eYbFQGFYVJsnYGwn6cOI3BgAdzMcin9YggAhiuL+xqM=,tag:/sPWGv4gue0vjc2nMk7wWA==,type:str]
-    db-password: ENC[AES256_GCM,data:ySkdvaGRCW9fMQqChs5l8dY4zCG19i4HOEinRC48mgLuh9WOrjd16T6jwFgiwB55,iv:+1gztJ22HbwgQJiolZMMWOcKIsxrBJpnruJdtxdooOU=,tag:8CfJGwyPTjDGyr0CvalXMQ==,type:str]
+    db-password: ENC[AES256_GCM,data:hBFpWXdpk1R5a23Pntup03t56NBRPS+SZU8CrHVb/ypIaIpgf7kjYtVy6nhTF6u0,iv:eBzoQgH5ZhWWpfKZ6AQmwaSal5QaiuN9gfexQ7GsVFs=,tag:j2k8EwhKDlj/Ud6O39ioVQ==,type:str]
-    access-key-encryption: ENC[AES256_GCM,data:1YAMLo1nVkpJWZc+KBZ7B4ia+Y8P+xMyOhHMcEGGJOXd3o+WC5FqirLBVh4=,iv:j28ozeaDhGWDjoaxFOhn2t9YbYE7L3ah/Mfi+i2DWwY=,tag:czG9Wxc/7Nsuh1W5iATOww==,type:str]
+    access-key-encryption: ENC[AES256_GCM,data:SKtAV4HxJ9pNYDqUK/1uOdI1rtMji3pTfprI2rhQxFQyhc6dFlhqI+OUEcY=,iv:rKCP+FLidcOltOwmMMcO5M8tz4qiifIMI+W5soGO2M8=,tag:HaHzkX/JI+9wmcg2SD9GAQ==,type:str]
-    oidc-providers-json: ENC[AES256_GCM,data:90yt5mIVWLVI2KrlJWpxhL/BMDpEgRhv7+BL+mBRt79diRvU7pN9vtzhZprAKzcU1sEywlT4EwFlsvJpYyo5Qbkp+ZVknbyhAlGvHhJOBHukOPWgf4LignLerO2GzhXjOOzFqWfZ4EyUxNMWTbSe/xc0zIVHDDNQiYplGVRK+E/ukiI7sD/Tn2RrbrdvpC5GyK0NXWRsXn9ffhE3fCoiRHXagj9i/q+uUrQI5EKpPobrDUAGqvKoBXTl/wJ4phnWg9IqtCTE/c3yA7fLLz4j5dCtPZNSVPT4TKDx8Mdv8CfCgziVicjG2s+yHuWpsQn8lf6nvdJmpYPvIqTtLWdxFWuD8Vfo6Qbq2mqPbKR+KmDOGUnjl4gbZdL1SkE9JiqBUobz7UXeLXwYzxMIsinOfYZLcCk7alrD5Yndj9JDxDJZQ54sMMBfVbxDl94W1VHy03RxAR46KKAbPaxOP5wH4Q4z3SDceNwmuGCDRGChRXiHhN5X2X0GqYCbc8E6GdkIMcpmD80e26ydEmAjw7F89mCF4F/yMzJNiG82qaZIn8HeqgGA8urous9r9bYJNlitYTYQ,iv:jbfVdrrZ9fGiOxhQeoTJfFg1QwiOMkoQyaadiVfxX/w=,tag:JkepcRs34EuEgNb4buTTDQ==,type:str]
+    oidc-client-secret: ENC[AES256_GCM,data:uCo7TNzbM2QKw9ie+xzyUG9MFKco1OAlwr5qPn/AGpNvo1YyXSfbZyyM0CLJ1NS821hGg5Zr/m3R/xpy84s3FQ==,iv:O8a4haskufpmorPaxOkJ1Q99Y5GTlzmDFJAsHS+irLk=,tag:VojJpA9jNMbNQmIe1Qr44w==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -29,14 +38,14 @@ sops:
         - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuRDlqRFJ0dk14UHluSGRP
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyTzF6NTBtcFVWK1Vtc0I2
-            S1pESy9PWlpMakU0aTkzNVhqUGdtbWNEdTJjCm5RNG14azJHc28rdEJ4L0RzR0xO
+            VGdySHBsRGgvc1h0VzgvQ01xYVBJV3g2Q1NRCjJyZHVYZGtuamo1MFJHWlBwWkxt
-            MGRocGZ1V1JZTENWQkN3VGZzamp2eTgKLS0tIGt6QkNNaW5hdTBJVHY0SVo5d3JX
+            cHA4VkY1TkRpN3M4UTdMOWZTbFVGemMKLS0tIGVNdDBrWVJveU56VjVBU2JWdHF4
-            elp0SWxHcjg0RWF3aW9mWTJHZ0lhcUEKc0rU5VilAVDleCwqq4SOF+Y9sniu1Pka
+            RGtaYStXYUpONTZjZCtmM0NyZ3Rka1UKIQpmQai8dz+uRAsXxyeuGaRlRwsFqIPQ
-            w/MGDA9THSB0b+YgWRWiX/KiimFqp0ZFODOJ9cHVwFGY71+VadPwOg==
+            9AfrZJcslYwlaK3ptm1oqaCdrlxcMar2nI7thSqzbLe+e1nzA2z7vQ==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-08-27T22:12:32Z"
+    lastmodified: "2026-09-07T07:01:50Z"
-    mac: ENC[AES256_GCM,data:nHftJTGnTr0O5h8yGXwgz3eflggXJfIXM3m8n3bzDJE/P4nvSpZzFeMb+sc2on+AL+t+SeKrIl8blzm33XsQ1lkaY85Zm+bc5cnED1g04BMDVmmK3brh/8w50Bp2wPEVzb4I01MDzhhyIbOYg7pFvZAH3/VuqvM1t6XmfF1NrOA=,iv:4KBa9PH/dfdGwLC16RenZIVB6BtSoGo6x3qt3h1R43I=,tag:KULiA1klP/RZMNigkloGDA==,type:str]
+    mac: ENC[AES256_GCM,data:Xutm1rQt57J4RILElD8FkNOsQT1HK5jZlPtqqLyZ/VqOoDwo+IS1wp7hROprlUvQC0bUXy79tRRIl5Q0CCOD+fvAGIdmade+vB7qXlYN6Q1441dGhuSTPvckso6FaB2ddRjKhKGLc/Z4bJoz9VaRNNxJNpWXIrENTg+I+QhNhlo=,iv:klJOrEQ6QN5dTGkQfGnbMbgCWsgWwZPb5YXDMeWeE34=,tag:fuJr4t3Tt3dT9u3/0SFo8Q==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

apps/wave-1/semaphore.yaml

@@ -110,7 +110,7 @@ spec:
         - name: SEMAPHORE_ADMIN_NAME
           value: Semaphore Admin
         - name: SEMAPHORE_ADMIN_EMAIL
-          value: admin@home.keithjackson.dev
+          value: admin@${BASE_DOMAIN}
         - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
           valueFrom:
             secretKeyRef:
@@ -118,11 +118,13 @@ spec:
               key: access-key-encryption
         - name: SEMAPHORE_PORT
           value: :3000
-        - name: SEMAPHORE_OIDC_PROVIDERS
+        - name: SEMAPHORE_OIDC_CLIENT_SECRET
           valueFrom:
             secretKeyRef:
               name: semaphore-bootstrap
-              key: oidc-providers-json
+              key: oidc-client-secret
+        - name: SEMAPHORE_OIDC_PROVIDERS
+          value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
         volumeMounts:
         - name: config
           mountPath: /etc/semaphore
@@ -225,13 +227,14 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: oidc
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - semaphore.home.keithjackson.dev
+  - semaphore.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -241,26 +244,6 @@ spec:
     - name: semaphore
       port: 3000
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: semaphore
-  namespace: semaphore
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: semaphore
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: semaphore
-  namespaceSelector:
-    matchNames:
-    - semaphore
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -275,11 +258,11 @@ spec:
   - name: semaphore.slis
     rules:
     - alert: semaphore-down
-      expr: up{job="semaphore"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"} == 0 or absent(kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"})
       for: 5m
       labels:
         severity: critical
         service: semaphore
       annotations:
         summary: semaphore is down
-        description: 'semaphore has been unreachable for 5m (SLI: up).'
+        description: 'semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/sonarr.yaml

@@ -35,6 +35,33 @@ spec:
     - protocol: TCP
       port: 53
 ---
+# PersistentVolume sonarr-media - static external NFS (bulk tier)
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: sonarr-media
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  capacity:
+    storage: 9Ti
+  accessModes:
+  - ReadWriteMany
+  persistentVolumeReclaimPolicy: Retain
+  storageClassName: ''
+  mountOptions:
+  - nfsvers=4.1
+  - hard
+  - noatime
+  - timeo=600
+  - retrans=2
+  nfs:
+    server: ${BULK_NFS_SERVER}
+    path: /mnt/Storage/plex
+  claimRef:
+    namespace: sonarr
+    name: sonarr-media
+---
 # PVC sonarr-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
@@ -51,6 +78,23 @@ spec:
     requests:
       storage: 5Gi
 ---
+# PVC sonarr-media (R-STATE)
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: sonarr-media
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  accessModes:
+  - ReadWriteMany
+  storageClassName: ''
+  volumeName: sonarr-media
+  resources:
+    requests:
+      storage: 9Ti
+---
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
 kind: Deployment
@@ -74,16 +118,25 @@ spec:
         homelab.keithjackson.dev/auth-mode: forward-auth
       annotations:
         homelab.keithjackson.dev/auth-mode: forward-auth
-        homelab.keithjackson.dev/auth-groups: media-users
+        homelab.keithjackson.dev/auth-groups: media-users,platform-admins
     spec:
       containers:
       - name: sonarr
         image: lscr.io/linuxserver/sonarr:4.0.15
         ports:
         - containerPort: 8989
+        env:
+        - name: PUID
+          value: '1000'
+        - name: PGID
+          value: '3000'
+        - name: TZ
+          value: Etc/UTC
         volumeMounts:
         - name: config
-          mountPath: /data/config
+          mountPath: /config
+        - name: media
+          mountPath: /media
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -91,16 +144,17 @@ spec:
         - --http-address=0.0.0.0:4180
         - --reverse-proxy=true
         - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
         - --upstream=http://127.0.0.1:8989
         - --email-domain=*
         - --scope=openid email profile
         - --oidc-groups-claim=groups
         - --allowed-group=media-users
+        - --allowed-group=platform-admins
         - --skip-provider-button=true
-        - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
+        - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
-        - --whitelist-domain=.home.keithjackson.dev
+        - --whitelist-domain=.${BASE_DOMAIN}
         - --cookie-secure=true
         - --pass-access-token=true
         - --set-xauthrequest=true
@@ -148,10 +202,15 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
+      securityContext:
+        fsGroup: 3000
       volumes:
       - name: config
         persistentVolumeClaim:
           claimName: sonarr-config
+      - name: media
+        persistentVolumeClaim:
+          claimName: sonarr-media
 ---
 # Service
 apiVersion: v1
@@ -288,14 +347,15 @@ metadata:
     app.kubernetes.io/name: sonarr
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
-    homelab.keithjackson.dev/auth-groups: media-users
+    homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
     namespace: gateway
     sectionName: https
   hostnames:
-  - sonarr.home.keithjackson.dev
+  - sonarr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -305,26 +365,6 @@ spec:
     - name: sonarr
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: sonarr
-  namespace: sonarr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: sonarr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: sonarr
-  namespaceSelector:
-    matchNames:
-    - sonarr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -339,11 +379,11 @@ spec:
   - name: sonarr.slis
     rules:
     - alert: sonarr-down
-      expr: up{job="sonarr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
       for: 5m
       labels:
         severity: critical
         service: sonarr
       annotations:
         summary: sonarr is down
-        description: 'sonarr has been unreachable for 5m (SLI: up).'
+        description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/tests/app-slis.test.yaml

@@ -0,0 +1,245 @@
+# promtool rule-unit tests for the wave-1 per-app `<app>-down` SLI rules.
+# Manifests under test: every PrometheusRule in apps/wave-1/ — nextcloud, radarr, semaphore, sonarr
+# (container/Deployment) and keycloak (StatefulSet, see below). Discovered by the SUITE convention;
+# the flattened manifests are the RENDERED ones Flux applies.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-1/tests/app-slis.test.yaml
+#
+# See apps/wave-0/tests/app-slis.test.yaml for the full history: these rules all read
+# `up{job="<app>"} == 0` until 2026-09-13 against a job no scrape target produced, and were
+# structurally inert on prod. Each is now pinned QUIET on the healthy state, FIRING on no-ready-
+# replica, and FIRING when the signal is ABSENT.
+#
+# KEYCLOAK IS THE INTERESTING ONE. Its server is a StatefulSet run by the Keycloak operator, NOT the
+# Deployment this ServiceDefinition would otherwise imply — so its rule reads
+# kube_statefulset_status_replicas_ready (catalog/keycloak.yaml declares
+# `observability.livenessWorkload: {kind: StatefulSet, name: keycloak}`). Had it been left to the
+# default, the `absent()` half would have matched forever and the alert would have been PERMANENTLY
+# FIRING, which is the other failure mode this repo has shipped to prod. Verified on prod
+# 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}
+# = 2, and the only Deployment in that namespace is keycloak-operator.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # ===========================================================================================
+  # 1. HEALTHY WAVE-1 -> every rule SILENT ("not permanently firing").
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 healthy - every app ready, keycloak StatefulSet at 2 replicas - all quiet
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '2+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '1+0x20'
+      # The namespace's OTHER workloads are present on purpose: keycloak-operator is a Deployment in
+      # the keycloak namespace, and keycloak-db is a StatefulSet in it. A rule that matched the
+      # namespace only (not the object) would be satisfied — or triggered — by these instead of by
+      # the IdP itself. That is the exact masking KeycloakDown had to be fixed for.
+      - series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
+        values: '3+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 2. THE IdP ITSELF IS DOWN while its operator and DB are fine -> ONLY keycloak-down fires.
+  #    This is the anti-masking assertion: the neighbours in that namespace stay ready.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 keycloak StatefulSet has no ready replica (operator + DB fine) - keycloak-down fires
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
+        values: '3+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: keycloak
+              namespace: keycloak
+              statefulset: keycloak
+            exp_annotations:
+              summary: "keycloak is down"
+              description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 3. EACH DEPLOYMENT-BACKED APP DOWN -> its own rule fires, keycloak stays quiet.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 nextcloud/radarr/semaphore/sonarr have no available replica - each own rule fires
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '2+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: nextcloud
+              namespace: nextcloud
+              deployment: nextcloud
+            exp_annotations:
+              summary: "nextcloud is down"
+              description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: radarr
+              namespace: radarr
+              deployment: radarr
+            exp_annotations:
+              summary: "radarr is down"
+              description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: semaphore
+              namespace: semaphore
+              deployment: semaphore
+            exp_annotations:
+              summary: "semaphore is down"
+              description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: sonarr
+              namespace: sonarr
+              deployment: sonarr
+            exp_annotations:
+              summary: "sonarr is down"
+              description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 4. THE SIGNAL VANISHES -> the rules FIRE. The defect class itself, asserted on: a rule that can
+  #    only fire on `== 0` goes SILENT when kube-state-metrics or the workload object disappears,
+  #    which is indistinguishable from healthy. Here NOTHING in wave-1 is scraped.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 all liveness series ABSENT (kube-state-metrics gone) - every rule fires
+    input_series:
+      # One unrelated series so the test has an input at all; no wave-1 liveness series exists.
+      - series: 'up{job="kube-state-metrics"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: keycloak
+              namespace: keycloak
+              statefulset: keycloak
+            exp_annotations:
+              summary: "keycloak is down"
+              description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: nextcloud
+              namespace: nextcloud
+              deployment: nextcloud
+            exp_annotations:
+              summary: "nextcloud is down"
+              description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: radarr
+              namespace: radarr
+              deployment: radarr
+            exp_annotations:
+              summary: "radarr is down"
+              description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: semaphore
+              namespace: semaphore
+              deployment: semaphore
+            exp_annotations:
+              summary: "semaphore is down"
+              description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: sonarr
+              namespace: sonarr
+              deployment: sonarr
+            exp_annotations:
+              summary: "sonarr is down"
+              description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."

apps/wave-2/seerr.yaml

@@ -223,7 +223,7 @@ spec:
     namespace: gateway
     sectionName: https
   hostnames:
-  - seerr.home.keithjackson.dev
+  - seerr.${BASE_DOMAIN}
   rules:
   - matches:
     - path:
@@ -233,26 +233,6 @@ spec:
     - name: seerr
       port: 5055
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: seerr
-  namespace: seerr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: seerr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: seerr
-  namespaceSelector:
-    matchNames:
-    - seerr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -267,11 +247,11 @@ spec:
   - name: seerr.slis
     rules:
     - alert: seerr-down
-      expr: up{job="seerr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})
       for: 5m
       labels:
         severity: critical
         service: seerr
       annotations:
         summary: seerr is down
-        description: 'seerr has been unreachable for 5m (SLI: up).'
+        description: 'seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-2/tests/app-slis.test.yaml

@@ -0,0 +1,69 @@
+# promtool rule-unit tests for the wave-2 per-app `<app>-down` SLI rule (seerr).
+# Manifest under test: apps/wave-2/seerr.yaml. Discovered by the SUITE convention; the flattened
+# manifest is the RENDERED one Flux applies.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-2/tests/app-slis.test.yaml
+#
+# seerr (Jellyseerr/Overseerr) exports NO Prometheus metrics of its own, so its `up` SLI could never
+# have been met by scraping the app — see apps/wave-0/tests/app-slis.test.yaml for the full history
+# of the inert `up{job="seerr"} == 0` rule this replaces. Pinned in both directions plus the
+# signal-vanishes case.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  - interval: 1m
+    name: wave-2 healthy - seerr has an available replica - quiet
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts: []
+
+  - interval: 1m
+    name: wave-2 seerr has no available replica - seerr-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: seerr
+              namespace: seerr
+              deployment: seerr
+            exp_annotations:
+              summary: "seerr is down"
+              description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+    # Under `for: 5m` a 2m-old condition must NOT yet be a firing alert, but the expression must
+    # already be selecting — the distinction an inert rule cannot make.
+    promql_expr_test:
+      - expr: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})'
+        eval_time: 2m
+        exp_samples:
+          - labels: 'kube_deployment_status_replicas_available{deployment="seerr", namespace="seerr"}'
+            value: 0
+
+  - interval: 1m
+    name: wave-2 seerr liveness series ABSENT - seerr-down still fires
+    input_series:
+      - series: 'up{job="kube-state-metrics"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: seerr
+              namespace: seerr
+              deployment: seerr
+            exp_annotations:
+              summary: "seerr is down"
+              description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."

bootstrap/flux/README.md

@@ -11,11 +11,69 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux
 ## Files
 | File | Purpose |
 |---|---|
-| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. `flux install` → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
+| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
+| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
 | `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
 | `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
 | `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). |
 
+## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster
+
+Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git
+and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01):
+
+```
+$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
+{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
+ "app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
+# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1
+```
+
+No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under
+`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux
+documents for hand-installed clusters ([Customize Flux
+manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)):
+
+```
+flux install --export …  ->  <tmp>/gotk-components.yaml     # regenerated from the pinned flux CLI
+components/kustomization.yaml -> <tmp>/kustomization.yaml   # the only hand-authored part (in git)
+kubectl kustomize <tmp> | kubectl apply -f -
+```
+
+Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION`
+(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations,
+more feature gates) is another entry in `components/kustomization.yaml`.
+
+### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)
+
+> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the
+> controller Deployment is outside Flux's inventory. Someone must run step 1.
+
+Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1:
+
+```bash
+# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
+d=$(mktemp -d)
+flux install --export --namespace=flux-system \
+  --components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
+cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
+kubectl kustomize "$d" | kubectl apply -f -      # add --server-side --force-conflicts on a field-manager conflict
+kubectl -n flux-system rollout status deploy/kustomize-controller
+```
+
+**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any
+reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing
+is lost. Confirm afterwards:
+
+```bash
+kubectl -n flux-system get deploy kustomize-controller \
+  -o jsonpath='{.spec.template.spec.containers[0].args}'   # must contain --feature-gates=StrictPostBuildSubstitutions=true
+flux get kustomizations -A                                  # all 11 must return to Ready
+```
+
+**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
+simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
+
 ## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
 
 The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):

bootstrap/flux/bootstrap.sh

@@ -1,6 +1,9 @@
 #!/usr/bin/env bash
 #
-# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
+# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
+# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
+# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
+# same script bootstraps dev and prod with no fork (R-ENV-02).
 #
 # Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
 # =============================================================================
@@ -33,7 +36,7 @@
 #   2. sops-age Secret  — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
 #                         decryption works on first pass (../sops/age-key-secret.md). This script
 #                         only VERIFIES it exists; it never handles key material.
-#   3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
+#   3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
 #                         GitRepository `flux-system` (source) -> Kustomization `platform` ->
 #                         `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
 #                         (chained dependsOn) + `apps-vms`, with cluster-vars overlay
@@ -52,7 +55,10 @@
 set -euo pipefail
 
 # --- config (env-overridable) ------------------------------------------------
-FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"     # dev is the ONLY Phase-1 build target
+# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
+# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
+# every existing caller — override it, never edit it.
+FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
 FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
 SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
 # The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
@@ -62,23 +68,66 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
 
-echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
+echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
 
 # --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
 for bin in flux kubectl; do
   command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
 done
 
-# --- 1. install Flux controllers ---------------------------------------------
+# --- 1. install Flux controllers (rendered through ./components) --------------
 # Default controllers + the image-reflector/image-automation pair (Phase-4 hands-off updates,
 # ADR-0009). `--components-extra` regenerates flux-system with all controllers and registers the
 # image.toolkit.fluxcd.io CRDs before any reconcile — so platform/image-automation's CR applies cleanly.
-if [ -n "${FLUX_COMPONENTS_EXTRA}" ]; then
+#
-  echo "==> [1/3] flux install (default controllers + extra: ${FLUX_COMPONENTS_EXTRA})"
+# We do NOT run `flux install` directly any more. The controllers need CUSTOMIZING (currently:
-  flux install --namespace="${FLUX_NAMESPACE}" --components-extra="${FLUX_COMPONENTS_EXTRA}"
+# --feature-gates=StrictPostBuildSubstitutions=true on kustomize-controller, so an undefined
+# postBuild variable FAILS the Kustomization instead of silently rendering an empty string — see
+# ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md).
+#
+# `flux install` has no flag for that, and because this repo does not use `flux bootstrap` the
+# controllers are not in git for a Flux Kustomization to patch. So we use the mechanism Flux
+# documents for hand-installed clusters ("Customize Flux manifests"):
+#
+#     flux install --export  >  gotk-components.yaml     # exact manifests `flux install` would apply
+#     kustomize overlay      =  ./components/kustomization.yaml (checked in — the ONLY hand-authored part)
+#     kubectl apply -k
+#
+# The export is regenerated from the PINNED flux CLI on every run (FLUX_VERSION,
+# infra/devcontainer/versions.env), so nothing is vendored and nothing can drift. The render dir is
+# a temp dir — the repo tree is never written to.
+#
+# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
+# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
+echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
+flux_install_args=(--namespace="${FLUX_NAMESPACE}")
+[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
+
+if [ "${FLUX_INSTALL_PLAIN:-0}" = "1" ]; then
+  echo "    WARNING: FLUX_INSTALL_PLAIN=1 — controller customization SKIPPED."
+  echo "             StrictPostBuildSubstitutions will NOT be enabled; substitution fails OPEN."
+  flux install "${flux_install_args[@]}"
 else
-  echo "==> [1/3] flux install (default controllers only)"
+  components_dir="${script_dir}/components"
-  flux install --namespace="${FLUX_NAMESPACE}"
+  [ -f "${components_dir}/kustomization.yaml" ] || {
+    echo "ERROR: ${components_dir}/kustomization.yaml missing — refusing to install uncustomized." >&2
+    echo "       (Override with FLUX_INSTALL_PLAIN=1 only if you accept fail-OPEN substitution.)" >&2
+    exit 1
+  }
+  render_dir="$(mktemp -d)"
+  trap 'rm -rf "${render_dir}"' EXIT
+  echo "    rendering flux install --export through ${components_dir}"
+  flux install --export "${flux_install_args[@]}" > "${render_dir}/gotk-components.yaml"
+  cp "${components_dir}/kustomization.yaml" "${render_dir}/kustomization.yaml"
+  # Fail fast if the overlay cannot build (e.g. a patch target that no longer exists after a Flux
+  # version bump) — better a hard stop than silently installing without the feature gate.
+  kubectl kustomize "${render_dir}" > "${render_dir}/rendered.yaml"
+  grep -q -- '--feature-gates=StrictPostBuildSubstitutions=true' "${render_dir}/rendered.yaml" || {
+    echo "ERROR: the StrictPostBuildSubstitutions patch did not apply to the rendered manifests." >&2
+    echo "       Check ./components/kustomization.yaml against this flux CLI version." >&2
+    exit 1
+  }
+  kubectl apply -f "${render_dir}/rendered.yaml"
 fi
 
 # --- 2. verify the out-of-band age key Secret exists -------------------------
@@ -91,7 +140,7 @@ if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null
   exit 1
 fi
 
-# --- 3. apply the authored dev entrypoint ------------------------------------
+# --- 3. apply the authored environment entrypoint ----------------------------
 echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
 kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
 

bootstrap/flux/components/kustomization.yaml

@@ -0,0 +1,64 @@
+---
+# Controller-level customization of the Flux CONTROLLERS themselves (the `flux-system` Deployments).
+#
+# WHY THIS DIRECTORY EXISTS
+# =========================
+# This repo does NOT use `flux bootstrap` (see ../README.md "Why `flux install` + apply"): the
+# reconciliation graph is authored by hand in clusters/base, and the controllers are installed
+# imperatively by `flux install`. Consequence, verified on the live dev cluster 2026-09-01:
+#
+#     kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
+#     -> NO kustomize.toolkit.fluxcd.io/name label, deployment.kubernetes.io/revision: 1
+#
+# i.e. **the controllers are not managed by Flux and are not in git.** There was therefore no
+# existing mechanism to customize them, and no amount of editing under clusters/** can reach them.
+#
+# This is the mechanism Flux documents for exactly that situation ("Customize Flux manifests" —
+# https://fluxcd.io/flux/installation/#customize-flux-manifests): export the component manifests
+# with `flux install --export`, overlay them with kustomize `patches:`, and apply the result.
+# `../bootstrap.sh` step 1 does that render — it copies THIS file next to a freshly exported
+# `gotk-components.yaml` in a temp dir and runs `kubectl apply -k`. Nothing is generated into the
+# repo, so the manifests can never drift from the pinned FLUX_VERSION
+# (infra/devcontainer/versions.env) the way a vendored copy would.
+#
+# Every future controller customization (vertical scaling, tolerations, extra feature gates,
+# --concurrent) belongs HERE, as another entry under `patches:`.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  # Written by `bootstrap.sh` from `flux install --export ...` at render time. NOT checked in.
+  - gotk-components.yaml
+patches:
+  # -------------------------------------------------------------------------------------------
+  # StrictPostBuildSubstitutions — make postBuild substitution FAIL CLOSED.
+  #
+  # Default behaviour is fail-OPEN: a `${VAR}` that is not defined in `cluster-vars` renders as the
+  # EMPTY STRING, silently. On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium
+  # HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry). With this
+  # gate the undefined variable FAILS the Kustomization instead, and Flux keeps the last-good state.
+  #
+  # PROVEN SAFE before enabling: every one of the 11 substituted paths was rendered with
+  # `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst --strict` and
+  # exited 0, with a negative control reproducing the incident. The permanent guard that keeps it
+  # true is the L0 gate conformance/tests/l0-static/flux-postbuild-substitution/.
+  #
+  # NOTE the gate does NOT catch default-providing forms (`${VAR:-d}`, `${VAR:?msg}`, `${VAR:=v}`) —
+  # Flux exempts them by design. That class is what the L0 gate covers; the two are complementary.
+  #
+  # This also pre-pays for the Flux 2.4.0 -> 2.9.x upgrade: strict substitution became DEFAULT-ON in
+  # kustomize-controller v1.9.0, so the upgrade flips this behaviour whether or not we opt in.
+  #
+  # JSON-6902 `add ... /args/-` (append), NOT a strategic merge: a container's `args` is an atomic
+  # list, so a strategic-merge patch would REPLACE every arg `flux install` generates
+  # (--events-addr, --watch-all-namespaces, --log-level, --log-encoding, --enable-leader-election)
+  # and silently drop any the CLI adds in a future version.
+  - target:
+      group: apps
+      version: v1
+      kind: Deployment
+      name: kustomize-controller
+      namespace: flux-system
+    patch: |-
+      - op: add
+        path: /spec/template/spec/containers/0/args/-
+        value: --feature-gates=StrictPostBuildSubstitutions=true

bootstrap/gitea/bootstrap-gitea.sh

@@ -1,11 +1,30 @@
 #!/usr/bin/env bash
 # bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
-# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
+# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
-# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
+# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
-# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
+# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
-# CONTRIBUTING.md).
+# using it now.
 #
-# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
+# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
+# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
+#
+#   PROD (bare metal, current use):  HOST_NAT_IP=192.168.2.132
+#       The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
+#       NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
+#       three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
+#       apply to prod, so do not "fix" prod by opening zones.
+#
+#   DEV (libvirt VMs, HISTORICAL):   HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
+#       *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
+#       bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
+#       recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
+#
+# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
+# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
+# author-only work and must never be fired off casually or "to see if it works": it starts a
+# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
+# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
+# it will then reconcile. Know which HEAD you are publishing before you run it.
 set -euo pipefail
 ENGINE="${ENGINE:-podman}"
 GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}"   # pinned; bump if needed
@@ -16,11 +35,24 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
 # subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
 DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
 DATA_DIR="podman volume ${DATA_VOLUME}"   # display label only (data lives in the named volume)
-# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
+# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
 # clone/redirect URLs are correct from inside the cluster.
-# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
+#
+# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
+# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
+# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
+# Pass it explicitly, e.g. for prod:   HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
+#
+# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
+# VM->host:3000 by default and had to be opened once with
 #   sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
-HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
+# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
+if [ -z "${HOST_NAT_IP:-}" ]; then
+  echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
+  echo "       prod: HOST_NAT_IP=192.168.2.132   (same LAN as the nodes, no NAT, no firewall change)" >&2
+  echo "       There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
+  exit 1
+fi
 ORG="${GITEA_ORG:-Home-Network}"
 REPO="${GITEA_REPO:-home-network-k3s-impl}"
 
@@ -75,8 +107,17 @@ cat <<EOF
 
 Gitea bootstrap source ready (${DATA_DIR}).
   UI (host):                     ${BASE}/${ORG}/${REPO}
-  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git   (clusters/dev)
+  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
+                                 (patch this url into the overlay being bootstrapped:
+                                  clusters/prod today; clusters/dev has already cut over)
   Admin creds (host, 0600):      ${ADMIN_CRED_FILE}
-Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
+Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
+firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
+must allow VM->host:${HTTP_PORT} —
   ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
+REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
+at this source will reconcile THAT revision.
+TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
+any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
+procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
 EOF

catalog/README.md

@@ -28,6 +28,16 @@ inherit every guarantee automatically.
   `auth.groups`; `native`/`none` are documented exceptions only.
 - Irreplaceable state (`class: longhorn` configs/DBs) must set `backup.policy` to an off-cluster
   schedule — never `reacquirable` (R-STATE-03). Only bulk media may be `reacquirable`.
+- **Shell braces must be escaped `$${...}`.** Anything rendered verbatim onto the workload — a
+  `lifecycle` postStart script, an `env` value — lands under `apps/wave-<N>/`, and every
+  `apps-wave-<N>` Flux Kustomization declares `postBuild.substituteFrom: cluster-vars`. Flux runs
+  envsubst over the WHOLE manifest at reconcile time, so an unescaped `${VAR:-}` meant for the
+  *shell* is replaced with the empty string before the container ever starts (this silently killed
+  gitea's and nextcloud's OIDC registration for days). Bare `$VAR` and `$(cmd)` are left alone and
+  need no escaping; a genuine environment knob is declared in `clusters/base/cluster-vars.yaml` and
+  written bare `${KNOB}`. Full contract: [`../clusters/base/apps-vms.yaml`](../clusters/base/apps-vms.yaml).
+  `render.py` REFUSES to emit a manifest that breaks this rule (Flux's *strict* substitution mode
+  cannot: it deliberately exempts the `${V:-}` / `${V:=x}` / `${V:?}` default-operator forms).
 
 ## Validate before commit — single command, no cluster (R-TEST-02)
 ```bash

catalog/gitea.yaml

@@ -24,9 +24,9 @@ env:
         name: gitea-bootstrap
         key: secret-key
   - name: GITEA__server__DOMAIN              # correct public clone/redirect URLs for the TLS-terminated host
-    value: gitea.home.keithjackson.dev
+    value: gitea.${BASE_DOMAIN}
   - name: GITEA__server__ROOT_URL
-    value: https://gitea.home.keithjackson.dev/
+    value: https://gitea.${BASE_DOMAIN}/
   - name: GITEA__actions__ENABLED            # Phase 6 CI/CD: declared (not implicit upstream default) so the
     value: "true"                            # in-cluster act_runner (ns gitea-actions) registers against it
   - name: GITEA__actions__DEFAULT_ACTIONS_URL  # resolve `uses: <org>/<action>@ref` from github.com (cluster
@@ -104,12 +104,24 @@ lifecycle:
           # persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
           # file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
           # usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
-          GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
           gitea_oidc_fail() {
             echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
           }
-          if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
+          # NOTE the `$$` escape below — it is load-bearing. apps-wave-0 declares postBuild.
+          # substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
+          # time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not
+          # a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS TRUE
+          # — making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on
+          # every boot since 8738871 (2026-08-27). SSO kept working only because the login source was
+          # already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.
+          # Flux STRICT substitution cannot catch this — it EXEMPTS default-operator forms by design.
+          # `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.
+          # Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why the
+          # `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.
+          # Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
+          if [ -z "$${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
             echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)"
           else
             # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).
@@ -150,7 +162,13 @@ lifecycle:
               gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
             fi
           fi
-exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
+          # A postStart hook MUST exit 0 — a non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but
+          # it is made EXPLICIT so the guarantee does not depend on whatever statement happens to
+          # be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
+          # the moment its OIDC guard was fixed and the block actually ran for the first time.
+          exit 0
+exposure: public                # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)
 storage:
   - name: data
@@ -160,8 +178,36 @@ storage:
                                 # the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
 dependencies: []                # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
 ingressRoutes:
-  - host: gitea.home.keithjackson.dev
+  - host: gitea.${BASE_DOMAIN}
     port: 3000                  # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
+# THIS IS WHAT KEEPS PROD'S GITOPS OUT OF A DEADLOCK — it is not an incidental allowance. Remove it
+# and prod cannot reconcile AT ALL, in a way that is very hard to diagnose from cold.
+#
+# Prod's Flux source is THIS SERVICE, not a public hostname:
+#   clusters/prod/kustomization.yaml -> GitRepository.spec.url =
+#   http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
+# chosen deliberately so prod's GitOps depends on NEITHER public DNS, Cloudflare, prod's own
+# Gateway, NOR ${BASE_DOMAIN} — a future domain change cannot sever prod from its own source. That
+# file documents the other half of this pair at length; the two must be read together.
+#
+# The only other ingress this entry renders is gitea-allow-gateway-ingress, which admits
+# `fromEntities: [ingress]` — the shared Gateway's Envoy proxy — and nothing else. Flux's
+# source-controller dials the ClusterIP DIRECTLY, so under the namespace default-deny (R-NET-01) its
+# clone was DROPPED (`wget` from source-controller timed out; srcReady=False, "unable to list
+# remote"). The deadlock that made this urgent: the HTTPRoute that would have served the new public
+# name after the 2026-09-13 apex flip lives in a DOWNSTREAM tier, which cannot apply while the
+# source is down — source down -> routes never rename -> source stays down. It was broken on
+# 2026-09-13 with a live `kubectl apply` of a hand-written gitea-allow-flux-ingress, i.e. with a
+# cluster object that existed OUTSIDE git. This declaration renders that object, so one
+# `kubectl delete` or one cluster rebuild can no longer put prod back in the deadlock.
+inClusterClients:
+  # `name` is pinned to `flux` (rather than defaulting to the namespace) so the rendered policy name
+  # equals the live object's, `gitea-allow-flux-ingress`, and Flux ADOPTS it in place: no
+  # near-duplicate policy beside it, and no window in which prod's own source has no allow.
+  - name: flux
+    namespace: flux-system      # matched by NAMESPACE alone — the flux controllers' pod labels belong
+                                # to upstream's manifests, not to this platform's contract
+    port: 3000                  # ONLY the Gitea HTTP port — the same single port the Gateway allow opens
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
   groups:

catalog/jackett.yaml

@@ -17,20 +17,40 @@ name: jackett
 kind: container
 source:
   image: lscr.io/linuxserver/jackett:latest
-exposure: internal              # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
     size: 2Gi
     class: longhorn             # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
+    mountPath: /config          # THE APP WRITES /config, NOT the renderer default /data/config.
+                                # LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
+                                # /config), so without this the Longhorn PVC mounted at /data/config sat
+                                # holding nothing but lost+found while the app wrote its entire
+                                # configuration to the container's EPHEMERAL writable layer — destroyed on
+                                # every pod recreation: restart, reschedule, image bump. Measured in-cluster
+                                # 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
+                                # dev=2053 on the node's own disk with [Sentry asp config.xml logs].
+                                # This is the SAME failure catalog/plex.yaml:54 already records fixing.
+                                # The live /config contents were copied onto each PVC before this landed,
+                                # so API keys and credentials other apps reference were preserved.
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: jackett.home.keithjackson.dev
+  - host: jackett.${BASE_DOMAIN}
     port: 9117                  # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
   groups:
     - media-users               # authenticated AND in an authorized group (R-IDP-06)
+    # platform-admins is a SUPERSET, not a widening. --allowed-group is repeatable and OR-ed, so
+    # listing both means "media consumers OR platform operators". Without it the media UIs are
+    # reachable by NOBODY the moment media-users is empty: on prod the realm's only user is the
+    # operator, who is in platform-admins only, so jackett/qbittorrent/radarr/sonarr answered an
+    # immediate 403 at the oauth2-proxy callback (no login prompt, because the Keycloak SSO session
+    # already existed) while pihole/crafty — which already gate on platform-admins — worked.
+    # This grants no real privilege: platform-admins already maps to cluster-admin via
+    # k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods.
+    - platform-admins           # operators of these services must not be locked out of them
 update:
   channel: stable               # R-UPD-01
   range: ">=0.22.0"             # stay on the Jackett 0.22.x stable line

catalog/keycloak.yaml

@@ -18,7 +18,7 @@ dependencies:
     port: 5432
     protocol: tcp
 ingressRoutes:
-  - host: keycloak.home.keithjackson.dev
+  - host: keycloak.${BASE_DOMAIN}
     port: 8080                # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
 auth:
   mode: native                # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.
@@ -31,5 +31,15 @@ observability:
   metrics: true
   slis:
     - up                      # IdP down/unhealthy is a critical alert (docs/10; R-OBS-01)
+  # WHAT "up" IS for Keycloak. The Keycloak OPERATOR runs the server as a StatefulSet named
+  # `keycloak` (pods keycloak-0/1), not as the Deployment this ServiceDefinition would otherwise
+  # imply — the operator owns the workload (platform/keycloak/keycloak-instance.yaml) and this SD
+  # is the catalog's declaration of intent for it. Without this override the rendered SLI would
+  # read a Deployment series that does not exist and its `absent()` half would fire FOREVER.
+  # Verified on prod 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak",
+  # statefulset="keycloak"} = 2, and the only Deployment in that namespace is keycloak-operator.
+  livenessWorkload:
+    kind: StatefulSet
+    name: keycloak
 backup:
   policy: daily-offcluster    # DR-critical: rebuilds from DB + git realm config (docs/15, docs/04; R-STATE-03)

catalog/nextcloud.yaml

@@ -50,13 +50,13 @@ env:
   #     the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
   #     though it speaks plain HTTP behind the proxy. ---
   - name: NEXTCLOUD_TRUSTED_DOMAINS
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
   - name: OVERWRITEHOST
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
   - name: OVERWRITEPROTOCOL
     value: https
   - name: OVERWRITECLIURL
-    value: https://nextcloud.home.keithjackson.dev
+    value: https://nextcloud.${BASE_DOMAIN}
   # --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
   #     `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
   #     (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
@@ -88,7 +88,7 @@ lifecycle:
           # provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
           # ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
           OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
-          NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
           nextcloud_oidc_fail() {
             echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
             { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
@@ -111,7 +111,19 @@ lifecycle:
           # every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.
           OCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \
             || echo "[nextcloud-oidc] WARN: failed to set allow_local_remote_servers" >&2
-          if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
+          # NOTE the `$$` escape below — it is load-bearing. apps-wave-1 declares postBuild.
+          # substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
+          # time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is
+          # not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS
+          # TRUE — so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never
+          # installing/enabling user_oidc nor registering the provider. SSO kept working only because
+          # the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with
+          # NO OIDC at all. Flux STRICT substitution cannot catch this — it EXEMPTS default-operator
+          # forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion
+          # for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why
+          # `--clientsecret-env` and the bare uses below always worked and only the guard broke.
+          # Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
+          if [ -z "$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
             echo "[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)"
             exit 0
           fi
@@ -191,7 +203,17 @@ lifecycle:
             [ "$missing" = 0 ] && break
             p=$((p + 1)); sleep 2
           done
-          [ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
+          if [ "$p" -ge 15 ]; then
+            nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
+          fi
+          # A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:
+          # the final statement used to be `[ "$p" -ge 15 ] && nextcloud_oidc_fail ...`, so on the
+          # SUCCESS path the test returned 1 and became the script's exit status. That was latent for
+          # as long as the OIDC guard above was dead code and exited 0 first; fixing the guard
+          # exposed it immediately. Every failure path here logs loudly and writes a marker file and
+          # deliberately does NOT crash the pod — the break-glass local admin must stay usable.
+          exit 0
 exposure: public                # edge port-forward + public DNS (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05) — app store, integrity checks
 storage:
@@ -201,12 +223,50 @@ storage:
     mountPath: /var/www/html    # 3-replica tier; irreplaceable, DR-critical (docs/04; R-STATE-02).
                                 # The whole html dir is persisted so config.php + user data survive a
                                 # rebuild (the default /data/<name> would strand them on ephemeral rootfs).
+  # THE LEGACY NEXTCLOUD DATASET (Storage/nextcloud, 270 G) — MOUNTED READ-ONLY, DELIBERATELY, AND
+  # NOT AS THIS INSTANCE'S DATA DIRECTORY. Read this before changing either decision.
+  #
+  # WHAT IS ACTUALLY IN THERE (measured 2026-09-07, not assumed):
+  #     /mnt/Storage/nextcloud/nextcloud/config          103 M
+  #     /mnt/Storage/nextcloud/nextcloud/config_old      3.4 G
+  #     /mnt/Storage/nextcloud/nextcloud/config_backup   267 G   <- 99% of the dataset
+  # There is NO `data/` directory. This is the OLD, non-k3s Nextcloud's configuration plus a backup
+  # blob — an ARCHIVE, not a live user-data tree.
+  #
+  # WHY READ-ONLY. Two independent reasons, either sufficient:
+  #   1. The old instance is STILL LIVE on it. The export is currently shared rw to 192.168.1.6 and
+  #      192.168.1.5, and the operator has said the existing non-k3s prod must remain active. Two
+  #      Nextcloud instances writing one instance's config directory, each with its own database and
+  #      therefore its own oc_filecache, is a corruption scenario with no clean recovery.
+  #   2. Nothing here needs writing. An archive is read.
+  # "Do not grant more than needed" therefore lands on `ro`. If the operator's intent is in fact a
+  # MIGRATION (read this data in, then decommission the old instance), that is a different and
+  # larger change than a mount, and it should be a deliberate one — flip `readOnly` and update the
+  # export at that point, not before. Enforced by the PV's nfs.readOnly, so it is a real `ro` mount.
+  #
+  # WHY NOT /var/www/html/data. That path is the LONGHORN PVC above and is this instance's own
+  # state; mounting a foreign tree into it would have Nextcloud scan and index an archive it does
+  # not own. A neutral path keeps the two completely separate. To expose it to users, add it as a
+  # Nextcloud External Storage (Local) mount pointing at this path — an in-app, per-user decision.
+  #
+  # subPath skips the redundant single-child directory: the dataset root holds exactly one entry,
+  # `nextcloud/`, so mounting that subtree puts config/ config_old/ config_backup/ at the top.
+  #
+  # UID: nothing to do. The tree is 0755 owned 1000:3000 and the mount is read-only, so www-data
+  # (uid 33) can read and traverse it. (fsGroup would be a no-op on an NFS volume in any case.)
+  - name: legacy-archive
+    class: bulk
+    export: /mnt/Storage/nextcloud   # the ZFS dataset Storage/nextcloud, on ${BULK_NFS_SERVER}
+    subPath: nextcloud
+    mountPath: /mnt/legacy-nextcloud
+    readOnly: true
+    size: 270Gi                      # ADVISORY ONLY on a static NFS PV — the dataset's used size.
 dependencies:
   - to: nextcloud-db            # explicit allow: Nextcloud -> its PostgreSQL (docs/13 graph; R-NET-02).
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
     protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: nextcloud.home.keithjackson.dev
+  - host: nextcloud.${BASE_DOMAIN}
     port: 80                    # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)

catalog/pihole.yaml

@@ -13,8 +13,95 @@ storage:
     size: 2Gi
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
+# ------------------------------------------------------------------------------------------------
+# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
+# ------------------------------------------------------------------------------------------------
+# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
+# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
+# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
+# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
+# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
+# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
+# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
+# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
+#
+# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
+# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
+# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
+# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
+# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
+# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
+# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
+# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
+# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
+# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
+# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
+# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
+#
+# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
+# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
+# configured through the UI is lost on the next reschedule — which is exactly what happened to the
+# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
+# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
+# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
+# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
+env:
+  # Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
+  # pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
+  # required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
+  # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
+  # enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
+  # nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
+  - name: FTLCONF_dns_listeningMode
+    value: "ALL"
+  # Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
+  # router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
+  # answers, and forwarding to a device that may forward back here invites a loop.
+  - name: FTLCONF_dns_upstreams
+    value: "8.8.8.8;8.8.4.4"
+  # Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
+  # because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
+  # externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
+  # announcement lease and the pihole pod land on different nodes, which is most of the time — and
+  # Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
+  # INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
+  # apparent client's budget, and a normal evening of browsing from a handful of machines would trip
+  # a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
+  # off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
+  # limit becomes meaningful again — and should be restored — if Cilium is ever switched to
+  # `loadBalancer.mode: dsr`, which preserves the client address.
+  - name: FTLCONF_dns_rateLimit_count
+    value: "0"
+  # THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
+  # every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
+  # so the first line is a wildcard for the whole zone and each later line is a specific override.
+  # PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
+  # line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
+  # changes nothing.
+  #
+  # A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
+  # `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
+  # ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
+  # platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
+  # behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
+  # repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
+  # is the annotation that means "this Service publishes its own hostname at its own address":
+  # apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
+  # platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
+  # resolver address, are the complete set of exceptions today.
+  #
+  # AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
+  # (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
+  # NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
+  #
+  # Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
+  # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
+  # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
+  # while resolution for the zone does not work. Confirmed both ways against the live image.
+  - name: FTLCONF_misc_dnsmasq_lines
+    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
 ingressRoutes:
-  - host: pihole.home.keithjackson.dev
+  - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
 auth:
   mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)

catalog/plex.yaml

@@ -4,21 +4,11 @@
 # auth: native (Plex accounts, NOT Keycloak — docs/13 + docs/15 documented exception).
 # Validates against schemas/servicedefinition.schema.json.
 #
-# BULK MEDIA (spec bulk-tier requirement, docs/04) — OPEN PREREQUISITE, intentionally NOT declared
+# BULK MEDIA — NOW ATTACHED (docs/04 bulk tier; the long-standing OQ-2 prerequisite, resolved for
-# here yet (the requirement ID is cited in the WS-APPS report, not as a token here: this change does
+# the STATIC case). The library is 8.33 T of ZFS on the Proxmox host at /mnt/Storage/plex, and it
-# NOT exercise the bulk tier, so it must not be selected as a "touched" requirement whose in-cluster
+# already exists: nothing is provisioned, an existing dataset is mounted. See
-# conformance test is still a Phase-1 gap).
+# docs/BULK-NFS-STORAGE.md for the layout, the export change and the uid analysis. (The bulk-tier
-#   The Plex library belongs on the `bulk` tier (docs/04 / docs/13 cross-cutting note: metadata DB
+# requirement ID is deliberately still not typed here — see that document's "gate" note.)
-#   on replicated Longhorn, large library on bulk). The `bulk` StorageClass exists
-#   (platform/longhorn/storageclasses.yaml) and the schema+renderer both support `class: bulk`, but
-#   that class is a documented STUB: OQ-2 (external NFS/SMB backing) is UNRESOLVED and the NFS CSI
-#   driver is not installed, so a `bulk` PVC will not bind and would leave the Plex pod Pending.
-#   To onboard the running Plex shell now (compute + config + public exposure), the media volume is
-#   omitted. Once OQ-2 is resolved and `bulk` provisions, attach the library with a one-line add:
-#     storage:
-#       - name: media
-#         class: bulk        # external mount; size omitted (sized outside the cluster)
-#   then re-render. See the WS-APPS report for the full bulk-media finding.
 name: plex
 kind: container
 source:
@@ -70,9 +60,34 @@ storage:
                               # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
                               # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
                               # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
+  # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
+  # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
+  # downloads/ movies/ photos/ tv/).
+  #
+  # WHY THE ROOT AND NOT THREE subPath MOUNTS. Nothing is gained by narrowing: the mount is `ro`,
+  # so no subPath could remove a privilege that matters, while three mounts would triple the NFS
+  # mounts per pod and mean a manifest change every time a top-level library is added.
+  #
+  # WHY READ-ONLY, when the export is `rw` and the other consumers write. Plex is a SERVER: it
+  # indexes and streams. Every write on that tree belongs to Sonarr and Radarr. Marking it here
+  # costs nothing and removes Plex from the set of things that can damage 8.33 T. Enforced by the
+  # PV's nfs.readOnly (a real `ro` mount), not only by the volumeMount flag.
+  #
+  # UID: no change needed. The tree is 0755 owned 1000:3000 throughout, so it is world-readable and
+  # world-traversable, and PLEX_UID/PLEX_GID above already make the process uid 1000 in any case.
+  # (fsGroup would have been a no-op here regardless — the in-tree NFS plugin does no ownership
+  # management. That is the trap the arrs' PUID/PGID exists to avoid; Plex never hits it.)
+  - name: media
+    class: bulk
+    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
+    mountPath: /media
+    readOnly: true
+    size: 9Ti                   # ADVISORY ONLY (nothing enforces a static NFS PV's capacity): the
+                                # dataset's 8.33 T used, rounded up, so `kubectl get pv` reports
+                                # something true. The real limit is the ZFS pool.
 dependencies: []              # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: plex.home.keithjackson.dev
+  - host: plex.${BASE_DOMAIN}
     port: 32400               # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
 auth:
   mode: native                # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)

catalog/qbittorrent.yaml

@@ -4,19 +4,18 @@
 # internal download client | egress VPN + kill-switch | config on Longhorn | auth forward-auth
 # (bare web UI, no native OIDC).
 #
-# BULK DOWNLOADS TIER — OPEN PREREQUISITE, intentionally NOT declared here (mirrors plex + sonarr/radarr):
+# BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record).
-#   docs/13 row 13 places qBittorrent's downloads/staging volume on the `bulk` tier (large, non-3x-
+#   This entry used to ship config-only, with a comment promising to attach the downloads volume "once
-#   replicated media handoff to the arr stack). The `bulk` StorageClass exists
+#   OQ-2 is resolved". OQ-2 was resolved for STATIC pre-existing datasets (schema 1.5.0 / renderer
-#   (platform/longhorn/storageclasses.yaml) and both the schema and renderer support `class: bulk`, BUT
+#   `class: bulk` + `export` -> a static, pre-bound NFS PV), and plex/sonarr/radarr/nextcloud all took
-#   it is a NON-PROVISIONING STUB pending OQ-2 (its NFS/SMB backing is REPLACE-ME and the NFS CSI driver
+#   their media mounts — but qBittorrent's was never added, so the DOWNLOAD CLIENT was the one member
-#   is not installed) — a `bulk` PVC would never bind and would leave the pod Pending. Declaring it now
+#   of the chain with no media volume at all. Measured on prod 2026-09-12:
-#   would also pull the bulk-tier requirement into the change's "touched" set, whose in-cluster
+#       Session\DefaultSavePath=/downloads/   Session\TempPath=/downloads/incomplete/
-#   conformance test is not yet scaffolded, blocking the merge gate. So the downloads volume is DEFERRED
+#       Downloads\SavePath=/downloads/       `ls /downloads` -> No such file or directory
-#   and this onboarding ships config-only. Once OQ-2 is resolved and `bulk` provisions, attach downloads
+#   i.e. the app was configured to write a path that DID NOT EXIST, in a container with no volume there.
-#   with a one-line add under storage:
+#   Anything it did manage to create would have been invisible to sonarr/radarr, lost on every pod
-#         - name: downloads
+#   recreation, and filling the NODE's ephemeral disk in the meantime. The `downloads` entry under
-#           class: bulk          # external mount; size omitted (sized outside the cluster)
+#   storage: below closes that. See docs/BULK-NFS-STORAGE.md and the bulk-tier L0 conformance test.
-#   then re-render. See the WS-APPS report for the full bulk-downloads finding.
 #
 # VPN-class (docs/13 row 13; docs/03 "VPN-class set"): ALL of this pod's public-internet egress
 # (peer/tracker/DHT traffic) MUST leave via the node PIA tunnel, never the WAN. `egress: vpn` renders
@@ -32,20 +31,423 @@ name: qbittorrent
 kind: container
 source:
   image: lscr.io/linuxserver/qbittorrent:5.1.2   # pinned stable tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
+# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
+# LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
+# 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
+# uid=911 gid=911, and /mnt/Storage/plex/downloads is 0755 owned 1000:1000. 0755 grants write to the
+# OWNER ONLY, so uid 911 would have mounted the export cleanly and then failed every write with
+# EACCES — the "mounts fine, cannot write" failure the sonarr entry warns about at length.
+env:
+  - name: PUID
+    value: "1000"               # owner uid of /mnt/Storage/plex/downloads (and of every dir under the export)
+  - name: PGID
+    value: "3000"               # the export's dominant group, matching sonarr/radarr: files this writes stay
+                                # readable to the EXISTING non-k3s consumers of the same dataset, and land
+                                # group-owned the same way the arrs' own writes do
+  # --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
+  # Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
+  # NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
+  # NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
+  # password is provided for this session: ..."). Two real consequences, not hypotheticals:
+  #   * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
+  #     tested valid against the then-current temporary password; both would have started failing at
+  #     the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
+  #     report a failed client, not a failed import).
+  #   * The human operator had no stable login: every restart meant reading the pod log.
+  # Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
+  # same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
+  - name: QBT_WEBUI_USERNAME
+    valueFrom:
+      secretKeyRef:
+        name: qbittorrent-bootstrap
+        key: webui-username
+  - name: QBT_WEBUI_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: qbittorrent-bootstrap
+        key: webui-password
+lifecycle:
+  # LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
+  #
+  # WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
+  # directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
+  # apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
+  # RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
+  # the container start rather than a clock, because the failure this closes IS a container start:
+  #   * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
+  #     PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
+  #     restore or rebuild in which the app serves a random password and both arrs are broken. The
+  #     hook has no such window: it runs before/with the very start that would otherwise regenerate.
+  #   * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
+  #     PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
+  #     then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
+  #     In-container, before the service starts, none of that exists.
+  #   * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
+  #     privilege than the CronJob alternative (R-SEC/least-privilege).
+  # The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
+  # keeps in its own state), and the schema documents this hook for exactly that use.
+  #
+  # AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
+  # BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
+  # would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
+  #   * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
+  #     credential — because the WebUI is not running yet. That assertion is the most valuable line
+  #     in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
+  #     writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
+  #   * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
+  #     mean a schema version bump plus a renderer change — a blast radius across every service's
+  #     golden — to buy something strictly weaker than what the hook already does.
+  # The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
+  # the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
+  #
+  # DURABILITY, the whole point — all three paths end with the SAME credential:
+  #   pod restart      -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
+  #   config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
+  #                       hook runs and re-asserts over whatever the snapshot contained.
+  #   rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
+  #                       credential into it before the service is serving. Nothing outside git.
+  #
+  # WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
+  # for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
+  # depend on network policy staying correct forever. The WebUI keeps demanding a password; that
+  # password is simply now known, fixed, and reproducible from git.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
+          # running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
+          #
+          # THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
+          # apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
+          # this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
+          # enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
+          # CLUSTER — but only there:
+          # envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
+          # apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
+          # development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;
+          # see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
+          # are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
+          # however it is run.
+          set -u
+          CONF=/config/qBittorrent/qBittorrent.conf
+          SVC=/run/service/svc-qbittorrent
+          # The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
+          # ServiceDefinition — which is the single place the renderer takes the containerPort, the
+          # Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
+          # env var for a lookup to read anyway.
+          # It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
+          # that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
+          # so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
+          # matched. Every curl below then got an unparseable URL, returned nothing, and the hook
+          # logged "the WebUI rejected the durable
+          # credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
+          # service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
+          # against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
+          PORT=8080
+          # A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
+          # that only echoes is effectively silent (gitea's hits the same wall and writes a marker
+          # file for it). Every outcome therefore also lands, timestamped, on the persistent config
+          # volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
+          # credential from git was asserted and accepted on this start.
+          log() {
+            echo "[qbt-webui-cred] $*"
+            printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
+              >> /config/qbittorrent-webui-cred.log 2>/dev/null || true
+          }
+
+          # (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
+          #     before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
+          #     racing to create one it would then not overwrite.
+          n=0
+          while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
+
+          waited=0
+          attempt=0
+          while [ "$attempt" -lt 3 ]; do
+            attempt=$((attempt + 1))
+
+            # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
+            #     10 = rewritten, anything else = could not write.
+            python3 - "$CONF" <<'PY'
+          import base64, hashlib, hmac, os, stat, sys
+
+          conf = sys.argv[1]
+          user = os.environ.get("QBT_WEBUI_USERNAME", "")
+          pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
+          if not user or not pw:
+              sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
+              raise SystemExit(1)
+
+          # qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
+          # iterations, 16-byte random salt, 64-byte digest, stored as
+          #   WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
+          # Get any one of them wrong and the app simply rejects every login.
+          ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
+          UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
+
+          def matches(raw):
+              """True iff the stored digest is one OUR password produces (salt is read from it)."""
+              v = raw.strip().strip('"')
+              if v.startswith("@ByteArray(") and v.endswith(")"):
+                  v = v[len("@ByteArray("):-1]
+              try:
+                  salt_b64, digest_b64 = v.split(":", 1)
+                  salt = base64.b64decode(salt_b64, validate=True)
+                  digest = base64.b64decode(digest_b64, validate=True)
+              except Exception:
+                  return False
+              if not salt or not digest:
+                  return False
+              return hmac.compare_digest(
+                  hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
+
+          try:
+              lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
+          except FileNotFoundError:
+              lines = []
+
+          def value_of(key):
+              for i, line in enumerate(lines):
+                  if line.startswith(key + "="):
+                      return i, line.split("=", 1)[1]
+              return -1, None
+
+          ui, uval = value_of(UKEY)
+          pi, pval = value_of(PKEY)
+          if uval == user and pval is not None and matches(pval):
+              raise SystemExit(0)          # converged - leave the file (and its salt) alone
+
+          salt = os.urandom(SALT_LEN)
+          digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
+          stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
+          desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
+
+          out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
+          if SECTION in out:
+              at = out.index(SECTION) + 1
+          else:
+              if out and out[-1].strip():
+                  out.append("")
+              out.append(SECTION)
+              at = len(out)
+          out[at:at] = desired
+
+          try:
+              os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
+              tmp = conf + ".webui-cred.new"
+              with open(tmp, "w", encoding="utf-8") as fh:
+                  fh.write("\n".join(out) + "\n")
+                  fh.flush()
+                  os.fsync(fh.fileno())
+              # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
+              # cost the app the ability to save its own settings, so carry the ownership over.
+              try:
+                  st = os.stat(conf)
+                  os.chown(tmp, st.st_uid, st.st_gid)
+                  os.chmod(tmp, stat.S_IMODE(st.st_mode))
+              except FileNotFoundError:
+                  os.chown(tmp, int(os.environ.get("PUID", "1000")),
+                           int(os.environ.get("PGID", "1000")))
+                  os.chmod(tmp, 0o664)
+              os.replace(tmp, conf)        # atomic: the app never reads a half-written conf
+          except OSError as exc:
+              sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
+              raise SystemExit(1)
+          raise SystemExit(10)
+          PY
+            rc=$?
+
+            if [ "$rc" = 0 ]; then
+              log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
+            elif [ "$rc" = 10 ]; then
+              log "asserted WebUI Username + Password_PBKDF2 into $CONF"
+              # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
+              # have read the old, credential-less conf and be serving a random temporary password.
+              # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
+              # Nothing to do when the service has not started yet - it will read the new file.
+              # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
+              # 2026-09-13 failure spent the entire verification budget before the first login and
+              # then fired attempts 2 and 3 with no wait left at all.
+              sv=0
+              while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
+                sv=$((sv + 1)); sleep 1
+              done
+              if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
+                log "qbittorrent-nox was already running - restarting it to load the credential"
+                s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
+                sleep 3
+              fi
+            else
+              log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
+              log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
+              exit 0
+            fi
+
+            # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
+            #     (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
+            #     at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
+            #
+            # "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
+            # conflating them is what made this hook lie once already (see the PORT comment). An
+            # unreachable WebUI is reported as unreachable, and the hook stops rather than spending
+            # its remaining login attempts to produce two more identical wrong diagnoses.
+            api_ok=0
+            while [ "$waited" -lt 120 ]; do
+              code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
+              if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
+              waited=$((waited + 1)); sleep 1
+            done
+            if [ "$api_ok" != 1 ]; then
+              log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
+              log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
+              log "never submitted. Do not rotate the credential on the strength of this line." >&2
+              exit 0
+            fi
+            body=$(curl -sS -m 10 \
+                     --data-urlencode "username=$QBT_WEBUI_USERNAME" \
+                     --data-urlencode "password=$QBT_WEBUI_PASSWORD" \
+                     "http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
+            if [ "$body" = "Ok." ]; then
+              log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
+              exit 0
+            fi
+            log "WebUI rejected the durable credential (login said '$body')" >&2
+            # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
+            # the running qbittorrent-nox is serving something it read from an EARLIER conf, so
+            # rewriting the same bytes on the next attempt would change nothing. The remedy is to
+            # make the process RE-READ the file, so restart the supervised service and let the next
+            # attempt re-verify. Without this the retry loop could never change its own outcome.
+            if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
+              log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
+              s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
+              sleep 3
+            fi
+          done
+
+          log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
+          # A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
+          # Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
+          # app, and the arrs' own download-client health check surfaces a broken credential.
+          exit 0
+podSecurityContext:
+  # For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
+  # on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
+  # the kubelet applies it, so /config becomes group-writable by the now-uid-1000 process without
+  # depending on the image chowning it. (This image DOES chown: init-qbittorrent-config runs
+  # `lsiown -R abc:abc /config`. Belt and braces — the live /config was 911:911 before this change,
+  # and losing write access to it would lose the WebUI password hash and the session state.)
+  fsGroup: 3000
+exposure: internal              # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
 egress: vpn                     # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
     size: 5Gi
     class: longhorn             # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
-dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
+    mountPath: /config          # THE APP WRITES /config, NOT the renderer default /data/config.
+                                # LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
+                                # /config), so without this the Longhorn PVC mounted at /data/config sat
+                                # holding nothing but lost+found while the app wrote its entire
+                                # configuration to the container's EPHEMERAL writable layer — destroyed on
+                                # every pod recreation: restart, reschedule, image bump. Measured in-cluster
+                                # 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
+                                # dev=2053 on the node's own disk with [Sentry asp config.xml logs].
+                                # This is the SAME failure catalog/plex.yaml:54 already records fixing.
+                                # The live /config contents were copied onto each PVC before this landed,
+                                # so API keys and credentials other apps reference were preserved.
+  # THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
+  # (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
+  # /downloads, which is precisely where this app is ALREADY configured to write.
+  #
+  # WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
+  #   * LEAST PRIVILEGE, AND IT COSTS NOTHING HERE. A download client has no business writing the
+  #     LIBRARY. The arrs need the root mount because THEY hardlink downloads/ -> tv/ or movies/ and
+  #     link() is a same-MOUNT operation (EXDEV otherwise) — their smallest common ancestor genuinely
+  #     IS the export root. qBittorrent performs no such cross-directory link: it writes into the
+  #     downloads tree and nothing else, so narrowing to that subtree removes movies/, tv/ and photos/
+  #     from this pod's mount namespace without removing any operation it actually performs.
+  #   * HARDLINKS STILL WORK, because they are not done here. The arrs still see ONE mount containing
+  #     both /media/downloads and /media/tv, so their link() stays intra-mount. Proven live 2026-09-12:
+  #     a file created from this pod under /downloads was hardlinked from the sonarr pod between
+  #     /media/downloads/... and /media/tv/... with st_nlink 1 -> 2 and no EXDEV.
+  #   * IT IS ONE MOUNT, so the L0 test `bulk-tier-distinct-from-replicated` assertion (6) — no
+  #     workload mounts one export twice — holds. That assertion is about TWO mounts inside ONE pod;
+  #     a single subPath mount is not the EXDEV trap, two are.
+  #   * The narrowing is a CLIENT/kubelet guarantee, stated honestly: the NFS server still exports the
+  #     whole dataset to this node, so the containment is the mount namespace, not the server's ACLs —
+  #     the same class of caveat docs/BULK-NFS-STORAGE.md §5 records for plex's readOnly.
+  #
+  # NOT readOnly — this is the one workload whose whole job is WRITING here.
+  #
+  # KNOWN, DELIBERATE SIDE EFFECT ON SHARED STATE: the image's init-qbittorrent-config runs
+  # `lsiown abc:abc /downloads` (non-recursive) whenever /downloads is a mount, so starting this pod
+  # chgrps /mnt/Storage/plex/downloads from 1000:1000 to 1000:3000 on the SERVER. Benign and verified:
+  # the directory is mode 0755, whose group bits (r-x) are identical to its other bits, so no consumer
+  # — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with
+  # non-cluster consumers, which should never be a surprise.
+  #
+  # PATH AGREEMENT WITH THE ARRS IS STILL *NOT* CLOSED — see DECISIONS-LOG 2026-09-12 "Remote Path
+  # Mappings". This pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
+  # /media/downloads/<name>. Same inode, different absolute path, so Completed Download Handling needs
+  # ONE Remote Path Mapping per arr (remote host `qbittorrent.qbittorrent.svc.cluster.local`, remote
+  # path /downloads/, local path /media/downloads/).
+  #
+  # THOSE MAPPINGS ARE NOW CONFIGURED AND VERIFIED ON PROD (2026-09-12), together with the qBittorrent
+  # download client itself — which did not exist in EITHER arr until then (both
+  # `GET /api/v3/downloadclient` returned `[]`, so completed-download handling had never been wired up
+  # at all). `downloadclient/testall` reports isValid=true in both, and Sonarr's queue was observed
+  # reporting outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it
+  # present — same torrent, same client.
+  #
+  # BUT THEY LIVE IN THE ARRS' OWN DATABASES (app state on their config PVCs), NOT IN GIT: a rebuild
+  # from git alone does NOT restore them and no gate here notices their absence. The alternative —
+  # repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead —
+  # buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was
+  # not chosen because it changes a running download client's save path, which the deferral above
+  # already shows is easy to get wrong. It is nonetheless the RECOMMENDED end state (it makes the
+  # mappings unnecessary and puts the mountPath half of the fix in git) and is awaiting an operator
+  # decision, not a technical blocker.
+  #
+  # RELATED OPEN BLOCKER: this app has no durable WebUI credential (no WebUI\Username /
+  # WebUI\Password_PBKDF2 in qBittorrent.conf), so v5 issues a NEW temporary `admin` password on every
+  # start. The arrs' saved download-client password therefore goes stale on every pod restart. See the
+  # same DECISIONS-LOG entry for the two fixes, neither of which was taken unilaterally.
+  - name: downloads
+    class: bulk
+    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export
+                                # plex (ro) and sonarr/radarr (rw) mount; one dataset, one set of inodes
+    subPath: downloads          # the handoff subtree ONLY: no movies/, no tv/, no photos/
+    mountPath: /downloads       # matches the app's EXISTING Session\DefaultSavePath / TempPath, so no
+                                # change to config-PVC app state is needed to make this take effect
+    size: 9Ti                   # ADVISORY ONLY on a static NFS PV (the ZFS pool is the real limit): the
+                                # dataset's 8.4 T used rounded up, stated the same way plex/sonarr/radarr
+                                # state it so `kubectl get pv` reports something true
+dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02).
+                                # STILL EMPTY WITH THE NFS MOUNT ABOVE, AND THAT IS CORRECT, NOT AN
+                                # OVERSIGHT. An NFS PersistentVolume is mounted by the NODE KERNEL in the
+                                # HOST netns, not by this pod: there is no pod-side socket to 2049, so no
+                                # CiliumNetworkPolicy can permit or deny it and none is needed. Verified by
+                                # construction — sonarr and radarr have mounted this same export read-write
+                                # for weeks and neither has any CNP naming the NFS server or port 2049
+                                # (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
+                                # egress gateway off this traffic (its selector is a podSelector), and
+                                # 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
+                                # platform/vpn-egress/egress-steering.yaml, "BULK NFS".
 ingressRoutes:
-  - host: qbittorrent.home.keithjackson.dev
+  - host: qbittorrent.${BASE_DOMAIN}
     port: 8080                  # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
   mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
   groups:
     - media-users               # authenticated AND in an authorized group (R-IDP-06)
+    # See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
+    # platform-admins here is a superset, not a widening (that group already carries cluster-admin
+    # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
+    # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
+    - platform-admins           # operators of these services must not be locked out of them
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the qBittorrent 5.x stable line

catalog/radarr.yaml

@@ -9,18 +9,90 @@ name: radarr
 kind: container
 source:
   image: lscr.io/linuxserver/radarr:5.21.1   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
+# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
+# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
+# which cannot write a 1000:3000-owned 0755 tree.
+env:
+  - name: PUID
+    value: "1000"               # owner uid of every directory under /mnt/Storage/plex
+  - name: PGID
+    value: "3000"               # dominant group there; keeps files this writes readable to the
+                                # EXISTING non-k3s consumers of the same export, which stay live
+  - name: TZ
+    value: Etc/UTC              # same safe default plex uses; the arrs schedule by local time
+podSecurityContext:
+  # For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
+  # does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
+  # what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
+  # /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
+  fsGroup: 3000
+exposure: internal              # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
     size: 5Gi
     class: longhorn             # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
-  # NOTE (bulk-media PREREQUISITE): docs/13 row 3 places the media library / downloads volume on the
+    mountPath: /config          # THE APP WRITES /config, NOT the renderer default /data/config.
-  # `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
+                                # LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
-  # but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
+                                # /config), so without this the Longhorn PVC mounted at /data/config sat
-  # volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
+                                # holding nothing but lost+found while the app wrote its entire
-  # therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Radarr deployable today.
+                                # configuration to the container's EPHEMERAL writable layer — destroyed on
+                                # every pod recreation: restart, reschedule, image bump. Measured in-cluster
+                                # 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
+                                # dev=2053 on the node's own disk with [Sentry asp config.xml logs].
+                                # This is the SAME failure catalog/plex.yaml:54 already records fixing.
+                                # The live /config contents were copied onto each PVC before this landed,
+                                # so API keys and credentials other apps reference were preserved.
+  # THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
+  # /media/movies (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
+  #
+  # *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Radarr imports by HARDLINKING or
+  # atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
+  # and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
+  #     /mnt/Storage/plex/movies/                      the library
+  #     /mnt/Storage/plex/downloads/complete/radarr/  where finished releases land
+  # — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
+  # them as two subPath volumes (subPath: movies at /media/movies, subPath: downloads elsewhere) creates
+  # TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Radarr then
+  # silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
+  # seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
+  # is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
+  # requirement, not a shortcut. (The cost is stated plainly: Radarr can also write /media/tv,
+  # which is sonarr's. Narrowing that would cost hardlinks, which is a worse trade.)
+  #
+  # *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
+  # renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
+  # config volume entirely; the pod would come up with its app database shadowed by an 8 T media
+  # tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
+  # the same export, so when qbittorrent is eventually given this mount too, the arrs and the
+  # download client will already agree on absolute paths — which is the other precondition for
+  # hardlinked imports (otherwise every import needs a Remote Path Mapping).
+  #
+  # SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
+  # is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
+  # with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
+  # Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
+  # documents having already been bitten by. Fixing it means starting from an empty config volume, so
+  # it belongs in its own change with the operator present.
+  #
+  # UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
+  # + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
+  # the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
+  # write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
+  # every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
+  #
+  # AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
+  # ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
+  # lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
+  # that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
+  # the image re-chowning /config for itself.
+  - name: media
+    class: bulk
+    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
+    mountPath: /media           # => /media/movies (Root Folder) and /media/downloads (import source)
+    size: 9Ti                   # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
+                                # rounded up, so `kubectl get pv` reports something true.
 dependencies:
   - to: qbittorrent             # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 8080
@@ -29,12 +101,17 @@ dependencies:
     port: 9117
     protocol: tcp
 ingressRoutes:
-  - host: radarr.home.keithjackson.dev
+  - host: radarr.${BASE_DOMAIN}
     port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
   groups:
     - media-users               # authenticated AND in the media group (R-IDP-06)
+    # See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
+    # platform-admins here is a superset, not a widening (that group already carries cluster-admin
+    # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
+    # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
+    - platform-admins           # operators of these services must not be locked out of them
 update:
   channel: stable               # R-UPD-01
   range: ">=5.0.0 <6.0.0"       # stay on the Radarr v5 stable line

catalog/seerr.yaml

@@ -57,7 +57,7 @@ dependencies:
     port: 32400
     protocol: tcp
 ingressRoutes:
-  - host: seerr.home.keithjackson.dev
+  - host: seerr.${BASE_DOMAIN}
     port: 5055                   # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: native                   # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)

catalog/semaphore.yaml

@@ -58,7 +58,7 @@ env:
   - name: SEMAPHORE_ADMIN_NAME
     value: Semaphore Admin
   - name: SEMAPHORE_ADMIN_EMAIL
-    value: admin@home.keithjackson.dev
+    value: admin@${BASE_DOMAIN}
   # --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
   #     SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
   - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
@@ -78,16 +78,34 @@ env:
   #     (util ConfigType `oidc_providers`, env-tagged SEMAPHORE_OIDC_PROVIDERS). The value is a single
   #     JSON blob for the `keycloak` provider — provider_url is the homelab realm issuer (auto-discovery),
   #     client_id=semaphore, redirect_url=/api/auth/oidc/keycloak/redirect, claims map to the realm's
-  #     token. It CONTAINS the confidential client secret, so the whole blob is held SOPS-encrypted in the
+  #     token. The name is NOT a kubelet service-link var (those are SEMAPHORE_SERVICE_HOST /
-  #     semaphore-bootstrap Secret (key oidc-providers-json; same secret value SOPS-held in
+  #     SEMAPHORE_PORT_*), so no collision. The LOCAL SEMAPHORE_ADMIN above stays break-glass (docs/15).
-  #     keycloak-realm-clients) and injected here by reference — never in cleartext. The name is NOT a
+  #
-  #     kubelet service-link var (those are SEMAPHORE_SERVICE_HOST / SEMAPHORE_PORT_*), so no collision.
+  #     PER-ENVIRONMENT URLS (2026-09-07 fix). The blob USED to live whole inside the SOPS-encrypted
-  #     The LOCAL SEMAPHORE_ADMIN above stays as the break-glass path (docs/15). ---
+  #     semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
-  - name: SEMAPHORE_OIDC_PROVIDERS
+  #     client secret. That hid two environment-specific, NON-SECRET URLs — the realm issuer and the
+  #     redirect URI — inside ciphertext, where Flux's ${BASE_DOMAIN} postBuild substitution CANNOT
+  #     reach them. Both were frozen at the apex `home.keithjackson.dev`, so PROD (BASE_DOMAIN =
+  #     prod.home.keithjackson.dev) pointed its OIDC at DEV's Keycloak — unroutable from prod's
+  #     192.168.2.0/24 (the identical root cause as the Headlamp login timeout; see
+  #     platform/headlamp/README.md). Dev was only ever "right" because dev IS the apex.
+  #
+  #     THE SPLIT: the URLs are public, so the blob is now a PLAINTEXT literal here and gets
+  #     ${BASE_DOMAIN}-substituted per environment like every other hostname. Only the confidential
+  #     client secret stays SOPS-encrypted, in its own key (`oidc-client-secret`), and is spliced back
+  #     in by the KUBELET's dependent-env-var expansion: an env `value` may reference an EARLIER env
+  #     var of the same container as $(NAME), and secretKeyRef-sourced vars are eligible. So
+  #     SEMAPHORE_OIDC_CLIENT_SECRET MUST STAY IMMEDIATELY ABOVE — the kubelet resolves env in list
+  #     order, and a reference to a not-yet-defined name is left as the LITERAL text $(...). It is
+  #     fail-loud in the way that matters: if the Secret key is missing the Pod never starts
+  #     (CreateContainerConfigError), rather than silently authenticating against nothing. ---
+  - name: SEMAPHORE_OIDC_CLIENT_SECRET
     valueFrom:
       secretKeyRef:
         name: semaphore-bootstrap
-        key: oidc-providers-json
+        key: oidc-client-secret
+  - name: SEMAPHORE_OIDC_PROVIDERS
+    value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
 # The semaphoreui/semaphore image runs as the non-root user UID 1001 / GID 0 (tini -> server-wrapper),
 # and on boot the wrapper writes its generated config.json into the config dir. A mounted Longhorn PVC
 # is root:root, so UID 1001 cannot write it -> the pod panics ("permission denied"). fsGroup=1001 makes
@@ -110,17 +128,19 @@ dependencies:
     port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
     protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: semaphore.home.keithjackson.dev
+  - host: semaphore.${BASE_DOMAIN}
     port: 3000                  # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
   mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
   groups:
     - platform-admins           # access scoped to an authorized group (R-IDP-06)
   # WIRED (native OIDC): the Keycloak `semaphore` confidential client (platform/keycloak/keycloak-realm-
-  # import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env (JSON blob held encrypted
+  # import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env — a PLAINTEXT JSON blob
-  # in the semaphore-bootstrap Secret, oidc-providers-json). The `groups` claim is exposed for
+  # (see `env` above) whose per-environment URLs are ${BASE_DOMAIN}-substituted, with the confidential
-  # authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the break-glass path so an IdP outage
+  # client secret spliced in at runtime from the semaphore-bootstrap Secret key `oidc-client-secret`.
-  # never locks the operator out (docs/15 "Availability caveat"). No secret is embedded here.
+  # The `groups` claim is exposed for authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the
+  # break-glass path so an IdP outage never locks the operator out (docs/15 "Availability caveat").
+  # No secret is embedded here.
 update:
   channel: stable               # R-UPD-01
   range: ">=2.19.0 <3.0.0"      # stay on the 2.x stable line

catalog/sonarr.yaml

@@ -9,18 +9,90 @@ name: sonarr
 kind: container
 source:
   image: lscr.io/linuxserver/sonarr:4.0.15   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
+# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
+# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
+# which cannot write a 1000:3000-owned 0755 tree.
+env:
+  - name: PUID
+    value: "1000"               # owner uid of every directory under /mnt/Storage/plex
+  - name: PGID
+    value: "3000"               # dominant group there; keeps files this writes readable to the
+                                # EXISTING non-k3s consumers of the same export, which stay live
+  - name: TZ
+    value: Etc/UTC              # same safe default plex uses; the arrs schedule by local time
+podSecurityContext:
+  # For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
+  # does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
+  # what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
+  # /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
+  fsGroup: 3000
+exposure: internal              # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                 # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
   - name: config
     size: 5Gi
     class: longhorn             # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
-  # NOTE (bulk-media PREREQUISITE): docs/13 row 2 places the media library / downloads volume on the
+    mountPath: /config          # THE APP WRITES /config, NOT the renderer default /data/config.
-  # `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
+                                # LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
-  # but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
+                                # /config), so without this the Longhorn PVC mounted at /data/config sat
-  # volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
+                                # holding nothing but lost+found while the app wrote its entire
-  # therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Sonarr deployable today.
+                                # configuration to the container's EPHEMERAL writable layer — destroyed on
+                                # every pod recreation: restart, reschedule, image bump. Measured in-cluster
+                                # 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
+                                # dev=2053 on the node's own disk with [Sentry asp config.xml logs].
+                                # This is the SAME failure catalog/plex.yaml:54 already records fixing.
+                                # The live /config contents were copied onto each PVC before this landed,
+                                # so API keys and credentials other apps reference were preserved.
+  # THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
+  # /media/tv (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
+  #
+  # *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Sonarr imports by HARDLINKING or
+  # atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
+  # and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
+  #     /mnt/Storage/plex/tv/                      the library
+  #     /mnt/Storage/plex/downloads/complete/tv-sonarr/  where finished releases land
+  # — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
+  # them as two subPath volumes (subPath: tv at /media/tv, subPath: downloads elsewhere) creates
+  # TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Sonarr then
+  # silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
+  # seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
+  # is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
+  # requirement, not a shortcut. (The cost is stated plainly: Sonarr can also write /media/movies,
+  # which is radarr's. Narrowing that would cost hardlinks, which is a worse trade.)
+  #
+  # *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
+  # renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
+  # config volume entirely; the pod would come up with its app database shadowed by an 8 T media
+  # tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
+  # the same export, so when qbittorrent is eventually given this mount too, the arrs and the
+  # download client will already agree on absolute paths — which is the other precondition for
+  # hardlinked imports (otherwise every import needs a Remote Path Mapping).
+  #
+  # SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
+  # is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
+  # with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
+  # Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
+  # documents having already been bitten by. Fixing it means starting from an empty config volume, so
+  # it belongs in its own change with the operator present.
+  #
+  # UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
+  # + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
+  # the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
+  # write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
+  # every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
+  #
+  # AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
+  # ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
+  # lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
+  # that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
+  # the image re-chowning /config for itself.
+  - name: media
+    class: bulk
+    export: /mnt/Storage/plex   # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
+    mountPath: /media           # => /media/tv (Root Folder) and /media/downloads (import source)
+    size: 9Ti                   # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
+                                # rounded up, so `kubectl get pv` reports something true.
 dependencies:
   - to: qbittorrent             # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
     port: 8080
@@ -29,12 +101,17 @@ dependencies:
     port: 9117
     protocol: tcp
 ingressRoutes:
-  - host: sonarr.home.keithjackson.dev
+  - host: sonarr.${BASE_DOMAIN}
     port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
   mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
   groups:
     - media-users               # authenticated AND in the media group (R-IDP-06)
+    # See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
+    # platform-admins here is a superset, not a widening (that group already carries cluster-admin
+    # via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
+    # immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
+    - platform-admins           # operators of these services must not be locked out of them
 update:
   channel: stable               # R-UPD-01
   range: ">=4.0.0 <5.0.0"       # stay on the Sonarr v4 stable line

ci/README.md

@@ -27,9 +27,31 @@ make -C ci test        # unit tests for these scripts themselves (ci/tests/*.tes
 ```
 
 The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
-(`kyverno test`), and **`conformance/run.sh --layer l0`** — the whole-repo L0 invariants a
+(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo
-per-resource policy cannot express (secret hygiene across every tracked file, least-privilege RBAC,
+L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
-spec-immutability enforcement). See `conformance/tests/README.md`.
+least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`.
+
+### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped
+
+R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the
+worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
+coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be
+selected *by* the requirements a change touches — because **the change most likely to break it
+attributes no requirement at all**. Requirement-scoped selection then filters the check out of
+exactly that change.
+
+This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
+3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
+`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that
+failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write
+a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"`
+and the runner unions every declaring test into the run. Adding the annotation is the only step.
+
+`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the
+tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the
+gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably,
+by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the
+full rationale and the incidents behind it.
 
 ## Exit-code contract
 
@@ -37,6 +59,47 @@ Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIP
 with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
 the gate stays honest in Phase 0 (no fake green).
 
+### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)
+
+**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It
+used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a
+warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
+`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule
+unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as
+merge-approved. Fail-open in the one component whose whole job is to fail closed.
+
+| verdict | exit | meaning |
+|---|---|---|
+| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. |
+| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. |
+| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. |
+| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. |
+
+Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.
+
+**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so
+`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A
+caller that must distinguish them invokes `./ci/gate.sh` directly.
+
+### Which skips block, and why — a skip is not one thing
+
+| stage | a skip there means | verdict |
+|---|---|---|
+| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
+| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
+| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
+
+That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
+
+`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml`
+sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls
+back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction
+cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI
+strictness.
+
+Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed
+stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate).
+
 ### The touched-requirement subset can also be *undeterminable* (exit 3)
 
 `touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
@@ -87,6 +150,8 @@ ci/
   lint.sh                  Stage: yaml/lint
   l0.sh                    Stage: schema validation + Kyverno L0 gate + L0 static conformance
   l1.sh                    Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
+                           (harness: platform/monitoring/tests/). Catches the two alerting bugs no
+                           other stage can see — permanently-firing and structurally INERT rules.
   conformance.sh           Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
   touched-requirements.sh  compute the R-* subset a change touches (drives the R-TEST-03 filter);
                            exits 3 LOUDLY rather than guessing when the diff base is unreliable
@@ -122,3 +187,15 @@ its owner delivers a runnable entrypoint.
 `CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and
 be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+)
 turns any unverified cluster-needing touched requirement into a hard block.
+
+All four switches are read through one `truthy()` in `lib.sh`, so they spell "on" identically
+(`true|1|yes|y|on`, any case) and — the direction that matters — **anything else is OFF**: a
+misspelled `CLUSTER_AVAILABLE=ture` leaves the capability undeclared instead of silently claiming
+it, which can only make the gate stricter about what it says it verified.
+
+| switch | default | effect |
+|---|---|---|
+| `CLUSTER_AVAILABLE` | `false` | declares a cluster; L2/L3 then run and are required |
+| `REQUIRE_CLUSTER_TESTS` | `false` | Phase 1+: an unverified cluster-needing touched requirement BLOCKS |
+| `REQUIRE_SPEC_VERIFICATION` | `false` | an unverifiable spec repo BLOCKS instead of UNDETERMINED |
+| `GATE_STRICT_SKIPS` | `auto` | `true` = a missing tool / unwired seam BLOCKS (CI); `auto` follows `GITHUB_ACTIONS`/`CI` |

ci/conformance.sh

@@ -47,7 +47,11 @@ fi
 # (conformance/lib/common.sh) now unions a comma/space list of layers, so this is a single
 # invocation with one aggregated report (no per-layer looping needed). Artifacts are then
 # COLLECTED from the fixed location below.
-"$CONFORMANCE_CMD" --layer l2,l3 "${req_args[@]}"
+# --include-scope: union the touched subset with every L2/L3 test DECLARING
+# `conformance.home.k3s/scope: whole-repo` (none today — wired once, here, so a future
+# whole-repo behavioural invariant needs only the annotation; see conformance/lib/common.sh).
+: "${WHOLE_REPO_SCOPE:=whole-repo}"
+"$CONFORMANCE_CMD" --layer l2,l3 --include-scope "$WHOLE_REPO_SCOPE" "${req_args[@]}"
 
 CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
 if [ -f "${CONF_ART}/report.junit.xml" ]; then

ci/gate.sh

@@ -7,6 +7,75 @@
 #   skip (3)  not run (cluster-gated stage w/ no cluster, or unwired seam) —
 #             NOT counted as green (HARD RULE 4: don't fake green)
 #
+# ...AND THREE OUTCOMES FOR THE GATE ITSELF (2026-09-13). Until today a stage that
+# SKIPPED only printed a warning ("— not green") and the gate then printed
+# `MERGE GATE: PASS` and exited 0 anyway. That is fail-open in the one component whose
+# entire job is to fail closed: a pinned tool missing from the runner, or a seam silently
+# becoming unwired, produced exit 0 and read as merge-approved. Observed live:
+# `prom-rules: skipped (missing tool) — not green` followed by `MERGE GATE: PASS`, and
+# `l0: skipped (kyverno CLI not installed)` likewise. The gate's own vocabulary already
+# had the answer — the 2026-08-31 "local false green" introduced UNDECIDED for exactly
+# this shape of non-answer — the offline stages just did not use it.
+#
+#   MERGE GATE: PASS           exit 0   every stage that was SUPPOSED TO RUN ran and was
+#                                       green, and the R-TEST-03 subset was checked.
+#   MERGE GATE: BLOCKED        exit 1   a stage FAILED, or a stage did not run WHERE IT
+#                                       MUST (see the per-stage rules below), or the
+#                                       decision input is unusable.
+#   MERGE GATE: UNDETERMINED   exit 3   nothing failed, but a stage did not run, so
+#                                       greenness was NOT ESTABLISHED. Never a pass;
+#                                       non-zero so no automation can read it as approval.
+#                                       `3` is this repo's existing "not run / not
+#                                       established" code (ci/lib.sh), now spoken by the
+#                                       gate as a whole and not only by its stages.
+#   MERGE GATE: NOT APPLICABLE exit 0   there is no branch under gate at all (HEAD is on
+#                                       base) — a push-to-main CI run. Stages were green;
+#                                       no subset was checked, and the headline says so.
+#
+# NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so
+# `make -C ci gate` collapses 1 and 3 into 2 — a human sees the headline and that is enough.
+# A caller that must tell BLOCKED from UNDETERMINED invokes ./ci/gate.sh directly; to tolerate
+# an UNDETERMINED run it writes `./ci/gate.sh || [ $? -eq 3 ]`, the same explicit idiom
+# .gitea/workflows/gate.yml already used for its spec-immutability pre-step and now uses for
+# the gate itself (with GATE_STRICT_SKIPS=true, so that tolerance can ONLY ever cover the
+# declared-environmental skip below — a missing tool on the runner is exit 1, untolerated).
+# TOLERATING A NON-ANSWER IS A VISIBLE, REVIEWED CHOICE IN THE CALLER. There is deliberately no
+# knob inside this script that turns UNDETERMINED into exit 0: that would be the fail-open this
+# change exists to remove, just relocated.
+#
+# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
+#
+#   lint / l0 / render-golden / l1   A skip here means a PINNED tool is absent or a wired
+#     seam went unwired. On a CI runner the pinned toolchain is installed for us by
+#     .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
+#     environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
+#     it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
+#     in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
+#
+#   spec-immutability (R-TEST-06)    A skip means the sibling spec repo is not checked out, so
+#     the spec COULD NOT BE VERIFIED. Never a pass — unverified is unverified — so it denies a
+#     PASS and yields UNDETERMINED. It is deliberately NOT upgraded to BLOCKED by CI strictness,
+#     because on the live runner it is not a defect anyone can fix from inside this repo: the
+#     workflow does not check the spec repo out at all (SPEC_REPO_SLUG is unwired — the blueprint
+#     lives in merge-gate.yaml), and blocking would leave the only enforced status check
+#     permanently red for a reason no PR could clear. So THIS is the one skip whose non-answer the
+#     CALLER may knowingly tolerate, and gate.yml does so in one visible line. Locally the
+#     remedy exists and the message names it: check the spec repo out beside this one, or point
+#     SPEC_REPO at it (infra/devcontainer/dev.sh mounts it automatically when it is a sibling).
+#     Set REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a
+#     hard block once that checkout is wired.
+#
+#   conformance (L2/L3)              A skip when CLUSTER_AVAILABLE!=true is NOT a defect and
+#     NOT an accident: it is the environment DECLARING that the capability is absent, which
+#     is the documented Phase-0 behaviour. A declared-inapplicable stage was never "supposed
+#     to run", so it does not deny a PASS — the touched requirements it would have covered
+#     are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that
+#     into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as
+#     it already did. That asymmetry is the whole distinction this file now draws:
+#     DECLARED-not-applicable may pass; COULD-NOT-RUN may not. A declared gap is still NAMED in
+#     the PASS headline (see DECLARED_GAPS below), so a PASS never reads as "everything was
+#     verified" when a layer was declared out of scope.
+#
 # Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
 # MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
 # available, the gate reports those requirements UNVERIFIED and — unless
@@ -20,6 +89,7 @@ source "${CI_DIR}/lib.sh"
 set +e   # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
 
 : "${REQUIRE_CLUSTER_TESTS:=false}"
+: "${REQUIRE_SPEC_VERIFICATION:=false}"
 
 declare -A OUTCOME
 run_stage() {
@@ -47,6 +117,13 @@ run_stage conformance   "${CI_DIR}/conformance.sh"
 echo >&2
 log "==== gate decision ===="
 BLOCK=0
+# Stages that did NOT run and are tolerated here — but never as green. A non-empty list
+# means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3.
+NOT_ESTABLISHED=()
+# Stages the environment DECLARED inapplicable (today: L2/L3 with no cluster declared). These
+# do not deny a PASS — nothing was supposed to run — but they are named IN the PASS headline, so
+# "PASS" is never mistaken for "everything was verified".
+DECLARED_GAPS=()
 
 # --- R-TEST-03 input: the touched-requirement subset --------------------------
 # Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
@@ -64,37 +141,80 @@ rm -f "$_reason_file"
 TOUCHED=()
 [ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
 UNDECIDED=0
+SUBSET_BROKEN=0
 if [ "$TOUCHED_RC" -eq 3 ]; then
   UNDECIDED=1
   echo >&2
   err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
   err "  determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
   summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
+elif [ "$TOUCHED_RC" -ne 0 ]; then
+  # Any OTHER non-zero exit is a hard defect in the subset computation itself — an
+  # incomplete path->requirement map (a changed path matches no row) or an invalid row
+  # would report exactly this. The reasoning is taken from the unlanded
+  # fix/touched-requirements-fail-closed (47068ee): gate.sh special-cased ONLY exit 3, so
+  # any other non-zero exit left TOUCHED empty and UNDECIDED=0, and the decision below
+  # printed "MERGE GATE: PASS" having applied no coverage gate at all. Unreachable on
+  # `main` today (touched-requirements.sh exits only 0 or 3) and wired here so it is
+  # already closed the moment a fail-closed path map makes exit 1 reachable. A subset that
+  # cannot be computed is NOT an empty subset.
+  SUBSET_BROKEN=1
+  echo >&2
+  err "R-TEST-03 DECISION INPUT BROKEN: touched-requirements.sh exited ${TOUCHED_RC}"
+  err "  (${TOUCHED_REASON:-unknown}). Its output above names the paths/rows to fix."
+  summary "- **R-TEST-03 subset UNUSABLE** (\`${TOUCHED_REASON:-unknown}\`, exit ${TOUCHED_RC}) — the coverage gate could not be applied."
 elif [ "${#TOUCHED[@]}" -gt 0 ]; then
   log "touched requirements: ${TOUCHED[*]}"
 fi
 
 # spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
-# A skip means the spec repo wasn't available to verify — surfaced, not green.
+# A skip means the spec repo wasn't available to verify — UNVERIFIED, never green: it
+# denies a PASS (exit 3) and, with REQUIRE_SPEC_VERIFICATION=true, blocks outright. See the
+# file header for why this one is not upgraded to a block by CI detection alone.
 case "${OUTCOME[spec-immutability]}" in
   fail) err  "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
-  skip) warn "spec-immutability: skipped (spec repo unavailable) — not verified" ;;
+  skip)
+    if truthy "$REQUIRE_SPEC_VERIFICATION"; then
+      err "spec-immutability: skipped (spec repo unavailable) and REQUIRE_SPEC_VERIFICATION=true -> blocks merge"
+      summary "- spec-immutability: **SKIPPED** (spec repo unavailable) — blocks merge (REQUIRE_SPEC_VERIFICATION=true)"
+      BLOCK=1
+    else
+      warn "spec-immutability: skipped (spec repo unavailable) — R-TEST-06 NOT VERIFIED, so"
+      warn "  this run cannot be a PASS. Check the spec repo out beside this one (or set"
+      warn "  SPEC_REPO) to verify it."
+      summary "- spec-immutability: **SKIPPED** (spec repo unavailable — R-TEST-06 not verified; not a pass)"
+      NOT_ESTABLISHED+=("spec-immutability")
+    fi ;;
   pass) ok   "spec-immutability: green" ;;
 esac
 
-# L0/L1 stages: any failure blocks. A SKIP here means a seam is unwired — in
+# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
-# Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
+# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
+# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
 # no-cluster reproducibility gate (canonical render == committed apps/ goldens):
 # a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
 for s in lint l0 render-golden l1; do
   case "${OUTCOME[$s]}" in
     fail) err  "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
-    skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;
+    skip)
+      if strict_skips; then
+        err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
+        err "  toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
+        err "  seam is on main — so this is a DEFECT, not an environment -> blocks merge"
+        summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
+        BLOCK=1
+      else
+        warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS"
+        summary "- ${s}: **SKIPPED** (unwired seam / missing tool) — not green"
+        NOT_ESTABLISHED+=("$s")
+      fi ;;
     pass) ok   "${s}: green" ;;
   esac
 done
 
-# L2/L3 conformance: cluster-gated.
+# L2/L3 conformance: cluster-gated. The ONLY stage whose non-execution can still be a PASS,
+# because CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent rather
+# than the stage failing to run where it could have — see the file header.
 case "${OUTCOME[conformance]}" in
   fail)
     err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
@@ -105,18 +225,22 @@ case "${OUTCOME[conformance]}" in
       err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
     elif [ "$UNDECIDED" -eq 1 ]; then
       warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
+      DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
     else
       if [ "${#TOUCHED[@]}" -gt 0 ]; then
         reqs=("${TOUCHED[@]}")
         warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
         warn "  (no cluster — Phase 0). These are NOT certified green."
         summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
-        if [ "$REQUIRE_CLUSTER_TESTS" = "true" ]; then
+        if truthy "$REQUIRE_CLUSTER_TESTS"; then
           err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
           BLOCK=1
+        else
+          DECLARED_GAPS+=("conformance-l2-l3(no cluster declared; ${#TOUCHED[@]} touched reqs unverified)")
         fi
       else
         skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
+        DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
       fi
     fi ;;
 esac
@@ -132,13 +256,34 @@ if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
   err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
   BLOCK=1
 fi
+# An incomplete/invalid path map is not an environment problem to skip past: the change under
+# gate contains paths whose requirements are UNKNOWN, so no subset can be certified.
+if [ "$SUBSET_BROKEN" -eq 1 ]; then
+  err "R-TEST-03 subset unusable (${TOUCHED_REASON:-unknown}) -> blocks merge"
+  BLOCK=1
+fi
 
 echo >&2
+# Precedence: BLOCKED (something is wrong) > UNDETERMINED (a stage did not run, so
+# greenness was not established) > NOT APPLICABLE (nothing to gate) > PASS. UNDETERMINED
+# outranks NOT APPLICABLE because "no branch under gate" AND "a stage did not run" is two
+# reasons this is not a green run, and the stronger, non-zero one must be the headline.
 if [ "$BLOCK" -ne 0 ]; then
   err "MERGE GATE: BLOCKED"
   summary ""
   summary "**Result: BLOCKED** — see failures above."
   exit 1
+elif [ "${#NOT_ESTABLISHED[@]}" -gt 0 ]; then
+  err "MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED."
+  err "  Stages that did not run: ${NOT_ESTABLISHED[*]}"
+  err "  Nothing FAILED, and nothing is certified either: a stage that did not run is not"
+  err "  green (HARD RULE 4). This is exit 3, never 0 — no automation may read it as"
+  err "  merge-approved. Fix the environment (install the pinned tool / wire the seam /"
+  err "  check out the spec repo) and run again, or accept it KNOWINGLY and explicitly:"
+  err "    ./ci/gate.sh || [ \$? -eq 3 ]"
+  summary ""
+  summary "**Result: UNDETERMINED** — these stages did not run, so nothing is certified: \`${NOT_ESTABLISHED[*]}\`. Not a pass (exit 3)."
+  exit 3
 elif [ "$UNDECIDED" -eq 1 ]; then
   warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
   warn "  The stages above ran and did not fail, but NO touched-requirement subset was"
@@ -148,8 +293,16 @@ elif [ "$UNDECIDED" -eq 1 ]; then
   summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
   exit 0
 else
-  ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
+  ok "MERGE GATE: PASS — every stage that was supposed to run RAN and was GREEN, and the"
+  ok "  R-TEST-03 touched-requirement subset was checked."
+  if [ "${#DECLARED_GAPS[@]}" -gt 0 ]; then
+    ok "  NOT verified here, because the environment DECLARED it out of scope (never a silent"
+    ok "  gap — Phase 0 by policy, see the header): ${DECLARED_GAPS[*]}"
     summary ""
-  summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
+    summary "**Result: PASS** — every stage that was supposed to run ran green. Declared out of scope (NOT verified): \`${DECLARED_GAPS[*]}\`."
+  else
+    summary ""
+    summary "**Result: PASS** — every stage that was supposed to run ran green; nothing was declared out of scope."
+  fi
   exit 0
 fi

ci/l0.sh

@@ -10,6 +10,18 @@
 #   Stage 3b L0     : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
 #                     a per-resource policy cannot express (secret hygiene across every
 #                     tracked file, least-privilege RBAC, spec-immutability enforcement).
+#                     Selection = the TOUCHED requirement subset (R-TEST-03) UNION every L0
+#                     test DECLARING `conformance.home.k3s/scope: whole-repo`. That union
+#                     replaces what used to be a hand-written "UNCONDITIONAL" stage per
+#                     whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e
+#                     bulk-tier): three near-identical 12-line rationales, three hardcoded
+#                     check.sh paths and three copies of the same rc 0/3/* case block, which
+#                     did not scale and — worse — failed SILENTLY, because whoever added the
+#                     next whole-repo invariant had to REMEMBER to hand-write a stage. Now the
+#                     test declares its own scope and the runner enforces it. The reasoning and
+#                     the incidents those stages recorded were moved INTO the tests they
+#                     describe (see each chainsaw-test.yaml's `scope` annotation) and the
+#                     general mechanism is documented in conformance/lib/common.sh.
 #
 # Integration seams are env-overridable (see ci/lib.sh + README). When an owning
 # subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
@@ -83,21 +95,45 @@ fi
 # binding, the spec-immutability gate's own behaviour) live in the conformance
 # harness's tests/l0-static subtree and run with no cluster. This stage executes them
 # through the same single-command runner CI and a laptop both use (R-TEST-02).
-log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0)"
+#
+# THE UNION, AND WHY IT IS NOT OPTIONAL (R-TEST-03's blind spot)
+# --------------------------------------------------------------
+# 3b scopes the conformance run to the TOUCHED requirements. That is right for an invariant
+# whose inputs live under the subtree its requirements map to. It is WRONG for one that
+# quantifies over the WHOLE tracked tree, or over subtrees (apps/, infra/, most of platform/)
+# that ci/touched-requirements.sh's coarse path map keys to NOTHING: for those, the change
+# most likely to break the invariant attributes no requirement at all, so requirement-scoped
+# selection filters the check out of EXACTLY that change. `--include-scope whole-repo` unions
+# in every L0 test that DECLARES that scope, so those run on every gate invocation.
+#
+# rc 3 IS NOT A PASS. The runner returns 3 when a scope-declared check SKIPPED (its subject
+# subtree is absent, or it has no check script): UNWIRED, not green, exactly as the deleted
+# per-check stages had it ("check SKIPPED (clusters/ absent) — not a pass"). It is mapped to
+# the gate's own rc 3 below, never to rc 0.
+: "${WHOLE_REPO_SCOPE:=whole-repo}"
+log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0 --include-scope ${WHOLE_REPO_SCOPE})"
 if ! entrypoint_ready "$CONFORMANCE_CMD"; then
   skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
   summary "- l0-static: **SKIPPED** (conformance runner missing)"
   if [ "$rc" -eq 0 ]; then rc=3; fi
 else
   load_touched_requirements
-  if "$CONFORMANCE_CMD" --layer l0 ${REQ_ARGS[@]+"${REQ_ARGS[@]}"}; then
+  l0_static_rc=0
-    ok "L0 static conformance checks passed"
+  "$CONFORMANCE_CMD" --layer l0 --include-scope "$WHOLE_REPO_SCOPE" \
-    summary "- l0-static: **passed**"
+    ${REQ_ARGS[@]+"${REQ_ARGS[@]}"} || l0_static_rc=$?
-  else
+  case "$l0_static_rc" in
-    err "L0 static conformance checks failed"
+    0) ok "L0 static conformance checks passed"
+       summary "- l0-static: **passed**" ;;
+    3) skip "an L0 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
+       summary "- l0-static: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
+       if [ "$rc" -eq 0 ]; then rc=3; fi ;;
+    *) err "L0 static conformance checks failed"
        summary "- l0-static: **FAILED**"
-    rc=1
+       rc=1 ;;
-  fi
+  esac
+  # Per-check step-summary lines, so the gate output still NAMES what ran — the generated
+  # replacement for the one hardcoded `summary` line each deleted stage carried.
+  emit_offline_summary
 fi
 
 case "$rc" in

ci/l1.sh

@@ -29,7 +29,18 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
 # [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
 # (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
 # touched subset, then COLLECT its artifacts into the CI artifact dirs.
-"$CONFORMANCE_CMD" --layer l1 ${req_args[@]+"${req_args[@]}"}
+# --include-scope: union the touched subset with every L1 test DECLARING
+# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
+# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
+# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
+# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
+# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
+# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
+# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
+: "${WHOLE_REPO_SCOPE:=whole-repo}"
+l1_rc=0
+"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
+  ${req_args[@]+"${req_args[@]}"} || l1_rc=$?
 
 CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
 if [ -f "${CONF_ART}/report.junit.xml" ]; then
@@ -39,5 +50,16 @@ if [ -d "${CONF_ART}/coverage" ]; then
   cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi
 
-ok "L1 rendering checks passed"
+case "$l1_rc" in
-summary "- l1-render: **passed**"
+  0) ok "L1 rendering checks passed"
+     summary "- l1-render: **passed**"
+     emit_offline_summary ;;
+  3) skip "an L1 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
+     summary "- l1-render: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
+     emit_offline_summary
+     exit 3 ;;
+  *) err "L1 rendering checks FAILED (rc=${l1_rc})"
+     summary "- l1-render: **FAILED**"
+     emit_offline_summary
+     exit "$l1_rc" ;;
+esac

ci/lib.sh

@@ -72,20 +72,83 @@ entrypoint_ready() {
   return 1
 }
 
-# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
+# truthy <value> — the ONE place the repo's "is this switch on" spelling lives, so
-# A cluster is 'available' only if explicitly declared. We NEVER auto-assume a
+# CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS
-# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
+# all agree. Anything not listed is OFF, and that direction matters: a misspelled value
-#   CLUSTER_AVAILABLE=true   -> L2/L3 run and are REQUIRED green
+# (CLUSTER_AVAILABLE=ture, REQUIRE_SPEC_VERIFICATION=yeah) leaves the guard OFF rather than
-#   CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
+# silently declaring a capability "available", so a typo can only make the gate stricter about
-cluster_available() {
+# what it claims to have verified, never looser.
-  case "${CLUSTER_AVAILABLE:-false}" in
+truthy() {
-    true|1|yes) return 0 ;;
+  case "${1:-}" in
+    true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;;
     *) return 1 ;;
   esac
 }
 
+# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
+# A cluster is 'available' only if explicitly DECLARED. We NEVER auto-assume a
+# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
+#   CLUSTER_AVAILABLE=true   -> L2/L3 run and are REQUIRED green
+#   CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
+# That DECLARATION is what makes an L2/L3 skip different IN KIND from every other skip —
+# see "WHICH SKIPS BLOCK, AND WHY" in ci/gate.sh.
+cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
+
+# --- Skip strictness: is this a CI runner, or a workstation? -------------------
+# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
+#
+#   CI       the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
+#            before the gate runs, and every seam is on `main`. A stage that skips here is a
+#            DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
+#   local    a missing pinned tool is a real, ordinary local condition -> the gate reports
+#            UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
+#
+# HOW STRICTNESS IS DECIDED — DECLARED first, detected only as a fallback:
+#
+#   1. GATE_STRICT_SKIPS=true|false is the AUTHORITY. .gitea/workflows/gate.yml sets `true` on
+#      the gate step, so on the one CI runner that matters strictness is a DECLARATION IN THE
+#      WORKFLOW, not an inference: the environment that KNOWS it is CI says so, and no
+#      env-var sniffing can get it wrong.
+#   2. `auto` (the default — anywhere the caller said nothing) falls back to GITHUB_ACTIONS /
+#      CI, which are set by the runner ITSELF (Gitea's act_runner, as well as GitHub). We do
+#      NOT sniff hostnames, paths or tool availability: a heuristic that guesses "this looks
+#      like CI" is exactly the unreliable inference this gate must not contain.
+#
+# WHY A WRONG ANSWER CANNOT FAIL OPEN: both branches are non-zero and NEITHER is a pass. A CI
+# run misread as local yields UNDETERMINED (exit 3) instead of BLOCKED (exit 1) — that still
+# fails the job unless its caller explicitly tolerates 3, which gate.yml does only knowingly
+# and only for the one declared-environmental skip (see its comment). A workstation misread as
+# CI yields BLOCKED instead of UNDETERMINED — harsher, never looser. Set GATE_STRICT_SKIPS=true
+# locally to rehearse CI strictness.
+: "${GATE_STRICT_SKIPS:=auto}"
+strict_skips() {
+  case "$GATE_STRICT_SKIPS" in
+    auto) truthy "${GITHUB_ACTIONS:-}" || truthy "${CI:-}" ;;
+    *)    truthy "$GATE_STRICT_SKIPS" ;;
+  esac
+}
+
 mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
 
+# --- Offline (L0/L1) per-check step summary -----------------------------------
+# conformance/lib/run_l1_offline.py writes one Markdown line per check it executed to
+# conformance/_artifacts/offline-summary.md. Splice those into $GITHUB_STEP_SUMMARY so the
+# gate's output still NAMES every check that ran and its verdict.
+#
+# WHY THIS EXISTS: ci/l0.sh used to carry a hand-written stage per whole-repo check, each with
+# its own `summary "- subst-safety: **passed**"` line. Those stages are gone (the tests declare
+# `conformance.home.k3s/scope: whole-repo` instead and the runner unions them in), so the
+# naming is generated for EVERY offline check rather than for the three that had a stage.
+: "${OFFLINE_SUMMARY_FILE:=${CONFORMANCE_DIR}/_artifacts/offline-summary.md}"
+emit_offline_summary() {
+  local f="${1:-$OFFLINE_SUMMARY_FILE}" line
+  [ -f "$f" ] || return 0
+  while IFS= read -r line; do
+    [ -n "$line" ] && summary "  $line"
+  done < "$f"
+  return 0
+}
+
 # --- Touched-requirement subset (R-TEST-03 input) ----------------------------
 # Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
 # (ready to splice into a conformance/run.sh invocation).
@@ -111,7 +174,16 @@ load_touched_requirements() {
     warn "  Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
     warn "  coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
   elif [ "$rc" -ne 0 ]; then
-    warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
+    # Any OTHER non-zero exit means the subset is UNUSABLE, not empty (today: a
+    # path->requirement map that cannot classify a changed path, once that map fails closed).
+    # The FULL layer still runs here as a superset — nothing goes untested — but this stage must
+    # not claim the SCOPED check happened, and ci/gate.sh turns the same exit into a BLOCKED
+    # merge decision. Wording carried over from the unlanded fix/touched-requirements-fail-closed
+    # (47068ee), whose gate-side half this branch lands.
+    warn "touched-requirements.sh exited ${rc} (subset UNUSABLE — see its output above)."
+    warn "  Running the FULL layer as a superset; the SCOPED R-TEST-03 check did NOT happen."
+    warn "  ci/gate.sh BLOCKS the merge on this: fix the path->requirement map in"
+    warn "  ci/touched-requirements.sh so every changed path is classified."
   elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
     log "touched requirements: ${TOUCHED_REQS[*]}"
     REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")

ci/render-keycloak-client-urls.py

@@ -0,0 +1,214 @@
+#!/usr/bin/env python3
+# =============================================================================
+# ci/render-keycloak-client-urls.py
+#
+# Render platform/keycloak/client-urls.yaml FROM platform/keycloak/keycloak-realm-import.yaml.
+#
+# WHY THIS EXISTS
+# ---------------
+# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
+# Done=True / Started=False it never re-runs, so a redirect URI changed in git is
+# never applied to the realm. The client-secret half of that failure class is
+# already healed by a level-triggered CronJob (platform/keycloak/client-secret-
+# reconciler.yaml). This renderer supplies the DESIRED STATE that the same CronJob
+# needs to heal the URL half.
+#
+# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
+# automountServiceAccountToken: false. It has NO Kubernetes API access at all, by
+# design (R-SEC-02) — so it cannot read the KeycloakRealmImport CR to learn what
+# the realm is supposed to look like. Its inputs must be MOUNTED. Hence a
+# ConfigMap.
+#
+# A hand-authored ConfigMap would be a SECOND SOURCE OF TRUTH for every public
+# hostname in the realm, free to disagree with the realm import — and "two
+# declarations that can disagree" is the same shape as the bug this whole effort
+# is closing. So the ConfigMap is GENERATED from the realm import and the
+# generation is GATED: conformance/tests/l0-static/keycloak-client-secret-
+# reconciler/check.sh runs this script with --check on every merge, and the merge
+# fails if the committed file is not what this renderer produces. That is the same
+# render==golden idiom catalog/render.py + ci/render-golden.sh already use for
+# apps/.
+#
+# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. If a URL in the realm import is
+# written as a Flux substitution token (https://grafana.${BASE_DOMAIN}/...), the
+# token is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files
+# are rendered by the SAME Flux Kustomization (./platform-config, whose
+# postBuild.substituteFrom reads cluster-vars), so both are substituted in the
+# same pass with the same values and cannot disagree about what BASE_DOMAIN is.
+# This renderer never parses, splits, or reconstructs a hostname, and never reads
+# cluster-vars — so there is no second place for the domain to be decided.
+#
+# USAGE
+#   python3 ci/render-keycloak-client-urls.py            # print the rendered file
+#   python3 ci/render-keycloak-client-urls.py --write     # write it (regenerate)
+#   python3 ci/render-keycloak-client-urls.py --check      # exit 1 if it has drifted
+# =============================================================================
+from __future__ import annotations
+
+import argparse
+import difflib
+import json
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:  # pragma: no cover
+    print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(__file__).resolve().parent.parent
+IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
+OUT_FILE = "platform/keycloak/client-urls.yaml"
+DATA_KEY = "client-urls.json"
+CONFIGMAP_NAME = "keycloak-client-urls"
+NAMESPACE = "keycloak"
+
+# The client fields this renderer exports, and therefore the ONLY fields the
+# reconciler may write. Every one of them is an environment-derived URL: it changes
+# when BASE_DOMAIN changes, and a stale value breaks OIDC login (a wrong
+# redirectUri) or silently widens trust (a stale wildcard left behind).
+#
+# DELIBERATELY ABSENT: `secret` (healed by the secret pass from the SOPS Secret,
+# and it must never appear in a ConfigMap), and every non-URL field —
+# protocolMappers, defaultClientScopes, attributes, flags. Those are not
+# environment-derived, so nothing re-renders them on a domain flip; keeping them
+# out keeps the reconciler's write surface as narrow as the problem.
+LIST_FIELDS = ("redirectUris", "webOrigins")
+STR_FIELDS = ("rootUrl", "baseUrl", "adminUrl")
+
+
+def render(root: Path) -> str:
+    """Return the full text of client-urls.yaml as it should be committed."""
+    docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
+    imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
+    if imp is None:
+        raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
+    clients = (imp.get("spec") or {}).get("realm", {}).get("clients") or []
+    if not clients:
+        # Fail closed. Emitting an empty desired state would make the reconciler
+        # believe every client is unmanaged and silently stop reconciling URLs.
+        raise SystemExit("FATAL: %s declares zero clients — refusing to render an "
+                         "empty desired state" % IMPORT_FILE)
+
+    want: dict[str, dict] = {}
+    for c in clients:
+        cid = c.get("clientId")
+        if not isinstance(cid, str) or not cid:
+            raise SystemExit("FATAL: %s has a client with no clientId" % IMPORT_FILE)
+        entry: dict[str, object] = {}
+        for f in LIST_FIELDS:
+            if f not in c:
+                continue                 # not declared => NOT MANAGED (never forced empty)
+            v = c[f]
+            if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
+                raise SystemExit("FATAL: %s client %r field %s is not a list of strings"
+                                 % (IMPORT_FILE, cid, f))
+            entry[f] = list(v)           # verbatim, order as authored
+        for f in STR_FIELDS:
+            if f not in c:
+                continue
+            v = c[f]
+            if not isinstance(v, str):
+                raise SystemExit("FATAL: %s client %r field %s is not a string"
+                                 % (IMPORT_FILE, cid, f))
+            entry[f] = v
+        if entry:
+            want[cid] = entry
+    if not want:
+        raise SystemExit("FATAL: no client in %s declares any of %s — refusing to render "
+                         "an empty desired state"
+                         % (IMPORT_FILE, ", ".join(LIST_FIELDS + STR_FIELDS)))
+
+    body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
+    indented = "\n".join(("    " + ln) if ln else "" for ln in body.splitlines())
+    return HEADER + indented + "\n"
+
+
+HEADER = """---
+# =============================================================================
+# GENERATED FILE — DO NOT EDIT BY HAND.
+#
+#   source:      platform/keycloak/keycloak-realm-import.yaml (spec.realm.clients)
+#   regenerate:  python3 ci/render-keycloak-client-urls.py --write
+#   gated by:    conformance/tests/l0-static/keycloak-client-secret-reconciler
+#                (runs this renderer with --check on every merge)
+#
+# WHAT THIS IS: the DESIRED STATE for every environment-derived client URL in the
+# `homelab` realm — redirectUris, webOrigins, rootUrl, baseUrl, adminUrl. The
+# client-secret reconciler CronJob mounts it and makes the live realm match it.
+#
+# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
+# once Done=True, so a redirect URI edited in git never reaches the realm. Prod's
+# stored redirect URIs were measured still pointing at the pre-migration hostname
+# while git declared the new one, and every OIDC login would have failed with a
+# redirect_uri mismatch. The reconciler is the level-triggered loop that closes
+# that; this file is what it reconciles TOWARDS.
+#
+# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of every public
+# hostname would be a second source of truth, free to disagree with the realm
+# import. It is rendered from the import and the render is gated, so the two
+# cannot drift. URL strings are copied VERBATIM — if the source uses a
+# ${BASE_DOMAIN} token it appears here unchanged, and both files are substituted
+# by the SAME Flux Kustomization (./platform-config), so the domain is decided in
+# exactly one place.
+#
+# A field that is NOT declared on a client in the realm import is simply ABSENT
+# here, and the reconciler leaves it alone. Absence means UNMANAGED, never
+# "force to empty".
+# =============================================================================
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: %s
+  namespace: %s
+  labels:
+    app.kubernetes.io/name: %s
+    app.kubernetes.io/part-of: keycloak
+data:
+  %s: |
+""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME, DATA_KEY)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    g = ap.add_mutually_exclusive_group()
+    g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
+    g.add_argument("--check", action="store_true",
+                   help="exit 1 if the committed file has drifted from the source")
+    args = ap.parse_args()
+
+    want = render(ROOT)
+    target = ROOT / OUT_FILE
+
+    if args.write:
+        target.write_text(want, encoding="utf-8")
+        print("wrote %s" % OUT_FILE)
+        return 0
+
+    if args.check:
+        if not target.is_file():
+            print("CHECK FAIL: %s does not exist. Run: python3 "
+                  "ci/render-keycloak-client-urls.py --write" % OUT_FILE, file=sys.stderr)
+            return 1
+        have = target.read_text(encoding="utf-8")
+        if have == want:
+            print("client-urls.yaml is in sync with %s" % IMPORT_FILE)
+            return 0
+        print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
+              "realm on a desired state the realm import does not declare.\n"
+              "Regenerate with: python3 ci/render-keycloak-client-urls.py --write\n"
+              % (OUT_FILE, IMPORT_FILE), file=sys.stderr)
+        for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
+                                         fromfile="committed " + OUT_FILE,
+                                         tofile="rendered from " + IMPORT_FILE):
+            sys.stderr.write(line)
+        return 1
+
+    sys.stdout.write(want)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

ci/tests/gate-decision.test.sh

@@ -0,0 +1,189 @@
+#!/usr/bin/env bash
+# ci/tests/gate-decision.test.sh — the merge gate's OWN decision table.
+#
+# WHY THIS SUITE EXISTS (2026-09-13). ci/gate.sh used to set BLOCK=1 for a FAILED stage but
+# only `warn` for a SKIPPED one, so it printed the words "— not green" and then concluded
+# `MERGE GATE: PASS` and exited 0. Observed live twice: `prom-rules` skipped for a missing
+# promtool, and `l0` skipped for a missing pinned kyverno CLI — both read by automation as
+# merge-approved. That is fail-open in the one component whose entire job is to fail closed,
+# and nothing tested it: every other ci/tests suite covers the INPUT (touched-requirements),
+# none covered the DECISION.
+#
+# WHAT IS UNDER TEST: the real ci/gate.sh and ci/lib.sh, copied into a throwaway CI_DIR beside
+# STUB stage scripts whose exit codes this suite dictates (STAGE_RC_<stage>) and a stub
+# touched-requirements.sh whose rc/reason it dictates (TR_RC / TR_REASON / TR_OUT). Stubbing the
+# stages is the point: this asserts how the gate REASONS about pass/fail/skip, independently of
+# what any stage happens to do on this machine — so it runs anywhere, with no tools, no cluster,
+# no network, in milliseconds.
+#
+# THE INVARIANT: `MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and
+# was GREEN. A stage that could not run is never a pass, and the exit code says so too.
+#
+# Run:  ci/tests/gate-decision.test.sh     (or `make -C ci test`)
+set -uo pipefail
+
+REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+TMP="$(mktemp -d)"
+trap 'rm -rf "$TMP"' EXIT
+CI="${TMP}/ci"; mkdir -p "$CI"
+REPO="${TMP}/repo"; mkdir -p "$REPO"
+
+cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
+
+# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
+for stage in spec-immutability lint l0 render-golden l1 conformance; do
+  cat > "${CI}/${stage}.sh" <<'STUB'
+#!/usr/bin/env bash
+n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
+exit "${!v:-0}"
+STUB
+  chmod +x "${CI}/${stage}.sh"
+done
+
+# Stub decision input. TR_RC (default 0), TR_REASON (written to $TOUCHED_REASON_FILE like the
+# real script does), TR_OUT (the touched requirement list on stdout).
+cat > "${CI}/touched-requirements.sh" <<'STUB'
+#!/usr/bin/env bash
+if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "${TR_REASON:-}" > "$TOUCHED_REASON_FILE"; fi
+[ -n "${TR_OUT:-}" ] && printf '%s\n' "$TR_OUT"
+exit "${TR_RC:-0}"
+STUB
+chmod +x "${CI}/touched-requirements.sh"
+
+# run_gate <VAR=VAL>... — run the gate with a clean, explicit environment. Sets OUT and RC.
+# GATE_STRICT_SKIPS is pinned false unless a case says otherwise, so the suite never depends on
+# whether the machine running it happens to look like CI.
+OUT="" RC=0
+run_gate() {
+  local summary_file="${TMP}/summary.md"; : > "$summary_file"
+  OUT="$(env -i PATH="$PATH" HOME="$HOME" \
+        REPO_ROOT="$REPO" GITHUB_STEP_SUMMARY="$summary_file" \
+        GATE_STRICT_SKIPS=false TR_OUT="R-TEST-03" \
+        "$@" bash "${CI}/gate.sh" 2>&1)"
+  RC=$?
+  SUMMARY="$(cat "$summary_file")"
+}
+
+# assert_verdict <headline> <expected-rc> <label>
+assert_verdict() {
+  local want="$1" want_rc="$2" label="$3"
+  if ! grep -qF "MERGE GATE: ${want}" <<<"$OUT"; then
+    bad "${label}: headline should be '${want}' — got: $(grep -oE 'MERGE GATE: [A-Z ]+' <<<"$OUT" | tail -1)"
+  elif [ "$RC" -ne "$want_rc" ]; then
+    bad "${label}: headline '${want}' correct but exit code ${RC} != ${want_rc}"
+  else
+    ok "${label} -> ${want}, exit ${want_rc}"
+  fi
+}
+# The load-bearing negative: nothing may say PASS, and nothing may exit 0.
+assert_not_pass() {
+  local label="$1"
+  if grep -qF "MERGE GATE: PASS" <<<"$OUT"; then bad "${label}: printed 'MERGE GATE: PASS'"
+  elif [ "$RC" -eq 0 ]; then bad "${label}: exited 0"
+  else ok "${label}: not a PASS, non-zero exit (${RC})"; fi
+}
+
+echo "== 1. the happy path: every stage green"
+run_gate
+assert_verdict PASS 0 "all stages green + a touched subset"
+grep -qF "nothing was declared out of scope" <<<"$SUMMARY" \
+  && ok "PASS summary states nothing was declared out of scope" \
+  || bad "PASS summary does not state the absence of declared gaps"
+
+echo
+echo "== 2. a FAILED stage blocks (unchanged behaviour, locked down)"
+for s in lint l0 render-golden l1 conformance spec-immutability; do
+  run_gate "STAGE_RC_${s//-/_}=1"
+  assert_verdict BLOCKED 1 "${s} FAILED"
+done
+
+echo
+echo "== 3. THE REGRESSION: a SKIPPED stage is not a PASS and not exit 0"
+for s in lint l0 render-golden l1; do
+  run_gate "STAGE_RC_${s//-/_}=3"
+  assert_not_pass "${s} SKIPPED (local)"
+  assert_verdict UNDETERMINED 3 "${s} SKIPPED (local)"
+  grep -qF "$s" <<<"$SUMMARY" && ok "${s}: the step summary names the stage that did not run" \
+                              || bad "${s}: step summary does not name the skipped stage"
+done
+
+echo
+echo "== 4. under STRICT skips (CI declares it) a could-not-run skip BLOCKS, never merely 3"
+for s in lint l0 render-golden l1; do
+  run_gate GATE_STRICT_SKIPS=true "STAGE_RC_${s//-/_}=3"
+  assert_verdict BLOCKED 1 "${s} SKIPPED with GATE_STRICT_SKIPS=true"
+done
+# ...which is what makes gate.yml's `|| [ $? -eq 3 ]` safe: in CI, exit 3 cannot be a missing
+# tool or an unwired seam. Only the declared-environmental skip below can produce it.
+run_gate GATE_STRICT_SKIPS=true STAGE_RC_l0=3 STAGE_RC_lint=3
+assert_verdict BLOCKED 1 "several stages SKIPPED under strict skips"
+
+echo
+echo "== 5. spec-immutability: unverified is never green, but is not upgraded to a block by CI"
+run_gate STAGE_RC_spec_immutability=3
+assert_not_pass "spec-immutability SKIPPED"
+assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED (local)"
+run_gate GATE_STRICT_SKIPS=true STAGE_RC_spec_immutability=3
+assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED under strict skips (deliberate exception)"
+run_gate REQUIRE_SPEC_VERIFICATION=true STAGE_RC_spec_immutability=3
+assert_verdict BLOCKED 1 "spec-immutability SKIPPED + REQUIRE_SPEC_VERIFICATION=true"
+
+echo
+echo "== 6. conformance L2/L3: a DECLARED-absent capability may still pass — and is NAMED"
+run_gate STAGE_RC_conformance=3
+assert_verdict PASS 0 "conformance SKIPPED, no cluster declared"
+grep -qF "conformance-l2-l3" <<<"$OUT" \
+  && ok "the PASS headline names the declared L2/L3 gap" \
+  || bad "the PASS headline hides the declared L2/L3 gap"
+grep -qF "conformance-l2-l3" <<<"$SUMMARY" \
+  && ok "the step summary names the declared L2/L3 gap" \
+  || bad "the step summary hides the declared L2/L3 gap"
+run_gate CLUSTER_AVAILABLE=true STAGE_RC_conformance=3
+assert_verdict BLOCKED 1 "conformance SKIPPED despite CLUSTER_AVAILABLE=true"
+run_gate REQUIRE_CLUSTER_TESTS=true STAGE_RC_conformance=3
+assert_verdict BLOCKED 1 "conformance SKIPPED + REQUIRE_CLUSTER_TESTS=true with a touched subset"
+# A misspelled switch must leave the guard OFF (truthy()), never half-on.
+run_gate CLUSTER_AVAILABLE=ture STAGE_RC_conformance=3
+assert_verdict PASS 0 "CLUSTER_AVAILABLE=ture is NOT 'available' (no cluster was declared)"
+run_gate CLUSTER_AVAILABLE=TRUE STAGE_RC_conformance=3
+assert_verdict BLOCKED 1 "CLUSTER_AVAILABLE=TRUE IS a declaration (skip then blocks)"
+
+echo
+echo "== 7. the undetermined touched subset: unchanged (2026-08-31 false-green paths)"
+run_gate TR_RC=3 TR_REASON=on-base TR_OUT=
+assert_verdict "NOT APPLICABLE" 0 "subset undeterminable, reason on-base, all stages green"
+run_gate TR_RC=3 TR_REASON=no-base TR_OUT=
+assert_verdict BLOCKED 1 "subset undeterminable, reason no-base"
+run_gate TR_RC=3 TR_REASON=diff-failed TR_OUT=
+assert_verdict BLOCKED 1 "subset undeterminable, reason diff-failed"
+# Precedence: a stage that did not run outranks "nothing to gate" — the stronger, non-zero
+# headline must win, or a push-to-main run could hide a missing tool behind NOT APPLICABLE.
+run_gate TR_RC=3 TR_REASON=on-base TR_OUT= STAGE_RC_l1=3
+assert_not_pass "on-base AND l1 SKIPPED"
+assert_verdict UNDETERMINED 3 "on-base AND l1 SKIPPED (UNDETERMINED outranks NOT APPLICABLE)"
+
+echo
+echo "== 8. an UNUSABLE subset blocks (carried over from 47068ee): any other non-zero exit"
+for rc in 1 2 4; do
+  run_gate "TR_RC=${rc}" TR_REASON=unclassified-path TR_OUT=
+  assert_verdict BLOCKED 1 "touched-requirements.sh exited ${rc} (subset unusable, not empty)"
+done
+run_gate TR_RC=1 TR_REASON=unclassified-path TR_OUT=
+grep -qF "UNUSABLE" <<<"$SUMMARY" && ok "the step summary calls the subset UNUSABLE" \
+                                  || bad "the step summary does not report an unusable subset"
+
+echo
+echo "== 9. combinations: a FAILURE outranks a non-run, and the headline is the worst news"
+run_gate STAGE_RC_l0=1 STAGE_RC_l1=3
+assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
+run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
+assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
+
+echo
+if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
+else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/tests/touched-requirements-test-attribution.test.sh

@@ -0,0 +1,164 @@
+#!/usr/bin/env bash
+# ci/tests/touched-requirements-test-attribution.test.sh — regression tests for DETECTION
+# RULE 3 in ci/touched-requirements.sh: a change anywhere under `conformance/tests/<test>/`
+# attributes THAT TEST'S OWN `conformance.home.k3s/R-*` labels.
+#
+# THE GAP THESE LOCK DOWN (2026-09-12). The coarse path map keys `conformance/*` to
+# R-TEST-01..04 — the requirements about the HARNESS — and says nothing about what a given
+# test PROVES. So weakening `conformance/tests/.../check.sh` attributed R-TEST-01..04 and
+# nothing else: the merge gate ran the harness's self-tests and NOT the weakened test's own
+# requirement subset. Weakening (or deleting) a test is the one change that most needs the
+# requirement it certifies to be re-verified, and it was the change least likely to select it.
+# The content scan cannot close this: it reads only ADDED/REMOVED lines, so it sees a change
+# to the LABEL BLOCK but not one to the test BODY — exactly the case that matters.
+#
+# Also covers the two path-map entries added in the same change (infra/* -> R-ENV-01, and
+# R-TEST-06 on .gitea/*|ci/*), which is how the two L0 checks that are DELIBERATELY not
+# whole-repo-scoped get selected by the paths that can break them.
+#
+# Same harness as the sibling suites: the real script, pointed at a THROWAWAY git repo via
+# the REPO_ROOT override. No network, no cluster, the working repo is never touched.
+#
+# Run:  ci/tests/touched-requirements-test-attribution.test.sh   (or `make -C ci test`)
+set -euo pipefail
+
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+SCRIPT="${CI_DIR}/touched-requirements.sh"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
+
+assert_present() {
+  if grep -qxF "$1" <<<"$2"; then ok "$3"
+  else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
+}
+assert_absent() {
+  if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))"
+  else ok "$3"; fi
+}
+
+TMP="$(mktemp -d)"
+trap 'rm -rf "$TMP"' EXIT
+
+git -C "$TMP" init -q -b main
+git -C "$TMP" config user.email [email protected]
+git -C "$TMP" config user.name t
+git -C "$TMP" config commit.gpgsign false
+
+# --- Baseline: two conformance tests, each declaring its own requirement labels ----------
+# TEST A: a flat l0-static test. TEST B: a test with a NESTED fixture dir, to prove the
+# walk-up from an arbitrary changed file to the enclosing test dir.
+mkdir -p "$TMP/conformance/tests/l0-static/alpha" \
+         "$TMP/conformance/tests/l2-behavioral/beta/fixtures" \
+         "$TMP/conformance/lib" "$TMP/infra" "$TMP/ci"
+
+cat > "$TMP/conformance/tests/l0-static/alpha/chainsaw-test.yaml" <<'YAML'
+# HEADER PROSE. This test used to also cover conformance.home.k3s/R-STALE-09: "true",
+# which is a REFERENCE in a comment, not a declaration, and must NOT be attributed.
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: alpha
+  labels:
+    conformance.home.k3s/R-ALPHA-01: "true"
+    conformance.home.k3s/R-ALPHA-02: "true"
+  annotations:
+    conformance.home.k3s/layer: "L0"
+spec: {}
+YAML
+# The check BODY deliberately contains NO R-ID text, so anything the scan finds for a
+# body-only edit came from rule 3 and not from the content scan.
+cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
+#!/usr/bin/env bash
+echo "assert something"
+SH
+
+cat > "$TMP/conformance/tests/l2-behavioral/beta/chainsaw-test.yaml" <<'YAML'
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: beta
+  labels:
+    conformance.home.k3s/R-BETA-07: "true"
+spec: {}
+YAML
+echo "seed: 1" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
+
+echo "harness helper" > "$TMP/conformance/lib/helper.sh"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
+
+# ========================================================================================
+echo "== case 1: a BODY-ONLY edit to a test attributes that test's own labels"
+git -C "$TMP" checkout -q -b body-edit
+cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
+#!/usr/bin/env bash
+echo "assert something weaker"
+SH
+git -C "$TMP" add -A && git -C "$TMP" commit -qm weaken
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-ALPHA-01" "$OUT" "body-only test edit attributes the test's own R-ALPHA-01"
+assert_present "R-ALPHA-02" "$OUT" "body-only test edit attributes the test's own R-ALPHA-02"
+assert_present "R-TEST-03"  "$OUT" "the conformance/* path map still applies (R-TEST-03)"
+assert_absent  "R-STALE-09" "$OUT" "an R-ID cited only in the test's HEADER COMMENT is NOT attributed"
+assert_absent  "R-BETA-07"  "$OUT" "an UNRELATED test's labels are NOT attributed"
+
+# ========================================================================================
+echo "== case 2: an edit to a NESTED file walks up to the enclosing test"
+git -C "$TMP" checkout -q -b nested-edit main
+echo "seed: 2" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm fixture
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-BETA-07" "$OUT" "nested fixture edit attributes the enclosing test's R-BETA-07"
+assert_absent  "R-ALPHA-01" "$OUT" "nested fixture edit does NOT attribute a sibling test's labels"
+
+# ========================================================================================
+echo "== case 3: DELETING a test still attributes what it used to certify"
+# Removing the only verifier of a requirement must not be the cheapest way to make the gate
+# stop checking it — so the labels are read from the BASE when HEAD no longer has them.
+git -C "$TMP" checkout -q -b delete-test main
+git -C "$TMP" rm -rq "$TMP/conformance/tests/l0-static/alpha"
+git -C "$TMP" commit -qm "delete the alpha test"
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-ALPHA-01" "$OUT" "deleting a test STILL attributes its R-ALPHA-01 (read from base)"
+assert_present "R-ALPHA-02" "$OUT" "deleting a test STILL attributes its R-ALPHA-02 (read from base)"
+
+# ========================================================================================
+echo "== case 4: a change OUTSIDE conformance/tests/ gains no test labels"
+git -C "$TMP" checkout -q -b lib-edit main
+echo "harness helper v2" > "$TMP/conformance/lib/helper.sh"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm lib
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-TEST-02"  "$OUT" "a conformance/lib edit still attributes the harness requirements"
+assert_absent  "R-ALPHA-01" "$OUT" "a conformance/lib edit attributes NO individual test's labels"
+assert_absent  "R-BETA-07"  "$OUT" "a conformance/lib edit attributes NO individual test's labels (B)"
+
+# ========================================================================================
+echo "== case 5: path-map entries for the two DELIBERATELY path-scoped L0 checks"
+# infra/* -> R-ENV-01 (dev-nodes-provisioned-as-code reads nothing but infra/).
+git -C "$TMP" checkout -q -b infra-edit main
+printf 'resource "x" "y" {}\n' > "$TMP/infra/main.tf"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm infra
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-ENV-01" "$OUT" "an infra/ edit attributes R-ENV-01 (was UNMAPPED: attributed nothing)"
+
+# ci/* -> ... R-TEST-06 (spec-immutability-enforced reads nothing but ci/ + the spec repo).
+git -C "$TMP" checkout -q -b ci-edit main
+printf 'echo hello\n' > "$TMP/ci/thing.sh"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm ci
+OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set:\n%s\n---\n' "$OUT"
+assert_present "R-TEST-06"  "$OUT" "a ci/ edit attributes R-TEST-06 (the spec-immutability test's own ID)"
+assert_present "R-GITOPS-01" "$OUT" "a ci/ edit still attributes R-GITOPS-01"
+
+echo
+if [ "$FAIL" -eq 0 ]; then
+  echo "touched-requirements test-attribution tests: PASS (${PASS} assertions)"; exit 0
+fi
+echo "touched-requirements test-attribution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
+exit 1

ci/tests/touched-requirements.test.sh

@@ -93,6 +93,141 @@ printf 'touched set (removal):\n%s\n---\n' "$OUT2"
 assert_absent  "R-DELDOC-01"  "$OUT2" "REMOVED whole-line comment R-ID is NOT attributed"
 assert_present "R-DELREAL-01"  "$OUT2" "REMOVED real-line R-ID IS attributed"
 
+# ---------------------------------------------------------------------------------------
+# PATH MAP — bootstrap/* (added 2026-09-13)
+#
+# WHY THIS IS LOCKED DOWN. path_requirements() keys `bootstrap/*` to R-STATE-03 +
+# R-GITOPS-01, and until 2026-09-13 the comment justifying it described
+# `bootstrap/standup.sh` as a file that carries the placeholder guards. That file has never
+# been on `main` (it landed only on unmerged prod-standup branches), so the comment was
+# stale and the entry looked like dead weight someone would tidy away. It is not dead: it
+# is what SELECTS conformance/tests/l0-static/placeholder-guard-coupling for the change
+# that eventually lands a placeholder guard under bootstrap/ — and that test discovers its
+# guard by content rather than by path, so it activates on exactly that change. Removing
+# the entry would let the change introducing a guard be gated by nothing, which is the
+# 2026-09-05 class. These assertions make the entry's deletion a test failure rather than a
+# silent loss of selection.
+#
+# NOTE the R-ID content scan must not be what produces these: `notes/` is outside the path
+# map precisely so the other cases isolate the scan, and here the file content cites no
+# R-ID at all, so any R-ID in the output came from the PATH.
+git -C "$TMP" checkout -q -b feature3 main
+mkdir -p "$TMP/bootstrap/flux"
+cat > "$TMP/bootstrap/flux/bootstrap.sh" <<'SH'
+#!/usr/bin/env bash
+# a bootstrap-path change that cites no requirement id of its own
+echo "flux bootstrap"
+SH
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch bootstrap/"
+OUT3="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (bootstrap/ path map):\n%s\n---\n' "$OUT3"
+assert_present "R-STATE-03"   "$OUT3" "bootstrap/* attributes R-STATE-03 (selects placeholder-guard-coupling)"
+assert_present "R-GITOPS-01"  "$OUT3" "bootstrap/* attributes R-GITOPS-01 (it IS the GitOps bootstrap path)"
+
+# And the neighbouring entry the same test leans on for its other half: clusters/* must
+# attribute R-ENV-02, which is what selects placeholder-guard-coupling's HALF A (the
+# authored-values sweep over clusters/<env> effective cluster-vars).
+git -C "$TMP" checkout -q -b feature4 main
+mkdir -p "$TMP/clusters/base"
+printf 'kind: ConfigMap\nmetadata:\n  name: cluster-vars\n' > "$TMP/clusters/base/cluster-vars.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch clusters/"
+OUT4="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
+assert_present "R-ENV-02"     "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
+assert_present "R-GITOPS-01"  "$OUT4" "clusters/* attributes R-GITOPS-01"
+
+# ---------------------------------------------------------------------------------------
+# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
+#
+# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
+# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
+# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
+# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
+# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
+# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
+#
+# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
+# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
+#
+# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
+# from the PATH and not from the content scan.
+
+# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
+# under test here, not just the presence of both arms.
+git -C "$TMP" checkout -q -b feature-apps-vms main
+mkdir -p "$TMP/apps/vms/demo"
+printf 'kind: VirtualMachine\nmetadata:\n  name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
+OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
+assert_present "R-PLAT-03"  "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
+assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
+assert_absent  "R-PLAT-02"  "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
+
+# A plain ServiceDefinition under apps/ gets the container-workload set.
+git -C "$TMP" checkout -q -b feature-apps main
+mkdir -p "$TMP/apps/wave-1"
+printf 'kind: ServiceDefinition\nmetadata:\n  name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
+OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
+assert_present "R-PLAT-02"  "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
+assert_present "R-NET-01"   "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
+assert_present "R-NET-06"   "$OUT_APPS" "apps/* attributes the egress-class requirements"
+assert_present "R-IDP-02"   "$OUT_APPS" "apps/* attributes the auth-mode requirements"
+
+# homepage / headlamp ARE the command-centre surface.
+git -C "$TMP" checkout -q -b feature-c2 main
+mkdir -p "$TMP/platform/headlamp"
+printf 'kind: Deployment\nmetadata:\n  name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
+OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
+assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
+assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
+
+# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
+git -C "$TMP" checkout -q -b feature-assembly main
+mkdir -p "$TMP/platform"
+printf 'resources:\n  - cilium\n  - longhorn\n' > "$TMP/platform/kustomization.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
+OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
+assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
+
+# ---------------------------------------------------------------------------------------
+# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
+#
+# The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match `R-C2-01` or
+# `R-C2-02` — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. Those are
+# the only two registry IDs whose workstream segment carries a digit, and the scan was blind
+# to BOTH, in the fail-OPEN direction: a change self-declaring a C2 requirement contributed
+# it to nobody, so the C2 conformance tests were filtered out of the merge gate. Found while
+# wiring conformance/tests/l0-static/c2-root-surface, which closes those two coverage gaps.
+#
+# `notes/` is outside path_requirements() on purpose, so the IDs below can only have come
+# from the CONTENT SCAN. A neighbouring all-letter ID is asserted too, so a regex that
+# somehow matched only the digit form would still fail here.
+git -C "$TMP" checkout -q -b feature5 main
+cat > "$TMP/notes/c2.yaml" <<'YAML'
+kind: Test
+metadata:
+  labels:
+    conformance.home.k3s/R-C2-01: "true"
+    conformance.home.k3s/R-C2-02: "true"
+    conformance.home.k3s/R-OBS-04: "true"
+YAML
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "declare R-C2-01 / R-C2-02"
+OUT5="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (digit in the workstream segment):\n%s\n---\n' "$OUT5"
+assert_present "R-C2-01"  "$OUT5" "an R-ID with a DIGIT in the workstream segment IS attributed (R-C2-01)"
+assert_present "R-C2-02"  "$OUT5" "the second such ID is attributed too (R-C2-02)"
+assert_present "R-OBS-04" "$OUT5" "an all-letter workstream segment still works (control)"
+# The pattern must not become so loose that a non-ID token is read as a requirement: a bare
+# numeric workstream is not a registry shape, and attributing one would WIDEN the gate on
+# noise (over-broad selection, never a false green, but noisy and misleading).
+assert_absent  "R-1-01"   "$OUT5" "a purely numeric workstream segment is not invented from thin air"
+
 echo
 if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
 else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/touched-requirements.sh

@@ -21,8 +21,12 @@
 #      the R-IDs that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011).
 #      Untouched lines and prose are NOT scanned; see changed_diff() for why.
 #   2. A coarse path->requirement fallback for changes that don't self-declare IDs.
+#   3. For a change anywhere under `conformance/tests/<test>/`: THAT TEST'S OWN requirement
+#      labels, read from its chainsaw-test.yaml. See test_requirements_for_path().
 # The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for
-# untagged changes so nothing silently escapes the gate.
+# untagged changes so nothing silently escapes the gate. (3) is the same principle applied
+# to the tests themselves: editing or weakening a test must be gated by the requirements
+# that test CERTIFIES, not merely by R-TEST-0x.
 set -euo pipefail
 CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # shellcheck disable=SC1091
@@ -173,6 +177,23 @@ path_requirements() {
   case "$1" in
     catalog/*)                 echo "R-GITOPS-02" ;;
     schemas/*)                 echo "R-GITOPS-02" ;;
+    # apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
+    # manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
+    # adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
+    # checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
+    # class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
+    # violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
+    # catch it at apply time" is post-merge, and the gate exists to catch it before.
+    #
+    # ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
+    # apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
+    # subject to default-deny (R-NET-01).
+    apps/vms/*)                echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
+    # Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
+    # workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
+    # (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
+    # default-deny (R-NET-01).
+    apps/*)                    echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
     platform/kyverno/*)        echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
     platform/cilium/*)         echo "R-NET-01 R-NET-02 R-NET-05" ;;
     platform/longhorn/*)       echo "R-STATE-01 R-STATE-02" ;;
@@ -180,14 +201,161 @@ path_requirements() {
     platform/externaldns/*)    echo "R-DNS-02" ;;
     platform/velero/*)         echo "R-STATE-03" ;;
     platform/monitoring/*)     echo "R-OBS-01 R-OBS-02" ;;
+    # platform/cnpg was UNMAPPED until 2026-09-06. It is the PostgreSQL DR path for keycloak-db /
+    # nextcloud-db / semaphore-db (barmanObjectStore WAL archiving + ScheduledBackup), so a change
+    # there is an R-IDP-05 / R-STATE-03 change; it now also carries the platform-health alert rules
+    # for that path (R-OBS-04). Without this line an edit under platform/cnpg/ contributed ZERO
+    # requirements unless it happened to retype an R-ID on a non-comment line, and escaped the gate.
+    platform/cnpg/*)           echo "R-IDP-05 R-STATE-03 R-OBS-04" ;;
+    # vpn-egress was UNMAPPED until 2026-09-06. platform/vpn-egress/egress-steering.yaml IS the
+    # R-NET-04 datapath (the rule that selects between the two egress paths) and the R-NET-05/06/07
+    # steering policy — it even self-declares those IDs in its `homelab.keithjackson.dev/requirements`
+    # annotation. But the content scan only reads ADDED/REMOVED lines, so any edit that does not
+    # happen to retype an R-ID (a selector change, an excludedCIDRs change, a comment) contributed
+    # ZERO requirements and escaped the gate entirely. Mapped explicitly; the annotation still wins
+    # when a diff does carry the tags.
+    platform/vpn-egress/*)     echo "R-NET-04 R-NET-05 R-NET-06 R-NET-07" ;;
+    # Keycloak is the whole SSO surface: the realm import (declarative identity config),
+    # the SOPS client secrets, and the reconciler that heals them. Added 2026-09-06 after
+    # the client-secret incident, in which editing platform/keycloak/ attributed NO
+    # requirement, so the L2 keycloak tests were filtered out of the merge gate's touched
+    # subset — the change that could break every OIDC login was gated by nothing.
+    # R-IDP-05 is in scope because platform/keycloak/ also carries the CNPG DB Cluster and
+    # its off-cluster backup credential, not only the realm.
+    platform/keycloak/*)       echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
+    platform/oauth2-proxy/*)   echo "R-IDP-02 R-IDP-06" ;;
     platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
+    # The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
+    # rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
+    # platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
+    # itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
+    # docs/12-command-center-ui.md), so editing them must select the C2 tests.
+    platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
+    # KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
+    platform/kubevirt/*)       echo "R-PLAT-03 R-OBS-01" ;;
+    # The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
+    # TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
+    platform/gateway/*)        echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
+    # CoreDNS is cluster DNS resolution (R-DNS-01).
+    platform/coredns/*)        echo "R-DNS-01" ;;
+    # crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
+    # its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
+    platform/crafty/*)         echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
+    # Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
+    platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
+    # runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
+    # resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
+    # is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
+    # runbook change is an observability change.
+    runbooks/*)                echo "R-OBS-04" ;;
+    # platform-config/ is the env-specific configuration Flux substitutes into and applies.
+    platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
+    # platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
+    # Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
+    # file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
+    # change. That is the most consequential single-line edit in the repo and it attributed nothing.
+    platform/kustomization.yaml) echo "R-GITOPS-01" ;;
+    # .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
+    # here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
+    .sops.yaml)                echo "R-SEC-01 R-GITOPS-01" ;;
     conformance/*)             echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
+    # infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
+    # nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
+    # infra/, yet an infra/ edit attributed ZERO requirements, so the one check that asserts
+    # the nodes are provisioned as code was filtered out of every change that could break it.
+    # That test's invariant IS path-scoped (it is deliberately NOT whole-repo-scoped; see its
+    # chainsaw-test.yaml), and the correct fix for a path-scoped invariant is a path-map entry.
+    infra/*)                   echo "R-ENV-01" ;;
     clusters/*)                echo "R-GITOPS-01 R-ENV-02" ;;
-    .gitea/*|ci/*)             echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
+    # bootstrap/ was UNMAPPED until 2026-09-06. The entry STAYS, but its original rationale
+    # was written against a file that is not here, and that is corrected on 2026-09-13:
+    # it claimed "bootstrap/standup.sh carries the placeholder guards that decide whether
+    # `velero` may be enabled against a real backup endpoint". bootstrap/standup.sh HAS NEVER
+    # BEEN ON `main` — it was not deleted (no deletion commit exists anywhere), it landed only
+    # on unmerged prod-standup branches. `git ls-tree main bootstrap/` lists README.md, flux/,
+    # gitea/, sops/. So R-STATE-03 here is ANTICIPATORY, not descriptive.
+    #
+    # WHY IT IS STILL RIGHT. conformance/tests/l0-static/placeholder-guard-coupling DISCOVERS
+    # its guard (any tracked *.sh defining `<name>_is_placeholder()`) instead of hardcoding a
+    # path, so the moment a standup guard lands under bootstrap/ that test's coupling half
+    # activates — and this entry is what selects the test for the change that lands it.
+    # Deleting the entry would mean the change introducing a placeholder guard is gated by
+    # nothing, which is the 2026-09-05 class (a prefix glob that stopped matching when the
+    # value it guards changed, in a different file) left to reintroduce itself. R-GITOPS-01 is
+    # descriptive today: bootstrap/flux/, gitea/ and sops/ ARE the GitOps bootstrap path.
+    # Locked down by ci/tests/touched-requirements.test.sh's path-map section.
+    bootstrap/*)               echo "R-STATE-03 R-GITOPS-01" ;;
+    # R-TEST-06 added 2026-09-12. The comment in changed_diff()'s rule 3 above records why it
+    # was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE and carried by no
+    # Chainsaw test, so attributing it made the coverage `--require` check false-block. It is now
+    # carried by conformance/tests/l0-static/spec-immutability-enforced — whose entire input
+    # surface is ci/ — so a ci/ change must attribute it, or the test that proves the
+    # spec-immutability mechanism is still load-bearing runs on nothing that can break it.
+    .gitea/*|ci/*)             echo "R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06" ;;
     *) : ;;
   esac
 }
 
+# ---------------------------------------------------------------------------------
+# A CHANGED TEST ATTRIBUTES THE REQUIREMENTS IT CERTIFIES (detection rule 3).
+#
+# THE GAP THIS CLOSES (2026-09-12). The path map keys `conformance/*` to
+# R-TEST-01..04 — the requirements about the HARNESS. It says nothing about what a given
+# test PROVES. So editing `conformance/tests/l2-behavioral/tenant-isolation/check.sh` to
+# weaken or delete an assertion attributed R-TEST-01..04 and nothing else: the merge gate
+# then ran the harness's self-tests and NOT the weakened test's own requirement subset.
+# Weakening a test is the one change that most needs the requirement it certifies to be
+# re-verified, and it was the change least likely to select it.
+#
+# The `conformance.home.k3s/R-*: "true"` labels on a test ARE the authoritative declaration of
+# what it verifies (lib/coverage.py reads the same labels); this reads them directly. Note the
+# content scan in main() only sees ADDED/REMOVED lines, so it catches a change to the LABEL
+# BLOCK but not a change to the test BODY — which is exactly the case that matters here.
+#
+# WHY IT READS HEAD *OR* THE BASE: a change that DELETES a test, or deletes its
+# chainsaw-test.yaml, must still attribute what that test used to certify — otherwise removing
+# the only verifier of a requirement is the cheapest way to make the gate stop checking it.
+# HEAD first (the post-change truth), base as the fallback (the deleted case). Fail-closed:
+# this function can only ADD requirements, never remove one.
+: "${CONF_TEST_LABEL_PREFIX:=conformance.home.k3s}"   # == CONF_LABEL_PREFIX, conformance/lib/common.sh
+
+file_text_head_or_base() {
+  local path="$1"
+  git -C "$REPO_ROOT" show "HEAD:${path}" 2>/dev/null && return 0
+  git -C "$REPO_ROOT" show "${BASE}:${path}" 2>/dev/null && return 0
+  return 0
+}
+
+# test_requirements_for_path <changed-path> -> space-separated R-IDs (possibly empty).
+# Walks UP from the changed file to the nearest ancestor directory carrying a
+# chainsaw-test.yaml, so it works for any file in a test dir (check.sh, mutations.py,
+# fixtures, the chainsaw-test.yaml itself) and for nested test layouts.
+test_requirements_for_path() {
+  local f="$1" dir yaml text
+  case "$f" in conformance/tests/*/*) ;; *) return 0 ;; esac
+  dir="${f%/*}"
+  while :; do
+    case "$dir" in
+      conformance/tests|conformance|.|"") return 0 ;;
+      */*) ;;
+      *) return 0 ;;
+    esac
+    yaml="${dir}/chainsaw-test.yaml"
+    text="$(file_text_head_or_base "$yaml")"
+    if [ -n "$text" ]; then
+      # Whole-line `#` comments are dropped first — same "reference, not declaration" rule the
+      # content scan applies (changed_diff() rule 3): a label shown inside a header example is
+      # documentation. Only a real YAML mapping line counts.
+      printf '%s\n' "$text" \
+        | grep -vE '^[[:space:]]*#' \
+        | sed -nE "s#^[[:space:]]*${CONF_TEST_LABEL_PREFIX}/(R-[A-Z0-9]+-[0-9]+):[[:space:]]*[\"']?true.*#\1#p" \
+        | tr '\n' ' '
+      return 0
+    fi
+    dir="${dir%/*}"
+  done
+}
+
 main() {
   local files reqs="" rc=0 base_sha head_sha local_main merge_base
 
@@ -221,9 +389,27 @@ main() {
   while IFS= read -r f; do
     [ -n "$f" ] || continue
     reqs+=" $(path_requirements "$f")"
+    # Detection rule 3: a change under conformance/tests/<test>/ attributes THAT TEST'S own
+    # requirement labels, so weakening a test is gated by what the test certifies.
+    reqs+=" $(test_requirements_for_path "$f")"
   done <<< "$files"
 
   # Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
+  #
+  # THE ID PATTERN IS `R-[A-Z0-9]+-[0-9]+`, NOT `R-[A-Z]+-[0-9]+`. The WORKSTREAM segment may
+  # contain a DIGIT: `R-C2-01` and `R-C2-02` (command & control UI) are the only two such IDs
+  # in conformance/requirements/registry.yaml today, and the stricter pattern silently failed
+  # to match BOTH of them — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is.
+  # Every other consumer in the repo already allows the digit: test_requirements_for_path()
+  # twenty lines above uses `R-[A-Z0-9]+-[0-9]+`, as do conformance/lib/coverage.py,
+  # conformance/lib/status.py and conformance/lib/run_l1_offline.py. This grep was the lone
+  # dissenter, and it dissented by UNDER-attributing — the fail-OPEN direction. A change that
+  # retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02 from a
+  # path the map keys elsewhere, contributed those IDs to NOBODY, so the C2 tests were
+  # filtered out of the merge gate. Same defect class as the map's `*) : ;;` arm: a miss that
+  # reads as "nothing touched". path_requirements() has no arm for platform/homepage/* or
+  # platform/headlamp/* either, so before this fix the ONLY route to R-C2-01/02 was rule 3
+  # (editing the C2 test itself). Regression-tested in ci/tests/touched-requirements.test.sh.
   # `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
   # `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so
   #   what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
@@ -237,7 +423,7 @@ main() {
     | grep -vE '^(\+\+\+|---)' \
     | sed -E 's/^[+-]//' \
     | grep -vE '^[[:space:]]*#' \
-    | grep -hoE 'R-[A-Z]+-[0-9]+' \
+    | grep -hoE 'R-[A-Z0-9]+-[0-9]+' \
     | tr '\n' ' ' || true)"
 
   # Normalize: split, unique, sort. Emit to stdout.

ci/versions.env

@@ -21,6 +21,15 @@ YAMLLINT_VERSION="1.35.1"
 KUBECTL_VERSION="1.36.4"
 YQ_VERSION="4.44.6"
 
+# Alert-rule unit tests (promtool test rules) — the L0 check
+# conformance/tests/l0-static/alert-rules-unit-tested runs every
+# platform/**/*.promtool-test.yaml through it. MUST stay in lockstep with
+# infra/devcontainer/versions.env (which bakes it into the image) AND with the Prometheus
+# version running on prod, because promtool IS the rule engine: verified 2026-09-12 as
+# v3.14.0-distroless via `kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`.
+# Shipped inside the prometheus release tarball, not as a separate release.
+PROMTOOL_VERSION="3.14.0"
+
 # ServiceDefinition JSON-Schema validation (schemas/validate.sh). Pip-installed;
 # pinned here so CI and a laptop validate against the exact same validator.
 CHECK_JSONSCHEMA_VERSION="0.29.4"

clusters/README.md

@@ -97,6 +97,27 @@ Kustomization: apps-wave-1  (./apps/wave-1, dependsOn apps-wave-0)
 Kustomization: apps-wave-N  (./apps/wave-N, dependsOn apps-wave-N-1)
 ```
 
+### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
+
+The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
+controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
+and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
+reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
+before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
+**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
+`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
+have existed for months); it cost 12h on prod's cold start.
+
+So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
+platform-config`), and a rule:
+
+> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
+> **controller at runtime** rather than by a manifest in git.
+
+`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
+`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
+profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
+
 Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
 exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
 (`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own

clusters/base/apps-vms.yaml

@@ -4,8 +4,11 @@
 # NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
 # import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
 # health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
-# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles
+# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
-# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern).
+# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
+# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
+# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
+# adoption concern).
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
@@ -19,6 +22,28 @@ spec:
   wait: false
   dependsOn:
     - name: platform-config
+    # LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
+    # storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
+    # decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
+    # (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
+    # live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
+    # with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
+    # to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
+    # conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
+    #
+    # HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
+    # derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
+    # status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
+    # it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
+    #
+    # This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
+    # deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
+    # dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
+    #
+    # The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
+    # edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
+    # cannot silently drop this tier's KubeVirt/StorageClass gate.
+    - name: platform-config-cdi
   sourceRef:
     kind: GitRepository
     name: flux-system

clusters/base/backup.yaml

@@ -34,6 +34,9 @@ spec:
     substituteFrom:
       - kind: ConfigMap
         name: cluster-vars
-  # SOPS decryption for velero-cloud.sops.yaml (the scoped MinIO `cloud` credential) is layered per
+  # SOPS decryption for the `cloud` INI credentials — velero-cloud.sops.yaml (dev's scoped MinIO key)
-  # environment by the overlay (clusters/dev patches THIS Kustomization by name), identical to how
+  # and velero-cloud-prod.sops.yaml (prod's Cloudflare R2 token) — is layered per environment by the
-  # headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. See clusters/dev/kustomization.yaml.
+  # overlay (clusters/dev and clusters/prod patch THIS Kustomization by name), identical to how
+  # headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. Both Secrets are applied on both
+  # clusters; ${BACKUP_CREDENTIALS_SUFFIX} (substituted from cluster-vars above) decides which one the
+  # velero Deployment actually mounts. See clusters/dev/kustomization.yaml.

clusters/base/cilium.yaml

@@ -0,0 +1,60 @@
+---
+# Flux Kustomization: CILIUM (the CNI) — reconciles ./platform/cilium.
+# Serves R-PLAT-01 (platform components reconciled by GitOps) and R-GITOPS-01/03.
+#
+# WHY CILIUM HAS ITS OWN KUSTOMIZATION (split out of `platform`, 2026-09-01)
+#
+# This is the MIRROR IMAGE of the oauth2-proxy / headlamp splits, not a repeat of them. Those are
+# leaves that must NEVER gate a tier, so they left `platform` and run `wait: false` below it.
+# Cilium is the one component that legitimately SHOULD gate everything — but inside the shared
+# `platform` Kustomization its health was indistinguishable from eleven unrelated components:
+#
+#   * `platform`'s Ready condition could not tell you "the CNI is down" from "longhorn is slow".
+#   * The only emergency valve was `flux suspend kustomization platform` — freezing the ENTIRE
+#     tier (storage, TLS, DNS, ingress, monitoring, Keycloak, KubeVirt) to stop touching the CNI.
+#     `flux suspend kustomization cilium` now freezes exactly the CNI and nothing else, which is
+#     the valve the 2026-09-01 substitution incident wanted and did not have.
+#   * `platform`'s 10m interval + 5m timeout are wrong for a CNI: a 3-node DaemonSet rollout with
+#     `wait: true` can exceed 5m, and re-rendering the CNI every 10m buys nothing.
+#
+# GATING IS PRESERVED, NOT WEAKENED: `wait: true` here plus `platform.dependsOn: [cilium]` means
+# nothing in the graph reconciles until the CNI is healthy — the same guarantee as before, now
+# expressed as an explicit edge instead of an accident of co-tenancy.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: cilium
+  namespace: flux-system
+spec:
+  # A CNI does not need a 10m re-render cadence. A git push still reconciles immediately —
+  # kustomize-controller is source-triggered; `interval` is only the drift-correction floor.
+  interval: 1h
+  retryInterval: 2m
+  # 10m (not the 5m `platform` uses): a 3-node Cilium DaemonSet rollout under `wait: true`,
+  # rolled ONE NODE AT A TIME (maxUnavailable: 1, P2), legitimately exceeds 5m.
+  timeout: 10m
+  prune: true
+  # TRUE, deliberately: this is the ONE component whose health should gate the whole platform.
+  wait: true
+  # Same revision-gating reason `platform` has it (2026-09-01 incident): `cluster-vars` is APPLIED
+  # by the root `flux-system` Kustomization, and this Kustomization SUBSTITUTES from it. `dependsOn`
+  # is revision-gated (kustomize-controller v1.4.0 `checkDependencies`), so this cannot render
+  # revision N of helmrelease.yaml against the revision N-1 `cluster-vars`. That race is exactly how
+  # `k8sServiceHost: ""` once reached the Cilium HelmRelease and took down 2/3 agents.
+  dependsOn:
+    - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/cilium
+  # MANDATORY, not decorative: platform/cilium/helmrelease.yaml references ${K3S_API_HOST}.
+  # kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without
+  # this stanza an undefined ${K3S_API_HOST} FAILS this Kustomization outright (it no longer
+  # renders empty). That is the intended fail-closed behaviour — but it means the stanza is
+  # load-bearing for the CNI coming up at all. Do not remove it.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # No `decryption` stanza: platform/cilium carries no SOPS-encrypted resource (unlike
+  # oauth2-proxy/headlamp/crafty/velero, whose overlays patch one in by name).

clusters/base/cluster-vars.yaml

@@ -17,9 +17,63 @@ data:
   CLUSTER_ENV: "base"                       # dev | prod  (overlays set this)
 
   # --- DNS / TLS ---
-  BASE_DOMAIN: "home.keithjackson.dev"      # platform hostname suffix (same both envs; split-horizon)
+  # THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
-  DNS_ZONE: "keithjackson.dev"              # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
+  # the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
-                                            # ExternalDNS domain-filter must be the zone, not the subdomain
+  # ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
+  #
+  # THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
+  # hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
+  # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
+  # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
+  # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
+  # subdomain, so the two clusters can no longer describe the same name.
+  #
+  #   dev   -> home.keithjackson.dev            (UNCHANGED — see below)
+  #   prod  -> prod.home.keithjackson.dev
+  #
+  # DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
+  # `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
+  # relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
+  # the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
+  # (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
+  # GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
+  # it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
+  # tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
+  BASE_DOMAIN: "home.keithjackson.dev"
+
+  # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
+  # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
+  # so it cannot be created, updated or deleted regardless of TXT ownership.
+  #
+  # DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
+  # apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
+  # that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
+  # record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
+  # from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
+  #
+  # PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
+  # (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
+  # chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
+  # — it can never delete another environment's records — and the fallback is to set prod's value
+  # back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
+  EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
+
+  # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
+  # neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
+  # record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
+  # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
+  # one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
+  #
+  # DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
+  # under owner `default`; changing dev's value would make dev's own external-dns stop recognising
+  # them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
+  #
+  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
+  # forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
+  # this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
+  # visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
+  EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
+
   ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
   GATEWAY_VIP: "192.168.1.53"                # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
                                             # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
@@ -42,15 +96,57 @@ data:
   #
   # Substitution coverage (verify before adding a knob here): every path that consumes these declares
   # postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
-  # (CILIUM_LB_POOL_*, S3_BACKUP_ENDPOINT), crafty (CRAFTY_L4_VIP), apps-wave-0 (S3_BACKUP_ENDPOINT),
+  # (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
-  # velero (S3_BACKUP_ENDPOINT), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an UNSUBSTITUTED path would
+  # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
-  # ship the literal string into the cluster.
+  # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
+  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
+  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
+  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
+  # UNSUBSTITUTED path would ship the literal string into the cluster.
+  #
+  # ADDING A KNOB IS SAFE UNDER StrictPostBuildSubstitutions ONLY BECAUSE OF THE ROOT EDGE. An
+  # UNDEFINED variable is a HARD Kustomization failure, so a manifest that references a knob the
+  # live ConfigMap does not yet carry breaks its whole tier. `clusters/base/platform.yaml` gained
+  # `dependsOn: [flux-system]` for exactly this: dependsOn is REVISION-gated, so no substituting
+  # tier can reconcile revision N until the ROOT Kustomization has APPLIED revision N — i.e. until
+  # the new cluster-vars key EXISTS. Every substituting consumer chains transitively through
+  # `platform`, so one edge covers the graph, and a new knob plus its first reference may ship in
+  # ONE commit. (A hand-run `kubectl apply -k clusters/<env>` bypasses that ordering entirely: apply
+  # the overlay BEFORE the tier that references the knob, or the tier fails until it retries.)
 
   # k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
-  # [parity] MUST equal the Ansible bootstrap's k3s_api_host (infra/ansible/roles/cilium/defaults/
+  #
-  # main.yml -> cilium-values.yaml.j2, already inventory-parameterized). Wrong value = Cilium cannot
+  # *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
-  # reach the API server on adopt and the CNI breaks. Port stays 6443 in both envs, so it is NOT a knob.
+  # 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
-  K3S_API_HOST: "192.168.1.50"
+  # control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
+  # live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
+  # own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
+  # Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
+  # outage you would notice. A virtual address that follows the control plane is the fix, and it is
+  # what kube-vip was stood up to provide.
+  #
+  #   dev  -> 192.168.1.49   (= kube_vip_address, group_vars/all.yml; base default, below)
+  #   prod -> 192.168.2.9    (= kube_vip_address, group_vars/k3s_prod/vars.yml)
+  #
+  # THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
+  # unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
+  # move. Read it as "the API host Cilium dials", which is what it always meant.
+  #
+  # [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
+  # (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
+  # already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
+  # the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
+  # already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
+  # block at the top of group_vars/all.yml.
+  #
+  # Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
+  # in both envs, so it is NOT a knob.
+  #
+  # TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
+  # both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
+  # https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
+  K3S_API_HOST: "192.168.1.49"
 
   # CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
   # L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
@@ -65,15 +161,84 @@ data:
   # HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
   UNIFI_L4_VIP: "192.168.1.54"               # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
   CRAFTY_L4_VIP: "192.168.1.55"              # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
+  # PiHole's DNS listener: udp/53 + tcp/53 for LAN CLIENTS (apps/wave-0/pihole-dns.yaml). It is a
+  # THIRD pinned L4 VIP for the same reason as the two above — the `shared` Gateway is an L7 HTTPS
+  # terminator and cannot express DNS at all (udp/53 especially). It must NOT be ${GATEWAY_VIP}:
+  # LAN clients are pointed at this address as their resolver, and it has to stay put across
+  # reschedules, so it is pinned by lbipam.cilium.io/ips rather than taken from the pool head.
+  #
+  # WHY A SEPARATE ADDRESS AND NOT THE GATEWAY VIP: every `<svc>.${BASE_DOMAIN}` name resolves TO
+  # the Gateway VIP. Serving DNS from the same address would be legal but conflates "the thing that
+  # answers where services live" with "a service", and it would put LAN resolution behind the
+  # Gateway Service's own lifecycle. Keep them separate.
+  #
+  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
+  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
+  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+
+  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
+  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
+  # from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
+  # CIDR, not to Cilium's `world` entity the way the unifi/crafty game-port allows are. It is the
+  # same /24 the node IPs and ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} come out of.
+  #
+  # dev  -> 192.168.1.0/24  (Proxmox LAN; nodes .50/.51/.52)
+  # prod -> 192.168.2.0/24  (prod LAN; nodes .10/.11/.12)
+  LAN_CIDR: "192.168.1.0/24"                 # ingress scope for the PiHole DNS listener
+
+  # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
+  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
+  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
+  # (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
+  # `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
+  # declared in the ServiceDefinition, because the path is identical in every environment and only
+  # the ADDRESS is not.
+  #
+  # WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
+  # The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
+  #   dev  nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
+  #        NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
+  #        /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
+  #        per the convention of the LAN block above.
+  #   prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
+  #        22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
+  #        the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
+  #        DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
+  # So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
+  # blast radius of shipping it unset, which is NOT the usual "fails to apply".
+  BULK_NFS_SERVER: "192.168.1.9"
 
   # Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
   # Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
   # nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
-  # Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR.
+  # Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR —
+  # and, measured 2026-09-05 from a pod INSIDE prod, it CANNOT reach it anyway (prod is on the SOHO
+  # network 192.168.2.0/24, which is pfSense's WAN, while this MinIO sits inside pfSense's LAN:
+  # `curl http://192.168.1.9:9000/minio/health/live` timed out after 15s). Prod uses Cloudflare R2.
   S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
 
+  # S3 region. Every S3 client REQUIRES one; MinIO ignores whatever it is; Cloudflare R2 demands
+  # exactly `auto` (its `head-bucket` reports the physical hint `WNAM`, which is NOT a usable region
+  # string). This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-
+  # accident for MinIO and wrong for anything else, which is why it is a knob now. The base default
+  # is dev's LIVE value, deliberately: dev's Velero has been signing requests with `us-east-1`
+  # against that MinIO since bootstrap, and this repo's rule is that base == the known-good dev
+  # value. Prod overrides to `auto`.
+  S3_BACKUP_REGION: "us-east-1"
+
   # --- storage (ADR-0003) ---
   LONGHORN_DEFAULT_REPLICA_COUNT: "3"       # default StorageClass replica count (3 on 3 nodes)
+  # Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
+  # replica is written into. It must be the path the NODE's own data disk is actually mounted at —
+  # roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
+  # fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
+  # Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
+  # working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
+  # on it — changing dev's value would orphan every replica, so the dev overlay restates it
+  # verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
+  # dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
+  # neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
+  LONGHORN_DATA_PATH: "/var/lib/longhorn"
 
   # --- observability sizing (docs/10) ---
   MONITORING_PROMETHEUS_RETENTION: "7d"     # dev short | prod long
@@ -113,6 +278,99 @@ data:
   VELERO_BACKUP_SCHEDULE: "0 3 * * *"        # cron; overlays may differ
   VELERO_RETENTION: "168h"                   # dev short | prod long
 
+  # ==========================================================================================
+  # OBJECT-STORE TENANCY — the three knobs that keep two clusters and two backup tools from
+  # writing into each other's data. THESE ARE SAFETY KNOBS, not preferences.
+  # ==========================================================================================
+  # Nothing about the object layout was per-environment before this: Velero's bucket was the
+  # hardcoded literal `velero` with no prefix, and every CNPG Cluster used the hardcoded
+  # `s3://cnpg` with barman's `serverName` defaulting to the CLUSTER NAME — and BOTH environments
+  # have a keycloak-db, a nextcloud-db and a semaphore-db. Point two clusters at one bucket under
+  # that layout and:
+  #   (a) each cluster's Velero ADOPTS the other's backups and TTL-DELETES them on retention; and
+  #   (b) two different Postgres servers stream WAL into ONE Barman prefix, which does not merely
+  #       mix files — it CORRUPTS THE CHAIN and destroys PITR for both. That is the worse half:
+  #       dev's DR restore is PROVEN (R-STATE-03) and this would silently un-prove it.
+  #
+  # WHY THERE IS NO SINGLE `S3_BACKUP_BUCKET`. The two consumers take differently-SHAPED values:
+  # the Velero chart wants `bucket` and `prefix` as separate fields, while CNPG's
+  # barmanObjectStore.destinationPath is ONE full s3:// URL whose first path segment IS the bucket.
+  # CNPG's bucket therefore already lives inside CNPG_BACKUP_PATH; adding a second knob that had to
+  # hold the same bucket name a second time would be a rotation waiting to drift — the exact class
+  # of duplication this repo has been bitten by. Three knobs, each shaped for its own consumer, and
+  # no two of them can contradict each other.
+  #
+  # BASE = DEV'S LIVE LAYOUT, BYTE-FOR-BYTE, AND MUST STAY THAT WAY. Dev's existing backups are at
+  # `s3://velero/` (bucket root, no prefix) and `s3://cnpg/<clusterName>/` on that MinIO. Giving dev
+  # a prefix, or renaming its buckets, would ORPHAN the very backups its proven restore depends on —
+  # the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default` and its LONGHORN_DATA_PATH stays
+  # /var/lib/longhorn. Verified: with an EMPTY prefix the Velero chart omits `objectStorage.prefix`
+  # from the rendered BackupStorageLocation entirely (helm template, chart 12.1.0), so dev's live
+  # BSL is unchanged by this parameterization rather than merely equivalent to it.
+  #
+  # PROD, which has no history to preserve, gets ONE bucket with TWO disjoint top-level prefixes:
+  #
+  #     s3://k3s-prod-backup/velero/…                 Velero (kopia repo + backup metadata)
+  #     s3://k3s-prod-backup/barman/<clusterName>/…   CNPG WAL + base backups
+  #
+  # The operator created exactly one R2 bucket, so velero-vs-barman separation has to come from the
+  # prefix rather than the bucket. The BUCKET ROOT IS DELIBERATELY LEFT EMPTY: if a prefix ever
+  # renders wrong, its objects land somewhere obviously bogus instead of silently colliding with a
+  # real tenant's data. Dev↔prod isolation is doubly independent — different endpoint AND different
+  # bucket — so neither cluster can reach the other's objects even with a mis-substituted prefix.
+  VELERO_BACKUP_BUCKET: "velero"             # dev: the `velero` MinIO bucket | prod: k3s-prod-backup
+  VELERO_BACKUP_PREFIX: ""                   # dev: bucket root (live layout) | prod: "velero"
+  CNPG_BACKUP_PATH: "s3://cnpg"              # dev: the `cnpg` MinIO bucket root | prod: s3://k3s-prod-backup/barman
+
+  # ------------------------------------------------------------------------------------------------
+  # WHICH CREDENTIAL SECRET THE BACKUP CONSUMERS READ (added 2026-09-07 — the per-env credential split)
+  # ------------------------------------------------------------------------------------------------
+  # The three knobs above say WHERE the backups go. This one says WHICH KEY signs the requests, and
+  # it exists because the endpoint moved per-environment while the credential did not.
+  #
+  # THE BUG IT FIXES. The four backup-credential Secrets — velero-cloud.sops.yaml and the three
+  # <db>-backup-credentials.sops.yaml — are SHARED BASE manifests: platform/ and apps/ are rendered
+  # by the same Flux tiers on both clusters, and cluster-vars substitution is the ONLY thing that
+  # differs (R-ENV-02). So writing prod's R2 token into those four files did not GIVE prod a
+  # credential, it REPLACED DEV'S: dev's Velero and all three dev CNPG Clusters would have started
+  # signing MinIO requests with an R2 key id. For Velero that is a broken BSL; for CNPG it is worse,
+  # because `archive_mode=on` means Postgres REFUSES TO RECYCLE a WAL segment it could not archive,
+  # so a 403 loop is a monotonically filling pgdata PVC ending in a PANIC — the same failure prod
+  # spent its whole bring-up in with a placeholder endpoint. One shared Secret also means rotating
+  # either environment's key silently breaks the other; that is the structural reason for the split,
+  # independent of dev's eventual decommission.
+  #
+  # THE MECHANISM. Each credential is now TWO Secret objects, both applied to BOTH clusters (they are
+  # base resources; nothing here forks a manifest per environment). The consumer names one:
+  #
+  #     velero:   credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
+  #     CNPG:     s3Credentials.*.name:  <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
+  #
+  # ONE knob, not four, ON PURPOSE. Four independent secret-name knobs could be updated three-
+  # quarters of the way, leaving one consumer pointed at the OTHER environment's key with nothing
+  # rendering wrong — and StrictPostBuildSubstitutions cannot catch that, because it only fails on an
+  # UNDEFINED variable, never on a defined-but-stale one. With a single suffix the four move together
+  # or not at all. The cost is that the rendered Secret name is not greppable from the consumer; the
+  # names are spelled out in full in each kustomization.yaml beside the resources.
+  #
+  # THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names,
+  # which are the objects that already exist and that the live dev cluster is running. A default
+  # naming a Secret that does not exist would be worse than the shared-Secret state it replaces:
+  # Velero's pod would sit in ContainerCreating forever on a missing `existingSecret` volume, and a
+  # CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — i.e. the disk-full clock
+  # again — so an unresolvable default breaks the referencing cluster HARDER than a wrong key does.
+  # Empty is also the value dev must carry, so base and dev agree, exactly like VELERO_BACKUP_PREFIX.
+  #
+  # An EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions: strict mode fails on a variable that is
+  # not DEFINED, not on one defined as "". VELERO_BACKUP_PREFIX above has shipped empty since the
+  # gate was enabled and proves it. Both overlays state this knob explicitly — dev "" and prod
+  # "-prod" — so neither inherits it silently.
+  #
+  # ADDING A NEW BACKUP CREDENTIAL: create BOTH halves (`x-credentials.sops.yaml` and
+  # `x-credentials-prod.sops.yaml`), list both in the kustomization, and reference
+  # `x-credentials${BACKUP_CREDENTIALS_SUFFIX}`. Never a single shared Secret again.
+  BACKUP_CREDENTIALS_SUFFIX: ""              # dev: "" (MinIO creds) | prod: "-prod" (R2 creds)
+
   # --- placement ---
   PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os"   # base no-op selector; prod may pin roles
   PLATFORM_NODE_SELECTOR_VALUE: "linux"

clusters/base/homepage.yaml

@@ -34,8 +34,12 @@ spec:
     kind: GitRepository
     name: flux-system
   path: ./platform/homepage
-  # Homepage references no ${VARS} today; the substitution is retained for parity with the other
+  # LOAD-BEARING, not parity decoration (the comment here used to say the opposite). This component
-  # tiers / future overlay knobs.
+  # references ${BASE_DOMAIN} in TWO places: httproute.yaml's apex hostname, and EVERY tile href in
+  # configmap.yaml. Without this substitution the token is applied LITERALLY — the apex route binds
+  # the host `${BASE_DOMAIN}`, which matches no Gateway listener, and every link on the operator's
+  # front door points at a host that does not exist — while Flux still reports this Kustomization
+  # Ready. It fails OPEN. Asserted by conformance/tests/l0-static/c2-root-surface (5).
   postBuild:
     substituteFrom:
       - kind: ConfigMap

clusters/base/kustomization.yaml

@@ -5,8 +5,24 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - flux-system.yaml     # GitRepository (reconciliation source)
+  # Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
+  # of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
+  # Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
+  # `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
+  # interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
+  # freeze the other eleven platform components. See clusters/base/cilium.yaml.
+  - cilium.yaml          # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
   - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
   - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
+  # SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
+  # manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
+  # datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
+  # platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
+  # keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
+  # CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
+  # cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
+  # as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
+  - platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
   # App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
   # Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
   # (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
@@ -34,6 +50,13 @@ resources:
   # reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same
   # non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform).
   - crafty.yaml          # Flux Kustomization -> ./platform/crafty (dependsOn platform-config)
+  # Alert runbooks — POST-platform tier (dependsOn platform-config, wait:false). Serves the 34
+  # runbooks the alert rules' `runbook_url` annotations point at, at <apex>/runbooks/<name>; before
+  # this tier existed that path was a 404 and every alert handed the operator a dead link. Same
+  # non-gating leaf rationale as headlamp/homepage/crafty. Carries NO postBuild substitution on
+  # purpose — it renders documentation containing `${...}` shell examples that envsubst would empty;
+  # see clusters/base/runbooks.yaml.
+  - runbooks.yaml        # Flux Kustomization -> ./runbooks (dependsOn platform-config)
   # Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup
   # controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
   # Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).

clusters/base/platform-config-cdi.yaml

@@ -0,0 +1,135 @@
+---
+# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
+# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
+#
+# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
+#
+# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
+# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
+# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
+# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
+# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
+# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
+# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
+#
+# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
+# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
+# graph order itself, so a cold cluster needs no human in the loop.
+#
+# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
+# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
+# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
+# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
+# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
+# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
+# kind. One edge, no pre-apply.
+#
+# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
+# and everything else waits on: putting CDI's health on that critical path is the mistake the
+# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
+# StorageProfiles that were in the wrong pass.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: platform-config-cdi
+  namespace: flux-system
+spec:
+  interval: 10m
+  # 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
+  # very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
+  # now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
+  # about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
+  # cache is one retryInterval. Keep it short.
+  retryInterval: 2m
+  # 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
+  # image to pull and no controller to become Available, so a slow apply here means a real error, not
+  # a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
+  timeout: 2m
+  # TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
+  # component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
+  # It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
+  # second path also declares. See the transition note at the bottom of this file.
+  prune: true
+  # TRUE, but understand exactly what it buys and what it does NOT.
+  #
+  # CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
+  # provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
+  # `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
+  # it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
+  # is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
+  # If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
+  # `apps-vms` waits on it.
+  #
+  # WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
+  # provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
+  # storageprofile-controller computes that status from our spec override plus inference. So Ready
+  # here means "the spec override is accepted by the API server", not "CDI has folded it into
+  # status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
+  # below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
+  # a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
+  wait: true
+  dependsOn:
+    # platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
+    # causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
+    # ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
+    # flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform-config-cdi
+  # NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
+  # no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
+  # oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
+  # clusters/dev or clusters/prod for this tier.
+  #
+  # substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
+  # substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
+  #   1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
+  #      whose Kustomization declares substituteFrom. Omitting it would move
+  #      cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
+  #      could ship as a literal string, precisely the class of bug that check exists to stop.
+  #   2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
+  #      with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
+  #      empty. Without the stanza there is no substitution at all and the literal ships silently.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
+  # Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
+  # Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
+  # StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
+  # the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
+  # dual-ownership fight — the objects are pruned before they are re-applied under this tier's
+  # `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
+  #
+  # Why that is acceptable here — and it is NOT the platform/cilium case:
+  #   * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
+  #     nothing else in the repo declares a StorageProfile. The stale set is exactly two
+  #     cluster-scoped CDI CRs.
+  #   * The objects do not actually go missing: CDI's storageprofile-controller creates a
+  #     StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
+  #     inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
+  #     absent".
+  #   * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
+  #     existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
+  #     is disturbed and no data path is touched.
+  #   * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
+  #     instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
+  #     permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
+  #     dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
+  #     created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
+  #     could.
+  #
+  # PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
+  # happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
+  # config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
+  # healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
+  # --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
+  # StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
+  #   `flux get kustomization platform-config` must be Ready BEFORE merging.
+  # If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
+  # temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
+  # this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
+  # `prune: true` in a second commit.

clusters/base/platform.yaml

@@ -11,8 +11,53 @@ spec:
   interval: 10m
   retryInterval: 2m
   timeout: 5m
-  prune: true          # drift correction: delete resources removed from git (R-GITOPS-03)
+  # Cilium was moved OUT of this tier into its own Kustomization (clusters/base/cilium.yaml)
+  # on 2026-09-01. `prune: true` is restored here now that the ownership transfer is VERIFIED
+  # live: Namespace/kube-system, the Cilium HelmRelease and the HelmRepository all carry
+  # `kustomize.toolkit.fluxcd.io/name=cilium`, this Kustomization's inventory no longer lists
+  # any of the three, and the `cilium` Kustomization's inventory holds exactly those three.
+  #
+  # LESSON WORTH KEEPING: `platform/cilium/namespace.yaml` was the ONLY declaration of
+  # Namespace/kube-system under ./platform. Moving that path out of a prune:true Kustomization
+  # made the stale set `delete namespace kube-system` — coredns, metrics-server, local-path AND
+  # Cilium. Before moving ANY platform/* component in future, check whether it is the sole
+  # declarer of a shared Namespace; that turns a component-scoped prune into a cluster-scoped one.
+  prune: true
   wait: true           # block `apps` until platform is healthy
+  # ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
+  #
+  # `cluster-vars` — the ConfigMap every substituting Kustomization reads — is APPLIED by the root
+  # `flux-system` Kustomization (clusters/<env>/sync.yaml), not by this one. Without an edge here,
+  # `platform` could render a manifest from revision N against the revision N-1 `cluster-vars`, so a
+  # NEWLY ADDED knob was undefined and (non-strict substitution) silently became the EMPTY STRING.
+  # That is how `k8sServiceHost: ""` reached the Cilium HelmRelease.
+  #
+  # `dependsOn` is REVISION-gated, not merely Ready-gated (verified against
+  # kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our version):
+  # when the dependency shares a GitRepository with the dependent, the dependent cannot reconcile
+  # revision N until the dependency has APPLIED revision N. Applying revision N is exactly what
+  # writes a newly-added `cluster-vars` knob — so this one edge closes the race.
+  #
+  # `platform` is the ONLY Kustomization in the graph with no `dependsOn`; every other substituting
+  # consumer (platform-config, apps-wave-0/1/2, apps-vms, oauth2-proxy, headlamp, homepage, crafty,
+  # velero) chains transitively through it. One edge therefore protects the whole graph.
+  #
+  # TRADE-OFF — READ THIS BEFORE YOU CURSE IT: the root is now a GATE. A malformed manifest
+  # ANYWHERE under the root's path (clusters/<env>, which includes clusters/base) fails the root
+  # Kustomization and therefore stops `platform` and EVERYTHING downstream, where before this edge
+  # the rest of the graph would have kept reconciling. That is deliberate and fail-closed: a broken
+  # entrypoint should stop the line, not let consumers render against a half-applied variable set.
+  # No deadlock risk — the root runs `wait: false`, so it reaches Ready on APPLY success and never
+  # health-gates its own children (see clusters/dev/sync.yaml).
+  dependsOn:
+    - name: flux-system   # the ROOT Kustomization (sync.yaml) that applies cluster-vars
+    # THE CNI GATES THE TIER. Cilium was split out of ./platform into its own Kustomization
+    # (clusters/base/cilium.yaml, wait:true) so its health has its own Ready condition and its own
+    # suspend valve. This edge preserves the ordering that co-tenancy used to provide implicitly:
+    # nothing in `platform` — and therefore nothing anywhere downstream — reconciles until the CNI
+    # is healthy. It is also revision-gated, which is what makes the prune-safety argument above
+    # deterministic rather than a race.
+    - name: cilium
   sourceRef:
     kind: GitRepository
     name: flux-system

clusters/base/runbooks.yaml

@@ -0,0 +1,57 @@
+---
+# Flux Kustomization: alert runbooks — reconciles ./runbooks.
+#
+# WHAT THIS SERVES. Every alert rule in this repo carries
+# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names.
+# Until this tier existed that path returned 404 on the live cluster, so every alert that reached a
+# human handed them a dead link at the moment they needed help. This Kustomization renders the
+# markdown under ./runbooks into a ConfigMap and a two-replica nginx that serves it at that exact
+# prefix. See runbooks/README.md and runbooks/serving/httproute.yaml.
+#
+# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false) — the same placement and
+# the same reasoning as homepage/headlamp/oauth2-proxy. It needs the wildcard TLS cert, the shared
+# Gateway (including the `https-apex` listener) and split-horizon DNS, all of which arrive with
+# `platform` + `platform-config`; and a leaf web server must never be able to stall the platform
+# chain, which is the whole lesson of DECISIONS-LOG's "Headlamp gating stall". wait:false guarantees
+# its health can never gate anything even if something is later mistakenly pointed dependsOn it.
+#
+# NO postBuild.substituteFrom — THE ONE DELIBERATE DIVERGENCE FROM EVERY OTHER TIER, AND THE MOST
+# IMPORTANT LINE IN THIS FILE.
+# This tier renders the runbook PROSE into a ConfigMap, and runbooks are full of shell: `$POD`,
+# `$(kubectl ...)`, and — unavoidably, since they document these very annotations — the literal text
+# `${BASE_DOMAIN}`. Flux's postBuild substitution runs envsubst over the ENTIRE built stream, and it
+# FAILS OPEN: an undefined `${X}` becomes the empty string while the Kustomization still reports
+# Ready (DECISIONS-LOG 2026-09-01 — that is how `k8sServiceHost` was emptied and Cilium broke on 2
+# of 3 dev nodes). Enabled here it would silently mutilate the documentation: every braced expansion
+# in every command example blanked, in a file nobody re-reads after writing. Worse, the L0 check
+# `conformance/tests/l0-static/flux-postbuild-substitution` scans only files reachable through
+# kustomization `resources:`, not `configMapGenerator` inputs, so the damage would be invisible to
+# the gate as well.
+# The fix is to have nothing to substitute. The only knob this tier would have wanted is the apex
+# hostname, and runbooks/serving/httproute.yaml avoids needing it by declaring NO `hostnames:` and
+# inheriting the `https-apex` listener's own hostname instead. So this tier is
+# environment-independent by construction and R-ENV-02 ("authored once, promoted dev -> prod") holds
+# without substitution rather than because of it.
+# IF A FUTURE KNOB EVER GENUINELY NEEDS SUBSTITUTION HERE: do not add substituteFrom to this
+# Kustomization. Split the serving manifests into their own substituted tier and leave the content
+# ConfigMap in an unsubstituted one, or the runbooks lose their shell examples.
+#
+# NO SOPS decryption stanza: this tier carries no encrypted Secret (markdown plus five plain
+# manifests), so — like homepage — neither clusters/dev nor clusters/prod patches it.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: runbooks
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config   # wildcard TLS + Gateway (incl. the https-apex listener) first
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./runbooks

clusters/dev/kustomization.yaml

@@ -106,8 +106,11 @@ patches:
           secretRef:
             name: sops-age
   # velero (backup controller) has its own post-platform Flux Kustomization (clusters/base/backup.yaml).
-  # It carries velero-cloud.sops.yaml (the scoped MinIO `cloud` credential), so it needs its own SOPS
+  # It carries BOTH halves of the per-env `cloud` INI credential — velero-cloud.sops.yaml (dev's scoped
-  # decryption stanza (same as headlamp/oauth2-proxy/crafty). The `sops-age` Secret is out-of-band (name only).
+  # MinIO key) and velero-cloud-prod.sops.yaml (prod's R2 token) — so it needs its own SOPS decryption
+  # stanza (same as headlamp/oauth2-proxy/crafty). Which one the Deployment mounts is chosen by
+  # BACKUP_CREDENTIALS_SUFFIX in the cluster-vars patch below, not here. The `sops-age` Secret is
+  # out-of-band (name only).
   - patch: |-
       apiVersion: kustomize.toolkit.fluxcd.io/v1
       kind: Kustomization
@@ -119,6 +122,23 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
+  # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
+  # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
+  # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
+  # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
+  # as load-bearing as the file itself. The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-vms
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   - patch: |-
       apiVersion: source.toolkit.fluxcd.io/v1
       kind: GitRepository
@@ -148,6 +168,33 @@ patches:
       - op: replace
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
+      # --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
+      # DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
+      # like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
+      # Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
+      # Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
+      # (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
+      # prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
+      # not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
+      # not a variable change: infra/PROD-PROVISIONING.md §10.
+      - op: replace
+        path: /data/BASE_DOMAIN
+        value: "home.keithjackson.dev"
+      # ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
+      # and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
+      # the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
+      - op: replace
+        path: /data/EXTERNALDNS_DOMAIN_FILTER
+        value: "keithjackson.dev"
+      # DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
+      # registry under owner `default` (the external-dns chart default, which is what both clusters
+      # silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
+      # BY this string: any other value and it stops recognising every record it manages, then
+      # re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
+      # whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
+      - op: replace
+        path: /data/EXTERNALDNS_TXT_OWNER_ID
+        value: "default"
       # --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
       # EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
       # equals the base default. These values are what the LIVE dev cluster is running right now, so
@@ -160,11 +207,25 @@ patches:
       - op: replace
         path: /data/GATEWAY_VIP
         value: "192.168.1.53"
-      # k3s API endpoint Cilium's kube-proxy replacement dials — the primary control-plane node
+      # k3s API endpoint Cilium's kube-proxy replacement dials — DEV'S CONTROL-PLANE VIP,
-      # (k3s-1). [parity] equals the Ansible inventory's k3s_api_host for dev.
+      # the address kube-vip floats, NOT the primary node .50.
+      # [parity] equals the Ansible inventory's k3s_apiserver_endpoint for dev
+      # (= kube_vip_address, group_vars/all.yml). See clusters/base/cluster-vars.yaml for why.
+      #
+      # *** THE ONE EXCEPTION TO "NEVER LET THIS BLOCK DRIFT FROM THE LIVE CLUSTER" ***
+      # Every other value in this block is what dev is running right now. This one is not:
+      # the live dev cilium-agents dial 192.168.1.50 (k3s-1). Changing it here is a
+      # DELIBERATE, OPERATOR-APPROVED move (2026-09-06) and reconciling it ROLLS THE CILIUM
+      # AGENT DAEMONSET on dev — one node at a time (updateStrategy.maxUnavailable: 1, kept
+      # in parity for exactly this reason). Do not push this to dev-live casually, and not
+      # while dev's DNS migration is mid-flight; the L2 datapath survives an agent restart
+      # but a wrong value here is the 2026-09-01 incident. .49 is in dev's serving-cert SANs
+      # already (k3s_tls_sans) and dev's own kubeconfig validates against it via
+      # tls-server-name, so TLS is not the risk — the risk is reachability, and .49 is on the
+      # same LAN as the nodes and ARP-announced by kube-vip on every server.
       - op: replace
         path: /data/K3S_API_HOST
-        value: "192.168.1.50"
+        value: "192.168.1.49"
       # Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
       # (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
       - op: replace
@@ -181,15 +242,81 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.1.55"
+      # Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
+      # (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
+      # Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
+      # Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
+      # end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
+      # change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
+      - op: replace
+        path: /data/BULK_NFS_SERVER
+        value: "192.168.1.9"
       # Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
       # barmanObjectStore and Velero's backupStorageLocation write to.
+      #
+      # DEV IS DELIBERATELY STAYING ON THIS MinIO while prod goes to Cloudflare R2 (2026-09-07).
+      # Prod could not use it at all — it cannot even route to it — but dev's move is a separate,
+      # operator-gated decision, because R2 would start dev on an EMPTY prefix: Velero would write
+      # a fresh kopia repository, CNPG would take new base backups, and the 30-day Barman window
+      # would rebuild from zero. Dev's DR restore is the one PROVEN recovery path in this repo
+      # (R-STATE-03); resetting its PITR window is not a side effect to take while fixing prod.
       - op: replace
         path: /data/S3_BACKUP_ENDPOINT
         value: "http://192.168.1.9:9000"
+      # DO NOT CHANGE THESE FOUR. They are dev's LIVE object layout, restated verbatim for the same
+      # reason as LONGHORN_DATA_PATH and EXTERNALDNS_TXT_OWNER_ID below/above: they equal the base
+      # defaults today, and a silent drift in base would ORPHAN dev's existing backups — the ones
+      # its proven restore depends on — without any manifest looking wrong.
+      #
+      # `us-east-1` is what dev's Velero has signed every request with since bootstrap; MinIO
+      # ignores the value, but changing it is a needless risk against a live, working backup path.
+      # An EMPTY VELERO_BACKUP_PREFIX means the bucket root, which is where dev's backups already
+      # are — and the Velero chart omits `objectStorage.prefix` entirely when it is empty (verified
+      # against chart 12.1.0), so dev's live BackupStorageLocation is byte-unchanged by this.
+      - op: replace
+        path: /data/S3_BACKUP_REGION
+        value: "us-east-1"
+      - op: replace
+        path: /data/VELERO_BACKUP_BUCKET
+        value: "velero"
+      - op: replace
+        path: /data/VELERO_BACKUP_PREFIX
+        value: ""
+      - op: replace
+        path: /data/CNPG_BACKUP_PATH
+        value: "s3://cnpg"
+      # WHICH CREDENTIAL SIGNS THOSE REQUESTS. Empty suffix = the UNSUFFIXED Secrets
+      # (velero-cloud-credentials, <db>-backup-credentials) — the scoped MinIO keys dev's Velero and
+      # its three CNPG Clusters are running RIGHT NOW, byte-identical to what is on the cluster. Prod
+      # renders "-prod" and reads the Cloudflare R2 token from the parallel `-prod` Secrets instead.
+      #
+      # RESTATED EXPLICITLY even though it equals the base default, for the same reason as
+      # EXTERNALDNS_TXT_OWNER_ID and the four backup-layout knobs above: the four Secrets are SHARED
+      # base manifests, so before this split prod's R2 token landed in the objects DEV reads. Dev
+      # signing MinIO with an R2 key id is 403 on every barman-cloud-wal-archive, and archive_mode=on
+      # turns that into a filling pgdata PVC — the disk-full clock, on the one cluster whose DR
+      # restore is PROVEN (R-STATE-03). This line is what keeps dev on its own credential.
+      #
+      # DEV IS NOT BEING BACKED UP GOING FORWARD and is slated for decommission (operator,
+      # 2026-09-07), so this is not protecting a future backup — it is protecting the EXISTING
+      # backups and the WAL chain until dev is retired. Do not "simplify" it away while dev runs.
+      - op: replace
+        path: /data/BACKUP_CREDENTIALS_SUFFIX
+        value: ""
       # dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
         value: "3"
+      # DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
+      # on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
+      # mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
+      # else would orphan every one of them (they are not recoverable from the old path by
+      # rescheduling). Restated explicitly even though it equals the base default, exactly like the
+      # LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
+      # its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
+      - op: replace
+        path: /data/LONGHORN_DATA_PATH
+        value: "/var/lib/longhorn"
       # smaller retention / volumes to keep the dev workstation footprint light.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION

clusters/prod/kustomization.yaml

@@ -5,11 +5,255 @@
 # When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
 # here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
 # values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
+#
+# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
+# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
+# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
+# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
+# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
+# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
+# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
+# done here: prod is design-only and its root object is the prod bring-up owner's call.)
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
   - ../base
+  - sync.yaml          # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
 patches:
+  # --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
+  # Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
+  # The `sops-age` Secret is created OUT OF BAND on the prod cluster
+  # (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
+  # Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: platform
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-0
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-1
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-wave-2
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: oauth2-proxy
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: headlamp
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: crafty
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: velero
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
+  # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
+  # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
+  # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
+  # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
+  # as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-vms
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
+  # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
+  #
+  # ############################################################################################
+  # ###                                                                                      ###
+  # ###   *** T E M P O R A R Y   B O O T S T R A P   S O U R C E  —  NOT PERMANENT ***       ###
+  # ###                                                                                      ###
+  # ###   The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside   ###
+  # ###   any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST   ###
+  # ###   be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this  ###
+  # ###   and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is  ###
+  # ###   a live single point of failure, not a settled design. See "CUTOVER" below.          ###
+  # ###                                                                                      ###
+  # ############################################################################################
+  #
+  # WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
+  # unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
+  # Two independent reasons, both consequences of the split-horizon design:
+  #   1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
+  #      GitOps source lives inside the very cluster slated for decommission. Gitea must move to
+  #      prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
+  #   2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
+  #      answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
+  #      192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
+  #      gitea.home.keithjackson.dev to itself and never reach a repository.
+  #
+  # WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
+  #   bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
+  #   operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
+  #   reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
+  #   change was needed. This is the same "local bootstrap source" pattern dev itself used before
+  #   its Phase-6 cutover (docs/05 "Source-of-truth").
+  #
+  # PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
+  # for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
+  # the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
+  # .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
+  #
+  # NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
+  # works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
+  #   curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
+  #   git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git  -> refs/heads/main
+  # If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
+  # THAT FIRST: a private repo here fails exactly that way.
+  #
+  # *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
+  #   WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
+  #         through prod's own Gateway VIP with a publicly-trusted cert.
+  #   DO:   1. push/mirror this repo into the prod in-cluster Gitea;
+  #         2. change the url below to
+  #            https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+  #            and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
+  #            clusters/dev/kustomization.yaml, is what "done" looks like);
+  #         3. confirm `flux get sources git -n flux-system` reconciles the new revision;
+  #         4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
+  #            exactly as dev retained its bootstrap Gitea.
+  #   UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
+  #   powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
+  #   it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
+  #
+  # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
+  # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
+  # CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
+  # NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
+  # Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
+  # is the revision you intend prod to run (`git ls-remote` above, against local `main`).
+  - patch: |-
+      apiVersion: source.toolkit.fluxcd.io/v1
+      kind: GitRepository
+      metadata:
+        name: flux-system
+        namespace: flux-system
+      spec:
+        # CUTOVER DONE 2026-09-06 (operator direction: "move the repo to prod"). Prod now reconciles
+        # from PROD'S OWN in-cluster Gitea. The workstation bootstrap container
+        # (http://192.168.2.132:3000) is retired as prod's source — it was a podman container on the
+        # operator's workstation holding a DHCP lease on WIFI, i.e. production's source of truth was a
+        # laptop. It is kept only as a recovery path (bootstrap/gitea/bootstrap-gitea.sh).
+        #
+        # THIS LINE IS SELF-REFERENTIAL AND THAT IS THE WHOLE DIFFICULTY. The root Kustomization
+        # renders ./clusters/prod, which contains this patch — so a live `kubectl patch` of
+        # GitRepository.spec.url is REVERTED on the next reconcile unless the authored value moves
+        # too. The cutover therefore had to be: (1) push the repo to prod's Gitea, (2) fast-forward
+        # its `main` to the exact commit prod was already running so the source swap changed nothing
+        # else, (3) live-patch the url to open a window, (4) land THIS commit through that window so
+        # the authored value agrees. Dev hit the identical trap on 2026-09-05
+        # (infra/PROD-PROVISIONING.md §10a "D0/D1 as executed").
+        #
+        # Verified before the switch: a throwaway pod INSIDE prod resolved this host through
+        # in-cluster CoreDNS to the Gateway VIP and cloned over the public LE cert with no CA
+        # supplied. The workstation being able to reach it proves nothing about pod egress.
+        # MOVED WITH THE APEX FLIP (2026-09-13) — and it had to move in the SAME fetch.
+        # This URL is a hardcoded literal, NOT templated on ${BASE_DOMAIN}, so the flip did not
+        # carry it. But the flip DOES rename prod's own HTTPRoute hostnames, so:
+        #   - external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev, and
+        #   - platform/coredns/coredns-custom.yaml answers the whole ${BASE_DOMAIN} zone with
+        #     ${GATEWAY_VIP}, so in-cluster resolution moves from prod.home. to home.
+        # Leaving this on the old name would therefore have severed prod from its OWN GitOps
+        # source moments after the flip reconciled — self-reference, the same class of trap as
+        # patching this url live and having the root Kustomization revert it. The fix has to land
+        # THROUGH THE OLD SOURCE, which is why it is in the same push as the flip rather than
+        # after it.
+        # FOLLOW-UP worth doing: point this at the in-cluster Service
+        # (gitea-http.gitea.svc.cluster.local) so prod's GitOps stops depending on public DNS and
+        # on its own Gateway at all. Deferred: it needs the TLS/credential path rethought.
+        # IN-CLUSTER SERVICE, NOT A PUBLIC HOSTNAME (2026-09-13). This is the durable fix for a
+        # deadlock the apex flip walked straight into, and it is worth stating precisely.
+        #
+        # The url used to be a public name. The flip renamed prod's HTTPRoute hostnames, so the
+        # name this GitRepository pointed at stopped serving — but the HTTPRoute that WOULD have
+        # served the new name lives in a DOWNSTREAM tier, which cannot reconcile while the source
+        # is broken. Source down -> routes never rename -> source stays down. Observed live:
+        # `srcReady=False`, "unable to list remote for https://gitea.home.keithjackson.dev".
+        #
+        # Pointing at the Service breaks the cycle permanently: prod's GitOps no longer depends on
+        # public DNS, on Cloudflare, on its own Gateway, or on ${BASE_DOMAIN} at all. A future
+        # domain change cannot sever prod from its own source again.
+        #
+        # Plain http is deliberate and safe here: it is cluster-internal traffic to a ClusterIP,
+        # TLS terminates at the Gateway for external clients only, the GitRepository has no
+        # secretRef (the repo is cloned anonymously), and every secret in the tree is SOPS-encrypted
+        # at rest regardless of transport.
+        url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
+        ref:
+          branch: main
   - target:
       kind: ConfigMap
       name: cluster-vars
@@ -21,9 +265,97 @@ patches:
       - op: replace
         path: /data/ACME_CLUSTER_ISSUER
         value: "letsencrypt-prod"
+      # ==========================================================================================
+      # PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
+      # ==========================================================================================
+      # PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
+      # home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
+      # public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
+      # of which already happened:
+      #   1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
+      #      default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
+      #   2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
+      #      wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
+      #      have raced each other's `_acme-challenge` TXT records on that one zone.
+      # Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
+      # record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
+      #
+      #   dev   -> home.keithjackson.dev         (UNCHANGED — dev is live; phase 2 moves it)
+      #   prod  -> prod.home.keithjackson.dev    (this knob)
+      #
+      # CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
+      # gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
+      # on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
+      # `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
+      # would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
+      #
+      # [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
+      # The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
+      # set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
+      # trusts a different Keycloak than the one prod actually runs.
+      - op: replace
+        path: /data/BASE_DOMAIN
+        # APEX AS OF 2026-09-13. Prod now serves home.keithjackson.dev directly; dev has been
+        # decommissioned and released the name. Prerequisites completed and VERIFIED before this
+        # flip, in this order:
+        #   1. dev's 18 A records released by its own external-dns (policy=sync, sources deleted)
+        #   2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand —
+        #      external-dns v0.21.0 removed only the `a-` prefixed form and then reported "All
+        #      records are already up to date", so it would NEVER have cleared them. They carried
+        #      `owner=default`, and prod runs `owner=prod`, so prod could not have claimed these
+        #      names while they existed. Confirmed gone at the AUTHORITY (betty.ns.cloudflare.com);
+        #      1.1.1.1 served stale cache for minutes afterwards and must not be trusted here.
+        #   3. dev's `*.home.keithjackson.dev` + apex Certificate deleted, so it stops renewing.
+        #      It was issued ~18d ago, so prod's request is #1 this week against LE's duplicate
+        #      limit of 5 per identical name set per week.
+        # NOTE the zone has wildcards `*.keithjackson.dev` AND `*.home.keithjackson.dev` pointing
+        # at the public IP 136.51.115.252 (legacy non-k3s prod). Dev's specific records were
+        # SHADOWING that wildcard; prod's new specific records shadow it again. The legacy stack
+        # lives on `*.keithjackson.dev` and is untouched by any of this.
+        value: "home.keithjackson.dev"
+      # SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
+      # OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
+      # TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
+      # ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
+      # the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
+      # matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
+      # records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
+      # EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
+      - op: replace
+        path: /data/EXTERNALDNS_DOMAIN_FILTER
+        # MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
+        # 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
+        # record then skipped it with "no hosted zone matching record DNS Name", so prod would
+        # publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
+        # (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
+        # (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
+        # That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
+        value: "keithjackson.dev"
+      # SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
+      # both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
+      # its own and, under policy: sync, deleted the records behind them. This value MUST differ from
+      # dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
+      # cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
+      - op: replace
+        path: /data/EXTERNALDNS_TXT_OWNER_ID
+        value: "prod"
       - op: replace
         path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
         value: "3"
+      # Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
+      # the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
+      # and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
+      # (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
+      # measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
+      # its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
+      #
+      # [parity] MUST equal node_baseline_longhorn_data_path in
+      # infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
+      # directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
+      # path on the OS filesystem instead of the data partition and /var fills up.
+      - op: replace
+        path: /data/LONGHORN_DATA_PATH
+        value: "/data/longhorn"
       # longer retention / larger volumes on prod NVMe.
       - op: replace
         path: /data/MONITORING_PROMETHEUS_RETENTION
@@ -34,8 +366,16 @@ patches:
       # Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
       # size cap before 30d if series volume is high, keeping the disk from filling either way.
       - op: replace
+      # LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for
+      # what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL,
+      # not the head's in-flight segments, and not compaction scratch, and a large compaction can need
+      # roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on
+      # /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every
+      # absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB
+      # at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history,
+      # grow the PVC rather than the fraction of it.
         path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
-        value: "45GiB"
+        value: "36GiB"
       - op: replace
         path: /data/MONITORING_LOKI_PVC_SIZE
         value: "50Gi"
@@ -89,14 +429,32 @@ patches:
       - op: replace
         path: /data/CILIUM_LB_POOL_STOP
         value: "192.168.2.30"
-      # --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
       # k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
-      # (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. It is owned
+      # (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
-      # Ansible-side (prod inventory k3s_api_host) and MUST equal that value — [parity] contract in
+      #
-      # platform/cilium/helmrelease.yaml. Not yet decided.
+      # *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
+      # read "Not yet decided" while shipping .10 — the founder's own address — and that is what
+      # the LIVE prod cluster is running: all three cilium-agents carry
+      # KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
+      # 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
+      # the datapath keeps running off Cilium's cache while no service or policy update lands, with
+      # nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
+      # control plane is, which is the whole reason kube-vip exists here.)
+      #
+      # It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
+      # this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
+      # equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
+      # that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
+      # `--server` join needs a host that is up before any VIP is floated. Two addresses, two
+      # variables, deliberately.
+      #
+      # .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
+      # LB-IPAM can never be handed the same IP — asserted by
+      # conformance/tests/l0-static/ha-control-plane-as-code.
       - op: replace
         path: /data/K3S_API_HOST
-        value: "192.168.2.10"
+        value: "192.168.2.9"
+      # --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
       # Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
       # prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
       # they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
@@ -107,10 +465,163 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.2.22"
-      # Off-cluster DR target. Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the
+      # PiHole's DNS listener VIP — udp/53 + tcp/53 for LAN clients (apps/wave-0/pihole-dns.yaml).
-      # network — a prod backup written there is not off-cluster DR and would mix two environments'
+      # 192.168.2.23 is the NEXT FREE address in the prod pool 192.168.2.20-.30, verified live
-      # PITR streams in one bucket. Operator supplies the prod endpoint (full URL incl. scheme+port).
+      # 2026-09-14: the CiliumLoadBalancerIPPool `default-pool` reported IPsTotal 11 / IPsUsed 3 /
+      # IPsAvailable 8, and the only three allocations are .20 (cilium-gateway-shared), .21
+      # (unifi-l4-tcp/-udp) and .22 (crafty-l4-tcp/-udp). ARP-probed from a LAN host in the same
+      # broadcast domain (arping 192.168.2.23, 3 probes, 0 responses), so nothing off-cluster holds
+      # it either, and it is inside the pool block the operator already asserted sits outside the
+      # SOHO router's DHCP scope.
+      - op: replace
+        path: /data/PIHOLE_DNS_VIP
+        value: "192.168.2.23"
+      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
+      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
+      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
+      - op: replace
+        path: /data/LAN_CIDR
+        value: "192.168.2.0/24"
+      # ------------------------------------------------------------------------------------------
+      # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
+      # ------------------------------------------------------------------------------------------
+      # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
+      # radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
+      #
+      # THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
+      # vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
+      # 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
+      # 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
+      # outcomes give DIFFERENT values here:
+      #     port-forward 192.168.2.102:2049 -> 192.168.1.9:2049   =>  "192.168.2.102"
+      #     a WAN->LAN route to 192.168.1.0/24                    =>  "192.168.1.9"
+      # Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
+      # ever became reachable — mounts something unintended. So it stays a placeholder.
+      #
+      # WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
+      # NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
+      # over v3 was in part this decision's cost: v3 would have needed four ports and three of them
+      # on ephemeral numbers.
+      #
+      # WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
+      # Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
+      # the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
+      # address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
+      # 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
+      # addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
+      #
+      # *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
+      # A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
+      # placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
+      # failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
+      # TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
+      # ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
+      # apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
+      # a revision containing the bulk volumes; the dev overlay renders a working value, which is
+      # where these manifests are meant to be proven first.
+      - op: replace
+        path: /data/BULK_NFS_SERVER
+        # 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
+        # host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
+        # reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
+        # keeps the base value 192.168.1.9.
+        #
+        # PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
+        #   - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
+        #   - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
+        #     carries no_root_squash, so a root write would have passed and proved nothing, while
+        #     LSIO images default to PUID 911 and would have gone Ready then failed every import
+        #     with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
+        #   - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
+        #     real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
+        # pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
+        # so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
+        value: "192.168.2.102"
+      # ==========================================================================================
+      # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
+      # ==========================================================================================
+      # Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the network and is NOT an
+      # option for prod, for two independent reasons:
+      #   1. UNREACHABLE. Prod is on the SOHO network 192.168.2.0/24 — pfSense's WAN — while that
+      #      MinIO is inside pfSense's LAN. Measured from a pod INSIDE prod (the workstation proves
+      #      nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` → connection
+      #      timed out after 15s. Reaching it would need a WAN→LAN rule that does not exist.
+      #   2. NOT OFF-CLUSTER DR. It runs on the same Proxmox host as dev's VMs, so it already shares
+      #      a physical failure domain with dev. R2 is genuinely off-site, and its free egress means
+      #      restore REHEARSALS cost nothing — R-STATE-03 has no financial disincentive to stay proven.
+      #
+      # THIS UNBLOCKS AN ACTIVE INCIDENT, not just a roadmap item. All three prod CNPG Clusters ship
+      # `archive_mode=on` with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, so barman-cloud-wal-archive
+      # exits 4 on every segment and Postgres — correctly — REFUSES TO RECYCLE WAL. Verified live
+      # 2026-09-07: keycloak-db / nextcloud-db / semaphore-db all report
+      # ContinuousArchiving=False/ContinuousArchivingFailing, and keycloak-db's 5 GiB PVC is at
+      # 579 MiB used / 4360 MiB free and climbing monotonically. A disk-full PANIC there takes the
+      # cluster's SSO database down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and
+      # forward-auth. Fixing the endpoint lets Postgres drain the whole backlog.
+      #
+      # ENDPOINT IS ACCOUNT-SCOPED, so it stays a prod-overlay value even though it is not a LAN
+      # address: this is the operator's Cloudflare account id, confirmed from inside a prod pod
+      # (`head-bucket` on k3s-prod-backup → BucketRegion WNAM; PUT and DELETE both succeed).
+      # It is NOT put in base, because base is dev's live MinIO and dev is deliberately STAYING
+      # there — moving dev to R2 would reset its PITR window and orphan the backups its PROVEN
+      # restore depends on. Two environments, two targets, isolated by URL *and* by bucket.
       - op: replace
         path: /data/S3_BACKUP_ENDPOINT
-        value: "TBD-PROD-S3-BACKUP-ENDPOINT"
+        value: "https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com"
+      # R2 requires the region string to be exactly `auto`. (`head-bucket` answers
+      # `{"BucketRegion":"WNAM"}` — that is a PHYSICAL LOCATION HINT, not a signing region; using it
+      # here produces AuthorizationHeaderMalformed.) Dev keeps the base `us-east-1` its live MinIO
+      # has always been signed with.
+      - op: replace
+        path: /data/S3_BACKUP_REGION
+        value: "auto"
+      # ------------------------------------------------------------------------------------------
+      # OBJECT TENANCY INSIDE THE ONE BUCKET. The operator created a SINGLE R2 bucket, so velero and
+      # barman cannot be separated by bucket the way they are on dev's MinIO (`velero` and `cnpg`).
+      # They are separated by disjoint TOP-LEVEL PREFIXES instead:
+      #
+      #     s3://k3s-prod-backup/velero/…                 Velero: kopia repo + backup metadata
+      #     s3://k3s-prod-backup/barman/<clusterName>/…   CNPG: WAL + base backups
+      #
+      # The bucket ROOT is left empty on purpose — a misrendered prefix then lands somewhere
+      # obviously wrong instead of silently colliding with a live tenant. Velero's retention TTL
+      # deletes objects under its own prefix only, so it can never reap a WAL segment; and barman's
+      # per-cluster serverName subdirectory keeps the three Postgres servers' chains apart, which
+      # matters because interleaving two servers' WAL into one prefix corrupts the chain outright.
+      # Dev cannot be reached from any of these: different endpoint AND different bucket names.
+      # ------------------------------------------------------------------------------------------
+      - op: replace
+        path: /data/VELERO_BACKUP_BUCKET
+        value: "k3s-prod-backup"
+      - op: replace
+        path: /data/VELERO_BACKUP_PREFIX
+        value: "velero"
+      - op: replace
+        path: /data/CNPG_BACKUP_PATH
+        value: "s3://k3s-prod-backup/barman"
+      # ------------------------------------------------------------------------------------------
+      # WHICH CREDENTIAL SIGNS THE REQUESTS TO THAT BUCKET (per-env credential split, 2026-09-07).
+      # ------------------------------------------------------------------------------------------
+      # The knobs above moved the ENDPOINT and the OBJECT LAYOUT per environment. The credential did
+      # not move with them, and it could not: the four backup-credential Secrets are SHARED BASE
+      # manifests, applied to both clusters by the same Flux tiers. Writing the R2 token into them
+      # gave prod a working key by TAKING DEV'S AWAY — dev would have signed MinIO requests with an
+      # R2 key id, and under archive_mode=on that is a filling pgdata PVC, not a missing backup.
+      #
+      # So each credential is now two objects, and this suffix selects prod's:
+      #     velero-cloud-credentials-prod           the R2 token in the AWS INI `cloud` shape
+      #     <db>-backup-credentials-prod            the same token as ACCESS_KEY_ID / ACCESS_SECRET_KEY
+      # for keycloak-db, nextcloud-db and semaphore-db. Both halves ship to both clusters (they are
+      # base resources); ONLY this value decides which one each consumer reads. It is a SUFFIX rather
+      # than four secret-name knobs deliberately — four could be updated three-quarters of the way,
+      # leaving one consumer on dev's MinIO key with nothing rendering wrong, and strict-mode
+      # substitution cannot catch a defined-but-stale variable.
+      #
+      # THE R2 TOKEN IS THE ONE PROVEN FROM INSIDE PROD (chore/r2-credentials 0a6a862): head-bucket on
+      # k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Rotating it now
+      # touches ONLY the four `-prod` files; dev's MinIO keys are untouched objects. That isolation is
+      # the structural reason for the split, independent of dev's eventual decommission.
+      - op: replace
+        path: /data/BACKUP_CREDENTIALS_SUFFIX
+        value: "-prod"
       # TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.

clusters/prod/sync.yaml

@@ -0,0 +1,32 @@
+---
+# Self-managing ROOT Kustomization for PROD (mirrors clusters/dev/sync.yaml).
+#
+# Reconciles ./clusters/prod from the GitRepository so the ENTIRE prod graph — the cluster-vars
+# ConfigMap, the GitRepository source, platform / platform-config / the app waves, and THIS object
+# itself — is applied from git on every push. dev/sync.yaml carried the note "a prod overlay needs
+# its own sync.yaml -> ./clusters/prod"; this is it.
+#
+# WHY IT IS NOW LOAD-BEARING, not just symmetry: `clusters/base/platform.yaml` gained
+# `dependsOn: [flux-system]` (2026-09-01) to close the post-build substitution race — a dependent
+# cannot reconcile revision N until the ROOT has APPLIED revision N. Without this object there is no
+# `flux-system` Kustomization on prod, so `platform` would block forever on a dependency that does
+# not exist, and NOTHING downstream would ever reconcile.
+#
+# `wait: false` is deliberate and is what keeps the dependsOn edge deadlock-free: the root reaches
+# Ready on apply success without health-gating its children (platform does its own wait/health
+# gating). Do not "harden" this to true.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: flux-system
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./clusters/prod

conformance/.gitignore

@@ -8,3 +8,22 @@ _artifacts/
 
 # Python bytecode from lib/ (status.py imports coverage.py); never committed.
 __pycache__/
+
+# --- THE ONE EXCEPTION: the committed run ledger (runs/) -------------------------------
+# `runs/<UTC>-<short-sha>/{report.junit.xml,run.log}` + `runs/ledger.jsonl` are the
+# machine-readable proof-of-pass record that lib/status.py's docstring asks for. They are
+# SOURCE, not regenerated artifacts: a run's evidence cannot be reconstructed after the
+# fact, so it must survive in git.
+#
+# The negations are needed because the ROOT .gitignore ignores `*.junit.xml` globally
+# (and `coverage/`). A deeper .gitignore overrides a shallower one, and nothing ignores
+# the `conformance/runs/` directory itself, so re-including the files here works (git
+# cannot re-include a file whose PARENT DIRECTORY is excluded — that is why this must not
+# live under _artifacts/).
+#
+# The run LOG is the load-bearing half: Chainsaw's JUnit carries no stdout, and the L2/L3
+# tests self-skip GREEN, so the sentinel token in run.log is the only thing separating
+# "proven" from "skipped". Never commit a report.junit.xml without its run.log.
+!runs/
+!runs/**
+!runs/**/*.junit.xml

conformance/README.md

@@ -22,6 +22,10 @@ behavioral / L3 fault-injection** (this harness).
 ./run.sh --lint                       # structural-validate test defs (no cluster)
 ./run.sh --strict-coverage            # hygiene gate: fail on tag/annotation drift
 ./run.sh --require-all                # phase-exit gate: every requirement must be covered
+./run.sh --record                     # run AND capture the evidence (see "Proof of pass")
+./run.sh --layer l0 --include-scope whole-repo   # union: the tagged subset PLUS every test
+                                      #   declaring `conformance.home.k3s/scope: whole-repo`
+./run.sh --layer l0 --print-selection # print the dirs the selection resolves to; run nothing
 ```
 
 **Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
@@ -31,6 +35,29 @@ behavioral / L3 fault-injection** (this harness).
 - `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
   those touched requirements are covered (a listed safety-critical one must have an L3 test).
 - `--require-all` — **phase-exit gate**: every registry requirement must be covered.
+- `--include-scope whole-repo` — **the escape hatch scoping needs**, because scoping has one
+  blind spot and it is the worst-shaped kind. Some L0 invariants quantify over the WHOLE
+  tracked tree (no plaintext secrets anywhere; no RBAC wildcard anywhere), or over subtrees
+  the coarse path→requirement map keys to **nothing** (`apps/`, `infra/`, most of
+  `platform/`). For those, *the change most likely to break the invariant attributes no
+  requirement at all*, so requirement-scoped selection filters the check out of exactly that
+  change. Such a test **declares** its scope:
+
+  ```yaml
+  metadata:
+    annotations:
+      conformance.home.k3s/scope: "whole-repo"
+  ```
+
+  and `--include-scope whole-repo` unions every declaring test into the selection. The CI
+  stages (`ci/l0.sh`, `ci/l1.sh`, `ci/conformance.sh`) all pass it, so **adding the
+  annotation is the only step** — there is nothing to remember and nothing to hand-write.
+  Two consequences, both deliberate: a declaring test always runs, and a **SKIP of a
+  declaring test is UNWIRED, not green** (the runner exits 3, which `ci/l0.sh` turns into the
+  gate's own rc 3 — never a pass). `tests/l0-static/whole-repo-invariants-wired` proves the
+  mechanism end-to-end and falsifiably, so it cannot rot silently. This replaced a
+  hand-written "UNCONDITIONAL" stage per check in `ci/l0.sh`; see `lib/common.sh` for the
+  full rationale and the incidents behind it.
 - Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01`
   state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map)
   are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
@@ -105,13 +132,89 @@ lib/
   common.sh                  tagging convention + test discovery/selection helpers
   coverage.py                requirement-coverage generator (stdlib only)
   status.py                  requirement-STATUS generator (stdlib only; --selftest)
+  record_run.py              run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
   emit_skipped_junit.py      Phase-0 skipped-JUnit shim (no cluster)
+runs/
+  ledger.jsonl               TRACKED append-only proof-of-pass ledger (one row per
+                             testcase per run) — the only source status.py trusts
+  <UTC>-<short-sha>/         TRACKED per-run evidence: run.log + report.junit.xml
 tests/
   l1-rendering/  l2-behavioral/  l3-fault-injection/  meta/   (see tests/README.md)
 incluster/
   chainsaw-job.yaml  rbac.yaml   in-cluster L2/L3 Job + scoped test SA (ADR-0011)
 ```
 
+## Proof of pass — the run ledger (`runs/`)
+
+`--status` used to report **PROVEN: 0** for all 51 requirements, and said so honestly:
+proof-of-pass was not recoverable from repo state. `_artifacts/` is gitignored and the
+root `.gitignore` kills `*.junit.xml` globally, so no run result ever reached git; the
+only record of a green run was DECISIONS-LOG.md prose. `runs/` is the fix `lib/status.py`
+asked for — **a committed, machine-readable run ledger**.
+
+```bash
+./run.sh --record                     # runs normally, and captures the evidence
+# -> conformance/runs/<UTC>-<short-sha>/run.log            (tee'd, carries the sentinels)
+# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml   (that run's JUnit)
+# -> conformance/runs/ledger.jsonl                         (one appended row per testcase)
+python3 lib/record_run.py --run-dir runs/<id> [--dry-run]  # or record an existing run dir
+python3 lib/record_run.py --selftest                       # deterministic, no cluster
+```
+
+**Recording is explicit, never implicit.** Plain `./run.sh` writes no ledger row: a run
+that silently records itself is how a bad run becomes a permanent false proof. **Commit
+the run dir and the ledger** — an uncommitted ledger proves nothing.
+
+### Why a committed JUnit alone would be WORSE than nothing
+
+The L2/L3 tests are runnable and **self-skipping**: when a precondition is missing (no
+`longhorn` StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token
+and `exit 0`, and their `check:` accepts either outcome —
+
+```yaml
+(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
+ || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
+```
+
+So **a self-skip exits 0 and Chainsaw records `passed` exactly as a real run does**, and
+Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`.
+A committed JUnit on its own would therefore record `DR_RESTORE_SKIPPED_OK` as *proof of
+R-STATE-03*: fake green, and worse than no ledger because it would look authoritative.
+
+**The sentinel token in `run.log` is the only thing that separates proven from skipped.**
+`record_run.py` derives each test's tokens from its own `chainsaw-test.yaml` (every
+`contains($stdout, 'TOKEN')`; `*_SKIP*_OK` is a skip token, the last non-skip token is the
+verdict token), so the table below is never hand-maintained:
+
+| Test | pass token (proof) | skip token (never proof) |
+|---|---|---|
+| `l3/node-loss` | `NODE_LOSS_VOLUME_AVAILABLE_OK` | `NODE_LOSS_SKIPPED_OK` |
+| `l3/dr-restore` | `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK` (+ `OFF_CLUSTER_TARGET_OK`, `BACKUPS_ARE_SCHEDULED_OK`) | `DR_RESTORE_SKIPPED_OK`, `DR_SCHEDULE_SKIPPED_OK` |
+| `l3/data-survival` | `DATA_SURVIVED_OK` | `DATA_SURVIVAL_SKIPPED_OK` |
+| `l3/vm-data-survival` | `VM_DATA_SURVIVED_OK` | `VM_DATA_SURVIVAL_SKIPPED_OK` |
+| `l3/hands-off-update` | `HANDS_OFF_UPDATE_OK` | `HANDS_OFF_UPDATE_SKIPPED_OK` |
+| `l2/vm-lifecycle` | `VM_LIFECYCLE_OK` | `VM_LIFECYCLE_SKIPPED_OK` |
+| `l2/vm-default-deny` | `VM_DEFAULT_DENY_OK` | `VM_DEFAULT_DENY_SKIPPED_OK` |
+
+A row reaches `proof: "proven"` on **one** path: JUnit `passed`, **every** declared pass
+token present in that test's log lines, and **no** skip token anywhere in them. A skip
+token beats a pass token (worst wins). Anything else is `skipped`, `failed`,
+`junit-skipped`, `unknown-no-sentinel-declared` or `unknown-sentinel-absent` — all
+non-green. L0/L1 need no sentinel because `lib/run_l1_offline.py` already renders their
+skips as JUnit `<skipped/>`, never as a green pass.
+
+### Staleness — a proof is a claim about a tree, not a badge
+
+`--status` honours a `proven` row only while the run's `git_sha` is a **descendant of the
+last commit touching that requirement's implementing manifests or its verifying test**.
+Otherwise the row degrades to **`stale-proof`**, which is not green. A run captured from a
+dirty working tree is recorded as `<sha>-dirty`, which no ancestry check can satisfy — so
+it never counts. Re-run with `--record` to refresh.
+
+A requirement reads `proven` only when **every** verifying test does; one skipped, failed
+or unrecorded test drags the whole requirement down. With an empty ledger `--status`
+behaves exactly as before: every requirement `not-derivable`, headline **PROVEN 0**.
+
 ## How CI invokes it
 CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
 

conformance/incluster/rbac.yaml

@@ -57,6 +57,13 @@ rules:
     verbs: [create, delete, get, list, watch, patch]
   - apiGroups: [""]
     resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
+    # LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment-
+    # agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable,
+    # oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run
+    # time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via
+    # chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings.
+    # If this rule is ever narrowed to the test's own namespace, those bindings fail and the
+    # tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap.
     verbs: [create, delete, get, list, watch, patch]
   - apiGroups: [""]
     resources: [pods/exec]

conformance/lib/common.sh

@@ -47,6 +47,62 @@ conf_dirs_for_requirements() {
   done < <(conf_all_test_files) | sort -u
 }
 
+# --- DECLARED SCOPE: whole-repo invariants -------------------------------------------
+# A test may also declare the SCOPE of the invariant it asserts:
+#
+#     metadata:
+#       annotations:
+#         conformance.home.k3s/scope: "whole-repo"
+#
+# WHY THIS EXISTS — the requirement-scoped-selection blind spot (R-TEST-03)
+# ------------------------------------------------------------------------
+# The merge gate runs the conformance subset tagged with the requirements a change
+# TOUCHES (ci/touched-requirements.sh -> `--requirements`). That is right for an invariant
+# whose input surface is the subtree its requirements map to. It is WRONG — silently, and in
+# the worst direction — for an invariant that quantifies over the WHOLE tracked tree, or over
+# subtrees (apps/, platform/<most>, infra/) that the coarse path->requirement map keys to
+# NOTHING. For those, the change most likely to break the invariant attributes no requirement
+# at all, so the requirement-scoped selection filters the check out of EXACTLY that change.
+# Three real incidents came out of that blind spot; each is recorded in the header of the
+# test that now guards it (flux-postbuild-substitution, keycloak-client-secret-reconciler,
+# bulk-tier-distinct-from-replicated).
+#
+# The repo's previous answer was a hand-written "UNCONDITIONAL" stage per such test in
+# ci/l0.sh — which does not scale and fails silently in the worst way: whoever adds the NEXT
+# whole-repo invariant must REMEMBER to hand-write a stage, and if they forget, the check runs
+# only when some unrelated requirement happens to be touched. The scope annotation replaces
+# that with a DECLARATION the runner enforces: `run.sh --include-scope whole-repo` unions
+# every test carrying it into the selection, so adding the annotation is the only step.
+#
+# TWO things follow from the annotation, and both are deliberate:
+#   1. the test is ALWAYS selected (union with the requirement-scoped subset), and
+#   2. a SKIP of that test is UNWIRED, NOT GREEN (lib/run_l1_offline.py --unwired-scope).
+#      A whole-repo invariant that reports "nothing to audit" is reporting that the subtree
+#      it audits has VANISHED, which is not a pass. This is the rc-3 semantics the bespoke
+#      stages encoded ("check SKIPPED (clusters/ absent) — not a pass"), preserved verbatim.
+export CONF_SCOPE_ANNOTATION="${CONF_LABEL_PREFIX}/scope"
+
+# Directories of tests DECLARING the given scope (e.g. "whole-repo"). The annotation is
+# matched on a real YAML mapping line only: a `#` comment cannot start with the key, so a
+# scope named in a test's header prose is documentation, not a declaration (the same
+# "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs).
+conf_dirs_for_scope() {
+  local scope="$1" f
+  [ -n "$scope" ] || return 0
+  while IFS= read -r f; do
+    [ -n "$f" ] || continue
+    if grep -Eq "^[[:space:]]*${CONF_SCOPE_ANNOTATION}:[[:space:]]*[\"']?${scope}[\"']?[[:space:]]*$" "$f"; then
+      dirname "$f"
+    fi
+  done < <(conf_all_test_files) | sort -u
+}
+
+# Union of conf_dirs_for_scope over a comma/space separated list of scopes.
+conf_dirs_for_scopes() {
+  local scopes="$1" s
+  for s in ${scopes//,/ }; do conf_dirs_for_scope "$s"; done | sort -u
+}
+
 # Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
 # Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
 # their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);

conformance/lib/record_run.py

@@ -0,0 +1,988 @@
+#!/usr/bin/env python3
+"""Append a conformance RUN LEDGER row per testcase — the committed proof-of-pass record.
+
+WHY THIS EXISTS
+---------------
+`lib/status.py` reports PROOF as `not-derivable` for every requirement, and names its own
+fix in its docstring: *"a committed, machine-readable run ledger"*. There was none, because
+`conformance/_artifacts/` is git-ignored and the root `.gitignore` kills `*.junit.xml` and
+`coverage/` globally, so nothing about a green run ever reached repo state. This tool is
+that ledger's writer: it reads one run's JUnit + run log out of `conformance/runs/<run-id>/`
+and appends one JSON object per testcase to `conformance/runs/ledger.jsonl`, which IS
+tracked (see conformance/.gitignore for the negations that let it survive).
+
+THE TRAP THIS TOOL EXISTS TO AVOID — READ BEFORE CHANGING ANYTHING
+------------------------------------------------------------------
+The L2/L3 tests are RUNNABLE / SELF-SKIPPING: when a precondition is absent (no `longhorn`
+StorageClass, no Velero CRDs, <3 Ready nodes) they echo a skip token and `exit 0`, and their
+`check:` block is deliberately written to accept either outcome, e.g. tests/l3-fault-
+injection/dr-restore:
+
+    (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
+     || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
+
+So a SELF-SKIP exits 0 and Chainsaw records `passed` EXACTLY as a real run does. And
+Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`
+(verified against the committed reports). Therefore:
+
+  *** A COMMITTED JUnit ALONE WOULD RECORD `DR_RESTORE_SKIPPED_OK` AS PROOF OF R-STATE-03. ***
+
+AND THE SECOND TRAP, WHICH IS THE MIRROR OF THE FIRST
+------------------------------------------------------
+Chainsaw echoes each script's SOURCE into an indented `=== COMMAND` block before running
+it, so `echo NODE_LOSS_SKIPPED_OK` — the skip branch that was NOT taken — is present in
+the log of every run of that test. A section-blind grep therefore reads a skip token out
+of the source of every skip-gated test and can never record `proven` for any of them: the
+ledger fails closed so completely that it records nothing and proves nothing. Tokens are
+consequently read from SECTIONS (see classify_sections): a pass token only from
+`=== STDOUT`, a skip token from any section that is not script source. Skip still beats
+pass — within genuine output, which is the only place either was ever evidence.
+
+That is the fake-green this project refuses, and it would be worse than no ledger at all
+because it would look authoritative. THE SENTINEL TOKEN IS THE ONLY THING THAT SEPARATES
+PROVEN FROM SKIPPED, AND IT EXISTS ONLY IN THE RUN LOG. Hence: this tool requires BOTH
+files, and for the Chainsaw layers it will NEVER emit `proof: "proven"` on a JUnit `passed`
+unless the log carries that test's own pass token and carries NO skip token.
+
+HOW THE TOKENS ARE OBTAINED (derived, never hand-listed)
+--------------------------------------------------------
+Hard-coding a token table would rot the moment a test is edited. Instead every token is
+parsed out of the test's own `chainsaw-test.yaml`: each `contains($stdout, 'TOKEN')` in a
+`check:` block is a declared sentinel. Classification is by the project's naming convention
+(`*_SKIP*_OK` is a skip token, anything else is a pass token), and the TERMINAL pass token
+— the last non-skip token in file order — is the test's verdict token. Edit a test's tokens
+and this tool follows automatically; delete them and the test simply becomes unprovable
+rather than silently proven.
+
+  node-loss   pass NODE_LOSS_VOLUME_AVAILABLE_OK              skip NODE_LOSS_SKIPPED_OK
+  dr-restore  pass DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK     skip DR_RESTORE_SKIPPED_OK
+              (+ OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK / DR_SCHEDULE_SKIPPED_OK)
+
+WHY L0/L1 NEED NO SENTINEL (and why that is not a loophole)
+-----------------------------------------------------------
+The offline layers are not run by Chainsaw. `lib/run_l1_offline.py` shells out to each
+dir's `check.sh` / `render-check.sh` and maps exit 3 to a JUnit `<skipped/>` element and
+any other non-zero to `<failure>`. A self-skip there is ALREADY a JUnit skip, never a green
+pass, so the JUnit is a truthful record on its own. Only the Chainsaw layers (L2/L3/META)
+can express "skipped" as "passed", and only they require a sentinel — see
+SENTINEL_REQUIRED_LAYERS.
+
+LEDGER ROW (conformance/runs/ledger.jsonl — append-only, one object per testcase per run)
+  ts_utc            run-dir timestamp, else the JUnit's mtime      (when)
+  git_sha           `git rev-parse HEAD`, suffixed `-dirty` when tracked files were
+                    modified — a proof captured from a dirty tree is NOT honoured by
+                    status.py, because what ran is not what is committed  (what)
+  env               --env / CONF_ENV (cluster overlay the run targeted)
+  chainsaw_version  CHAINSAW_VERSION from versions.env (the pin IS the truth)
+  kube_version      `kubectl version` server gitVersion, else "unknown"
+  layer             L0|L1|L2|L3|META, from the test's `layer` annotation / JUnit property
+  test              testcase name (== the test dir basename, asserted by convention)
+  requirements[]    the `conformance.home.k3s/R-*` labels on the test — the SAME
+                    authoritative source coverage.py uses, not the JUnit classname
+  junit_result      passed | failed | error | skipped
+  sentinel          the DECIDING token grepped from run.log, or null
+  proof             proven | skipped | failed | junit-skipped
+                    | unknown-no-sentinel-declared | unknown-sentinel-absent
+  run_id            the runs/<dir> name (provenance: which log backs this row)
+  note              one line saying WHY this row landed on that proof state
+
+`proven` is emitted on exactly one path (see decide()); every other path is non-green.
+
+RECORDING IS EXPLICIT, NEVER IMPLICIT
+-------------------------------------
+Invoke this tool by hand, or via `run.sh --record`. A run that silently records itself is
+how a bad run becomes a permanent false proof, so `run.sh` without `--record` writes no
+ledger row at all.
+
+EXIT CODES (ci/lib.sh convention: 0 pass / 1 fail / 3 skipped-not-green)
+  0  rows appended.
+  1  a real failure: unparsable JUnit, unreadable test tree, unwritable ledger.
+  3  the inputs a ledger row needs are absent (no report.junit.xml, or no run.log).
+     NOTHING is recorded and this is NEVER green — a run we cannot evidence is not a run.
+
+Pure standard library, matching the harness's no-pip-dependency posture (coverage.py,
+status.py). PyYAML lives in catalog/render.py; the harness does not depend on it.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import time
+import xml.etree.ElementTree as ET
+from pathlib import Path
+
+_LIB = Path(__file__).resolve().parent
+if str(_LIB) not in sys.path:
+    sys.path.insert(0, str(_LIB))
+# Importing a sibling would otherwise litter conformance/lib/__pycache__/ into the working
+# tree, and the lint stage scans untracked files. Same guard as status.py.
+sys.dont_write_bytecode = True
+import coverage as cov  # noqa: E402  — reuse the registry/test parsers, don't fork them
+
+CONF_ROOT = _LIB.parent
+REPO_ROOT = CONF_ROOT.parent
+DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
+
+# The ONLY layers exempt from needing a pass sentinel: the offline ones, whose skips
+# lib/run_l1_offline.py already renders as JUnit <skipped/> rather than as a green pass
+# (see docstring). Written as an EXEMPT set, not a REQUIRED set, so that an unrecognised
+# or new layer fails CLOSED — it needs a sentinel — instead of quietly becoming provable
+# by JUnit alone.
+SENTINEL_EXEMPT_LAYERS = frozenset({"L0", "L1"})
+
+# Layer from the test's subtree. Authoritative over the annotation: coverage.py's layer
+# regex stops at the first non-[L/A/M/0-9] character, so `layer: "META"` parses as "M" —
+# harmless for its own report, but here a mis-read layer decides whether a sentinel is
+# required at all, so it is derived from the path exactly as emit_skipped_junit.py does.
+_LAYER_BY_SUBTREE = (("/l0-static/", "L0"), ("/l1-rendering/", "L1"),
+                     ("/l2-behavioral/", "L2"), ("/l3-fault-injection/", "L3"),
+                     ("/meta/", "META"))
+
+
+def layer_of(path: Path) -> str:
+    rel = str(path)
+    for frag, layer in _LAYER_BY_SUBTREE:
+        if frag in rel:
+            return layer
+    return ""
+
+# A declared sentinel: `contains($stdout, 'TOKEN')` inside a Chainsaw `check:` block.
+# Both quote styles are accepted; the repo uses single quotes throughout.
+_SENTINEL_RE = re.compile(r"contains\(\s*\$stdout\s*,\s*['\"]([A-Za-z0-9_]+)['\"]\s*\)")
+
+# Naming convention for a SKIP token (tests/l3-fault-injection/*: NODE_LOSS_SKIPPED_OK,
+# DR_RESTORE_SKIPPED_OK, DR_SCHEDULE_SKIPPED_OK, VM_DATA_SURVIVAL_SKIPPED_OK, ...).
+# Deliberately broad: mis-classifying a PASS token as a skip only ever loses a proof;
+# mis-classifying a SKIP token as a pass would manufacture one.
+_SKIP_TOKEN_RE = re.compile(r"_SKIP")
+
+# A Chainsaw log line header, as it really appears (Go `t.Log` prefixes the source site and
+# pads it with backspaces, and the fields are ANSI-coloured — both are stripped first):
+#     l.go:52: | 23:51:18 | node-loss | volume-stays-available... | SCRIPT | LOG |
+# Field 2 is the test name; it is how output is attributed back to a testcase. The
+# `<file>.go:<line>:` prefix is optional so a redirected/plain log still attributes.
+_LOG_HEADER_RE = re.compile(
+    r"^\s*(?:[\w./-]+\.go:\d+:\s*)?"
+    r"\|\s*\d{1,2}:\d{2}:\d{2}(?:\.\d+)?\s*\|\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*\|")
+
+# ANSI colour + the backspace padding Go writes ahead of a `t.Log` continuation. Stripped
+# before ANY parsing; neither ever appears at the head of a line, so indentation survives.
+_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
+
+
+def _strip_noise(line: str) -> str:
+    return _ANSI_RE.sub("", line).replace("\b", "")
+
+
+# ===================================================================== output sections
+# THE BUG THIS SECTION MACHINERY EXISTS TO KILL — READ BEFORE CHANGING ANYTHING
+# -----------------------------------------------------------------------------
+# Chainsaw echoes the script it is ABOUT TO RUN, verbatim, in an indented `=== COMMAND`
+# block, and only afterwards prints what the script actually wrote in `=== STDOUT`. Every
+# skip-gated test contains its own skip token as SOURCE TEXT (`echo NODE_LOSS_SKIPPED_OK`),
+# so a raw whole-log grep finds that token in EVERY run of that test — including runs that
+# never took the skip branch. Combined with decide()'s (correct) rule that a skip token
+# outranks a pass token, a section-blind scan can never record `proven` for ANY test that
+# has a skip gate: the ledger fails closed so hard it records nothing. Observed on
+# runs/20260901T235117Z-e9a4b66, where NODE_LOSS_SKIPPED_OK appears twice as source at
+# lines 51/62 (`=== COMMAND`) and never in output, yet the run genuinely proved R-STATE-02.
+#
+# So tokens are attributed to SECTIONS, never to raw lines:
+#     l.go:52: | 23:51:18 | node-loss | step | CMD    | RUN |     <- header, no section
+#         === COMMAND                                            <- marker (indent 8)
+#         /usr/bin/sh -c set -euo pipefail                       <- source, indent >= 8
+#           echo NODE_LOSS_SKIPPED_OK                            <- SOURCE, not output
+#     l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |    <- indent 4 => section ends
+#         === STDOUT                                             <- marker
+#         NODE_LOSS_VOLUME_AVAILABLE_OK                          <- the REAL emission
+#
+# A marker is an INDENTED `=== NAME` on a line of its own. Go's own test markers
+# (`=== RUN   chainsaw`, `=== PAUSE`, `=== CONT`) are at column 0 and carry a test name
+# after the verb, so they match neither the indent nor the end-of-line anchor; they are
+# additionally named in _GO_TEST_MARKERS so a future indented variant still cannot open a
+# section that would swallow the whole log.
+_SECTION_MARKER_RE = re.compile(r"^(\s+)=== ([A-Z][A-Z0-9_]*)\s*$")
+_GO_TEST_MARKERS = frozenset({"RUN", "PAUSE", "CONT"})
+
+# Pseudo-section for a log that contains NO section markers at all (the plain/redirected
+# logs the earlier ledger rows were written from, and the selftest fixtures). Such a log
+# never echoes script source, so there is nothing to be confused by and the legacy
+# whole-log scan is exactly as safe as it was — narrowing is only needed where a COMMAND
+# echo exists. Lines OUTSIDE a section in a log that DOES use sections are attributed to
+# "" and are evidence for nothing.
+UNSECTIONED = "*"
+
+# A pass token counts ONLY where Chainsaw's own `contains($stdout, ...)` check looks:
+# real stdout. Anything else — source echo, stderr, an unrecognised future section — can
+# never manufacture a proof.
+PASS_SECTIONS = frozenset({"STDOUT", UNSECTIONED})
+
+# Sections that carry SOURCE rather than output. A skip token is honoured in every OTHER
+# section (STDOUT, STDERR, any section name added by a later Chainsaw release), because
+# over-reading a skip only ever COSTS a proof, while over-reading a pass would fabricate
+# one. Unknown section names therefore fail closed in both directions.
+SOURCE_SECTIONS = frozenset({"COMMAND"})
+
+
+def classify_sections(log_text: str) -> list[tuple[str, str]]:
+    """[(section, line)] for every line of a run log.
+
+    `section` is the enclosing `=== NAME` block, "" for a line inside no block, or
+    UNSECTIONED for every line when the log declares no sections at all. A block runs from
+    its marker until the first non-blank line indented LESS than the marker (blank lines do
+    not end it) or until the next marker; the marker line itself is content of nothing.
+    """
+    rows: list[tuple[str, str]] = []
+    cur, cur_indent, sectioned = "", 0, False
+    for raw in log_text.splitlines():
+        line = _strip_noise(raw)
+        m = _SECTION_MARKER_RE.match(line)
+        if m and m.group(2) not in _GO_TEST_MARKERS:
+            sectioned = True
+            cur, cur_indent = m.group(2), len(m.group(1))
+            rows.append(("", line))
+            continue
+        if cur:
+            if not line.strip() or len(line) - len(line.lstrip()) >= cur_indent:
+                rows.append((cur, line))
+                continue
+            cur = ""
+        rows.append(("", line))
+    if not sectioned:
+        return [(UNSECTIONED, line) for _, line in rows]
+    return rows
+
+
+def scan_texts(lines) -> tuple[str, str]:
+    """(text a PASS token may be read from, text a SKIP token may be read from).
+
+    Accepts either [(section, line)] pairs from classify_sections() or a plain list of
+    strings, which is treated as an unsectioned log (legacy whole-log scanning).
+    """
+    pass_lines, skip_lines = [], []
+    for item in lines:
+        section, line = item if isinstance(item, tuple) else (UNSECTIONED, item)
+        if section in PASS_SECTIONS:
+            pass_lines.append(line)
+        if section and section not in SOURCE_SECTIONS:
+            skip_lines.append(line)
+    return "\n".join(pass_lines), "\n".join(skip_lines)
+
+_RUN_ID_TS_RE = re.compile(r"^(\d{8}T\d{6}Z)-")
+
+
+# ========================================================================= sentinels
+def parse_sentinels(test_yaml: Path) -> dict:
+    """Declared sentinel tokens of one Chainsaw test, derived from the test file itself.
+
+    Returns {"pass": [...], "skip": [...], "terminal_pass": str|None}. `terminal_pass` is
+    the LAST non-skip token in file order — the test's verdict token, i.e. the one its
+    final `check:` block asserts (dr-restore -> DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK).
+    A test with no `contains($stdout, ...)` assertion declares no sentinel and is reported
+    unprovable rather than proven; see decide().
+    """
+    try:
+        text = test_yaml.read_text()
+    except (OSError, UnicodeDecodeError):
+        return {"pass": [], "skip": [], "terminal_pass": None}
+    toks = _SENTINEL_RE.findall(text)
+    passes, skips = [], []
+    for t in toks:
+        (skips if _SKIP_TOKEN_RE.search(t) else passes).append(t)
+    # De-duplicate, preserving order (dr-restore names DR_RESTORE_SKIPPED_OK twice).
+    def uniq(xs: list[str]) -> list[str]:
+        return list(dict.fromkeys(xs))
+    passes, skips = uniq(passes), uniq(skips)
+    return {"pass": passes, "skip": skips,
+            "terminal_pass": passes[-1] if passes else None}
+
+
+def sentinel_map(tests_dir: Path) -> dict[str, dict]:
+    """{test-name: sentinels} for every test in the tree.
+
+    Keyed on the Chainsaw `metadata.name`, which is what lands in the JUnit `<testcase
+    name=...>`. This repo keeps that equal to the test's directory basename; the mapping
+    is by name, so it stays correct if that ever drifts.
+    """
+    out: dict[str, dict] = {}
+    for f in sorted(tests_dir.rglob("chainsaw-test.yaml")):
+        meta = cov.parse_test(f)
+        name = meta["name"] or f.parent.name
+        s = parse_sentinels(f)
+        s["dir"] = str(f.parent)
+        s["layer"] = layer_of(f) or meta["layer"]
+        s["requirements"] = meta["labels"]
+        out[name] = s
+    return out
+
+
+# =============================================================================== log
+def attribute_log(log_text: str, known: set[str]) -> tuple[dict[str, list[tuple[str, str]]],
+                                                            bool]:
+    """Split a run log into per-testcase buckets of (section, line) pairs.
+
+    Chainsaw prefixes its log lines with `| HH:MM:SS | <test> | <step> | ...` and prints a
+    script's sections underneath that header, so the test name is carried by a state
+    machine: a header naming a KNOWN test switches the current bucket, every following line
+    lands in it. Requiring the name to be known is what stops run.sh's own `[run.sh]`
+    chatter (and any other pipe-shaped line) from opening a bogus bucket.
+
+    Section labels come from classify_sections() and travel WITH the lines, so a bucket
+    still knows which of its lines were script source and which were real output.
+
+    Returns (buckets, attributed) where `attributed` is False when the log contained no
+    recognisable header at all — the signal for the degraded global scan in decide().
+    """
+    buckets: dict[str, list[tuple[str, str]]] = {}
+    cur: str | None = None
+    attributed = False
+    for section, line in classify_sections(log_text):
+        m = _LOG_HEADER_RE.match(line)
+        if m and m.group(1) in known:
+            cur = m.group(1)
+            attributed = True
+        if cur is not None:
+            buckets.setdefault(cur, []).append((section, line))
+    return buckets, attributed
+
+
+def unique_tokens(smap: dict[str, dict]) -> set[str]:
+    """Tokens claimed by EXACTLY ONE test in the tree.
+
+    Only these may be used by the degraded global scan: `DNS_OK` is echoed by two different
+    tests, so in an unattributable log it can never be evidence for either; but
+    `NODE_LOSS_VOLUME_AVAILABLE_OK` names precisely one test and still can.
+    """
+    owners: dict[str, set[str]] = {}
+    for name, s in smap.items():
+        for t in s["pass"] + s["skip"]:
+            owners.setdefault(t, set()).add(name)
+    return {t for t, o in owners.items() if len(o) == 1}
+
+
+# ============================================================================ decide
+def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
+           scope: str) -> tuple[str | None, str, str]:
+    """The ONE place a testcase becomes `proven` — or, far more often, does not.
+
+    Returns (sentinel, proof, note).
+
+    Ordering is worst-first ON PURPOSE, so that ambiguity can only ever cost a proof:
+      1. JUnit failure/error            -> failed        (a red run is recorded as red)
+      2. JUnit <skipped/>               -> junit-skipped  (never green; ci/lib.sh rule)
+      3. ANY declared skip token seen   -> skipped        (BEATS a green JUnit and beats
+           IN REAL OUTPUT                                  the pass token: if any gated
+                                                           step self-skipped, the run did
+                                                           not prove the requirement)
+      4. no sentinel declared           -> proven only for L0/L1, whose skips are already
+                                           JUnit skips; unknown-no-sentinel-declared for
+                                           the Chainsaw layers
+      5. EVERY declared pass token seen -> proven
+      6. anything else                  -> unknown-sentinel-absent
+    Step 5 requires ALL pass tokens, not just the terminal one: every Chainsaw step here
+    asserts `pass || skip`, so a missing pass token means a step did not reach its proven
+    branch, whatever the JUnit says.
+
+    Steps 3 and 5 read from SECTION-SCOPED text, never from raw lines: a pass token counts
+    only in `=== STDOUT`, a skip token counts in any section that is not script source.
+    That narrows WHERE a token is read from and changes nothing about what it means — skip
+    still outranks pass, and it outranks it within genuine output, which is the only place
+    either of them was ever evidence. `lines` may be [(section, line)] pairs or plain
+    strings (an unsectioned log; see scan_texts / UNSECTIONED).
+    """
+    if junit_result in ("failed", "error"):
+        return (None, "failed", f"JUnit recorded {junit_result}")
+    if junit_result == "skipped":
+        return (None, "junit-skipped",
+                "JUnit <skipped/> — the test did not execute; a skip is never green")
+
+    pass_text, skip_text = scan_texts(lines)
+    seen_skip = [t for t in sent["skip"] if t in skip_text]
+    if seen_skip:
+        return (seen_skip[0], "skipped",
+                f"run log OUTPUT carries the SELF-SKIP token {seen_skip[0]!r} — the test "
+                f"exited 0 without exercising the behaviour, so this is NOT proof")
+
+    if not sent["pass"] and not sent["skip"]:
+        if layer not in SENTINEL_EXEMPT_LAYERS:
+            return (None, "unknown-no-sentinel-declared",
+                    f"{layer or 'unknown-layer'} test declares no contains($stdout, ...) "
+                    f"sentinel, so a green JUnit cannot be told apart from a self-skip — "
+                    f"refusing to call it proven")
+        return (None, "proven",
+                f"{layer} offline check exited 0; run_l1_offline.py renders its skips as "
+                f"JUnit <skipped/>, so a green JUnit is truthful without a sentinel")
+
+    missing = [t for t in sent["pass"] if t not in pass_text]
+    if not missing:
+        return (sent["terminal_pass"], "proven",
+                f"run log OUTPUT carries every declared pass token "
+                f"({', '.join(sent['pass'])}) and no skip token [{scope}]")
+    return (None, "unknown-sentinel-absent",
+            f"run log OUTPUT lacks pass token(s) {', '.join(missing)} for this test "
+            f"[{scope}] — refusing to infer proof from the JUnit alone")
+
+
+# ============================================================================= junit
+def junit_cases(path: Path) -> list[dict]:
+    """[{name, layer, requirements, result}] for every <testcase> in a JUnit report."""
+    root = ET.parse(path).getroot()
+    cases: list[dict] = []
+    for c in root.iter("testcase"):
+        if c.find("failure") is not None:
+            result = "failed"
+        elif c.find("error") is not None:
+            result = "error"
+        elif c.find("skipped") is not None:
+            result = "skipped"
+        else:
+            result = "passed"
+        props = {p.get("name"): (p.get("value") or "")
+                 for p in c.iter("property") if p.get("name")}
+        reqs = sorted(set(cov.RID_RE.findall(props.get("requirements", ""))))
+        layer = (props.get("layer") or "").upper()
+        if not layer:
+            layer = (c.get("classname") or "").split(":", 1)[0].upper()
+        cases.append({"name": c.get("name") or "?", "layer": layer,
+                      "requirements": reqs, "result": result})
+    return cases
+
+
+# =============================================================================== git
+def git_sha(repo: Path) -> str:
+    """HEAD, suffixed `-dirty` when TRACKED files differ from it.
+
+    Untracked files are deliberately ignored, matching status.py's `tracked_files()`
+    posture (untracked scratch is not repo state) — and because a `--record` run creates
+    untracked `runs/<id>/` files as it goes, which would otherwise mark every run dirty.
+    A `-dirty` sha is not a valid rev, so status.py's ancestry check cannot resolve it and
+    the proof degrades to `stale-proof`: correct, since what ran is not what is committed.
+    """
+    try:
+        sha = subprocess.run(["git", "-C", str(repo), "rev-parse", "HEAD"],
+                             capture_output=True, text=True, check=True).stdout.strip()
+    except (subprocess.CalledProcessError, OSError):
+        return "unknown"
+    # EXCLUDE conformance/runs/. It is TRACKED, so `--record` writing ledger.jsonl dirtied the
+    # tree and every LATER run in the same session was stamped `-dirty` — recording one test
+    # poisoned the next, and check.sh refuses to honour a dirty proven row. That cost a second
+    # live fault-injection run on prod to work around. The docstring above already states the
+    # intent for the untracked runs/<id>/ half of this same artifact; the ledger is the tracked
+    # half. Mirrors status.py's IMPL_EXCLUDE_PREFIXES, which lists this path for the same reason.
+    dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--",
+                            ".", ":(exclude)conformance/runs/"],
+                           capture_output=True).returncode != 0
+    return f"{sha}-dirty" if dirty else sha
+
+
+def pinned_chainsaw_version(conf_root: Path) -> str:
+    """CHAINSAW_VERSION from versions.env — the pin is the single source of truth."""
+    f = conf_root / "versions.env"
+    if f.is_file():
+        m = re.search(r'^CHAINSAW_VERSION="?([^"\n]+)"?', f.read_text(), re.M)
+        if m:
+            return m.group(1)
+    return "unknown"
+
+
+def context_env() -> str:
+    """The env a run TARGETED, derived from the live kube context — never guessed.
+
+    WHY THIS IS NOT A DEFAULT STRING. `--env` used to fall back to the literal "dev"
+    (`os.environ.get("CONF_ENV", "dev")`), and CONF_ENV is set nowhere in this repo, so EVERY
+    recorded proof was stamped `env: dev` regardless of the cluster it ran against. That was
+    harmless while dev was the only cluster. Dev was decommissioned 2026-09-12 and prod is now
+    the only one, so on 2026-09-13 a genuine prod DR-restore proof was recorded as `env: dev` —
+    a proof that lies about where it was proven is worse than no proof, because the ledger's
+    whole purpose is to say what was demonstrated WHERE. Caught before it was committed.
+
+    Derived from `kubectl config current-context` (`k3s-prod` -> `prod`), which cannot drift
+    from the cluster the run actually talked to. Undeterminable is "unknown", NEVER a cluster
+    name: an unreadable context must not silently attribute a proof to an environment.
+    """
+    try:
+        p = subprocess.run(["kubectl", "config", "current-context"],
+                           capture_output=True, text=True, timeout=15)
+    except (OSError, subprocess.TimeoutExpired):
+        return "unknown"
+    if p.returncode != 0:
+        return "unknown"
+    ctx = (p.stdout or "").strip()
+    if not ctx:
+        return "unknown"
+    # contexts in this repo are named k3s-<env>; anything else is recorded verbatim so an
+    # unexpected context is visible in the ledger rather than being coerced into a known name.
+    return ctx[len("k3s-"):] if ctx.startswith("k3s-") else ctx
+
+
+def server_kube_version() -> str:
+    """Server gitVersion via `kubectl version` (read-only), else "unknown"."""
+    for args in (["kubectl", "version", "-o", "json"],
+                 ["kubectl", "version", "--output=json"]):
+        try:
+            p = subprocess.run(args, capture_output=True, text=True, timeout=15)
+        except (OSError, subprocess.TimeoutExpired):
+            continue
+        if p.returncode != 0 or not p.stdout.strip():
+            continue
+        try:
+            v = json.loads(p.stdout).get("serverVersion") or {}
+        except json.JSONDecodeError:
+            continue
+        if v.get("gitVersion"):
+            return v["gitVersion"]
+    return "unknown"
+
+
+def run_timestamp(run_dir: Path, junit: Path) -> str:
+    """UTC stamp: the run-dir name's own `<UTC>-<sha>` prefix, else the JUnit's mtime."""
+    m = _RUN_ID_TS_RE.match(run_dir.name)
+    if m:
+        t = m.group(1)
+        return f"{t[0:4]}-{t[4:6]}-{t[6:8]}T{t[9:11]}:{t[11:13]}:{t[13:15]}Z"
+    return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(junit.stat().st_mtime))
+
+
+# ============================================================================ record
+def build_rows(run_dir: Path, tests_dir: Path, env: str, chainsaw_version: str,
+               kube_version: str, sha: str) -> list[dict]:
+    junit = run_dir / "report.junit.xml"
+    log = run_dir / "run.log"
+    smap = sentinel_map(tests_dir)
+    log_text = log.read_text(errors="replace")
+    buckets, attributed = attribute_log(log_text, set(smap))
+    uniq = unique_tokens(smap)
+    all_pairs = classify_sections(log_text)
+    sectioned = any(sec not in ("", UNSECTIONED) for sec, _ in all_pairs)
+    section_note = ("tokens read from real-output sections only"
+                    if sectioned else "log declares no === sections; whole-log scan")
+    ts = run_timestamp(run_dir, junit)
+
+    rows: list[dict] = []
+    for case in junit_cases(junit):
+        name = case["name"]
+        sent = smap.get(name, {"pass": [], "skip": [], "terminal_pass": None,
+                               "layer": case["layer"], "requirements": []})
+        layer = sent.get("layer") or case["layer"] or "?"
+        # Requirements come from the test's LABELS (coverage.py's authoritative source);
+        # the JUnit property is the fallback for a testcase with no matching test file.
+        reqs = sent.get("requirements") or case["requirements"]
+
+        if name in buckets:
+            lines, scope = buckets[name], f"log scoped to this testcase; {section_note}"
+        elif not attributed:
+            # Degraded mode: the log had no recognisable Chainsaw header (a redirected or
+            # reformatted log). Fall back to a whole-log scan, but ONLY for tokens owned by
+            # exactly one test, so a token another test emitted can never prove this one.
+            lines = all_pairs
+            sent = dict(sent,
+                        **{"pass": [t for t in sent["pass"] if t in uniq],
+                           "skip": [t for t in sent["skip"] if t in uniq]})
+            sent["terminal_pass"] = sent["pass"][-1] if sent["pass"] else None
+            scope = ("DEGRADED: unattributable log, globally-unique tokens only; "
+                     + section_note)
+        else:
+            lines, scope = [], "no log lines attributed to this testcase"
+
+        sentinel, proof, note = decide(layer, case["result"], sent, lines, scope)
+        rows.append({
+            "ts_utc": ts,
+            "git_sha": sha,
+            "env": env,
+            "chainsaw_version": chainsaw_version,
+            "kube_version": kube_version,
+            "layer": layer,
+            "test": name,
+            "requirements": reqs,
+            "junit_result": case["result"],
+            "sentinel": sentinel,
+            "proof": proof,
+            "run_id": run_dir.name,
+            "note": note,
+        })
+    return rows
+
+
+def append_rows(ledger: Path, rows: list[dict]) -> None:
+    """Append-only, one compact JSON object per line. Never rewrites existing rows."""
+    ledger.parent.mkdir(parents=True, exist_ok=True)
+    with ledger.open("a", encoding="utf-8") as fh:
+        for r in rows:
+            fh.write(json.dumps(r, sort_keys=True) + "\n")
+
+
+# ========================================================================== selftest
+def _selftest() -> int:
+    """Deterministic, no-cluster self-test of every decision rule (house pattern:
+    `status.py --selftest`, `catalog/render.py --selftest`). Throwaway tree only.
+
+    The load-bearing case is `SKIP sentinel does NOT produce proven`: it is the single
+    assertion standing between this ledger and a permanent false proof of R-STATE-03.
+    """
+    import tempfile
+    ok = True
+
+    def check(label: str, cond: bool) -> None:
+        nonlocal ok
+        if not cond:
+            sys.stderr.write(f"record_run.py selftest: FAIL — {label}\n")
+            ok = False
+
+    # --- token derivation, against the dr-restore/node-loss shape ---------------------
+    with tempfile.TemporaryDirectory() as td:
+        root = Path(td)
+        tdir = root / "tests" / "l3-fault-injection" / "dr-restore"
+        tdir.mkdir(parents=True)
+        (tdir / "chainsaw-test.yaml").write_text(
+            "apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n"
+            "  name: dr-restore\n  labels:\n"
+            "    conformance.home.k3s/R-STATE-03: \"true\"\n  annotations:\n"
+            "    conformance.home.k3s/requirements: \"R-STATE-03\"\n"
+            "    conformance.home.k3s/layer: \"L3\"\nspec:\n  steps:\n"
+            "    - script:\n        content: 'echo OFF_CLUSTER_TARGET_OK'\n"
+            "        check:\n"
+            "          (contains($stdout, 'OFF_CLUSTER_TARGET_OK')"
+            " || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n"
+            "    - script:\n        content: 'echo x'\n        check:\n"
+            "          (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')"
+            " || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n")
+        s = parse_sentinels(tdir / "chainsaw-test.yaml")
+        check("pass tokens derived from the test file",
+              s["pass"] == ["OFF_CLUSTER_TARGET_OK",
+                            "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"])
+        check("skip token classified by the *_SKIP*_OK convention, de-duplicated",
+              s["skip"] == ["DR_RESTORE_SKIPPED_OK"])
+        check("terminal pass token is the LAST non-skip token",
+              s["terminal_pass"] == "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK")
+
+        smap = sentinel_map(root / "tests")
+        check("sentinel_map keys on metadata.name and carries layer + labels",
+              smap["dr-restore"]["layer"] == "L3"
+              and smap["dr-restore"]["requirements"] == ["R-STATE-03"])
+
+    # --- decide(): the SKIP-is-never-proven contract ----------------------------------
+    l3 = {"pass": ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"],
+          "skip": ["DR_RESTORE_SKIPPED_OK"],
+          "terminal_pass": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"}
+
+    # *** THE CASE THIS WHOLE FILE EXISTS FOR ***
+    sent, proof, _ = decide("L3", "passed", l3, ["DR_RESTORE_SKIPPED_OK"], "t")
+    check("SKIP sentinel on a GREEN JUnit is 'skipped', NEVER 'proven'",
+          proof == "skipped" and sent == "DR_RESTORE_SKIPPED_OK")
+
+    _, proof, _ = decide("L3", "passed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
+    check("pass sentinel on a green JUnit is 'proven'", proof == "proven")
+
+    _, proof, _ = decide("L3", "passed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK",
+                          "DR_RESTORE_SKIPPED_OK"], "t")
+    check("a skip token BEATS a pass token in the same log (worst wins)",
+          proof == "skipped")
+
+    _, proof, _ = decide("L3", "passed", l3, ["nothing relevant here"], "t")
+    check("green JUnit with no sentinel in the log is unknown, not proven",
+          proof == "unknown-sentinel-absent")
+
+    _, proof, _ = decide("L3", "passed", l3, [], "t")
+    check("green JUnit with NO log lines attributed is unknown, not proven",
+          proof == "unknown-sentinel-absent")
+
+    _, proof, _ = decide("L3", "failed", l3,
+                         ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
+    check("a failing JUnit outranks a pass token in the log", proof == "failed")
+
+    _, proof, _ = decide("L3", "skipped", l3, [], "t")
+    check("JUnit <skipped/> records junit-skipped, never green",
+          proof == "junit-skipped")
+
+    multi = {"pass": ["A_OK", "B_OK"], "skip": [], "terminal_pass": "B_OK"}
+    _, proof, _ = decide("L3", "passed", multi, ["B_OK"], "t")
+    check("every declared pass token must be present, not just the terminal one",
+          proof == "unknown-sentinel-absent")
+    _, proof, _ = decide("L3", "passed", multi, ["A_OK", "B_OK"], "t")
+    check("all pass tokens present == proven", proof == "proven")
+
+    none = {"pass": [], "skip": [], "terminal_pass": None}
+    _, proof, _ = decide("L2", "passed", none, ["whatever"], "t")
+    check("a Chainsaw test declaring no sentinel is NOT proven by a green JUnit",
+          proof == "unknown-no-sentinel-declared")
+    _, proof, _ = decide("L1", "passed", none, [], "t")
+    check("L0/L1 offline checks are proven by JUnit (their skips are JUnit skips)",
+          proof == "proven")
+    _, proof, _ = decide("L0", "skipped", none, [], "t")
+    check("an L0 JUnit skip is still never green", proof == "junit-skipped")
+    _, proof, _ = decide("META", "passed", none, [], "t")
+    check("META (a Chainsaw layer) needs a sentinel like L2/L3",
+          proof == "unknown-no-sentinel-declared")
+    _, proof, _ = decide("", "passed", none, [], "t")
+    check("an unknown/blank layer fails CLOSED — sentinel required",
+          proof == "unknown-no-sentinel-declared")
+
+    # --- SECTION SCOPING: the COMMAND echo is SOURCE, never evidence -------------------
+    # Regression cases for the ledger's first real L3 run, which recorded a genuine PASS as
+    # `skipped` because `echo NODE_LOSS_SKIPPED_OK` — the skip branch NOT taken — is echoed
+    # inside `=== COMMAND` on every run of a skip-gated test.
+    nl = {"pass": ["NODE_LOSS_VOLUME_AVAILABLE_OK"],
+          "skip": ["NODE_LOSS_SKIPPED_OK"],
+          "terminal_pass": "NODE_LOSS_VOLUME_AVAILABLE_OK"}
+    _CMD = ("    l.go:52: | 23:51:18 | node-loss | step | CMD    | RUN |\n"
+            "        === COMMAND\n"
+            "        /usr/bin/sh -c set -euo pipefail\n"
+            "        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then\n"
+            "          echo NODE_LOSS_SKIPPED_OK\n"
+            "\n"
+            "          exit 0\n"
+            "        fi\n"
+            "        echo NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+            "    l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |\n")
+
+    ran = classify_sections(_CMD + "        === STDOUT\n"
+                            "        ready nodes: 3\n"
+                            "        NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+                            "    l.go:52: | 23:52:42 | node-loss | step | SCRIPT | DONE |\n")
+    sent, proof, _ = decide("L3", "passed", nl, ran, "t")
+    check("skip token ONLY in the COMMAND echo + pass token in STDOUT is PROVEN",
+          proof == "proven" and sent == "NODE_LOSS_VOLUME_AVAILABLE_OK")
+
+    selfskipped = classify_sections(_CMD + "        === STDOUT\n"
+                                    "        SKIP: 'longhorn' StorageClass absent\n"
+                                    "        NODE_LOSS_SKIPPED_OK\n")
+    sent, proof, _ = decide("L3", "passed", nl, selfskipped, "t")
+    check("skip token GENUINELY in STDOUT is still 'skipped' (the guarantee holds)",
+          proof == "skipped" and sent == "NODE_LOSS_SKIPPED_OK")
+
+    neither = classify_sections(_CMD + "        === STDOUT\n        ready nodes: 3\n")
+    _, proof, _ = decide("L3", "passed", nl, neither, "t")
+    check("both tokens in COMMAND and neither in STDOUT is NOT proven",
+          proof == "unknown-sentinel-absent")
+
+    both_out = classify_sections("        === STDOUT\n"
+                                 "        NODE_LOSS_VOLUME_AVAILABLE_OK\n"
+                                 "        NODE_LOSS_SKIPPED_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, both_out, "t")
+    check("skip still BEATS pass when both are in genuine output", proof == "skipped")
+
+    stderr_skip = classify_sections("        === STDERR\n        NODE_LOSS_SKIPPED_OK\n"
+                                    "        === STDOUT\n"
+                                    "        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, stderr_skip, "t")
+    check("a skip token in a non-source section (STDERR) still costs the proof",
+          proof == "skipped")
+
+    stderr_pass = classify_sections("        === STDERR\n"
+                                    "        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    _, proof, _ = decide("L3", "passed", nl, stderr_pass, "t")
+    check("a pass token OUTSIDE stdout cannot prove anything",
+          proof == "unknown-sentinel-absent")
+
+    # --- classify_sections: markers, extents, and Go's own `=== RUN` --------------------
+    gorows = classify_sections("=== RUN   chainsaw\n=== PAUSE chainsaw\n"
+                               "=== CONT  chainsaw/node-loss\n"
+                               "        === COMMAND\n        echo NODE_LOSS_SKIPPED_OK\n"
+                               "    l.go:52: | 1:00:00 | node-loss | s | SCRIPT | DONE |\n"
+                               "--- PASS: chainsaw/node-loss (131.09s)\n")
+    secs = [sec for sec, _ in gorows]
+    check("Go's column-0 `=== RUN/PAUSE/CONT` never open a section",
+          secs[:3] == ["", "", ""])
+    check("a section covers its indented body and ends at a less-indented line",
+          secs[3] == "" and secs[4] == "COMMAND" and secs[5] == "" and secs[6] == "")
+    check("a log with no === markers stays wholly scannable (UNSECTIONED)",
+          {sec for sec, _ in classify_sections("plain line\n  another\n")} == {UNSECTIONED})
+
+    # --- the preserved run itself, as a fixture ----------------------------------------
+    # runs/20260901T235117Z-e9a4b66 IS the bug: NODE_LOSS_SKIPPED_OK twice in `=== COMMAND`
+    # (source), never in output; NODE_LOSS_VOLUME_AVAILABLE_OK once in `=== STDOUT`.
+    fixture = CONF_ROOT / "runs" / "20260901T235117Z-e9a4b66"
+    if (fixture / "run.log").is_file() and (fixture / "report.junit.xml").is_file():
+        rows = build_rows(fixture, CONF_ROOT / "tests", "dev", "v0.2.12", "v1.36.4", "sha")
+        row = next((r for r in rows if r["test"] == "node-loss"), None)
+        check("preserved real L3 run records node-loss/R-STATE-02 as proven",
+              row is not None and row["proof"] == "proven"
+              and row["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK"
+              and row["requirements"] == ["R-STATE-02"])
+        check("preserved real L3 run attributes (no DEGRADED fallback needed)",
+              row is not None and "DEGRADED" not in row["note"])
+        pairs = classify_sections((fixture / "run.log").read_text(errors="replace"))
+        secs = {sec for sec, line in pairs if "NODE_LOSS_SKIPPED_OK" in line}
+        check("every NODE_LOSS_SKIPPED_OK in the preserved log is COMMAND source",
+              secs == {"COMMAND"})
+        secs = {sec for sec, line in pairs
+                if line.strip() == "NODE_LOSS_VOLUME_AVAILABLE_OK"}
+        check("the real emission in the preserved log is in STDOUT", secs == {"STDOUT"})
+
+    # --- log attribution --------------------------------------------------------------
+    log = ("[run.sh] Coverage report first\n"
+           "    | 20:41:32 | node-loss | step | SCRIPT | RUN |\n"
+           "        NODE_LOSS_SKIPPED_OK\n"
+           "    | 20:44:01 | dr-restore | step | SCRIPT | RUN |\n"
+           "        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK\n")
+    b, attributed = attribute_log(log, {"node-loss", "dr-restore"})
+    check("log lines are attributed to the testcase whose header preceded them",
+          attributed
+          and any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["node-loss"])
+          and not any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["dr-restore"]))
+    check("a pass token does NOT leak across testcase boundaries",
+          not any("DR_RESTORED" in l for _, l in b["node-loss"]))
+    check("an unsectioned log keeps whole-log scanning (nothing narrowed away)",
+          all(sec == UNSECTIONED for sec, _ in b["node-loss"]))
+
+    # Chainsaw's real headers carry a Go `t.Log` source prefix and ANSI colour; the earlier
+    # regex matched neither, so every genuine Chainsaw log silently fell into DEGRADED mode.
+    real_hdr = ("    l.go:52: \x08\x08\x08| 23:51:18 | \x1b[32;1mnode-loss\x1b[0;22m | "
+                "\x1b[32;1mstep\x1b[0;22m | SCRIPT | LOG |\n"
+                "        === STDOUT\n        NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+    b2, attributed_real = attribute_log(real_hdr, {"node-loss"})
+    check("a REAL Chainsaw header (go source prefix + ANSI) attributes, not degrades",
+          attributed_real and "node-loss" in b2
+          and ("STDOUT", "        NODE_LOSS_VOLUME_AVAILABLE_OK") in b2["node-loss"])
+    _, attributed2 = attribute_log("no chainsaw headers at all\n", {"node-loss"})
+    check("an unattributable log is reported as such (degraded mode)", not attributed2)
+
+    smap2 = {"a": {"pass": ["DNS_OK", "A_ONLY_OK"], "skip": []},
+             "b": {"pass": ["DNS_OK"], "skip": []}}
+    check("only tokens owned by exactly one test may be used in degraded mode",
+          unique_tokens(smap2) == {"A_ONLY_OK"})
+
+    # --- end-to-end: a self-skipped L3 run must land in the ledger as `skipped` --------
+    with tempfile.TemporaryDirectory() as td:
+        root = Path(td)
+        tdir = root / "tests" / "l3-fault-injection" / "node-loss"
+        tdir.mkdir(parents=True)
+        (tdir / "chainsaw-test.yaml").write_text(
+            "kind: Test\nmetadata:\n  name: node-loss\n  labels:\n"
+            "    conformance.home.k3s/R-STATE-02: \"true\"\n  annotations:\n"
+            "    conformance.home.k3s/requirements: \"R-STATE-02\"\n"
+            "    conformance.home.k3s/layer: \"L3\"\nspec:\n  steps:\n"
+            "    - script:\n        content: 'echo hi'\n        check:\n"
+            "          (contains($stdout, 'NODE_LOSS_VOLUME_AVAILABLE_OK')"
+            " || contains($stdout, 'NODE_LOSS_SKIPPED_OK')): true\n")
+        rd = root / "runs" / "20260901T120000Z-abc1234"
+        rd.mkdir(parents=True)
+        (rd / "report.junit.xml").write_text(
+            '<testsuites><testsuite><testcase classname="L3:R-STATE-02" '
+            'name="node-loss"><properties>'
+            '<property name="requirements" value="R-STATE-02"/>'
+            '<property name="layer" value="L3"/></properties></testcase>'
+            "</testsuite></testsuites>")
+        (rd / "run.log").write_text(
+            "  | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
+            "      SKIP: 'longhorn' StorageClass absent\n"
+            "      NODE_LOSS_SKIPPED_OK\n")
+        rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
+        check("end-to-end: Chainsaw 'passed' + skip token => proof 'skipped'",
+              len(rows) == 1 and rows[0]["junit_result"] == "passed"
+              and rows[0]["proof"] == "skipped"
+              and rows[0]["sentinel"] == "NODE_LOSS_SKIPPED_OK")
+        check("end-to-end: row carries ts from the run-dir name and the R-ID label",
+              rows[0]["ts_utc"] == "2026-09-01T12:00:00Z"
+              and rows[0]["requirements"] == ["R-STATE-02"])
+
+        (rd / "run.log").write_text(
+            "  | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
+            "      NODE_LOSS_VOLUME_AVAILABLE_OK\n")
+        rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
+        check("end-to-end: Chainsaw 'passed' + pass token => proof 'proven'",
+              rows[0]["proof"] == "proven"
+              and rows[0]["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK")
+
+        ledger = root / "ledger.jsonl"
+        append_rows(ledger, rows)
+        append_rows(ledger, rows)
+        check("ledger is append-only JSONL (one parsable object per line)",
+              len(ledger.read_text().strip().splitlines()) == 2
+              and json.loads(ledger.read_text().splitlines()[0])["test"] == "node-loss")
+
+    if not ok:
+        return 1
+    sys.stderr.write("record_run.py selftest: PASS\n")
+    return 0
+
+
+# ============================================================================== main
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--run-dir", type=Path,
+                    help="conformance/runs/<UTC>-<sha>/ holding report.junit.xml + run.log")
+    ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
+                    help=f"append-only ledger (default {DEFAULT_LEDGER})")
+    ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
+    ap.add_argument("--repo", type=Path, default=REPO_ROOT)
+    ap.add_argument("--env", default=os.environ.get("CONF_ENV") or context_env(),
+                    help="env the run targeted; derived from the kube context when "
+                         "unset (never defaulted to a cluster name)")
+    ap.add_argument("--chainsaw-version", default=None,
+                    help="override the versions.env pin (recorded verbatim)")
+    ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
+                    help="override the `kubectl version` probe (recorded verbatim)")
+    ap.add_argument("--dry-run", action="store_true",
+                    help="print the rows that WOULD be appended; write nothing")
+    ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
+    args = ap.parse_args()
+
+    if args.selftest:
+        return _selftest()
+    if args.run_dir is None:
+        sys.stderr.write("[record] --run-dir is required (or use --selftest)\n")
+        return 1
+
+    run_dir = args.run_dir.resolve()
+    junit, log = run_dir / "report.junit.xml", run_dir / "run.log"
+    # BOTH files or nothing. The JUnit alone cannot tell a self-skip from a real pass, so
+    # recording from it alone is exactly the false proof this tool exists to prevent.
+    missing = [p.name for p in (junit, log) if not p.is_file()]
+    if missing:
+        sys.stderr.write(
+            f"[record] {run_dir} is missing {', '.join(missing)} — nothing recorded. "
+            f"The JUnit alone cannot distinguish a self-skip from a real pass "
+            f"(see module docstring). SKIPPED (not green).\n")
+        return 3
+
+    try:
+        rows = build_rows(
+            run_dir, args.tests, args.env,
+            args.chainsaw_version or pinned_chainsaw_version(CONF_ROOT),
+            args.kube_version or server_kube_version(),
+            git_sha(args.repo))
+    except ET.ParseError as e:
+        sys.stderr.write(f"[record] {junit} is not parsable JUnit: {e}\n")
+        return 1
+    except OSError as e:
+        sys.stderr.write(f"[record] cannot read the run inputs: {e}\n")
+        return 1
+
+    if not rows:
+        sys.stderr.write(f"[record] {junit} contains no <testcase> — nothing to record. "
+                         f"SKIPPED (not green).\n")
+        return 3
+
+    for r in rows:
+        print(f"[record] {r['layer']:4s} {r['proof']:28s} {r['test']} "
+              f"({','.join(r['requirements']) or 'untagged'}) "
+              f"junit={r['junit_result']} sentinel={r['sentinel'] or '—'}")
+    counts: dict[str, int] = {}
+    for r in rows:
+        counts[r["proof"]] = counts.get(r["proof"], 0) + 1
+    print(f"[record] {len(rows)} testcase(s): "
+          + ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
+
+    if args.dry_run:
+        print("[record] --dry-run: ledger NOT written.")
+        return 0
+    try:
+        append_rows(args.ledger, rows)
+    except OSError as e:
+        sys.stderr.write(f"[record] cannot append to {args.ledger}: {e}\n")
+        return 1
+    print(f"[record] appended {len(rows)} row(s) -> {args.ledger}")
+    print("[record] COMMIT the run dir and the ledger; an uncommitted ledger proves nothing.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

conformance/lib/run_l1_offline.py

@@ -19,6 +19,30 @@ Exit 0 iff every selected offline check passes.
 
 The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
 carries; a dir with neither is reported SKIPPED (never a pass).
+
+DECLARED SCOPE, AND WHY A SKIP IS NOT ALWAYS HARMLESS (--unwired-scope)
+----------------------------------------------------------------------
+Exit 0 here means "every selected check passed"; a check that exits 3 is SKIPPED and, by
+default, does not fail the run (a scaffold with no check script, an audit whose subject
+subtree does not exist yet). That default is WRONG for a WHOLE-REPO invariant. Those are the
+checks ci/l0.sh used to run as hand-written "UNCONDITIONAL" stages, and each of those stages
+mapped rc 3 to the GATE's rc 3 with the words "not a pass":
+
+    3) skip "substitution check SKIPPED (clusters/ absent) — not a pass"
+       summary "- subst-safety: **SKIPPED** (nothing to audit)"
+       if [ "$rc" -eq 0 ]; then rc=3; fi
+
+A whole-repo check reporting "nothing to audit" is reporting that the tree it audits has
+VANISHED — which is a louder problem than the check failing, not a quieter one. So:
+
+  --unwired-scope SCOPE   a test DECLARING `<prefix>/scope: SCOPE` whose check SKIPS (rc 3,
+                          or no check script at all) makes this runner exit 3 instead of 0.
+                          Hard failures still win (exit 1); rc 3 never masks one.
+
+Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing failed but a
+scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it
+into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now,
+instead of once per hand-written stage.
 """
 from __future__ import annotations
 
@@ -36,6 +60,24 @@ LABEL_LINE_RE_TMPL = r"^\s*{prefix}/(R-[A-Z0-9]+-[0-9]+)\s*:\s*[\"']?true"
 CHECK_SCRIPTS = ("render-check.sh", "check.sh")
 
 
+def scope_of(test_yaml: Path, annotation: str) -> str:
+    """The scope a test DECLARES, or "" when it declares none.
+
+    Matched on a real YAML mapping line: a `#` comment cannot begin with the annotation key,
+    so a scope named in the test's header prose is documentation, not a declaration — the
+    same "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs.
+    Mirrors conf_dirs_for_scope() in lib/common.sh; they must agree.
+    """
+    if not test_yaml.exists():
+        return ""
+    rx = re.compile(r"^\s*" + re.escape(annotation) + r"\s*:\s*[\"']?([A-Za-z0-9_.-]+)[\"']?\s*$")
+    for line in test_yaml.read_text().splitlines():
+        m = rx.match(line)
+        if m:
+            return m.group(1)
+    return ""
+
+
 def layer_of(d: Path) -> str:
     """Layer for the JUnit properties, from the subtree the test lives in."""
     rel = str(d)
@@ -59,25 +101,38 @@ def main() -> int:
     ap = argparse.ArgumentParser(description=__doc__)
     ap.add_argument("--junit", type=Path, required=True)
     ap.add_argument("--label-prefix", default="conformance.home.k3s")
+    ap.add_argument("--scope-annotation", default="conformance.home.k3s/scope",
+                    help="annotation key a test uses to DECLARE its invariant's scope")
+    ap.add_argument("--unwired-scope", action="append", default=[], metavar="SCOPE",
+                    help="a test declaring this scope that SKIPS is UNWIRED, not green "
+                         "(makes this runner exit 3); repeatable")
+    ap.add_argument("--summary-out", type=Path, default=None,
+                    help="write one Markdown line per case here, for the CI step summary")
     ap.add_argument("dirs", nargs="+", type=Path)
     args = ap.parse_args()
 
-    cases = []  # (name, reqs, dir, rc, output)
+    strict_scopes = set(args.unwired_scope)
+
+    cases = []  # (name, reqs, dir, rc, output, scope)
     for d in args.dirs:
         check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
         name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
+        scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation)
         if check is None:
             cases.append((name, reqs, d, 3,
-                          f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}"))
+                          f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
+                          scope))
             continue
         proc = subprocess.run(
             ["bash", str(check)], capture_output=True, text=True
         )
         out = (proc.stdout + proc.stderr).strip()
-        cases.append((name, reqs, d, proc.returncode, out))
+        cases.append((name, reqs, d, proc.returncode, out, scope))
 
     failures = sum(1 for c in cases if c[3] not in (0, 3))
     skipped = sum(1 for c in cases if c[3] == 3)
+    # A SKIP of a scope-declared check is UNWIRED, not green (see the module docstring).
+    unwired = [c for c in cases if c[3] == 3 and c[5] in strict_scopes]
 
     suite = ET.Element(
         "testsuite",
@@ -89,7 +144,8 @@ def main() -> int:
             "skipped": str(skipped),
         },
     )
-    for name, reqs, d, rc, out in cases:
+    summary_lines = []
+    for name, reqs, d, rc, out, scope in cases:
         tc = ET.SubElement(
             suite,
             "testcase",
@@ -100,17 +156,33 @@ def main() -> int:
         ET.SubElement(props, "property", {"name": "requirements", "value": ",".join(reqs)})
         ET.SubElement(props, "property", {"name": "layer", "value": layer_of(d)})
         ET.SubElement(props, "property", {"name": "path", "value": str(d)})
+        ET.SubElement(props, "property", {"name": "scope", "value": scope})
         if rc == 3:
             ET.SubElement(tc, "skipped", {"message": out or "no offline check script"})
         elif rc != 0:
             fail = ET.SubElement(tc, "failure", {"message": f"offline check exit {rc}"})
             fail.text = out
         status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
+        mark = f" scope={scope}" if scope else ""
         print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
-              f"({','.join(reqs) or 'untagged'})",
+              f"({','.join(reqs) or 'untagged'}){mark}",
               file=sys.stderr)
         if rc not in (0, 3):
             print(out, file=sys.stderr)
+        # One step-summary line per case, naming what ran — the generated replacement for the
+        # hand-written `summary "- subst-safety: **passed**"` line each deleted stage carried.
+        tag = f" ({scope})" if scope else ""
+        if rc == 0:
+            verdict = "**passed**"
+        elif rc == 3:
+            verdict = ("**SKIPPED** (UNWIRED, not green)" if scope in strict_scopes
+                       else "**SKIPPED** (nothing to audit)")
+        else:
+            verdict = f"**FAILED** (exit {rc})"
+        summary_lines.append(f"- {layer_of(d).lower()}/{name}{tag}: {verdict}")
+        if rc == 3 and scope in strict_scopes:
+            print(f"[offline] {name}: SKIPPED, and it declares scope={scope!r} — "
+                  f"UNWIRED, NOT GREEN. {out}", file=sys.stderr)
 
     root = ET.Element(
         "testsuites",
@@ -126,9 +198,19 @@ def main() -> int:
     args.junit.parent.mkdir(parents=True, exist_ok=True)
     xml = minidom.parseString(ET.tostring(root, "utf-8")).toprettyxml(indent="  ")
     args.junit.write_text(xml)
-    print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s)) -> "
+    if args.summary_out is not None:
-          f"{args.junit}", file=sys.stderr)
+        args.summary_out.parent.mkdir(parents=True, exist_ok=True)
-    return 1 if failures else 0
+        args.summary_out.write_text("\n".join(summary_lines) + ("\n" if summary_lines else ""))
+
+    print(f"[offline] wrote JUnit ({len(cases)} case(s), {failures} failure(s), "
+          f"{skipped} skip(s), {len(unwired)} unwired) -> {args.junit}", file=sys.stderr)
+    if failures:
+        return 1
+    if unwired:
+        print("[offline] UNWIRED (declared scope, SKIPPED — not a pass): "
+              + ", ".join(c[0] for c in unwired), file=sys.stderr)
+        return 3
+    return 0
 
 
 if __name__ == "__main__":

conformance/lib/status.py

@@ -15,26 +15,42 @@ This generator derives, from the REPOSITORY ALONE, three orthogonal axes per req
                        from a Flux Kustomization entrypoint (i.e. actually reconciled)?
   2. VERIFICATION    — is there a requirement-tagged test, at which layer, and is it an
                        EXECUTABLE test or a TODO scaffold?
-  3. PROOF           — has that test actually passed?  *** DELIBERATELY NOT DERIVED. ***
+  3. PROOF           — has that test actually passed?  Derived ONLY from the committed
+                       run ledger, and ONLY as far as that ledger's evidence reaches.
 
-WHAT THIS TOOL REFUSES TO DERIVE (and why)
+PROOF — WHAT CHANGED, AND WHAT DID NOT
-------------------------------------------
+--------------------------------------
-PROOF-OF-PASS IS NOT DERIVABLE FROM REPO STATE. Nothing tracked in git records a test
+This report used to say `not-derivable` for every requirement, because nothing tracked in
-result: `conformance/_artifacts/` and `results/` are git-ignored, and the only prose
+git recorded a test result and the only prose record of a green run lived in
-record of a green run lives in DECISIONS-LOG.md — hand-maintained, unstructured, and
+DECISIONS-LOG.md — hand-maintained, unstructured, exactly the source the operator ruled
-exactly the kind of source the operator has ruled out. The project's own culture is that
+out. The fix this docstring asked for now exists: `conformance/runs/ledger.jsonl`, an
-"merged" is never "done" (agents/definition-of-done.md, R-TEST-03), so this report will
+append-only ledger written by `lib/record_run.py` from a run's JUnit **and its run log**
-NOT infer, guess, or default a requirement to green. Every requirement's proof state is
+(see conformance/.gitignore for the negations that keep both in git).
-reported as `not-derivable`, and the headline says so. If an untracked JUnit artifact
-happens to be lying in the workspace it is surfaced ONLY as a quarantined advisory with
-its mtime — never counted, never mixed into a status.
 
-The honest fix is a committed, machine-readable run ledger; until that exists, the most
+THE REFUSAL POSTURE IS UNCHANGED. Evidence replaced "not derivable" only where evidence
-this report can say is "wired and testable", never "proven".
+exists; nowhere is optimism substituted for it:
+
+  * A ledger row reaches `proven` only if `record_run.py` saw that test's own PASS
+    sentinel token in the run log. The L2/L3 tests SELF-SKIP GREEN — Chainsaw records
+    `passed` on `DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries no
+    stdout — so a skip token maps to `skipped` and can never become proof.
+  * A proof is honoured only while it still describes the current code: if any of the
+    requirement's implementing manifests or verifying tests changed in a commit that is
+    NOT an ancestor of the run's `git_sha`, the row degrades to `stale-proof`, which is
+    not green. A run recorded from a dirty tree (`<sha>-dirty`) is never honoured.
+  * A requirement with no ledger row, or whose verifying tests did not ALL report proven,
+    stays `not-derivable` / `unknown` — never rounded up.
+  * The untracked `_artifacts/report.junit.xml` is STILL only a quarantined advisory with
+    its mtime. It is not the ledger, it is not evidence, and it feeds no status.
+
+With an empty or absent ledger this report behaves exactly as it did before: every
+requirement `not-derivable`, headline PROVEN 0, "the most this report can say is wired and
+testable, never proven".
 
 READS (all read-only)
   requirements/registry.yaml     the authoritative R-* mirror  (via lib/coverage.py)
   tests/**/chainsaw-test.yaml    requirement tags + layer + executable annotation
+  runs/ledger.jsonl              COMMITTED proof-of-pass rows (via lib/record_run.py)
   clusters/<env>/               Flux entrypoint -> the reconciliation graph
   `git ls-files`                 TRACKED files only; untracked scratch is not repo state
   ../home-network-k3s/docs/00-vision-and-scope.md   OPTIONAL registry<->spec parity check
@@ -92,7 +108,14 @@ FLUX_API = "kustomize.toolkit.fluxcd.io"
 #   conformance/tests — those are the TESTS; they are the VERIFICATION axis, counted there.
 #   ci/tests          — gate unit-test fixtures embed example R-IDs as DATA.
 #   registry.yaml     — the requirement mirror itself, not an implementation of anything.
-IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/")
+# `conformance/runs/` is excluded for a subtler reason than the test dirs: the ledger and the
+# preserved run logs CONTAIN the requirement IDs they are evidence FOR. Counting them as
+# implementing evidence makes `last_change` move every time proof is recorded, so a row's
+# git_sha always predates it and EVERY proof degrades to stale-proof the instant it is
+# committed — recording the proof destroys it. The ledger is the proof record; it must not
+# participate in the freshness computation about itself. (Found live 2026-09-01 on the first
+# PROVEN row.)
+IMPL_EXCLUDE_PREFIXES = ("conformance/tests/", "ci/tests/", "conformance/runs/")
 IMPL_EXCLUDE_PATHS = ("conformance/requirements/registry.yaml",)
 
 
@@ -274,6 +297,109 @@ _EXECUTABLE_RE = re.compile(
 _BLOCKED_RE = re.compile(
     r"^\s*" + re.escape(LABEL_PREFIX) + r"/blocked-by:\s*(.+)$", re.M)
 
+# ---------------------------------------------------------- scaffold-marker detection
+# WHY THIS IS NOT A WHOLE-FILE `"TODO" in text`
+# ---------------------------------------------------
+# It used to be, and that mis-reported a SAFETY-CRITICAL requirement. A test whose SUBJECT
+# is placeholder detection has to name the idioms it hunts for:
+# `l0-static/placeholder-guard-coupling/check.sh` carries
+# `SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", ...]` — its detection vocabulary, as data.
+# A whole-file substring test read that as the test declaring itself unfinished, classified
+# a 32-assertion executable check as a scaffold, raised a permanent `--strict` annotation
+# conflict, and (because the same file is tagged R-STATE-03) reported R-STATE-03 as
+# `exec=1 scaffold=1`. A previous engineer worked around it by rewording prose.
+#
+# The rule below is derived from the corpus (43 tests), not invented. Every genuine
+# scaffold in this tree — `l2-behavioral/onboarding-completeness`, `l3-fault-injection/
+# alerting`, `l3-fault-injection/gitops-drift`, and no others — declares itself in BOTH of
+# two structural positions, and no executable test does either:
+#
+#   1. a `# STATUS: ... scaffold` line in the file's documentation header. This is the
+#      repo's DEDICATED declaration slot (tests/README.md): 3 of 43 tests say
+#      `STATUS: scaffold` and they are exactly the three scaffolds; 19 others declare
+#      `STATUS: EXECUTABLE` / `IMPLEMENTED` / `RUNNABLE`; the rest declare nothing.
+#   2. a TODO/FIXME marker in the BODY that CARRIES A MESSAGE — `echo "TODO: assert PVC
+#      bound ..."`, `TODO(T-4.1): hermetic SMTP capture sink`. A marker that carries a
+#      message is a note about work that is missing. A marker that is the ENTIRE content of
+#      a quoted literal (`"TODO",` in a vocabulary list) carries no message: it is the
+#      marker as a *datum*, which is what a placeholder DETECTOR necessarily contains.
+#
+# Both are still evaluated over the body of `chainsaw-test.yaml` AND of any
+# `render-check.sh` / `check.sh`, and either one still OUTRANKS an
+# `executable: "true"` annotation (see classify_test) — the invariant that an annotation
+# must never outrank the body is unchanged. Ambiguity still only ever DOWNGRADES: an
+# unrecognised marker shape, a marker in body code outside a quoted literal, or a marker
+# carrying any trailing text all classify as scaffold.
+#
+# KNOWN, ACCEPTED IMPRECISION (stated rather than hidden): a test that embeds a marker with
+# text attached but as data — e.g. `re.compile(r"TODO|FIXME")`, or a single string holding
+# several idioms — is still called a scaffold. That is the conservative direction, and the
+# author's remedy is the same structural discipline the corpus already uses: keep each
+# idiom a standalone literal in a list.
+
+# Vocabulary: only the idioms that DECLARE unfinished work. Deliberately excludes `TBD`,
+# `PLACEHOLDER`, `XXX`, `REPLACE-ME` — tests in this tree legitimately carry those in
+# executable code as data (`l0-static/egress-paths-selected-by-rules/check.sh` compares
+# authored values against `"TBD"`; `l0-static/no-plaintext-secrets/check.sh` matches
+# `placeholder.*`), and none of them is used here to mark a placeholder body.
+# Case-insensitive: no English word contains either token, and the corpus carries no
+# lower-case occurrence, so folding case only ever ADDS detection (`echo "todo: ..."`).
+_MARKER_RE = re.compile(r"\b(?:TODO|FIXME)\b", re.I)
+
+# The dedicated declaration slot, on a comment line, case-insensitive. The declared state
+# must be the FIRST token after `STATUS:` — a `STATUS:` line declares exactly one state,
+# and all three real scaffolds read `# STATUS: scaffold — <what is missing>`. Anchoring it
+# there is what keeps `# STATUS: EXECUTABLE ... this is not a scaffold` from matching; a
+# loose `.*scaffold` would turn any prose that merely uses the word into a declaration.
+_STATUS_SCAFFOLD_RE = re.compile(
+    r"^[ \t]*#[ \t]*STATUS:[ \t]*(?:scaffold|placeholder)\b", re.M | re.I)
+
+# A quoted literal whose entire content is one marker word: the marker as data.
+_MARKER_AS_DATUM_RE = re.compile(r"(['\"])[ \t]*(?:TODO|FIXME)[ \t]*\1", re.I)
+
+
+def _split_doc_header(text: str) -> tuple[str, str]:
+    """Split a test file into (documentation header, body).
+
+    Both file kinds in this tree open with a banner comment block that is PROSE about the
+    test: the Given/When/Then, the incident being closed, the mandatory "what this does NOT
+    prove" list (tests/README.md, L0 authoring rule 1). Prose necessarily discusses
+    placeholder idioms when the test hunts them, so an incidental mention there is
+    documentation, not a placeholder body.
+
+    The body starts at the first line that is neither blank nor a `#` comment:
+      *.sh    -> `set -euo pipefail` (the shebang is a comment)
+      *.yaml  -> `apiVersion: chainsaw.kyverno.io/v1alpha1`
+    Comments BELOW that point are body, deliberately: a scaffold marks a missing fixture
+    with an inline comment (`l3-fault-injection/alerting`: `file: fixtures/smtp-sink.yaml
+    # TODO(T-4.1): hermetic SMTP capture sink`), and that must keep counting.
+    """
+    lines = text.splitlines(keepends=True)
+    for i, ln in enumerate(lines):
+        s = ln.strip()
+        if s and not s.startswith("#"):
+            return "".join(lines[:i]), "".join(lines[i:])
+    return text, ""          # no executable line at all: it is all header, body is empty
+
+
+def scaffold_marker(text: str) -> str:
+    """Why `text` declares itself a placeholder body, or "" if it does not.
+
+    Returns a short human-readable reason (used verbatim in the hygiene report) so the
+    defect is actionable rather than a bare path.
+    """
+    head, body = _split_doc_header(text)
+    m = _STATUS_SCAFFOLD_RE.search(head) or _STATUS_SCAFFOLD_RE.search(body)
+    if m:
+        return m.group(0).strip().lstrip("#").strip()[:80]
+    # Blank out markers held as pure data before looking for a marker that has a message.
+    m2 = _MARKER_RE.search(_MARKER_AS_DATUM_RE.sub('""', body))
+    if m2:
+        line = body[:m2.start()].count("\n")
+        snippet = body.splitlines()[line].strip()
+        return f"placeholder marker in body: {snippet[:80]}"
+    return ""
+
 
 def classify_test(t: dict) -> dict:
     """Decide EXECUTABLE vs SCAFFOLD for one discovered test, conservatively.
@@ -281,14 +407,18 @@ def classify_test(t: dict) -> dict:
     A tagged test is only worth counting if it can actually assert something. Rules, in
     order, chosen so ambiguity NEVER upgrades a test:
 
-      scaffold             the body contains TODO markers (a placeholder), *even if* it
+      scaffold             the test DECLARES itself a placeholder — a `# STATUS: scaffold`
-                           is annotated executable — an annotation must never outrank the
+                           header line, or a TODO/FIXME marker carrying a message in the
-                           body. That mismatch is also reported as a hygiene defect.
+                           body (see `scaffold_marker` for the rule and the corpus it was
-      executable           annotated `conformance.home.k3s/executable: "true"` and no TODOs.
+                           derived from) — *even if* it is annotated executable. An
-      executable-untagged  no TODOs and a real runnable body, but the annotation is
+                           annotation must never outrank the body. That mismatch is also
-                           missing (true of the L1 render tests, which execute a
+                           reported as a hygiene defect.
-                           render-check.sh + golden diff, of the L0 static checks, and
+      executable           annotated `conformance.home.k3s/executable: "true"` and no
-                           of the META self-check).
+                           scaffold declaration.
+      executable-untagged  no scaffold declaration and a real runnable body, but the
+                           annotation is missing (true of the L1 render tests, which
+                           execute a render-check.sh + golden diff, of the L0 static
+                           checks, and of the META self-check).
                            Counted as executable, and flagged as an annotation gap.
       scaffold             anything else.
     """
@@ -299,16 +429,26 @@ def classify_test(t: dict) -> dict:
     # downgrades the test to a scaffold — an annotation must never outrank the body,
     # whichever file the body lives in.
     extra = ""
+    scripts: list[tuple[str, str]] = []
     for script in ("render-check.sh", "check.sh"):
         sp = p.parent / script
         if sp.is_file():
-            extra += sp.read_text()
+            body = sp.read_text()
+            extra += body
+            scripts.append((script, body))
     declared = bool(_EXECUTABLE_RE.search(text))
-    has_todo = "TODO" in text or "TODO" in extra
+    # Each file is classified on ITS OWN structure — the yaml and the script have different
+    # header blocks, so they must not be concatenated before the header/body split.
+    reason = scaffold_marker(text)
+    for script, body in scripts:
+        if reason:
+            break
+        r = scaffold_marker(body)
+        reason = f"{script}: {r}" if r else ""
     bm = _BLOCKED_RE.search(text)
     has_body = bool(extra) or bool(re.search(r"^\s+- (script|assert|apply):", text, re.M))
 
-    if has_todo:
+    if reason:
         state = "scaffold"
     elif declared:
         state = "executable"
@@ -325,7 +465,8 @@ def classify_test(t: dict) -> dict:
         "name": t["name"], "path": rel, "layer": t["layer"],
         "state": state,
         "declared_executable": declared,
-        "annotation_conflict": bool(declared and has_todo),
+        "annotation_conflict": bool(declared and reason),
+        "scaffold_reason": reason,
         "blocked_by": bm.group(1).strip().strip('"') if bm else "",
     }
 
@@ -343,6 +484,169 @@ def spec_requirement_ids(spec_root: Path) -> set[str] | None:
     return set(RID_RE.findall(doc.read_text()))
 
 
+# ========================================================================== ledger
+# PROOF axis. Everything here reads `conformance/runs/ledger.jsonl` (written ONLY by an
+# explicit `lib/record_run.py` / `run.sh --record` invocation) and downgrades relentlessly.
+# Nothing in this section may ever turn absent, ambiguous or aged evidence into green.
+
+DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
+
+# Severity ranking, WORST FIRST. A requirement's proof is the MINIMUM over its verifying
+# tests, so one failed/skipped/missing test drags the whole requirement down and a single
+# proven test can never carry the others. `stale-proof` ranks below `proven` on purpose:
+# it is a real past pass that no longer describes the current code, and it is NOT green.
+PROOF_ORDER = {
+    "failed": 0,              # a ledger row recorded a JUnit failure/error
+    "unknown": 1,             # ran, but the log could not evidence pass vs self-skip
+    "not-derivable": 2,       # no ledger row for this test at all (the pre-ledger default)
+    "junit-skipped": 3,       # JUnit <skipped/> — did not execute
+    "skipped": 4,             # SELF-SKIPPED green: exited 0 without exercising the behaviour
+    "stale-proof": 5,         # proven at a sha the implementation/test has since moved past
+    "proven": 6,              # the ONLY green state
+}
+# record_run.py's two "we cannot tell" states collapse to one reported state here; the
+# per-test detail in status.json keeps the distinction for whoever has to fix it.
+_PROOF_ALIASES = {
+    "unknown-no-sentinel-declared": "unknown",
+    "unknown-sentinel-absent": "unknown",
+}
+
+
+def read_ledger(path: Path) -> list[dict]:
+    """Parse the append-only JSONL ledger. A malformed line is SKIPPED, never guessed at.
+
+    Absent or empty file -> [] , which reproduces this report's pre-ledger behaviour
+    exactly (every requirement `not-derivable`, headline PROVEN 0).
+    """
+    if not path.is_file():
+        return []
+    rows: list[dict] = []
+    for line in path.read_text(errors="replace").splitlines():
+        line = line.strip()
+        if not line or line.startswith("#"):
+            continue
+        try:
+            obj = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(obj, dict) and obj.get("test"):
+            rows.append(obj)
+    return rows
+
+
+def latest_by_test(rows: list[dict]) -> dict[str, dict]:
+    """Most recent ledger row per test name (ts_utc, then file order as the tiebreak).
+
+    Latest wins rather than best-ever: a test that has since gone back to self-skipping
+    must stop reading as proven.
+    """
+    latest: dict[str, dict] = {}
+    for i, r in enumerate(rows):
+        key = r["test"]
+        prev = latest.get(key)
+        if prev is None or (r.get("ts_utc", ""), i) >= (prev.get("ts_utc", ""),
+                                                       prev["_i"]):
+            latest[key] = {**r, "_i": i}
+    return latest
+
+
+class _GitOracle:
+    """Small cached wrapper for the two git questions the staleness rule asks."""
+
+    def __init__(self, repo: Path):
+        self.repo = repo
+        self._last: dict[tuple[str, ...], str | None] = {}
+        self._anc: dict[tuple[str, str], bool] = {}
+
+    def _git(self, *args: str) -> subprocess.CompletedProcess:
+        return subprocess.run(["git", "-C", str(self.repo), *args],
+                              capture_output=True, text=True)
+
+    def last_change(self, paths: tuple[str, ...]) -> str | None:
+        """Commit that last touched any of `paths`; None if git knows none of them."""
+        if not paths:
+            return None
+        if paths in self._last:
+            return self._last[paths]
+        p = self._git("log", "-1", "--format=%H", "--", *paths)
+        sha = p.stdout.strip() if p.returncode == 0 else ""
+        self._last[paths] = sha or None
+        return self._last[paths]
+
+    def is_ancestor(self, older: str, newer: str) -> bool:
+        """True iff `older` is an ancestor of (or equal to) `newer`.
+
+        False whenever either rev is unresolvable — an unknown or `-dirty` sha (a run
+        recorded from a modified working tree) can never satisfy the freshness test, which
+        is the intended fail-closed direction.
+        """
+        key = (older, newer)
+        if key in self._anc:
+            return self._anc[key]
+        ok = self._git("merge-base", "--is-ancestor", older, newer).returncode == 0
+        self._anc[key] = ok
+        return ok
+
+
+def derive_proof(git: _GitOracle, rid: str, verifiers: list[dict],
+                 impl_files: list[dict], latest: dict[str, dict]) -> dict:
+    """PROOF for one requirement, from the ledger, with the staleness rule applied.
+
+    STALENESS RULE — a proof is a claim about a specific tree, not a permanent badge:
+      relevant paths = every implementing artifact this report found for the requirement
+                       + every verifying test's directory
+      last_change    = `git log -1 -- <relevant paths>`
+      A row's `proven` is honoured only if `last_change` is an ANCESTOR of the row's
+      `git_sha`, i.e. the run happened at or after the newest change to the code it
+      claims to prove. Otherwise it degrades to `stale-proof` (not green). A `-dirty`
+      sha, a sha this clone does not have (rebased/dropped), or a path git has never
+      seen (an uncommitted test) all fail the ancestry test and degrade the same way.
+
+    The requirement's state is the MINIMUM over its verifying tests: every verifying test
+    must be proven and fresh before the requirement reads `proven`. A test with no ledger
+    row contributes `not-derivable`, so a partially-recorded run cannot manufacture green.
+    """
+    paths = sorted({f["path"] for f in impl_files}
+                   | {os.path.dirname(v["path"]) for v in verifiers if v.get("path")})
+    last_change = git.last_change(tuple(paths))
+
+    per_test: list[dict] = []
+    for v in verifiers:
+        row = latest.get(v["name"])
+        if row is None:
+            per_test.append({"test": v["name"], "state": "not-derivable",
+                             "reason": "no ledger row — this test has never been "
+                                       "recorded by run.sh --record / record_run.py"})
+            continue
+        base = _PROOF_ALIASES.get(row.get("proof", ""), row.get("proof", "unknown"))
+        if base not in PROOF_ORDER:
+            base = "unknown"
+        state, reason = base, row.get("note", "")
+        if base == "proven":
+            sha = row.get("git_sha", "")
+            if last_change and not git.is_ancestor(last_change, sha):
+                state = "stale-proof"
+                reason = (f"proven at {sha[:12]}, but the implementing manifests / test "
+                          f"changed in {last_change[:12]}, which that run predates")
+        per_test.append({"test": v["name"], "state": state, "reason": reason,
+                         "run_id": row.get("run_id"), "ts_utc": row.get("ts_utc"),
+                         "git_sha": row.get("git_sha"), "env": row.get("env"),
+                         "sentinel": row.get("sentinel"),
+                         "junit_result": row.get("junit_result")})
+
+    if not per_test:
+        state = "not-derivable"
+        reason = "no requirement-tagged test exists to be proven"
+    else:
+        worst = min(per_test, key=lambda d: PROOF_ORDER[d["state"]])
+        state, reason = worst["state"], worst["reason"]
+        if state == "not-derivable" and not reason:
+            reason = "no committed ledger row for this requirement's verifying test(s)"
+
+    return {"state": state, "reason": reason, "last_change_sha": last_change,
+            "evidence_paths": paths, "per_test": per_test}
+
+
 # ========================================================================== build
 IMPL_ORDER = {"none": 0, "off-cluster": 1, "unwired": 2, "reconciled-partial": 3,
               "reconciled": 4}
@@ -369,7 +673,13 @@ LABELS = {
 
 def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, list[dict]],
           applied: set[str], gitops_roots: set[str], spec_ids: set[str] | None,
-          env: str) -> dict:
+          env: str, ledger_rows: list[dict] | None = None) -> dict:
+    # PROOF axis inputs. `ledger_rows=None` / `[]` reproduces the pre-ledger behaviour:
+    # `latest` is empty, so every requirement derives `not-derivable` exactly as before.
+    ledger_rows = ledger_rows or []
+    latest = latest_by_test(ledger_rows)
+    git = _GitOracle(repo)
+
     classified = [classify_test(t) for t in tests]
     by_req_tests: dict[str, list[dict]] = {r["id"]: [] for r in registry}
     for t, c in zip(tests, classified):
@@ -439,17 +749,19 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
                 "blocked_by": sorted({t["blocked_by"] for t in ts if t["blocked_by"]}),
                 "tests": ts,
             },
-            # NEVER derived. See the module docstring.
+            # Derived ONLY from the committed ledger, and only as far as it reaches.
-            "proof": {"state": "not-derivable",
+            # With no ledger row this stays `not-derivable`, as it always was.
-                      "reason": "no committed, machine-readable record of a test run "
+            "proof": derive_proof(git, rid, ts, ev, latest),
-                                "exists in the repo (_artifacts/ and results/ are "
-                                "git-ignored); DECISIONS-LOG.md prose is not a signal"},
             "status": LABELS[(impl_state, verify_state)],
         }
 
     # --- hygiene defects this tool owns -------------------------------------------
     hygiene = {
         "annotation_conflicts": [t["path"] for t in classified if t["annotation_conflict"]],
+        # Same set, with WHY each one was called a placeholder body, so the fix is obvious
+        # from the report instead of requiring the reader to re-derive the rule.
+        "annotation_conflict_reasons": {t["path"]: t["scaffold_reason"]
+                                       for t in classified if t["annotation_conflict"]},
         "executable_untagged": [t["path"] for t in classified
                                 if t["state"] == "executable-untagged"],
         "registry_not_in_spec": [],
@@ -492,8 +804,19 @@ def build(repo: Path, registry: list[dict], tests: list[dict], impl: dict[str, l
             and v["verification"]["state"] == "executable"),
         "safety_critical_total": count(lambda v: v["safety_critical"]),
         "safety_critical_without_executable_l3": len(safety_scaffold),
-        "proven": 0,
+        # PROVEN is now a real count — but only of requirements whose EVERY verifying
-        "proven_derivable": False,
+        # test has a committed ledger row that saw its pass sentinel, at a sha the
+        # implementation has not moved past. `stale-proof` and `skipped` are counted
+        # separately and are NOT green.
+        "proven": count(lambda v: v["proof"]["state"] == "proven"),
+        "proof_stale": count(lambda v: v["proof"]["state"] == "stale-proof"),
+        "proof_skipped": count(lambda v: v["proof"]["state"] == "skipped"),
+        "proof_failed": count(lambda v: v["proof"]["state"] == "failed"),
+        "proof_unknown": count(lambda v: v["proof"]["state"] == "unknown"),
+        "proof_not_derivable": count(
+            lambda v: v["proof"]["state"] in ("not-derivable", "junit-skipped")),
+        "ledger_rows": len(ledger_rows),
+        "proven_derivable": bool(ledger_rows),
         "coverage_style_covered": count(lambda v: v["verification"]["state"] != "none"),
     }
     summary["uncited_by_any_reconciled_manifest"] = count(
@@ -552,16 +875,39 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
     o.append(f"| Safety-critical requirements whose L3 test is a scaffold | "
              f"**{s['safety_critical_without_executable_l3']}** of "
              f"{s['safety_critical_total']} |")
+    if s["proven_derivable"]:
+        o.append(f"| **PROVEN (every verifying test passed, evidenced)** | "
+                 f"**{s['proven']}** |")
+        o.append(f"| Proof exists but is STALE (code changed after the run) | "
+                 f"{s['proof_stale']} |")
+        o.append(f"| Verifying test SELF-SKIPPED green (not proof) | {s['proof_skipped']} |")
+        o.append(f"| Verifying test FAILED in the recorded run | {s['proof_failed']} |")
+        o.append(f"| Ran, but the log could not evidence pass vs self-skip | "
+                 f"{s['proof_unknown']} |")
+        o.append(f"| No committed ledger row yet | {s['proof_not_derivable']} |")
+        o.append("")
+        o.append(f"> **Proof comes from `conformance/runs/ledger.jsonl`** "
+                 f"({s['ledger_rows']} row(s)), written only by an explicit "
+                 f"`run.sh --record` / `lib/record_run.py` invocation. A row reaches "
+                 f"`proven` ONLY when the run log carried that test's own pass sentinel "
+                 f"token — the L2/L3 tests self-skip GREEN (Chainsaw records `passed` on "
+                 f"`DR_RESTORE_SKIPPED_OK` exactly as on a real run, and its JUnit carries "
+                 f"no stdout), so a skip token maps to `skipped` and can never become "
+                 f"proof. A proof is honoured only while the implementing manifests and "
+                 f"the test itself have not changed since the recorded `git_sha`; "
+                 f"otherwise it degrades to `stale-proof`, which is not green.\n")
+    else:
         o.append(f"| **PROVEN (a test actually passed)** | **{s['proven']} — NOT DERIVABLE** |")
         o.append("")
-    o.append("> **Proof is deliberately not derived.** Nothing tracked in git records a "
+        o.append("> **No proof is derivable: `conformance/runs/ledger.jsonl` is absent or "
-             "test result: `conformance/_artifacts/` and `results/` are git-ignored, and "
+                 "empty.** The ledger is the committed, machine-readable run record; until "
-             "the only record of a green run is DECISIONS-LOG.md prose — hand-maintained, "
+                 "a run is captured with `run.sh --record`, nothing tracked in git records "
-             "which is the very thing this report replaces. Per "
+                 "a test result (`conformance/_artifacts/` and `results/` are git-ignored, "
-             "`agents/definition-of-done.md` a requirement is done only when its tagged "
+                 "and DECISIONS-LOG.md prose is hand-maintained — the very thing this "
-             "test **passes**, so the strongest claim this report can make is "
+                 "report replaces). Per `agents/definition-of-done.md` a requirement is "
-             "*wired and testable*. Closing the gap needs a committed, machine-readable "
+                 "done only when its tagged test **passes**, so the strongest claim this "
-             "run ledger; until then, treat every row below as unproven.\n")
+                 "report can make is *wired and testable*; treat every row below as "
+                 "unproven.\n")
     o.append(f"> For contrast, `coverage.md` counts **{s['coverage_style_covered']}** "
              f"requirements as covered — that count includes the "
              f"{s['verify_scaffold_only']} whose only verifying test is a TODO scaffold.\n")
@@ -612,7 +958,15 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
             if ver["blocked_by"]:
                 vcell += f" · blocked-by: {'; '.join(ver['blocked_by'])}"
         sc = " ⚠safety" if v["safety_critical"] else ""
-        o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | not-derivable |")
+        pr = v["proof"]
+        if pr["state"] == "not-derivable":
+            pcell = "not-derivable"
+        else:
+            runs = sorted({d["run_id"] for d in pr["per_test"] if d.get("run_id")})
+            pcell = f"**{pr['state']}**" if pr["state"] == "proven" else pr["state"]
+            if runs:
+                pcell += f" ({', '.join(runs)})"
+        o.append(f"| {rid}{sc} | **{v['status']}** | {icell} | {vcell} | {pcell} |")
 
     if rep["safety_critical_without_executable_l3"]:
         o.append("\n## Safety-critical requirements whose L3 fault test is a SCAFFOLD "
@@ -678,8 +1032,10 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
     else:
         o.append("- registry ↔ spec parity: **NOT CHECKED** (spec tree not available; "
                  "this is a skip, not a pass)")
-    o.append(f"- tests annotated executable but containing TODO bodies: "
+    o.append("- tests annotated executable but declaring a placeholder body: "
-             f"{h['annotation_conflicts'] or 'none'}")
+             + (", ".join(f"`{k}` ({v})"
+                          for k, v in sorted(h["annotation_conflict_reasons"].items()))
+                if h["annotation_conflicts"] else "none"))
     o.append(f"- executable tests missing the `{LABEL_PREFIX}/executable` annotation: "
              f"{h['executable_untagged'] or 'none'}")
 
@@ -698,25 +1054,82 @@ def render_md(rep: dict, nodes: list[dict], advisory: dict | None) -> str:
              "reconciled |")
     o.append("| verifying test | `conformance.home.k3s/R-*` label on a Chainsaw test "
              "(the authoritative tag, same source as `coverage.md`) | exact |")
-    o.append("| executable vs scaffold | TODO markers in the test body outrank the "
+    o.append("| executable vs scaffold | a `# STATUS: scaffold` header line, or a "
-             "`executable` annotation; a real body without the annotation still counts "
+             "TODO/FIXME marker CARRYING A MESSAGE in the body (a marker held as a bare "
-             "and is flagged | conservative — ambiguity never upgrades |")
+             "quoted datum — a placeholder detector's own vocabulary — is not one); "
-    o.append("| **test actually passed** | **NOT DERIVED** — nothing in git records a "
+             "either outranks the `executable` annotation, and a real body without the "
-             "run result | refused |")
+             "annotation still counts and is flagged | conservative — ambiguity never "
+             "upgrades |")
+    o.append("| **test actually passed** | the committed `runs/ledger.jsonl` row for "
+             "every verifying test: JUnit result **plus** that test's own pass sentinel "
+             "token grepped from the run log | exact where a run was recorded, refused "
+             "everywhere else — a self-skip token or a missing/unrecognised sentinel "
+             "NEVER yields `proven` |")
+    o.append("| proof still current | the row's `git_sha` must be a descendant of the "
+             "last commit touching the requirement's implementing manifests or its test "
+             "| conservative — an unresolvable or `-dirty` sha degrades to `stale-proof` |")
     o.append("| cluster liveness (pods/CRs healthy) | **NOT DERIVED** — no cluster is "
              "contacted; `reconciled` means Flux *would* apply it, not that it is "
              "healthy | refused |")
     o.append("| phase/board status | **NOT DERIVED** — `agents/task-board.md` is stale "
              "and is not a signal | refused |")
 
+    o.append("\n## Run ledger — the committed proof-of-pass evidence\n")
+    if not s["proven_derivable"]:
+        o.append("- `conformance/runs/ledger.jsonl` holds no rows. Capture a run with "
+                 "`./run.sh --record` (recording is never implicit: a run that silently "
+                 "records itself is how a bad run becomes a permanent false proof), then "
+                 "commit `conformance/runs/<run-id>/` **with its `run.log`** and the "
+                 "ledger. The log is the load-bearing half — Chainsaw's JUnit carries no "
+                 "stdout, so without it a self-skip is indistinguishable from a pass.")
+    else:
+        o.append(f"- {s['ledger_rows']} row(s) in `conformance/runs/ledger.jsonl`. "
+                 f"Per-test evidence for every non-`not-derivable` requirement below; "
+                 f"full rows (env, chainsaw/kube versions, sentinel, note) are in "
+                 f"`status.json` under `by_requirement.<R-ID>.proof.per_test`.\n")
+        o.append("| Requirement | Proof | Test | Ledger state | Sentinel | Run |")
+        o.append("|---|---|---|---|---|---|")
+        for rid, v in rep["by_requirement"].items():
+            pr = v["proof"]
+            if pr["state"] == "not-derivable":
+                continue
+            for d in pr["per_test"]:
+                o.append(f"| {rid} | {pr['state']} | `{d['test']}` | {d['state']} | "
+                         f"`{d.get('sentinel') or '—'}` | {d.get('run_id') or '—'} |")
+        stale = [(rid, v) for rid, v in rep["by_requirement"].items()
+                 if v["proof"]["state"] == "stale-proof"]
+        if stale:
+            o.append("\n### Stale proofs — a real pass that no longer describes this tree\n")
+            o.append("These requirements DID pass, but their implementing manifests or "
+                     "their verifying test changed in a commit the recorded run predates. "
+                     "A stale proof is not green; re-run with `--record`.\n")
+            for rid, v in stale:
+                pr = v["proof"]
+                o.append(f"- **{rid}** — {pr['reason']}")
+        skipped = [(rid, v) for rid, v in rep["by_requirement"].items()
+                   if v["proof"]["state"] == "skipped"]
+        if skipped:
+            o.append("\n### SELF-SKIPPED green — recorded, and deliberately NOT proof\n")
+            o.append("Chainsaw reported `passed` for these, because the test exited 0 "
+                     "after emitting its skip token: a precondition (Longhorn "
+                     "StorageClass, Velero CRDs, >=3 Ready nodes) was absent, so the "
+                     "behaviour was never exercised. This is exactly the row a "
+                     "JUnit-only ledger would have recorded as green.\n")
+            for rid, v in skipped:
+                toks = ", ".join(sorted({d.get("sentinel") or "?"
+                                         for d in v["proof"]["per_test"]
+                                         if d["state"] == "skipped"}))
+                o.append(f"- **{rid}** — skip token(s): `{toks}`")
+
     o.append("\n## Advisory — untracked local run artifact (NOT counted)\n")
     if advisory:
         o.append(f"- `{advisory['path']}` (untracked, git-ignored), last written "
                  f"{advisory['mtime']}: {advisory['tests']} testcase(s), "
                  f"{advisory['failures']} failure(s), {advisory['skipped']} skipped.")
         o.append(f"- requirement IDs it names: {', '.join(advisory['requirements']) or '—'}")
-        o.append("- This is one developer's local workspace, not repo state. It is shown "
+        o.append("- This is one developer's local workspace, not repo state, and it is NOT "
-                 "for orientation only and contributes to NO status above.")
+                 "the ledger. It is shown for orientation only and contributes to NO "
+                 "status above — a JUnit alone cannot tell a self-skip from a pass.")
     else:
         o.append("- none present.")
     return "\n".join(o) + "\n"
@@ -841,6 +1254,186 @@ def _selftest() -> int:
               classify_test({"name": "s", "path": str(f), "layer": "L2"})["state"]
               == "scaffold")
 
+        # === THE BUG THIS DETECTOR WAS REWRITTEN FOR ==============================
+        # A whole-file `"TODO" in text` mis-classified `l0-static/placeholder-guard-
+        # coupling` — a 32-assertion executable check whose SUBJECT is placeholder
+        # detection, so it must name the idioms it hunts — as a scaffold, and through it
+        # reported the SAFETY-CRITICAL R-STATE-03 as `exec=1 scaffold=1`. These
+        # assertions pin BOTH directions: a marker held as DATA must not downgrade, and a
+        # real placeholder body must still be caught. Both halves are load-bearing; the
+        # second is what stops this fix from degrading into "always executable".
+        yml = ("metadata:\n  annotations:\n"
+               "    conformance.home.k3s/executable: \"true\"\n"
+               "spec:\n  steps:\n    - script:\n"
+               "        content: bash ./check.sh\n")
+        f.write_text(yml)
+        detector = td2 / "check.sh"
+        # Shaped exactly like the real check: a prose header that discusses the idioms,
+        # then executable code holding them as a vocabulary list.
+        detector.write_text(
+            "#!/usr/bin/env bash\n"
+            "# L0 static check — placeholder vocabulary\n"
+            "#   * SEED — loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER)\n"
+            "# STATUS: EXECUTABLE. A pure offline assertion script.\n"
+            "set -euo pipefail\n"
+            "python3 - <<'EOF'\n"
+            "SEED = [\"TBD\", \"TODO\", \"FIXME\", \"PLACEHOLDER\"]\n"
+            "assert any(w in open('/etc/hostname').read().upper() for w in SEED) is False\n"
+            "EOF\n")
+        c = classify_test({"name": "s", "path": str(f), "layer": "L0"})
+        check("a marker in HEADER PROSE does not downgrade an executable check",
+              c["state"] == "executable" and not c["annotation_conflict"])
+        check("a marker held as a bare quoted DATUM does not downgrade (the R-STATE-03 bug)",
+              c["scaffold_reason"] == "")
+
+        # ... and the protection is intact: a real placeholder body is still caught, in
+        # each of the three shapes the corpus actually uses.
+        detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
+                            "echo \"TODO: assert the PVC is bound\"\n")
+        c = classify_test({"name": "s", "path": str(f), "layer": "L0"})
+        check("a TODO carrying a message in check.sh IS still a scaffold",
+              c["state"] == "scaffold" and c["annotation_conflict"])
+        check("the scaffold reason names the file and the offending line",
+              c["scaffold_reason"].startswith("check.sh:")
+              and "TODO: assert the PVC is bound" in c["scaffold_reason"])
+        detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
+                            "# TODO(T-4.1) write the hermetic sink fixture\ntrue\n")
+        check("a TODO(T-x.y) roadmap marker in an inline body comment IS still a scaffold",
+              classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"]
+              == "scaffold")
+        detector.write_text("#!/usr/bin/env bash\nset -euo pipefail\n"
+                            "echo \"FIXME the extraction before trusting this\"\n")
+        check("an unpunctuated FIXME carrying trailing prose IS still a scaffold",
+              classify_test({"name": "s", "path": str(f), "layer": "L0"})["state"]
+              == "scaffold")
+        # The dedicated declaration slot outranks a body that would otherwise look real —
+        # this is how all three genuine scaffolds in the tree declare themselves.
+        detector.unlink()
+        f.write_text("# L3 FAULT-INJECTION SCAFFOLD — x\n"
+                     "# STATUS: scaffold — needs T-1.4 (Flux bootstrapped).\n" + yml)
+        c = classify_test({"name": "s", "path": str(f), "layer": "L3"})
+        check("a `# STATUS: scaffold` header declaration IS a scaffold, body notwithstanding",
+              c["state"] == "scaffold" and c["annotation_conflict"])
+        check("`# STATUS: EXECUTABLE` is not read as a scaffold declaration",
+              scaffold_marker("# STATUS: EXECUTABLE. `check.sh` is a pure offline "
+                              "assertion script; covers no scaffold.\nset -e\n") == "")
+
+    # ---------------------------------------------------------------- PROOF / ledger
+    # These are the assertions that stand between this report and a fake green.
+    check("empty ledger parses to no rows (pre-ledger behaviour is preserved)",
+          read_ledger(Path("/nonexistent/ledger.jsonl")) == [])
+
+    def row(**kw):
+        base = {"ts_utc": "2026-09-01T12:00:00Z", "git_sha": "deadbeef", "env": "dev",
+                "chainsaw_version": "v0.2.12", "kube_version": "v1.36.4", "layer": "L3",
+                "test": "dr-restore", "requirements": ["R-STATE-03"],
+                "junit_result": "passed", "sentinel": None, "proof": "skipped",
+                "run_id": "20260901T120000Z-deadbee", "note": ""}
+        base.update(kw)
+        return base
+
+    verifiers = [{"name": "dr-restore",
+                  "path": "conformance/tests/l3-fault-injection/dr-restore/"
+                          "chainsaw-test.yaml"}]
+
+    with tempfile.TemporaryDirectory() as td:
+        gr = Path(td)
+        env = {**os.environ, "GIT_AUTHOR_NAME": "t", "GIT_AUTHOR_EMAIL": "t@t",
+               "GIT_COMMITTER_NAME": "t", "GIT_COMMITTER_EMAIL": "t@t"}
+
+        def git(*a):
+            return subprocess.run(["git", "-C", str(gr), *a], capture_output=True,
+                                  text=True, env=env)
+
+        git("init", "-q", "-b", "main")
+        (gr / "platform").mkdir()
+        (gr / "platform" / "velero.yaml").write_text("kind: X  # R-STATE-03\n")
+        git("add", "-A"); git("commit", "-qm", "one")
+        sha_a = git("rev-parse", "HEAD").stdout.strip()
+        (gr / "platform" / "velero.yaml").write_text("kind: X  # R-STATE-03 v2\n")
+        git("add", "-A"); git("commit", "-qm", "two")
+        sha_b = git("rev-parse", "HEAD").stdout.strip()
+
+        oracle = _GitOracle(gr)
+        impl = [{"path": "platform/velero.yaml"}]
+
+        # *** THE CASE THIS WHOLE FEATURE EXISTS FOR ***
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="skipped", junit_result="passed",
+                                              sentinel="DR_RESTORE_SKIPPED_OK",
+                                              git_sha=sha_b)]))
+        check("a SELF-SKIP row on a GREEN JUnit is 'skipped', NEVER 'proven'",
+              pr["state"] == "skipped")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b,
+                                              sentinel="DR_RESTORED_OK")]))
+        check("a proven row at the CURRENT sha is honoured", pr["state"] == "proven")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_a,
+                                              sentinel="DR_RESTORED_OK")]))
+        check("STALENESS: proven at a sha the implementation has moved past degrades "
+              "to stale-proof", pr["state"] == "stale-proof")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha=f"{sha_b}-dirty",
+                                              sentinel="DR_RESTORED_OK")]))
+        check("a run recorded from a DIRTY tree is never honoured as proof",
+              pr["state"] == "stale-proof")
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(proof="proven", git_sha="0" * 40)]))
+        check("a git_sha this clone cannot resolve degrades, never upgrades",
+              pr["state"] == "stale-proof")
+
+        for bad, expect in (("failed", "failed"),
+                            ("junit-skipped", "junit-skipped"),
+                            ("unknown-sentinel-absent", "unknown"),
+                            ("unknown-no-sentinel-declared", "unknown"),
+                            ("something-new-from-the-future", "unknown")):
+            pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                              latest_by_test([row(proof=bad, git_sha=sha_b)]))
+            check(f"ledger state {bad!r} maps to {expect!r}, never to proven",
+                  pr["state"] == expect)
+
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl, {})
+        check("no ledger row at all stays not-derivable",
+              pr["state"] == "not-derivable")
+
+        # Two verifying tests, only one proven: the requirement must NOT read proven.
+        two = verifiers + [{"name": "data-survival",
+                            "path": "conformance/tests/l3-fault-injection/"
+                                    "data-survival/chainsaw-test.yaml"}]
+        pr = derive_proof(oracle, "R-STATE-03", two, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b)]))
+        check("a requirement is proven only when EVERY verifying test is",
+              pr["state"] == "not-derivable")
+        pr = derive_proof(oracle, "R-STATE-03", two, impl,
+                          latest_by_test([row(proof="proven", git_sha=sha_b),
+                                          row(test="data-survival", proof="skipped",
+                                              git_sha=sha_b)]))
+        check("worst-of wins across verifying tests (skip drags proven down)",
+              pr["state"] == "skipped")
+
+        # Latest row wins, not best-ever: a test that regressed to self-skipping must
+        # stop reading as proven.
+        pr = derive_proof(oracle, "R-STATE-03", verifiers, impl,
+                          latest_by_test([row(ts_utc="2026-09-01T10:00:00Z",
+                                              proof="proven", git_sha=sha_b),
+                                          row(ts_utc="2026-09-02T10:00:00Z",
+                                              proof="skipped", git_sha=sha_b)]))
+        check("the LATEST ledger row wins, not the best one ever recorded",
+              pr["state"] == "skipped")
+
+        # A malformed ledger line is skipped, never guessed at.
+        lf = gr / "ledger.jsonl"
+        lf.write_text('{"test":"a","proof":"proven"}\nnot json at all\n\n'
+                      '{"proof":"proven"}\n')
+        rows = read_ledger(lf)
+        check("malformed / test-less ledger lines are dropped, not guessed at",
+              len(rows) == 1 and rows[0]["test"] == "a")
+
     if not ok:
         return 1
     sys.stderr.write("status.py selftest: PASS\n")
@@ -865,6 +1458,10 @@ def main() -> int:
                     help="exit 1 on defects this tool owns: registry/spec drift, "
                          "executable-annotation conflicts, or a safety-critical "
                          "requirement whose only L3 test is a scaffold")
+    ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
+                    help=f"committed proof-of-pass ledger written by lib/record_run.py "
+                         f"(default {DEFAULT_LEDGER}). Absent or empty => every "
+                         f"requirement stays `not-derivable`, as before the ledger existed")
     ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
     args = ap.parse_args()
 
@@ -895,7 +1492,12 @@ def main() -> int:
     applied, nodes = reconciled_files(repo, entry)
     gitops_roots = {n["resolved"].split("/", 1)[0] for n in nodes if n["exists"]}
     impl = scan_implementation(repo, tracked_files(repo))
-    rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env)
+    ledger_rows = read_ledger(args.ledger)
+    rep = build(repo, registry, tests, impl, applied, gitops_roots, spec_ids, args.env,
+                ledger_rows)
+    rep["ledger"] = {"path": os.path.relpath(args.ledger, REPO_ROOT),
+                     "rows": len(ledger_rows),
+                     "present": args.ledger.is_file()}
     rep["flux_graph"] = nodes
     rep["gitops_roots"] = sorted(gitops_roots)
     rep["reconciled_file_count"] = len(applied)
@@ -915,7 +1517,11 @@ def main() -> int:
           f"(+{s['wired_or_partial_and_testable'] - s['wired_and_testable']} partial) | "
           f"scaffold-only {s['verify_scaffold_only']} | "
           f"safety-critical w/o executable L3 "
-          f"{s['safety_critical_without_executable_l3']} | PROVEN: not derivable")
+          f"{s['safety_critical_without_executable_l3']} | "
+          + (f"PROVEN: {s['proven']} (stale {s['proof_stale']}, self-skipped "
+             f"{s['proof_skipped']}, failed {s['proof_failed']}, unknown "
+             f"{s['proof_unknown']}) from {s['ledger_rows']} ledger row(s)"
+             if s["proven_derivable"] else "PROVEN: not derivable (empty run ledger)"))
     print(f"[status] wrote {args.out/'status.md'} and {args.out/'status.json'}")
 
     rc = 0
@@ -926,8 +1532,11 @@ def main() -> int:
                              f"{h['registry_not_in_spec']} {h['spec_not_in_registry']}\n")
             rc = 1
         if h["annotation_conflicts"]:
-            sys.stderr.write(f"[status] STRICT: tests annotated executable with TODO "
+            sys.stderr.write(
-                             f"bodies: {h['annotation_conflicts']}\n")
+                "[status] STRICT: tests annotated executable that declare a placeholder "
+                "body: " + "; ".join(f"{k} ({v})" for k, v in
+                                     sorted(h["annotation_conflict_reasons"].items()))
+                + "\n")
             rc = 1
         if rep["safety_critical_without_executable_l3"]:
             sys.stderr.write(f"[status] STRICT: safety-critical requirement(s) whose L3 "

conformance/run.sh

@@ -10,16 +10,48 @@
 #   ./run.sh --requirements R-NET-01,R-NET-06   # merge-gate subset (tagged)
 #   ./run.sh --layer l2               # one layer (l0|l1|l2|l3|meta|all; l0/l1 need no cluster)
 #   ./run.sh --requirements R-NET-01,R-NET-02 --layer l2   # intersect: tagged AND in layer
+#   ./run.sh --layer l0 --requirements R-X --include-scope whole-repo
+#                                     # UNION: the tagged subset PLUS every test that
+#                                     #   DECLARES `conformance.home.k3s/scope: whole-repo`.
+#                                     #   A whole-repo invariant must run on EVERY gate
+#                                     #   invocation — see lib/common.sh's scope block for
+#                                     #   why, and why a SKIP of one is exit 3, not green.
 #   ./run.sh --coverage-only          # regenerate coverage report only (no cluster)
 #   ./run.sh --status                 # DERIVED STATUS report only (no cluster): per
 #                                     #   requirement, is it implemented, is it actually
 #                                     #   reconciled by Flux, is its test executable or a
-#                                     #   scaffold. Proof-of-pass is deliberately NOT
+#                                     #   scaffold, and whether the committed run ledger
-#                                     #   derived — see lib/status.py.
+#                                     #   (runs/ledger.jsonl) actually proves it — see
+#                                     #   lib/status.py. No ledger row => not-derivable.
 #   ./run.sh --lint                   # structural-validate test defs (no cluster)
+#   ./run.sh --print-selection        # print the test dirs the CURRENT selection resolves
+#                                     #   to, one per line, and exit. Runs no check. This is
+#                                     #   how a test can PROVE the selection includes it
+#                                     #   (tests/l0-static/whole-repo-invariants-wired) without
+#                                     #   executing the suite and recursing into itself.
 #   ./run.sh --strict-coverage        # hygiene gate: fail on tag/annotation drift
 #   ./run.sh --requirements R-NET-01  # ALSO asserts those IDs are covered (merge gate)
 #   ./run.sh --require-all            # phase-exit: every requirement must be covered
+#   ./run.sh --record                 # RUN AND CAPTURE: tee the whole run to
+#                                     #   runs/<UTC>-<short-sha>/run.log, copy the JUnit
+#                                     #   beside it, and append one ledger row per
+#                                     #   testcase to runs/ledger.jsonl (lib/record_run.py)
+#
+# --record — PROOF-OF-PASS, CAPTURED AT RUN TIME (why it is a separate flag)
+# --------------------------------------------------------------------------
+# Without it, nothing about a green run ever reaches repo state: _artifacts/ is gitignored
+# and the root .gitignore kills *.junit.xml globally, which is why `--status` reports
+# PROVEN 0 for all 51 requirements. `--record` writes the one tracked thing that fixes
+# that — `conformance/runs/` (see conformance/.gitignore for the negations).
+#
+# RECORDING IS DELIBERATELY EXPLICIT. A run that silently records itself is how a bad run
+# becomes a permanent false proof, so the plain `./run.sh` writes no ledger row at all.
+#
+# BOTH FILES ARE KEPT, and the LOG is the load-bearing one. The L2/L3 tests self-skip
+# GREEN: when a precondition is missing they echo e.g. `DR_RESTORE_SKIPPED_OK` and exit 0,
+# and Chainsaw records `passed` exactly as on a real run. Chainsaw's JUnit carries no
+# stdout, so the sentinel token in run.log is the ONLY thing that separates proven from
+# skipped. A JUnit committed on its own would record a skip as proof of R-STATE-03.
 #
 # Artifacts (always written under conformance/_artifacts/, gitignored):
 #   _artifacts/report.junit.xml        JUnit (Chainsaw output, or skipped-scaffold shim)
@@ -28,6 +60,11 @@
 #   _artifacts/status/status.md        human requirement-status report (--status)
 #   _artifacts/status/status.json      machine requirement-status report (--status)
 #
+# Run ledger (TRACKED — commit these; --record only):
+#   runs/<UTC>-<short-sha>/run.log            full tee'd run output (carries the sentinels)
+#   runs/<UTC>-<short-sha>/report.junit.xml   the JUnit for that same run
+#   runs/ledger.jsonl                         append-only, one row per testcase per run
+#
 # Phase 0: no cluster exists. The behavior tests are valid scaffolds that cannot
 # execute yet, so run.sh emits a *skipped* JUnit and a coverage report, and exits
 # green — the harness is proven reproducible while later phases fill in behavior.
@@ -47,27 +84,31 @@ STATUS_OUT="${ARTIFACTS}/status"
 JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
 PYTHON="${PYTHON:-python3}"
 
+# Every argument as given, so `--record` can re-exec this script unchanged (minus itself)
+# under `tee`. Captured BEFORE parsing so nothing is normalized away.
+ORIG_ARGS=("$@")
+
 LAYER="all"
 REQS=""
+INCLUDE_SCOPES=""   # --include-scope: scopes unioned into the selection (see lib/common.sh)
 MODE="run"          # run | coverage-only | status | lint
 STRICT_COVERAGE=0
 REQUIRE_ALL=0
 FORCE_NO_CLUSTER=0
+RECORD=0
 
 # Documented, out-of-scope coverage gaps exempt from the completeness gate: requirements
 # owned by another workstream that WS-TEST does not yet scaffold. Listed here so the
 # exemption is explicit and reviewable, not silent.
-#   R-ENV-02   (dev->prod promote) — owned by WS-INFRA/WS-GITOPS; unscaffolded (a GAP in
+#
-#                                    coverage.md). The coarse path->requirement map keys
+# EMPTY as of 2026-09-01. R-ENV-02 and R-GITOPS-01 were the last two entries — exempted
-#                                    clusters/* AND ci/* to it and the R-ENV-02 doc token
+# because they were unscaffolded while the coarse path->requirement map keys clusters/*
-#                                    appears in platform/kustomization.yaml's header, so ANY
+# and ci/* to them, so any overlay/CI edit would have false-blocked. They now have a real
-#                                    infra/overlay/CI edit (e.g. enabling a component, bumping
+# verifying L0 test (tests/l0-static/flux-postbuild-substitution), which asserts the
-#                                    SPEC_PIN) would otherwise false-block on it.
+# substitution mechanism that IS the "authored once, promoted" contract, so they are
-#   R-GITOPS-01 (state in git/Flux) — same: owned by WS-GITOPS, unscaffolded, and the ci/*
+# covered rather than exempt and the completeness gate can hold them to it. That test
-#                                    path-map keys to it, so editing any ci/ file (SPEC_PIN
+# names its own limits in its header: it proves the STATIC property (no manifest renders
-#                                    adoption included) would false-block.
+# an undefined variable), not the live-cluster half of either requirement.
-# These are visible GAPs, not hidden — removing them from this list is the right move the
-# moment WS-TEST adds their verifying tests.
 #
 # FOLLOW-UP (WS-TEST) — HALF DONE. The debt was "make the matcher stop keying on doc-header
 # tokens so real requirements need not be exempted". The CONTENT-SCAN half is now fixed in
@@ -83,20 +124,25 @@ FORCE_NO_CLUSTER=0
 # R-STATE-02 (node-loss) WAS listed here (owned by WS-STORAGE, "delivered with Longhorn/infra")
 # but now has a real verifying L3 test — tests/l3-fault-injection/node-loss — so it is no
 # longer a gap and has been removed from this list; the coverage report counts it as covered.
-DOCUMENTED_GAPS="R-ENV-02 R-GITOPS-01"
+DOCUMENTED_GAPS=""
 
-usage() { sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; }
+# Print the header block (line 2 through the closing `####` rule, which is dropped) as
+# help. Bounded by the rule rather than a fixed line number so the header can grow.
+usage() { sed -n '2,/^#####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; }
 
 while [ $# -gt 0 ]; do
   case "$1" in
     --layer)          LAYER="$2"; shift 2 ;;
     --requirements)   REQS="$(conf_normalize_reqs "$2")"; shift 2 ;;
+    --include-scope)  INCLUDE_SCOPES="${INCLUDE_SCOPES:+${INCLUDE_SCOPES},}$2"; shift 2 ;;
     --coverage-only)  MODE="coverage-only"; shift ;;
     --status)         MODE="status"; shift ;;
     --lint)           MODE="lint"; shift ;;
+    --print-selection) MODE="print-selection"; shift ;;
     --strict-coverage) STRICT_COVERAGE=1; shift ;;
     --require-all)    REQUIRE_ALL=1; shift ;;
     --no-cluster)     FORCE_NO_CLUSTER=1; shift ;;
+    --record)         RECORD=1; shift ;;
     -h|--help)        usage; exit 0 ;;
     *) echo "unknown arg: $1" >&2; usage; exit 2 ;;
   esac
@@ -106,9 +152,69 @@ mkdir -p "${ARTIFACTS}" "${COVERAGE_OUT}" "${STATUS_OUT}"
 
 log() { printf '\033[1m[run.sh]\033[0m %s\n' "$*"; }
 
+###############################################################################
+# --record: capture this run as committed evidence, then hand it to the ledger.
+#
+# Implemented as a ONE-LEVEL re-exec of this same script under `tee`, so the run itself is
+# byte-for-byte the ordinary run (same code path, same artifacts) and the log is the exact
+# operator-visible output — no second, differently-behaving "recording mode" to drift.
+# CONF_RECORD_CHILD guards against recursion.
+#
+# The recorder runs REGARDLESS of the run's exit status: a red run must be recorded as red
+# (that is evidence too), and run.sh still exits with the child's real rc, so no gate is
+# softened by recording. `${PIPESTATUS[0]}` is what carries that rc past `tee`.
+###############################################################################
+if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
+  case "${MODE}" in
+    run) ;;
+    # --record is about EVIDENCE OF A RUN. --coverage-only/--status/--lint execute no
+    # test, so a ledger row from one of them would be a row about nothing.
+    *) echo "--record captures a TEST RUN; --${MODE} executes none." >&2; exit 2 ;;
+  esac
+  RUN_SHA="$(git -C "${CONF_ROOT}" rev-parse --short HEAD 2>/dev/null || echo nogit)"
+  RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${RUN_SHA}"
+  RUN_DIR="${CONF_ROOT}/runs/${RUN_ID}"
+  mkdir -p "${RUN_DIR}"
+  # Drop any JUnit left behind by a PREVIOUS run before starting, so this run dir can only
+  # ever capture a report THIS run produced. Copying a stale artifact would pair someone
+  # else's result with this run's log — a fabricated pairing, which is precisely the class
+  # of false proof the ledger exists to prevent.
+  rm -f "${JUNIT_OUT}"
+  # Strip --record from the argv handed to the child; everything else passes through.
+  CHILD_ARGS=()
+  for a in ${ORIG_ARGS[@]+"${ORIG_ARGS[@]}"}; do
+    [ "$a" = "--record" ] || CHILD_ARGS+=("$a")
+  done
+  log "RECORD: capturing this run to runs/${RUN_ID}/ (run.log + report.junit.xml)"
+  set +e
+  CONF_RECORD_CHILD=1 "${BASH_SOURCE[0]}" ${CHILD_ARGS[@]+"${CHILD_ARGS[@]}"} 2>&1 \
+    | tee "${RUN_DIR}/run.log"
+  RUN_RC=${PIPESTATUS[0]}
+  set -e
+  if [ -f "${JUNIT_OUT}" ]; then
+    cp "${JUNIT_OUT}" "${RUN_DIR}/report.junit.xml"
+  else
+    log "RECORD: no ${JUNIT_OUT} was produced — the ledger step will SKIP (never green)."
+  fi
+  # Explicit, separate step: read the JUnit + the log and append one row per testcase.
+  # rc 3 = inputs absent (nothing recorded, not green); it must not mask the run's own rc.
+  # CONF_LEDGER exists so the harness's own tests can exercise this path without writing
+  # to the real ledger; operators should never set it.
+  "${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
+    --run-dir "${RUN_DIR}" ${CONF_ENV:+--env "$CONF_ENV"} \
+    --ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
+  log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
+  log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
+  exit "${RUN_RC}"
+fi
+
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
-  local flags=(--allow-gap "${DOCUMENTED_GAPS}")
+  # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
+  # requirement is exempt, and handing the flag an empty string would be a silent no-op
+  # that reads like an exemption in the CLI log.
+  local flags=()
+  [ -n "${DOCUMENTED_GAPS}" ] && flags+=(--allow-gap "${DOCUMENTED_GAPS}")
   [ "${STRICT_COVERAGE}" = 1 ] && flags+=(--strict)
   # A tagged subset run is also the merge gate: the touched requirements MUST be covered.
   [ -n "${REQS}" ] && flags+=(--require "${REQS}")
@@ -133,7 +239,7 @@ gen_status() {
     --registry "${CONF_ROOT}/requirements/registry.yaml" \
     --tests "${CONF_ROOT}/tests" \
     --out "${STATUS_OUT}" \
-    --env "${CONF_ENV:-dev}" \
+    ${CONF_ENV:+--env "$CONF_ENV"} \
     "$@"
 }
 
@@ -159,11 +265,38 @@ lint_tests() {
 }
 
 # --- Select which test directories Chainsaw should run --------------------------------
+# Requirement-scoped selection, then the DECLARED-SCOPE union on top:
+#
+#   (--requirements ∩ --layer)  ∪  (--include-scope ∩ --layer)
+#
+# The left half is R-TEST-03's touched subset. The right half is the fix for its blind
+# spot: an invariant that quantifies over the whole tracked tree cannot be selected BY the
+# requirements a change touches, because the change most likely to break it touches none of
+# them. See lib/common.sh's scope block for the incidents. The union is fail-closed by
+# construction — it can only ADD tests to a run, never remove one.
 select_dirs() {
-  # --requirements AND --layer together => intersection (tests carrying the tag
+  {
-  # AND living in that layer subtree). Either alone keeps its prior meaning, so
+    requirement_scoped_dirs
-  # this is backward compatible (LAYER defaults to "all"). Both helpers emit
+    # The scope union is still LAYER-bounded: `--layer l0` must not pull a whole-repo-scoped
-  # sorted, de-duplicated dirs, so `comm -12` yields the intersection.
+    # L2 test into an offline run. With no --layer (all), nothing is bounded away.
+    if [ -n "${INCLUDE_SCOPES}" ]; then
+      if [ "${LAYER}" != all ]; then
+        comm -12 \
+          <(conf_dirs_for_scopes "${INCLUDE_SCOPES}") \
+          <(conf_dirs_for_layer "${LAYER}")
+      else
+        conf_dirs_for_scopes "${INCLUDE_SCOPES}"
+      fi
+    fi
+  } | sort -u
+}
+
+# The pre-existing requirement/layer selection, unchanged:
+# --requirements AND --layer together => intersection (tests carrying the tag
+# AND living in that layer subtree). Either alone keeps its prior meaning, so
+# this is backward compatible (LAYER defaults to "all"). Both helpers emit
+# sorted, de-duplicated dirs, so `comm -12` yields the intersection.
+requirement_scoped_dirs() {
   if [ -n "${REQS}" ] && [ "${LAYER}" != all ]; then
     comm -12 \
       <(conf_dirs_for_requirements "${REQS}") \
@@ -304,6 +437,12 @@ layers_are_offline_only() {
   return 0
 }
 
+# Markdown one-liner per executed offline check, for the CI step summary. Written next to
+# the JUnit; ci/l0.sh + ci/l1.sh splice it into $GITHUB_STEP_SUMMARY so the gate output still
+# NAMES every check that ran (that is what the deleted per-check gate stages did, one hard-coded
+# `summary` line each — now generated for all of them, whole-repo-scoped ones marked as such).
+: "${OFFLINE_SUMMARY_OUT:=${ARTIFACTS}/offline-summary.md}"
+
 run_offline_checks() {
   local -a dirs
   mapfile -t dirs < <(offline_selected_dirs)
@@ -311,14 +450,30 @@ run_offline_checks() {
     log "No L0/L1 offline checks match the selection (layer=${LAYER} reqs='${REQS}')."
     return 0
   fi
+  # Declared scopes are passed through so the runner knows WHICH cases must not be allowed to
+  # report a SKIP as green (rc 3). See lib/common.sh's scope block and run_l1_offline.py.
+  local -a scope_args=() s
+  for s in ${INCLUDE_SCOPES//,/ }; do scope_args+=(--unwired-scope "$s"); done
   log "Running ${#dirs[@]} offline L0/L1 check(s) (no cluster; docs/14)."
+  [ -n "${INCLUDE_SCOPES}" ] && log "  unconditional (declared scope ${INCLUDE_SCOPES}): $(conf_dirs_for_scopes "${INCLUDE_SCOPES}" | xargs -n1 basename 2>/dev/null | tr '\n' ' ')"
   "${PYTHON}" "${CONF_ROOT}/lib/run_l1_offline.py" \
-    --junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" "${dirs[@]}"
+    --junit "${JUNIT_OUT}" --label-prefix "${CONF_LABEL_PREFIX}" \
+    --scope-annotation "${CONF_SCOPE_ANNOTATION}" \
+    --summary-out "${OFFLINE_SUMMARY_OUT}" \
+    ${scope_args[@]+"${scope_args[@]}"} "${dirs[@]}"
 }
 
 ###############################################################################
 # Main
 ###############################################################################
+# --print-selection is deliberately handled FIRST: it resolves the selection and exits,
+# writing nothing and running no check. A test that asserts "the gate's selection cannot
+# filter me out" needs to ask that question without executing the suite it is part of.
+if [ "${MODE}" = "print-selection" ]; then
+  select_dirs
+  exit 0
+fi
+
 log "Coverage report first (always)"
 gen_coverage
 
@@ -348,17 +503,26 @@ esac
 # L0 static + L1 rendering are cluster-independent (docs/14): run the selected offline
 # checks and treat their result as authoritative REGARDLESS of cluster availability. An
 # offline check failure is a real failure (nothing cluster-gated about it), so it always blocks.
+# rc 3 is the harness-wide SKIPPED/not-run code (ci/README.md): NOT a pass, NOT a fail. The
+# offline runner returns it when a test carrying a --unwired-scope scope SKIPPED — a whole-repo
+# invariant reporting "nothing to audit" means the subtree it audits is gone, which the gate
+# must see as UNWIRED. It is propagated verbatim so ci/l0.sh can set the gate's own rc=3,
+# exactly as the deleted per-check stages did.
 OFFLINE_RC=0
 run_offline_checks || OFFLINE_RC=$?
 if layers_are_offline_only; then
   # Offline-only run (the ci/l0.sh + ci/l1.sh merge-gate paths): the offline result IS the result.
-  if [ "${OFFLINE_RC}" -eq 0 ]; then
+  case "${OFFLINE_RC}" in
-    log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md"
+    0) log "Done (offline ${LAYER}). JUnit: ${JUNIT_OUT} | Coverage: ${COVERAGE_OUT}/coverage.md" ;;
-  else
+    3) log "Offline ${LAYER}: a scope-declared check SKIPPED — UNWIRED, not green (rc=3). JUnit: ${JUNIT_OUT}" ;;
-    log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}"
+    *) log "Offline ${LAYER} checks FAILED (rc=${OFFLINE_RC}). JUnit: ${JUNIT_OUT}" ;;
-  fi
+  esac
   exit "${OFFLINE_RC}"
 fi
+if [ "${OFFLINE_RC}" -eq 3 ]; then
+  log "Offline L0/L1: a scope-declared check SKIPPED — UNWIRED, not green (rc=3)."
+  exit 3
+fi
 if [ "${OFFLINE_RC}" -ne 0 ]; then
   log "Offline L0/L1 checks FAILED (rc=${OFFLINE_RC}) — blocking (not cluster-gated)."
   exit "${OFFLINE_RC}"

conformance/runs/20260901T235117Z-e9a4b66/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="node-loss" classname="" time="131.090121" />
+  </testsuite>
+</testsuites>

conformance/runs/20260901T235117Z-e9a4b66/run.log

@@ -0,0 +1,232 @@
+[run.sh] Coverage report first (always)
+[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/node-loss
+=== PAUSE chainsaw/node-loss
+=== CONT  chainsaw/node-loss
+    l.go:52: | 23:51:18 | node-loss | @chainsaw                               | CREATE    | OK    | v1/Namespace @ chainsaw-fond-marlin
+    l.go:52: | 23:51:18 | node-loss | volume-stays-available-across-node-loss | TRY       | BEGIN |
+    l.go:52: | 23:51:18 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -euo pipefail
+        NS="$NAMESPACE"
+        
+        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
+        READY="$(kubectl get nodes \
+          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
+          | grep -c '^True$' || true)"
+        echo "ready nodes: ${READY}"
+        if [ "${READY:-0}" -lt 3 ]; then
+          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        CANARY="node-loss-canary-$(date +%s%N)"
+        
+        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
+        #    writer pod that writes the canary and flushes it to the replicated volume. --
+        kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: survivor
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        apply_pod() {
+          kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: $1
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 3600"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: survivor
+        EOF
+        }
+        
+        apply_pod writer
+        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
+        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
+        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
+        
+        # The victim node is the one the attached volume currently lives on.
+        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
+        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
+        echo "victim node (currently hosts the attached volume): $VICTIM"
+        
+        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
+        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
+        # runner is least-privilege and may not have longhorn.io read rights) — the
+        # data-availability assertion below is the authoritative proof, not this.
+        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        if [ -n "$PVNAME" ]; then
+          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
+            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
+        fi
+        
+        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
+        #    Label it with THIS test's unique namespace so `finally` can always find and
+        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
+        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
+        #    pods on the node are never touched.)
+        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
+        kubectl cordon "$VICTIM"
+        echo "cordoned + labelled victim node: $VICTIM"
+        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
+        echo "evicted writer off the victim node; volume detached from $VICTIM"
+        
+        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
+        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
+        #    the volume there from its replicas on the other nodes. --
+        apply_pod reader
+        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
+        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
+        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
+        if [ "$READER_NODE" = "$VICTIM" ]; then
+          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
+          exit 1
+        fi
+        
+        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
+        echo "volume available + data intact from a surviving node"
+        
+        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
+        # (bounded) for robustness to settle back to available (healthy|degraded — both
+        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
+        # Non-fatal for the same least-privilege reason as the pre-fault log.
+        if [ -n "$PVNAME" ]; then
+          for i in $(seq 1 30); do
+            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
+            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
+            echo "post-fault robustness[$i]=$ROB"
+            case "$ROB" in healthy|degraded) break ;; esac
+            sleep 10
+          done
+        fi
+        
+        echo NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        ready nodes: 3
+        persistentvolumeclaim/survivor created
+        pod/writer created
+        pod/writer condition met
+        wrote canary: node-loss-canary-1788306678898809444
+        victim node (currently hosts the attached volume): k3s-dev-2
+        pre-fault robustness=healthy replicas=3
+        node/k3s-dev-2 labeled
+        node/k3s-dev-2 cordoned
+        cordoned + labelled victim node: k3s-dev-2
+        pod "writer" deleted from chainsaw-fond-marlin namespace
+        evicted writer off the victim node; volume detached from k3s-dev-2
+        pod/reader created
+        pod/reader condition met
+        reader scheduled on surviving node: k3s-dev-1 (victim was k3s-dev-2)
+        recovered='node-loss-canary-1788306678898809444' expected='node-loss-canary-1788306678898809444'
+        volume available + data intact from a surviving node
+        post-fault robustness[1]=healthy
+        NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | TRY       | END   |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | FINALLY   | BEGIN |
+    l.go:52: | 23:52:42 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -uo pipefail
+        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
+          kubectl uncordon "${n#node/}" || true
+          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
+          echo "restored node to service: ${n#node/}"
+        done
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        node/k3s-dev-2 uncordoned
+        node/k3s-dev-2 unlabeled
+        restored node to service: k3s-dev-2
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
+        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
+        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        pod "reader" deleted from chainsaw-fond-marlin namespace
+        persistentvolumeclaim "survivor" deleted from chainsaw-fond-marlin namespace
+        cleanup best-effort; chainsaw namespace teardown finalizes
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 23:52:43 | node-loss | volume-stays-available-across-node-loss | FINALLY   | END   |
+    l.go:52: | 23:52:43 | node-loss | @chainsaw                               | CLEANUP   | BEGIN |
+    l.go:52: | 23:52:43 | node-loss | @chainsaw                               | DELETE    | OK    | v1/Namespace @ chainsaw-fond-marlin
+    l.go:52: | 23:53:29 | node-loss | @chainsaw                               | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/node-loss (131.09s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260902T001907Z-fdc3265/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="dr-restore" classname="" time="149.053701" />
+  </testsuite>
+</testsuites>

conformance/runs/20260902T001907Z-fdc3265/run.log

@@ -0,0 +1,671 @@
+[run.sh] Coverage report first (always)
+[coverage] 42/51 requirements have a scaffolded test (82.4%); 9 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/dr-restore
+=== PAUSE chainsaw/dr-restore
+=== CONT  chainsaw/dr-restore
+    l.go:52: | 00:19:07 | dr-restore | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-cunning-hen
+    l.go:52: | 00:19:07 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | BEGIN |
+    l.go:52: | 00:19:07 | dr-restore | assert-backup-target-is-off-cluster       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        
+        # ---- helpers -------------------------------------------------------
+        # Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
+        # fail loudly; an absent API is a genuine precondition gap and may skip.
+        die_or_skip() {
+          case "$2" in
+            *orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
+              echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
+              echo "$2" >&2
+              exit 1 ;;
+            *)
+              echo "SKIP: $1 not available on this cluster -- $2"
+              echo DR_RESTORE_SKIPPED_OK
+              exit 0 ;;
+          esac
+        }
+        
+        ip2int() {
+          _q="$(echo "$1" | tr '.' ' ')"
+          set -- $_q
+          [ $# -eq 4 ] || { echo -1; return 0; }
+          echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
+        }
+        
+        # in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
+        in_cidr() {
+          case "$2" in *"/"*) ;; *) return 1 ;; esac
+          _net="${2%/*}"; _len="${2#*/}"
+          case "$_len" in ''|*[!0-9]*) return 1 ;; esac
+          [ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
+          _i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
+          [ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
+          [ "$_len" -eq 0 ] && return 0
+          _d=1; _s=0
+          while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
+          [ $(( _i / _d )) -eq $(( _n / _d )) ]
+        }
+        
+        # ---- Gate 1: the replicated storage tier the DR-critical data lives on --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
+          2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
+        if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
+          echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        echo "backup storage locations:"
+        echo "$BSL_LINES"
+        
+        VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
+        
+        # ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
+        # File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
+        # deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
+        # ready node-agent no volume data can be captured, so the drill is not runnable.
+        VREADY=0; NAREADY=0
+        for i in $(seq 1 12); do
+          VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
+          NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
+          echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
+          if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
+          sleep 10
+        done
+        if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
+          echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Pick the BSL this drill will use: prefer the default, and it MUST be
+        #      Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
+        #      gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
+        #      wait for first-validation on a cold start), it does not skip. --------
+        BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
+        for i in $(seq 1 12); do
+          BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+            -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
+          CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
+          [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+          if [ -n "$CAND" ]; then
+            BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
+            BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+            BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
+            BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
+            break
+          fi
+          echo "poll $i: no Available BackupStorageLocation yet"
+          sleep 10
+        done
+        if [ -z "$BSL_NAME" ]; then
+          echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
+          echo "      The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
+          echo "$BSL_LINES" >&2
+          exit 1
+        fi
+        echo "using BSL ${BSL_NS}/${BSL_NAME}  bucket=${BSL_BUCKET}  endpoint=${BSL_URL}"
+        
+        # ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
+        if [ -z "$BSL_URL" ]; then
+          echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
+          echo "      the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
+          echo "      test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
+          exit 1
+        fi
+        HOST="${BSL_URL#*://}"      # strip scheme
+        HOST="${HOST%%/*}"          # strip path
+        HOST="${HOST%%:*}"          # strip port
+        echo "backup endpoint host: $HOST"
+        
+        case "$HOST" in
+          localhost|*.svc|*.svc.cluster.local|*.cluster.local)
+            echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
+            echo "      An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
+            exit 1 ;;
+        esac
+        
+        # Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
+        # off-cluster property -- that is a failure, never a silent pass.
+        case "$HOST" in
+          *[!0-9.]*)
+            # Endpoint given as a name. Try the resolvers that may exist in the
+            # runner image, in order; if none is available or none resolves, the
+            # test FAILS below rather than guessing.
+            EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
+          *) EPIP="$HOST" ;;
+        esac
+        if [ -z "$EPIP" ]; then
+          echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
+          echo "      the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
+          exit 1
+        fi
+        echo "backup endpoint resolves to: $EPIP"
+        
+        # ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
+        case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
+        
+        NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for a in $NODE_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
+        
+        SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $SVC_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
+        
+        POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $POD_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
+        
+        POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for c in $POD_CIDRS; do
+          case "$c" in *:*) continue ;; esac      # IPv6 prefixes are out of scope here
+          if in_cidr "$EPIP" "$c"; then
+            echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
+        
+        # Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
+        # runner may not read): the checks above are the authoritative ones.
+        SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        if [ -n "$SVC_CIDRS" ]; then
+          for c in $SVC_CIDRS; do
+            case "$c" in *:*) continue ;; esac
+            if in_cidr "$EPIP" "$c"; then
+              echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
+              exit 1
+            fi
+          done
+          echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
+        else
+          echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
+        fi
+        
+        echo "OFF_CLUSTER_TARGET_OK  bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | LOG   |
+        === STDOUT
+        backup storage locations:
+        velero default Available aws true http://192.168.1.9:9000 velero
+        poll 1: velero readyReplicas=1 node-agent numberReady=3
+        using BSL velero/default  bucket=velero  endpoint=http://192.168.1.9:9000
+        backup endpoint host: 192.168.1.9
+        backup endpoint resolves to: 192.168.1.9
+        endpoint is not any node address (checked 6 node addresses)
+        endpoint is not any Service ClusterIP (checked 81)
+        endpoint is not any Pod IP (checked 186)
+        endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
+        endpoint is outside every ServiceCIDR (10.43.0.0/16 )
+        OFF_CLUSTER_TARGET_OK  bsl=velero/default endpoint=192.168.1.9 bucket=velero
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | DONE  |
+    l.go:52: | 00:19:11 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | END   |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | TRY       | BEGIN |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        SCHED="$(kubectl get schedules.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
+        if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
+          # Distinguish "Velero absent" (skip, in lock-step with step 1) from
+          # "Velero present but nothing scheduled" (a real R-STATE-03 defect).
+          if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
+            if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
+              echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
+              echo DR_SCHEDULE_SKIPPED_OK
+              exit 0
+            fi
+            echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
+            echo "      R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
+            exit 1
+          fi
+          echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
+          echo DR_SCHEDULE_SKIPPED_OK
+          exit 0
+        fi
+        echo "velero schedules:"
+        echo "$SCHED"
+        # A Schedule that exists but is paused/disabled backs nothing up.
+        if echo "$SCHED" | grep -q 'phase=Enabled'; then
+          echo BACKUPS_ARE_SCHEDULED_OK
+        else
+          echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
+          exit 1
+        fi
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | LOG   |
+        === STDOUT
+        velero schedules:
+        velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
+        velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-01T03:00:27Z
+        BACKUPS_ARE_SCHEDULED_OK
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | DONE  |
+    l.go:52: | 00:19:11 | dr-restore | assert-backups-are-scheduled              | TRY       | END   |
+    l.go:52: | 00:19:11 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | BEGIN |
+    l.go:52: | 00:19:11 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"          # scratch source namespace (destroyed by design)
+        DST="${NS}-dr-dst"          # clean restore target (must not pre-exist)
+        BK="${NS}-dr-backup"
+        RS="${NS}-dr-restore"
+        OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
+        
+        # ---- Re-derive the preconditions (steps do not share shell state). Every
+        #      skip below mirrors step 1 exactly, so the whole test skips or runs
+        #      as a unit -- it can never half-run. -----------------------------
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
+        CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
+        [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+        if [ -z "$CAND" ]; then
+          echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        VNS="$(echo "$CAND" | cut -d' ' -f1)"
+        BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+        if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
+          echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
+        
+        # A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
+        # be free. If either is lying around from an aborted run, remove it before we
+        # claim "clean" -- and fail loudly if it will not go, because a half-deleted
+        # namespace would silently invalidate the destroy proof below.
+        kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
+        
+        # ============================ GIVEN ==============================
+        # A DR-critical stateful workload with known data on the replicated tier.
+        CANARY="dr-canary-$(date +%s%N)"
+        kubectl create namespace "$SRC"
+        kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: dr-data
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: dr-workload
+          labels:
+            app: dr-workload
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 7200"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: dr-data
+        EOF
+        
+        kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
+        printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
+        echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
+        
+        ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
+        [ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
+        echo "original in-cluster PersistentVolume: $ORIG_PV"
+        
+        # ======================= WHEN (part 1): back up off-cluster =======
+        # Created as a Velero Backup CR -- exactly what `velero backup create` does.
+        # defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
+        # the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
+        # ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
+        # own GC still removes this drill's objects from the bucket.
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+          name: ${BK}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          includedNamespaces: [${SRC}]
+          storageLocation: ${BSL_NAME}
+          snapshotVolumes: false
+          defaultVolumesToFsBackup: true
+          ttl: 2h0m0s
+        EOF
+        
+        BPHASE=""
+        for i in $(seq 1 90); do
+          BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: backup phase=${BPHASE:-<pending>}"
+          case "${BPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${BPHASE:-}" != "Completed" ]; then
+          echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
+          exit 1
+        fi
+        echo "backup $BK Completed to ${BSL_NAME}"
+        
+        # ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
+        #      kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
+        PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
+                 -l velero.io/backup-name="$BK" \
+                 -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
+        echo "pod volume backups:"
+        echo "$PVB"
+        if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
+          echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
+          echo "      Resource-only backups cannot restore DR-critical DATA." >&2
+          exit 1
+        fi
+        # A line is "<name> <phase> <snapshotID>"; require Completed AND a
+        # non-empty snapshot ID (an empty ID means nothing reached the repository).
+        GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
+        if [ "${GOOD_PVB:-0}" -lt 1 ]; then
+          echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
+          echo "      the volume data did not reach the off-cluster repository." >&2
+          exit 1
+        fi
+        echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
+        
+        # ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
+        # Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
+        # underlying PersistentVolume (and with it the Longhorn replicas). After this
+        # there is no in-cluster source the restore could quietly read from.
+        kubectl delete namespace "$SRC" --wait=true --timeout=420s
+        for i in $(seq 1 30); do
+          if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for namespace $SRC to disappear"
+          sleep 10
+        done
+        if kubectl get namespace "$SRC" >/dev/null 2>&1; then
+          echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
+          exit 1
+        fi
+        echo "source namespace destroyed"
+        
+        # The longhorn StorageClass reclaims Delete, so the PV should already be gone;
+        # force it if a Retain policy (or a stuck finalizer) left it behind, because a
+        # surviving PV would make the restore prove nothing about off-cluster recovery.
+        kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
+        for i in $(seq 1 30); do
+          if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
+          sleep 10
+        done
+        if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
+          echo "      An in-cluster copy of the data still exists, so a successful restore" >&2
+          echo "      would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
+          exit 1
+        fi
+        echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
+        
+        # Best-effort corroboration (the least-privilege runner may not read longhorn.io):
+        # the Longhorn Volume CR and its replicas should be gone too.
+        if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
+          exit 1
+        fi
+        echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
+        
+        # ============================ RESTORE ============================
+        # Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
+        # the catalog's "restored into a clean state".
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Restore
+        metadata:
+          name: ${RS}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+          includedNamespaces: [${SRC}]
+          namespaceMapping:
+            ${SRC}: ${DST}
+          existingResourcePolicy: none
+        EOF
+        
+        RPHASE=""
+        for i in $(seq 1 90); do
+          RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: restore phase=${RPHASE:-<pending>}"
+          case "${RPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${RPHASE:-}" != "Completed" ]; then
+          echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        echo "restore $RS Completed into $DST"
+        
+        # ============================= THEN ==============================
+        # The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
+        # `restore-wait` init container that blocks until the node-agent has written the
+        # volume contents back, so Ready here already implies the data landed.
+        if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
+          echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
+          kubectl -n "$DST" describe pod dr-workload >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        
+        NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
+        [ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
+        if [ "$NEW_PV" = "$ORIG_PV" ]; then
+          echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
+          echo "      That is an in-cluster survival, not an off-cluster restore." >&2
+          exit 1
+        fi
+        echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
+        
+        GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        if [ "$GOT" != "$CANARY" ]; then
+          echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
+          exit 1
+        fi
+        
+        echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        velero ns=velero bsl=default src=chainsaw-cunning-hen-dr-src dst=chainsaw-cunning-hen-dr-dst
+        namespace/chainsaw-cunning-hen-dr-src created
+        namespace/chainsaw-cunning-hen-dr-src labeled
+        persistentvolumeclaim/dr-data created
+        pod/dr-workload created
+        pod/dr-workload condition met
+        seeded canary: dr-canary-1788308352223263220
+        original in-cluster PersistentVolume: pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745
+        backup.velero.io/chainsaw-cunning-hen-dr-backup created
+        poll 1: backup phase=InProgress
+        poll 2: backup phase=InProgress
+        poll 3: backup phase=Completed
+        backup chainsaw-cunning-hen-dr-backup Completed to default
+        pod volume backups:
+        chainsaw-cunning-hen-dr-backup-l95lw Completed 47fa2737115e17142fb41857cef4e779
+        volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
+        namespace "chainsaw-cunning-hen-dr-src" deleted
+        source namespace destroyed
+        original PersistentVolume pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 is GONE -- no in-cluster copy of the data remains
+        (longhorn Volume CR pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745 not present -- either deleted or not readable by this identity)
+        restore.velero.io/chainsaw-cunning-hen-dr-restore created
+        poll 1: restore phase=InProgress
+        poll 2: restore phase=InProgress
+        poll 3: restore phase=InProgress
+        poll 4: restore phase=Completed
+        restore chainsaw-cunning-hen-dr-restore Completed into chainsaw-cunning-hen-dr-dst
+        pod/dr-workload condition met
+        restored PVC bound to PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
+        restored onto a NEW volume (pvc-63c47973-9c7f-4423-a27e-caa008026e49 != pvc-b9523c15-770e-4fc5-8130-ca2e9d9da745) -- the data came from outside the cluster
+        recovered='dr-canary-1788308352223263220' expected='dr-canary-1788308352223263220'
+        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+        === STDERR
+        Defaulted container "app" out of: app, restore-wait (init)
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | END   |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | BEGIN |
+    l.go:52: | 00:21:16 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -u
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
+        
+        VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
+        
+        # Velero objects first: a DeleteBackupRequest makes Velero remove the backup
+        # DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
+        if [ -n "$VNS" ]; then
+          kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
+          if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
+            kubectl apply -f - <<EOF || true
+        apiVersion: velero.io/v1
+        kind: DeleteBackupRequest
+        metadata:
+          name: ${BK}-delete
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+        EOF
+            for i in $(seq 1 18); do
+              kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
+              sleep 10
+            done
+            # Backstop if Velero could not process the request (the 2h ttl also GCs it).
+            kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
+            kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
+          fi
+          echo "velero objects for $NS cleaned up (ns $VNS)"
+        fi
+        
+        # Scratch namespaces (both, in case the run aborted at any point).
+        for n in "$DST" "$SRC"; do
+          kubectl delete namespace "$n" --ignore-not-found --wait=false || true
+        done
+        # Anything left labelled as ours (defensive; the two names above are the only
+        # namespaces this test ever creates).
+        kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
+        
+        # Released PVs from either namespace's PVCs would otherwise leak storage.
+        for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
+          kubectl delete pv "$pv" --ignore-not-found --wait=false || true
+          echo "released leftover PersistentVolume: $pv"
+        done
+        
+        echo "dr-restore teardown complete (best-effort; every object was created by this test)"
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        restore.velero.io "chainsaw-cunning-hen-dr-restore" deleted from velero namespace
+        deletebackuprequest.velero.io/chainsaw-cunning-hen-dr-backup-delete created
+        velero objects for chainsaw-cunning-hen cleaned up (ns velero)
+        namespace "chainsaw-cunning-hen-dr-dst" deleted
+        namespace "chainsaw-cunning-hen-dr-dst" deleted
+        persistentvolume "pvc-63c47973-9c7f-4423-a27e-caa008026e49" deleted
+        released leftover PersistentVolume: pvc-63c47973-9c7f-4423-a27e-caa008026e49
+        dr-restore teardown complete (best-effort; every object was created by this test)
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 00:21:28 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | END   |
+    l.go:52: | 00:21:28 | dr-restore | @chainsaw                                 | CLEANUP   | BEGIN |
+    l.go:52: | 00:21:28 | dr-restore | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-cunning-hen
+    l.go:52: | 00:21:36 | dr-restore | @chainsaw                                 | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/dr-restore (149.05s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260913T014321Z-19fc8bc/report.junit.xml

@@ -0,0 +1,12 @@
+<?xml version="1.0" ?>
+<testsuites name="home-k3s-conformance" tests="1" failures="0" errors="0" skipped="0">
+  <testsuite name="home-k3s-conformance L0/L1 (offline)" tests="1" failures="0" errors="0" skipped="0">
+    <testcase classname="L1:R-NET-05,R-NET-06,R-NET-07" name="egress-class-render">
+      <properties>
+        <property name="requirements" value="R-NET-05,R-NET-06,R-NET-07"/>
+        <property name="layer" value="L1"/>
+        <property name="path" value="/work/.claude/conf-clone/conformance/tests/l1-rendering/egress-class-render"/>
+      </properties>
+    </testcase>
+  </testsuite>
+</testsuites>

conformance/runs/20260913T014321Z-19fc8bc/run.log

@@ -0,0 +1,7 @@
+[run.sh] Coverage report first (always)
+[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
+[run.sh] Running 1 offline L0/L1 check(s) (no cluster; docs/14).
+[offline] L1  PASS egress-class-render (R-NET-05,R-NET-06,R-NET-07)
+[offline] wrote JUnit (1 case(s), 0 failure(s)) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml
+[run.sh] Done (offline l1). JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260913T014437Z-19fc8bc/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="vpn-killswitch" classname="" time="174.632779" />
+  </testsuite>
+</testsuites>

conformance/runs/20260913T014437Z-19fc8bc/run.log

@@ -0,0 +1,237 @@
+[run.sh] Coverage report first (always)
+[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-NET-06').
+[run.sh] LIGHT pass: 1 test(s), parallel=2 (pinned v0.2.12)
+Version: 0.2.12
+Loading config (/work/.claude/conf-clone/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-light'
+- ReportPath '/work/.claude/conf-clone/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 2
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- vpn-killswitch (/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/vpn-killswitch
+=== PAUSE chainsaw/vpn-killswitch
+=== CONT  chainsaw/vpn-killswitch
+    l.go:52: | 01:44:38 | vpn-killswitch | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-credible-bird
+    l.go:52: | 01:44:38 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | TRY       | BEGIN |
+    l.go:52: | 01:44:38 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        kubectl label namespace "$NAMESPACE" \
+          pod-security.kubernetes.io/enforce=privileged \
+          pod-security.kubernetes.io/warn=privileged \
+          pod-security.kubernetes.io/audit=privileged \
+          --overwrite
+        echo "NS_PRIVILEGED_OK"
+    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | LOG   |
+        === STDOUT
+        namespace/chainsaw-credible-bird labeled
+        NS_PRIVILEGED_OK
+    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | DONE  |
+    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | RUN   | v1/Pod @ chainsaw-credible-bird/egress-internet
+    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CREATE    | OK    | v1/Pod @ chainsaw-credible-bird/egress-internet
+    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | DONE  | v1/Pod @ chainsaw-credible-bird/egress-internet
+    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | RUN   | v1/Pod @ chainsaw-credible-bird/egress-vpn
+    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CREATE    | OK    | v1/Pod @ chainsaw-credible-bird/egress-vpn
+    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | DONE  | v1/Pod @ chainsaw-credible-bird/egress-vpn
+    l.go:52: | 01:44:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | RUN   | v1/Pod @ chainsaw-credible-bird/egress-internet
+    l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | DONE  | v1/Pod @ chainsaw-credible-bird/egress-internet
+    l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | RUN   | v1/Pod @ chainsaw-credible-bird/egress-vpn
+    l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | DONE  | v1/Pod @ chainsaw-credible-bird/egress-vpn
+    l.go:52: | 01:44:41 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        ns="$NAMESPACE"
+        get_ip() {
+          kubectl -n "$ns" exec "$1" -c curl -- \
+            curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
+        }
+        wan="" vpn=""
+        for i in $(seq 1 12); do
+          [ -n "$wan" ] || wan="$(get_ip egress-internet)"
+          [ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
+          if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
+          [ "$wan" = "$vpn" ] && vpn=""   # not steered yet; re-poll
+          sleep 5
+        done
+        echo "wan=$wan vpn=$vpn"
+        [ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
+        [ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
+        [ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
+        echo "TUNNEL_HEALTHY_OK"
+    l.go:52: | 01:44:43 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | LOG   |
+        === STDOUT
+        wan=136.51.115.252 vpn=158.173.241.11
+        TUNNEL_HEALTHY_OK
+    l.go:52: | 01:44:43 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | DONE  |
+    l.go:52: | 01:44:43 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | TRY       | END   |
+    l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | TRY       | BEGIN |
+    l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | APPLY     | RUN   | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
+    l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CREATE    | OK    | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
+    l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | APPLY     | DONE  | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
+    l.go:52: | 01:44:43 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | ASSERT    | RUN   | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
+    l.go:52: | 01:44:48 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | ASSERT    | DONE  | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
+    l.go:52: | 01:44:48 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -u
+        ns="$NAMESPACE"
+        wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
+                 curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
+        for i in $(seq 1 8); do
+          out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
+                   curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
+          if [ -n "$out" ]; then
+            if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
+              echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
+            fi
+            echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
+          fi
+          echo "attempt $i: vpn-class blackholed (no response), good"
+          sleep 3
+        done
+        echo "NO_LEAK_OK"
+    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | LOG   |
+        === STDOUT
+        attempt 1: vpn-class blackholed (no response), good
+        attempt 2: vpn-class blackholed (no response), good
+        attempt 3: vpn-class blackholed (no response), good
+        attempt 4: vpn-class blackholed (no response), good
+        attempt 5: vpn-class blackholed (no response), good
+        attempt 6: vpn-class blackholed (no response), good
+        attempt 7: vpn-class blackholed (no response), good
+        attempt 8: vpn-class blackholed (no response), good
+        NO_LEAK_OK
+    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | DONE  |
+    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        ns="$NAMESPACE"
+        out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
+                 curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
+        echo "internet_ip=$out"
+        [ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
+        echo "INTERNET_UNAFFECTED_OK"
+    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | LOG   |
+        === STDOUT
+        internet_ip=136.51.115.252
+        INTERNET_UNAFFECTED_OK
+    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | DONE  |
+    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | TRY       | END   |
+    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | FINALLY   | BEGIN |
+    l.go:52: | 01:46:18 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        ns="$NAMESPACE"
+        # Distinct Job name (tunnel-restart) so it never collides with the
+        # tunnel-stop Job. Left-aligned manifest so the block scalar yields
+        # valid YAML at column 0 at runtime.
+        cat <<'JOB' | kubectl -n "$ns" apply -f -
+        apiVersion: batch/v1
+        kind: Job
+        metadata:
+          name: tunnel-restart
+          labels: { app: vpn-killswitch-fault }
+        spec:
+          backoffLimit: 3
+          activeDeadlineSeconds: 90
+          ttlSecondsAfterFinished: 600
+          template:
+            metadata:
+              labels: { app: vpn-killswitch-fault }
+            spec:
+              restartPolicy: Never
+              hostPID: true
+              nodeSelector:
+                homelab.keithjackson.dev/egress-gateway: "true"
+              tolerations:
+                - operator: Exists
+              containers:
+                - name: nsenter
+                  image: busybox:1.37.0
+                  securityContext:
+                    privileged: true
+                  command: ["nsenter","-t","1","-m","-u","-i","-n","-p","--","systemctl","start","vpn-egress-tunnel"]
+        JOB
+        # Block teardown until the restart actually completed on the host — a
+        # leaked-down tunnel is unacceptable. `wait` exits non-zero on failure
+        # or timeout, which (set -e) fails the step and surfaces the problem.
+        kubectl -n "$ns" wait --for=condition=complete --timeout=110s job/tunnel-restart
+        echo "TUNNEL_SELF_HEAL_OK"
+    l.go:52: | 01:46:21 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | LOG   |
+        === STDOUT
+        job.batch/tunnel-restart created
+        job.batch/tunnel-restart condition met
+        TUNNEL_SELF_HEAL_OK
+    l.go:52: | 01:46:21 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | DONE  |
+    l.go:52: | 01:46:21 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | FINALLY   | END   |
+    l.go:52: | 01:46:21 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY       | BEGIN |
+    l.go:52: | 01:46:21 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        ns="$NAMESPACE"
+        wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
+                 curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
+        vpn=""
+        for i in $(seq 1 24); do
+          vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
+                   curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
+          if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
+          vpn=""
+          sleep 5
+        done
+        echo "recovered_vpn=$vpn wan=$wan"
+        [ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
+        [ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
+        echo "TUNNEL_RECOVERED_OK"
+    l.go:52: | 01:46:25 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT    | LOG   |
+        === STDOUT
+        recovered_vpn=158.173.241.11 wan=136.51.115.252
+        TUNNEL_RECOVERED_OK
+    l.go:52: | 01:46:25 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT    | DONE  |
+    l.go:52: | 01:46:25 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY       | END   |
+    l.go:52: | 01:46:25 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CLEANUP   | BEGIN |
+    l.go:52: | 01:46:25 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | DELETE    | OK    | batch/v1/Job @ chainsaw-credible-bird/tunnel-stop
+    l.go:52: | 01:46:25 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CLEANUP   | END   |
+    l.go:52: | 01:46:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CLEANUP   | BEGIN |
+    l.go:52: | 01:46:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | DELETE    | OK    | v1/Pod @ chainsaw-credible-bird/egress-vpn
+    l.go:52: | 01:46:56 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | DELETE    | OK    | v1/Pod @ chainsaw-credible-bird/egress-internet
+    l.go:52: | 01:47:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CLEANUP   | END   |
+    l.go:52: | 01:47:26 | vpn-killswitch | @chainsaw                                 | CLEANUP   | BEGIN |
+    l.go:52: | 01:47:26 | vpn-killswitch | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-credible-bird
+    l.go:52: | 01:47:33 | vpn-killswitch | @chainsaw                                 | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/vpn-killswitch (174.63s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+[run.sh] HEAVY pass: no heavy tests selected.
+merge_junit: merged 1 report(s) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260913T014923Z-9d0814a/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="vpn-killswitch" classname="" time="173.473422" />
+  </testsuite>
+</testsuites>

conformance/runs/20260913T014923Z-9d0814a/run.log

@@ -0,0 +1,237 @@
+[run.sh] Coverage report first (always)
+[coverage] 49/51 requirements have a scaffolded test (96.1%); 2 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md and /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-NET-06').
+[run.sh] LIGHT pass: 1 test(s), parallel=2 (pinned v0.2.12)
+Version: 0.2.12
+Loading config (/work/.claude/conf-clone/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-light'
+- ReportPath '/work/.claude/conf-clone/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 2
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- vpn-killswitch (/work/.claude/conf-clone/conformance/tests/l3-fault-injection/vpn-killswitch)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/vpn-killswitch
+=== PAUSE chainsaw/vpn-killswitch
+=== CONT  chainsaw/vpn-killswitch
+    l.go:52: | 01:49:24 | vpn-killswitch | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-beloved-toad
+    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | TRY       | BEGIN |
+    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        kubectl label namespace "$NAMESPACE" \
+          pod-security.kubernetes.io/enforce=privileged \
+          pod-security.kubernetes.io/warn=privileged \
+          pod-security.kubernetes.io/audit=privileged \
+          --overwrite
+        echo "NS_PRIVILEGED_OK"
+    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | LOG   |
+        === STDOUT
+        namespace/chainsaw-beloved-toad labeled
+        NS_PRIVILEGED_OK
+    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | DONE  |
+    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | RUN   | v1/Pod @ chainsaw-beloved-toad/egress-internet
+    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CREATE    | OK    | v1/Pod @ chainsaw-beloved-toad/egress-internet
+    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | DONE  | v1/Pod @ chainsaw-beloved-toad/egress-internet
+    l.go:52: | 01:49:24 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | RUN   | v1/Pod @ chainsaw-beloved-toad/egress-vpn
+    l.go:52: | 01:49:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CREATE    | OK    | v1/Pod @ chainsaw-beloved-toad/egress-vpn
+    l.go:52: | 01:49:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | APPLY     | DONE  | v1/Pod @ chainsaw-beloved-toad/egress-vpn
+    l.go:52: | 01:49:25 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | RUN   | v1/Pod @ chainsaw-beloved-toad/egress-internet
+    l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | DONE  | v1/Pod @ chainsaw-beloved-toad/egress-internet
+    l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | RUN   | v1/Pod @ chainsaw-beloved-toad/egress-vpn
+    l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | ASSERT    | DONE  | v1/Pod @ chainsaw-beloved-toad/egress-vpn
+    l.go:52: | 01:49:26 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        ns="$NAMESPACE"
+        get_ip() {
+          kubectl -n "$ns" exec "$1" -c curl -- \
+            curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true
+        }
+        wan="" vpn=""
+        for i in $(seq 1 12); do
+          [ -n "$wan" ] || wan="$(get_ip egress-internet)"
+          [ -n "$vpn" ] || vpn="$(get_ip egress-vpn)"
+          if [ -n "$wan" ] && [ -n "$vpn" ] && [ "$wan" != "$vpn" ]; then break; fi
+          [ "$wan" = "$vpn" ] && vpn=""   # not steered yet; re-poll
+          sleep 5
+        done
+        echo "wan=$wan vpn=$vpn"
+        [ -n "$wan" ] || { echo "FAIL: no WAN IP from internet-class pod" >&2; exit 1; }
+        [ -n "$vpn" ] || { echo "FAIL: vpn-class egress not working at baseline" >&2; exit 1; }
+        [ "$wan" != "$vpn" ] || { echo "FAIL: vpn-class == WAN at baseline (not tunneled)" >&2; exit 1; }
+        echo "TUNNEL_HEALTHY_OK"
+    l.go:52: | 01:49:27 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | LOG   |
+        === STDOUT
+        wan=136.51.115.252 vpn=158.173.241.11
+        TUNNEL_HEALTHY_OK
+    l.go:52: | 01:49:27 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | SCRIPT    | DONE  |
+    l.go:52: | 01:49:27 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | TRY       | END   |
+    l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | TRY       | BEGIN |
+    l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | APPLY     | RUN   | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
+    l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CREATE    | OK    | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
+    l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | APPLY     | DONE  | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
+    l.go:52: | 01:49:27 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | ASSERT    | RUN   | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
+    l.go:52: | 01:49:31 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | ASSERT    | DONE  | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
+    l.go:52: | 01:49:31 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -u
+        ns="$NAMESPACE"
+        wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
+                 curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
+        for i in $(seq 1 8); do
+          out="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
+                   curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
+          if [ -n "$out" ]; then
+            if [ -n "$wan" ] && [ "$out" = "$wan" ]; then
+              echo "LEAK: vpn-class reached the WAN IP ($out) with the tunnel DOWN" >&2; exit 1
+            fi
+            echo "LEAK: vpn-class reached the internet ($out) with the tunnel DOWN" >&2; exit 1
+          fi
+          echo "attempt $i: vpn-class blackholed (no response), good"
+          sleep 3
+        done
+        echo "NO_LEAK_OK"
+    l.go:52: | 01:51:01 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | LOG   |
+        === STDOUT
+        attempt 1: vpn-class blackholed (no response), good
+        attempt 2: vpn-class blackholed (no response), good
+        attempt 3: vpn-class blackholed (no response), good
+        attempt 4: vpn-class blackholed (no response), good
+        attempt 5: vpn-class blackholed (no response), good
+        attempt 6: vpn-class blackholed (no response), good
+        attempt 7: vpn-class blackholed (no response), good
+        attempt 8: vpn-class blackholed (no response), good
+        NO_LEAK_OK
+    l.go:52: | 01:51:01 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | DONE  |
+    l.go:52: | 01:51:01 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        ns="$NAMESPACE"
+        out="$(kubectl -n "$ns" exec egress-internet -c curl -- \
+                 curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
+        echo "internet_ip=$out"
+        [ -n "$out" ] || { echo "FAIL: internet-class egress broke while tunnel down (must be unaffected)" >&2; exit 1; }
+        echo "INTERNET_UNAFFECTED_OK"
+    l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | LOG   |
+        === STDOUT
+        internet_ip=136.51.115.252
+        INTERNET_UNAFFECTED_OK
+    l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | DONE  |
+    l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | TRY       | END   |
+    l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | FINALLY   | BEGIN |
+    l.go:52: | 01:51:02 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        ns="$NAMESPACE"
+        # Distinct Job name (tunnel-restart) so it never collides with the
+        # tunnel-stop Job. Left-aligned manifest so the block scalar yields
+        # valid YAML at column 0 at runtime.
+        cat <<'JOB' | kubectl -n "$ns" apply -f -
+        apiVersion: batch/v1
+        kind: Job
+        metadata:
+          name: tunnel-restart
+          labels: { app: vpn-killswitch-fault }
+        spec:
+          backoffLimit: 3
+          activeDeadlineSeconds: 90
+          ttlSecondsAfterFinished: 600
+          template:
+            metadata:
+              labels: { app: vpn-killswitch-fault }
+            spec:
+              restartPolicy: Never
+              hostPID: true
+              nodeSelector:
+                homelab.keithjackson.dev/egress-gateway: "true"
+              tolerations:
+                - operator: Exists
+              containers:
+                - name: nsenter
+                  image: busybox:1.37.0
+                  securityContext:
+                    privileged: true
+                  command: ["nsenter","-t","1","-m","-u","-i","-n","-p","--","systemctl","start","vpn-egress-tunnel"]
+        JOB
+        # Block teardown until the restart actually completed on the host — a
+        # leaked-down tunnel is unacceptable. `wait` exits non-zero on failure
+        # or timeout, which (set -e) fails the step and surfaces the problem.
+        kubectl -n "$ns" wait --for=condition=complete --timeout=110s job/tunnel-restart
+        echo "TUNNEL_SELF_HEAL_OK"
+    l.go:52: | 01:51:05 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | LOG   |
+        === STDOUT
+        job.batch/tunnel-restart created
+        job.batch/tunnel-restart condition met
+        TUNNEL_SELF_HEAL_OK
+    l.go:52: | 01:51:05 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | SCRIPT    | DONE  |
+    l.go:52: | 01:51:05 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | FINALLY   | END   |
+    l.go:52: | 01:51:05 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY       | BEGIN |
+    l.go:52: | 01:51:05 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        ns="$NAMESPACE"
+        wan="$(kubectl -n "$ns" exec egress-internet -c curl -- \
+                 curl -fsS --max-time 10 https://api.ipify.org 2>/dev/null || true)"
+        vpn=""
+        for i in $(seq 1 24); do
+          vpn="$(kubectl -n "$ns" exec egress-vpn -c curl -- \
+                   curl -fsS --max-time 8 https://api.ipify.org 2>/dev/null || true)"
+          if [ -n "$vpn" ] && [ "$vpn" != "$wan" ]; then break; fi
+          vpn=""
+          sleep 5
+        done
+        echo "recovered_vpn=$vpn wan=$wan"
+        [ -n "$vpn" ] || { echo "FAIL: vpn-class egress did not recover after tunnel restart" >&2; exit 1; }
+        [ "$vpn" != "$wan" ] || { echo "FAIL: vpn-class egress recovered onto the WAN (not tunneled)" >&2; exit 1; }
+        echo "TUNNEL_RECOVERED_OK"
+    l.go:52: | 01:51:08 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT    | LOG   |
+        === STDOUT
+        recovered_vpn=158.173.241.11 wan=136.51.115.252
+        TUNNEL_RECOVERED_OK
+    l.go:52: | 01:51:08 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | SCRIPT    | DONE  |
+    l.go:52: | 01:51:08 | vpn-killswitch | when-tunnel-recovers-connectivity-returns | TRY       | END   |
+    l.go:52: | 01:51:08 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CLEANUP   | BEGIN |
+    l.go:52: | 01:51:08 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | DELETE    | OK    | batch/v1/Job @ chainsaw-beloved-toad/tunnel-stop
+    l.go:52: | 01:51:08 | vpn-killswitch | when-force-tunnel-down-then-no-leak       | CLEANUP   | END   |
+    l.go:52: | 01:51:08 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CLEANUP   | BEGIN |
+    l.go:52: | 01:51:08 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | DELETE    | OK    | v1/Pod @ chainsaw-beloved-toad/egress-vpn
+    l.go:52: | 01:51:39 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | DELETE    | OK    | v1/Pod @ chainsaw-beloved-toad/egress-internet
+    l.go:52: | 01:52:10 | vpn-killswitch | given-healthy-tunnel-and-two-workloads    | CLEANUP   | END   |
+    l.go:52: | 01:52:10 | vpn-killswitch | @chainsaw                                 | CLEANUP   | BEGIN |
+    l.go:52: | 01:52:10 | vpn-killswitch | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-beloved-toad
+    l.go:52: | 01:52:18 | vpn-killswitch | @chainsaw                                 | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/vpn-killswitch (173.47s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+[run.sh] HEAVY pass: no heavy tests selected.
+merge_junit: merged 1 report(s) -> /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/.claude/conf-clone/conformance/_artifacts/report.junit.xml | Coverage: /work/.claude/conf-clone/conformance/_artifacts/coverage/coverage.md