fix(ci): CLUSTER_AVAILABLE=true would have FAKED A GREEN over 54 skipped tests

Wiring L2/L3 into CI was a tracked next step, described in the plan as needing only
vars.CLUSTER_AVAILABLE. Attempting it surfaced that the variable does not enable L2/L3 at
all -- it only changes which way the stage fails, and before this commit one of those ways
was a false pass.

MEASURED, before the guard:
  CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
  "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
  "[ ok ] L2/L3 conformance passed for touched subset"

run.sh exits 0 in scaffold mode when it cannot reach a cluster, marking every test
<skipped>. That is correct and honest FOR run.sh. It is catastrophic in ci/conformance.sh,
which reaches that code only because the environment DECLARED a cluster available -- so it
reported a pass over 54 tests that never executed. One Gitea variable away from a fake green
on a gate that is now an ENFORCING required status check, and nothing caught it because
nobody had ever set the variable.

The CI runner holds no cluster credential by design (automountServiceAccountToken: false),
so the local run this stage attempts can never succeed. The in-cluster trigger path that
would make L2/L3 actually run is still unimplemented, and is blocked on a privilege-
escalation design decision that is the operator's to make (plan section 4: any principal
that can `create jobs` in ns conformance can run arbitrary code as the privileged
conformance-runner SA, because Kubernetes places no `use` check on serviceAccountName).

The stage now BLOCKS (exit 1) when a cluster is declared but every test skipped, which is
what the merge-gate decision matrix already said should happen ("conformance skipped while
CLUSTER_AVAILABLE=true -> BLOCK"). Verified: rc=1, naming the count and the reason.

The plan document claimed CLUSTER_AVAILABLE was "the only remaining gate". Corrected in
place with the measurement and an explicit do-not-set-this-yet.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/README.md

@@ -40,9 +40,15 @@ and ADR-0011: **CI holds no broad kubeconfig.** L2/L3 run as an in-cluster **Cha
 *collects* its artifacts, using a narrow trigger credential referenced **by secret name only**
 (`secrets.CONFORMANCE_TRIGGER_KUBECONFIG`) — never a plaintext secret, never a cluster-admin kubeconfig.
 
-## Honest SKIP — never fake green (Phase 0)
+## Honest SKIP — never fake green
 
-**No cluster exists in Phase 0.** The gate is deliberately built so unavailable checks **SKIP loudly,
+**NOTE (2026-09-29):** this section is written in the past's voice — "Phase 0", "no cluster
+exists". **A cluster has existed for weeks** and Flux has been reconciling against it. The
+skip logic below is still correct and still honest, but read it as describing a **live
+coverage gap** (L2/L3 is not wired into CI) rather than an expected phase. Its being
+phrased as a phase is part of why the gap went unexamined.
+
+Originally: The gate is deliberately built so unavailable checks **SKIP loudly,
 they do not pass**:
 
 - The `conformance` job is `if: ${{ vars.CLUSTER_AVAILABLE == 'true' }}` — with no cluster it is
@@ -77,21 +83,29 @@ Both dirs are git-ignored (see [`.gitignore`](../../.gitignore)).
 ## Triggers & branch model (CONTRIBUTING.md)
 
 - `pull_request` → `main`: **PRs are the enforced merge gate into `main`.**
-- `push` → `feature/**`, `fix/**`, `docs/**`, `chore/**`: fast feedback on the branch. **No work on `main`.**
+- `push` → `main` only. The per-branch prefix list was removed: `act_runner` runs at capacity 1,
+  so a branch push fires a redundant run that serialises *ahead* of the PR run for the same tree.
+  **No work on `main`** — and since 2026-09-29 that is enforced, not conventional: `main` is a
+  protected branch and a direct push is rejected by the pre-receive hook (verified, not assumed).
 
-## Transition: generic-CI-now → Gitea-Actions-after-Gitea-is-up
+## Transition: COMPLETE (2026-09-29)
 
-Per [`CONTRIBUTING.md`](../../../home-network-k3s/CONTRIBUTING.md) and
-[`docs/05` §"local bootstrap → self-hosted Gitea"](../../../home-network-k3s/docs/05-gitops-and-delivery.md):
+This section used to describe a migration as pending. It has happened. Gitea is live at
+`gitea.home.keithjackson.dev`, the runner `incluster-dind` is registered and online, and
+`gate.yml` runs on every PR.
 
-1. **Now (interim):** there is **no remote git host** on this network. This workflow file is dormant —
-   but the **identical gate runs locally today** via `make -C ci gate` (R-TEST-02). Development is
-   local feature-branch → `main`, the gate enforced by convention + the local runner.
-2. **Gitea comes online early** (one of the first services onboarded) at `gitea.home.keithjackson.dev`.
-3. **CI moves to Gitea Actions:** this workflow activates unchanged. The only operational steps are
-   enabling a Gitea Actions runner and setting `vars.CLUSTER_AVAILABLE=true` +
-   `secrets.CONFORMANCE_TRIGGER_KUBECONFIG` once a dev/test cluster exists. PRs then become the
-   **platform-enforced** merge gate — the same no-work-in-main rule, now enforced by machine.
+**What took a month to notice:** the migration was recorded as done on 2026-08-31 without
+anyone checking the platform. In reality `main` had **no branch protection** and **zero
+runners were registered** until 2026-09-29. Every "the gate enforces this" statement in
+between rested on an assertion nobody verified. PR #1 — the first pull request this
+repository ever had — sat `pending` for 30.6 minutes because nothing existed to run it.
+
+So: **do not read this file, or the DECISIONS-LOG, as evidence about the live platform.**
+Query the API. `infra/gitea/` holds the scripts and the measured state.
+
+Still genuinely outstanding: `vars.CLUSTER_AVAILABLE` and
+`secrets.CONFORMANCE_TRIGGER_KUBECONFIG` are unset, so L2/L3 conformance does not run in
+CI. That is a real coverage gap, not a phase.
 
 ## Configuration (set in Gitea; referenced by name only)
 

.gitea/workflows/gate.yml

@@ -44,16 +44,31 @@
 ---
 name: gate
 
+# TRIGGERS — deliberately only `pull_request` into main, plus `push` to main itself.
+#
+# There is NO per-branch push trigger. There used to be a prefix list (feature/** fix/**
+# docs/** chore/**) for "fast feedback on the branch", and it was a net negative here:
+#
+#   * `act_runner` runs at capacity 1. A branch push and its PR fire the SAME gate over the
+#     SAME tree, and the redundant run SERIALISES AHEAD of the PR run — doubling the wall
+#     clock on the one result that actually gates the merge. That cost real time during the
+#     2026-09-29 plex claim, where the loop is racing a short-lived token.
+#   * The list silently did not match this repo's own convention. Branches here are `feat/**`,
+#     which was absent, so the "fast feedback" never fired for the most common prefix anyway —
+#     a mismatch nobody noticed for a month because its only symptom was a MISSING run.
+#     Any hand-maintained prefix list has that failure mode; deleting it removes the class.
+#   * It is redundant. CLAUDE.md already requires `./ci/gate.sh` in the pinned toolbox before
+#     you push (74s locally), and the PR gate re-runs it authoritatively. A third run of the
+#     same checks buys nothing.
+#
+# `push: [main]` is KEPT: post-merge main must still be verified, and it is the only run for
+# which no PR context exists.
 on:
   pull_request:
     branches: [main]
   push:
     branches:
       - main
-      - 'feature/**'
-      - 'fix/**'
-      - 'docs/**'
-      - 'chore/**'
 
 concurrency:
   group: gate-${{ github.ref }}

.gitea/workflows/merge-gate.yaml

@@ -11,11 +11,14 @@
 #                                          v
 #                                     merge-gate decision  [R-TEST-03: touched subset green]
 #
-# Transition (CONTRIBUTING.md / docs/05 §"local bootstrap -> self-hosted Gitea"):
-# Today there is no Gitea host, so this file is dormant. The SAME gate is runnable
-# now via `make -C ci gate` (R-TEST-02). Once Gitea is self-hosted and Gitea
-# Actions is enabled, this workflow becomes the ENFORCED PR merge gate into main —
-# no code change required, only turning on the runner and setting CLUSTER_AVAILABLE.
+# STATUS (corrected 2026-09-29): Gitea IS hosted and Gitea Actions IS enabled; the runner
+# `incluster-dind` is online and gate.yml is the live PR gate. The text that used to sit here
+# ("Today there is no Gitea host, so this file is dormant") was stale by weeks -- it was written
+# as a plan and never revisited once the plan completed.
+#
+# THIS file is still dormant, but for the reason in the DORMANT block below -- its spec checkout
+# and L2/L3 seams are unwired -- NOT because there is no host. Do not read a stale plan as a
+# statement about the live platform; query the API instead (see infra/gitea/).
 #
 # CI holds NO broad kubeconfig. L2/L3 run as an in-cluster Chainsaw Job whose own
 # scoped ServiceAccount holds cluster access (docs/14, R-SEC-02); CI uses only a

.standup-dns/dns-20260914-pre-externaldns-scope.txt

@@ -0,0 +1,97 @@
+# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev`
+# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied.
+#
+# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers,
+# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not
+# merely left alone. That is a one-way, outward-facing operation against a live authoritative
+# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the
+# reconcile removes can be restored by hand from this file. It is the same discipline as
+# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable
+# change rather than to a migration.
+#
+# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones/<id>/dns_records, all pages) using the
+# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the
+# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the
+# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the
+# devcontainer, which has previously produced false "no records" readings — the API is the only
+# reading that is both complete and authoritative.
+#
+# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the
+# TTL and the proxy flag, not just the value.
+#
+#   <type>|<name>|<content>|<ttl>|<proxied>
+#
+# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here
+# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address.
+#
+# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry
+# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which
+# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy
+# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS
+# cannot touch them regardless of what this branch does:
+#
+#   *.keithjackson.dev            -> 136.51.115.252   legacy stack (the zone wildcard)
+#   brickbreak.keithjackson.dev   -> 20.236.242.14    unrelated, off-prem
+#   dev.brickbreak…               -> 20.236.242.14    unrelated, off-prem
+#
+# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in
+# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of
+# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does
+# NOT cover `<svc>.home.keithjackson.dev`: when a specific `<svc>.home` record is removed, that
+# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended
+# end state for `internal` services and it is also precisely why Pi-hole must already be
+# answering the LAN authoritatively before this branch merges.
+#
+# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog
+# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its
+# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed
+# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an
+# HTTPRoute that already had a published record produced
+# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile
+# interval (~60s) later, and nothing else in the zone moved.
+#
+# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its
+# own): recreate the A record with its value/ttl/proxied from this file, then recreate the
+# matching `a-<name>` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod,
+# external-dns/resource=…` string is what re-establishes ownership. An A record restored without
+# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value
+# points the registry at a source object that does not exist.
+A|*.keithjackson.dev|136.51.115.252|1|false
+A|auth.home.keithjackson.dev|192.168.2.20|1|false
+A|brickbreak.keithjackson.dev|20.236.242.14|1|false
+A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false
+A|crafty.home.keithjackson.dev|192.168.2.20|1|false
+A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false
+A|gitea.home.keithjackson.dev|192.168.2.20|1|false
+A|grafana.home.keithjackson.dev|192.168.2.20|1|false
+A|headlamp.home.keithjackson.dev|192.168.2.20|1|false
+A|home.keithjackson.dev|192.168.2.20|1|false
+A|jackett.home.keithjackson.dev|192.168.2.20|1|false
+A|keycloak.home.keithjackson.dev|192.168.2.20|1|false
+A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false
+A|pihole.home.keithjackson.dev|192.168.2.20|1|false
+A|plex.home.keithjackson.dev|192.168.2.20|1|false
+A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false
+A|radarr.home.keithjackson.dev|192.168.2.20|1|false
+A|seerr.home.keithjackson.dev|192.168.2.20|1|false
+A|semaphore.home.keithjackson.dev|192.168.2.20|1|false
+A|sonarr.home.keithjackson.dev|192.168.2.20|1|false
+A|unifi.home.keithjackson.dev|192.168.2.21|1|false
+TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false
+TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false
+TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false
+TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false
+TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false
+TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false
+TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false
+TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false
+TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false
+TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false
+TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false
+TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false
+TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false
+TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false
+TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false
+TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false
+TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false
+TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false

CLAUDE.md

@@ -0,0 +1,147 @@
+# CLAUDE.md — how work lands in this repo
+
+Read this before doing anything durable. It is the delivery contract for AI coding
+sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
+takes precedence on *what* to work on; this file governs *how* it lands.
+
+## 1. Never push to `main`. Ever.
+
+Every durable change reaches `main` exactly one way:
+
+    branch  ->  push the branch  ->  open a PR  ->  CI gate green  ->  human approval  ->  merge
+
+There is no other path. Specifically:
+
+- **Do not `git push origin main`.**
+- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
+- **Do not merge your own PR.** A human approves and clicks merge.
+- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
+  anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
+
+> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
+> That wording is **stale** — it predates the PR flow. It describes what Flux does
+> (main is what the cluster follows), not what you are permitted to do. Honour this
+> file, not that line. Fixing those runbooks is tracked work.
+
+## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
+
+Verified against the live Gitea on 2026-09-29:
+
+| Expected | Actual |
+|---|---|
+| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
+| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
+| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
+| Approval required before merge | ✅ `required_approvals: 1` |
+
+All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
+true. They are true now because each was measured, and the push rejection was provoked rather
+than assumed. **Re-measure before relying on any of them** — see §2b.
+
+Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
+CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
+
+**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
+the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
+strength of a check you have not seen report — and note that a **skipped** Gitea job
+reports as `success`, so a green tick is not by itself evidence that anything ran.
+
+## 2b. Never infer platform state from this repository
+
+**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
+Both describe what someone believed at the time of writing, and both have been
+confidently wrong for a month at a stretch.
+
+On 2026-09-29 the log stated the merge gate was a required status check with a runner
+online. The API said `branch_protections: []` and **zero** runners. Everything downstream
+of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
+was wrong, and nothing in the tree could have revealed it, because the tree was the source
+of the error.
+
+**So, before asserting that any gate, protection, runner, secret, or workload is in a
+given state: query the thing itself.**
+
+| Claim about... | Check |
+|---|---|
+| branch protection / required checks | `GET /api/v1/repos/{owner}/{repo}/branches/main` |
+| a runner exists | `GET /api/v1/admin/actions/runners` |
+| CI actually ran | the commit's `/statuses` — and read the **description**; a *skipped* Gitea job reports `success` with "Has been skipped" |
+| a workload / Secret / env | `kubectl --context k3s-prod ...` (read-only is fine for agents) |
+| Flux applied your commit | compare `.status.lastAppliedRevision` against the GitRepository's artifact revision |
+
+Quote the measured field, not the prose. If you cannot measure it, say so explicitly —
+"unverified" is a legitimate report, and a far better one than a confident guess that
+matches the docs.
+
+**A green tick is not proof anything ran.** Read what actually executed.
+
+## 3. Gate before you push
+
+    ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
+
+Exit codes are a contract — do not flatten them:
+
+| rc | Meaning |
+|---|---|
+| 0 | PASS |
+| 1 | BLOCKED — something failed |
+| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
+
+`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
+beside this one; that single skip is expected and tolerated. Any *other* skip is a
+real gap in your run.
+
+**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
+so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
+
+### Running the gate from a git worktree
+
+Two unrelated failures, both with known workarounds — put both in any agent brief:
+
+1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
+   and exists only in the main checkout. Do **not** copy it (it is a credential).
+   Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
+   normally.
+2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
+   gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
+   the `ENGINE` hook — a shim that intercepts `run` and re-execs
+   `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
+   `tracked_files()` is this bug, not a broken tree.
+
+## 4. Branch naming
+
+Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
+trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
+The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
+
+## 5. Cluster access
+
+`kubectl` does not work on the operator workstation; the toolchain lives only in the
+pinned container. Always name the context explicitly:
+
+    ./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+`dev.sh` merges both clusters and picks a current-context **per invocation**; that
+default has already changed once. Never rely on it. Do not run two `dev.sh`
+invocations concurrently.
+
+**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
+
+## 6. Secrets
+
+- **Never `sops -d` a file into a terminal, a log, or a transcript.**
+- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
+  list key names or object names only. A password has been leaked into a transcript
+  this way before.
+- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
+  the single creation-rules file; do not add a second one in a subtree.
+- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
+  flow. Do not design around committing one — read the expiry from the issuing page and
+  treat a burnt token as free to retry.
+
+## 7. Before you finish
+
+Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
+verified, and what you did **not** verify. State gaps plainly — an honest
+"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
+burned by the difference.

README.md

@@ -38,22 +38,56 @@ ci/             the gate as scripts (lint · schema · Kyverno L0 · L1 render
 
 ## Status
 
-Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
-open** — `clusters/prod` is a design-only overlay.
+Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in
+progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the
+apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration:
+dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
+hardware rather than on Proxmox VMs.
+
+**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring +
+alerts live*:
+
+| criterion | state |
+|---|---|
+| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
+| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
+| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only |
+
+**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 —
+the first requirement in this repo to reach that state. Both of its verifiers are recorded:
+`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`)
+and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn`
+class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy**
+(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
+*different* PersistentVolume — so an in-cluster survival cannot pass as a restore.
+
+**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned
+dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
+exercise rather than something to run in passing.
+
+Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a
+bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
 
 | Phase | Delivered |
 |---|---|
 | 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
-| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
-| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
+| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
+| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
 | 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
-| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
+| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
 | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
 | 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
+| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
 
-**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
-(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
-the full report is `conformance/_artifacts/coverage/coverage.md`.
+**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
+2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with
+`./conformance/run.sh --coverage-only`; the full report is
+`conformance/_artifacts/coverage/coverage.md`.
+
+Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and
+*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current
+SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the
+second.
 
 The running narrative — decisions, root causes, incidents, follow-ups — is
 [`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.

apps/vms/unifi/30-virtualmachine.yaml

@@ -41,10 +41,41 @@ spec:
       # disk KubeVirt cannot migrate and eviction would hard-stop the guest.
       evictionStrategy: LiveMigrate
       domain:
+        # =====================================================================================
+        # FIRMWARE UUID == THE CLOUD-INIT INSTANCE-ID. THIS IS THE RE-RUN LEVER. READ BEFORE EDIT.
+        # =====================================================================================
+        # MEASURED on prod 2026-09-14, not assumed:
+        #   * the NoCloud seed KubeVirt builds for this VM carries
+        #       {"instance-id":"de5c3338-6d32-4956-89d1-12c8272b2c07","local-hostname":"unifi"}
+        #     (read out of noCloud.iso inside virt-launcher-unifi-gp594), and that value was
+        #     EXACTLY `spec.template.spec.domain.firmware.uuid`, which KubeVirt's mutating webhook
+        #     had stamped onto the VM OBJECT — so it is PERSISTENT, not per-VMI (the VMI's own uid
+        #     was b532fd7c-…, a different value). It is not a hash of the VM name either: the four
+        #     standard uuid5 namespaces over "unifi" / "unifi.unifi" all produce something else.
+        #   * cloud-init's NoCloud datasource decides "is this the same instance?" by comparing the
+        #     cached instance-id against the guest's DMI product_uuid — which IS this firmware UUID.
+        #
+        # CONSEQUENCE, and it is the whole reason the previous unattended attempt could never have
+        # worked: EDITING `userData` BELOW AND RESTARTING THE VM CHANGES NOTHING. cloud-init sees an
+        # unchanged instance-id, restores its CACHED datasource, never re-reads the seed, and skips
+        # every per-instance module (write_files, runcmd). The new payload is silently ignored.
+        #
+        # So the UUID is pinned HERE, in git, and BUMPING IT IS THE SUPPORTED WAY to make the guest
+        # process a new cloud-init payload: change the trailing hex, commit, restart the VM. Pinning
+        # it also makes a rebuild reproducible — a fresh apply would otherwise take a random
+        # webhook-generated UUID and give the guest a different identity every time.
+        #
+        # The value below is DELIBERATELY NOT the de5c3338-…c07 the live guest booted with: that
+        # difference is what makes the next boot a NEW cloud-init instance and installs the
+        # `install-uos` helper + its systemd unit. `serial` is pinned to the value the webhook
+        # already stamped, so this change does not also churn the guest's DMI serial.
+        firmware:
+          uuid: "de5c3338-6d32-4956-89d1-12c8272b2c11"
+          serial: "24bb12ac-d311-45f6-a75b-90a9e1b63e18"
         # FLAG (verify-at-bringup): Ubuntu 24.04 amd64 cloud images are hybrid BIOS+UEFI bootable,
         # so KubeVirt's default SeaBIOS SHOULD boot cleanly (unlike the Fedora containerdisk, which
-        # is UEFI-only and needed an explicit efi block). If the guest drops to emergency/no-boot,
-        # add:  firmware: { bootloader: { efi: { secureBoot: false } } }
+        # is UEFI-only and needed an explicit efi block). RESOLVED at bring-up: boots as shipped;
+        # the guest reports Ubuntu 24.04.4 LTS.
         cpu:
           cores: 2
         memory:
@@ -58,9 +89,28 @@ spec:
             - name: cloudinit
               disk:
                 bus: virtio
+            # The operator's installer URL (+ optional SSH key), delivered as a tiny read-only
+            # iso9660 disk — see the `uosdata` volume near the bottom of this file.
+            - name: uosdata
+              disk:
+                bus: virtio
           interfaces:
+            # MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
+            # KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
+            # /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
+            # that file while the NoCloud instance-id is unchanged — and the instance-id here is
+            # domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
+            # handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
+            # administratively DOWN with no address and no DHCP attempt.
+            # That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
+            # device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
+            # (because the per-instance modules were skipped), yet UOS is installed and
+            # qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
+            # Pinning the MAC to the value the current VMI already holds makes the guest's netplan
+            # match permanently, so this cannot recur on the next recreation.
             - name: default
               masquerade: {}
+              macAddress: a6:d7:c0:b3:72:fa
               # Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
               # these and (b) surfaces them as containerPorts on the virt-launcher pod so the
               # LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
@@ -83,41 +133,112 @@ spec:
         - name: rootdisk
           dataVolume:
             name: unifi-rootdisk
+        # =====================================================================================
+        # THE OPERATOR'S ONE FIELD — SOPS Secret `unifi-uos-installer`, delivered as a DISK.
+        # =====================================================================================
+        # KubeVirt exposes a Secret as a small read-only iso9660 disk whose volume label we choose;
+        # each Secret key becomes a file on it. The guest mounts it by label (LABEL=UOSDATA) and
+        # reads `installer-url` plus an optional `console-password`. The Secret ships ENCRYPTED with
+        # REPLACE-ME placeholders, and the guest-side script treats anything that is not an https
+        # URL as "not provided yet" and exits 0 — so the VM boots and stays healthy until the
+        # operator fills it in. See unifi-uos-installer.sops.yaml.example next to this file.
+        #
+        # `console-password` exists because THERE IS CURRENTLY NO WAY INTO THIS GUEST. The stock
+        # Ubuntu cloud image ships a locked `ubuntu` account, this cloud-init never installed a key,
+        # and SSH could not help anyway: the masquerade interface above forwards ONLY the declared
+        # ports, and 22 is not one of them — probed from inside virt-launcher on 2026-09-14, the
+        # guest answers RST on 22 exactly as it does on the UniFi ports. The serial console is the
+        # only door, and a console needs a password. Set it and `virtctl -n unifi console unifi`
+        # works; `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` already gives
+        # read-only console output with no virtctl at all.
+        #
+        # WHY A SEPARATE DISK AND NOT `cloudInitNoCloud.secretRef` (which would also lift the
+        # 2048-byte inline cap): secretRef moves the ENTIRE cloud-config into ciphertext. Two
+        # concrete costs. (a) The shell below stops being reviewable in a diff and falls outside the
+        # L0 dollar-collapse guard, which can only read plaintext. (b) Flux runs its postBuild
+        # envsubst over the DECRYPTED resources, so the escape contract in
+        # clusters/base/apps-vms.yaml would still govern text nobody can see in git — the exact
+        # shape of the 2026-09-13 nextcloud-db incident. Keeping the script in plaintext and putting
+        # ONLY the operator's two values behind SOPS keeps both properties.
+        #
+        # A CHANGED SECRET REACHES THE GUEST ON THE NEXT VMI START with NO firmware-UUID bump: the
+        # disk is rebuilt from the Secret every time the VMI starts, and the systemd unit below
+        # re-reads it on EVERY boot instead of relying on cloud-init (which will not re-run — see
+        # the firmware block above). The UUID bump is a ONE-TIME step, to get that unit installed.
+        - name: uosdata
+          secret:
+            secretName: unifi-uos-installer
+            volumeLabel: UOSDATA
         - name: cloudinit
           cloudInitNoCloud:
-            # INLINE (kept < KubeVirt's 2048-byte userData limit — so NO SSH key here; add one back via
-            # userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
-            # signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
-            # Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
-            # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
-            # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
-            # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
-            # as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
-            # by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
-            # an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
-            # and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
-            # flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
-            # text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
-            # wording in clusters/base/apps-vms.yaml overstated it.
+            # INLINE, and it MUST stay <= 2048 bytes. That ceiling was MEASURED against the
+            # DEPLOYED KubeVirt v1.9.0 rather than quoted: a server-side dry-run apply of an
+            # otherwise-identical VM accepted 2048 bytes of userData and rejected 2049 with
+            # "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for
+            # larger data." The payload below is ~1.5 KB; keep additions small.
+            #
+            # WHAT IT DOES: installs qemu-guest-agent (the KubeVirt AgentConnected signal, which is
+            # also the proof cloud-init ran) and preps Podman for UOS, then writes `install-uos`
+            # plus a systemd unit that runs it on EVERY boot. The unit — not cloud-init — is what
+            # makes a later URL change take effect, because cloud-init will not re-run per-instance
+            # modules while the instance-id is unchanged.
+            #
+            # `install-uos` IS IDEMPOTENT AND WILL NOT REINSTALL OVER A WORKING CONTROLLER. It bails
+            # out the moment systemd knows a `uosserver.service` — which the live guest ALREADY has:
+            # the serial console log for the current boot shows both `uosserver.service` and
+            # `uosserver-updater.service` starting, and the root filesystem is 6.3 GB of 40 GB used.
+            # UOS is installed here; see the README for what is actually wrong (the guest lost its
+            # IP). A stamp file at /var/lib/uos-installed is the second, weaker guard for the case
+            # where the unit name changes in some future UOS release.
+            #
+            # ENVSUBST: apps-vms declares postBuild.substituteFrom (for ${UNIFI_L4_VIP} in
+            # 50-service-l4.yaml), so Flux's envsubst DOES run over this text. ${BASE_DOMAIN} below
+            # is a deliberate substitution. Everything else is bare `$name` or `$(cmd)`, both of
+            # which envsubst leaves untouched — verified against the live VMI, whose rendered
+            # userData still carries its shell variables verbatim. There is deliberately NO
+            # brace-form shell expansion left in this payload, so nothing here needs escaping; if
+            # you add one, escape it per the contract in clusters/base/apps-vms.yaml, and NEVER
+            # write an indexed expansion.
             userData: |-
               #cloud-config
               hostname: unifi
               fqdn: unifi.${BASE_DOMAIN}
               package_update: true
               package_upgrade: false
-              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
+              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl]
               write_files:
                 - path: /usr/local/sbin/install-uos
                   permissions: '0755'
                   content: |
-                    #!/usr/bin/env bash
+                    #!/bin/bash
                     set -euo pipefail
-                    if [ -z "$${UOS_URL:-}" ]; then
-                      UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
-                    fi
-                    [ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
-                    t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
+                    d=/run/uos; mkdir -p $d
+                    mountpoint -q $d || mount -o ro LABEL=UOSDATA $d
+                    p=$(tr -d '\r\n' < $d/console-password || true)
+                    case "$p" in ""|REPLACE-ME) ;; *) echo "ubuntu:$p" | chpasswd ;; esac
+                    systemctl cat uosserver.service >/dev/null 2>&1 && exit 0
+                    s=/var/lib/uos-installed
+                    [ -e $s ] && exit 0
+                    u=$(tr -d '\r\n' < $d/installer-url || true)
+                    case "$u" in https://*) ;; *) echo "UOS installer-url not set yet"; exit 0;; esac
+                    t=$(mktemp)
+                    curl -fL --retry 5 --retry-delay 10 -o $t "$u"
+                    chmod +x $t
+                    yes y | $t install || yes y | $t
+                    rm -f $t; date -Is > $s
+                - path: /etc/systemd/system/uos-install.service
+                  content: |
+                    [Unit]
+                    Description=Install UniFi OS Server
+                    Wants=network-online.target
+                    After=network-online.target
+                    [Service]
+                    Type=oneshot
+                    ExecStart=/usr/local/sbin/install-uos
+                    StandardOutput=journal+console
+                    [Install]
+                    WantedBy=multi-user.target
               runcmd:
-                - [ systemctl, enable, --now, qemu-guest-agent ]
-                - [ systemctl, enable, --now, podman.socket ]
-                - [ bash, -lc, "/usr/local/sbin/install-uos 2>&1 | tee /var/log/uos-install.log > /dev/console" ]
+                - [systemctl, enable, --now, qemu-guest-agent]
+                - [systemctl, enable, --now, podman.socket]
+                - [systemctl, enable, --now, uos-install.service]

apps/vms/unifi/50-service-l4.yaml

@@ -21,8 +21,61 @@
 # k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
 #
 # selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
-# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
-# VIP pinned to the node running the VM (also what L2 announcement does).
+# into the guest.
+#
+# -------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
+# -------------------------------------------------------------------------------------------
+# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
+# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
+# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
+#
+# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
+#    per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
+#    matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
+#    MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
+#    `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
+#    announcing node that holds no backend has nowhere to send the packet and answers ICMP port
+#    unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
+#    (the leases were 8 days old); the operator's controller UI and device inform path were down.
+#    It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
+#    deleting the Lease forces a re-election — and that re-election is random, so it can pick a
+#    backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
+#    reference implementation this file now follows.
+#
+# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
+#    interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
+#    SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
+#    controller has always seen the pod-side masquerade gateway as the source, never the LAN
+#    client. There is a SECOND translation below the Service, and it already destroyed the very
+#    thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
+#    it is giving up an address the guest could not see in the first place.
+#
+#    Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
+#    MAC (and the device reports its own IP inside the inform payload), not the packet source,
+#    and guest-portal authorization runs on the gateway appliance, not on this controller.
+#
+# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
+# is what makes this survive a reschedule and a rebuild.
+#
+# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
+# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
+# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
+# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
+# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
+# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
+# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
+# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
+# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
+#
+# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
+# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
+# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
+# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
+# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
+# fix would be a NEW restriction that could silently break a remote site's device informing over
+# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
+# change, made deliberately and tested against the device fleet.
 #
 # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
 # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
@@ -49,7 +102,7 @@ metadata:
     external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:
@@ -72,7 +125,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:

apps/vms/unifi/README.md

@@ -1,18 +1,48 @@
 # UniFi Network controller (UniFi OS Server) — VM onboarding
 
-> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
-> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
-> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
-> or install cannot stall the app tier.
+> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
+> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
+> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
+> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
+> cannot stall the app tier.
 >
-> **Status (verified 2026-08-26):**
-> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
-> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
-> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
-> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
-> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
->   (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
->   The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
+> **Status (re-verified against live prod 2026-09-14):**
+> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
+> - CDI import — **Succeeded** (100%).
+> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
+>   `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
+> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
+> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
+> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
+>   `:8080` refuse connections.
+
+## What is actually wrong (2026-09-14)
+
+The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
+That was wrong, and acting on it would have meant reinstalling over a working controller.
+
+Evidence, all read-only, from the live cluster:
+
+| Claim | How it was checked | Result |
+|---|---|---|
+| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
+| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
+| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
+| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
+| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
+
+So the controller is installed and its units start; nothing downstream of the guest's own network
+stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
+~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
+UOS taking over host networking, but **that has not been proven and cannot be, from outside the
+guest.** Diagnosing it needs a shell, which is the second problem below.
+
+**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
+no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
+nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
+ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
+guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
+new Secret is for.
 
 ## What this is
 
@@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
 | `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
 | `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
 | `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
-| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
+| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
 | `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
-| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
+| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
+| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
+| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
 | `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
 
-The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
-serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
-this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
+The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
+`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
+version. Browser access is via the L4 VIP. Re-add it if/when that lands.
 
-### Cloud-init delivery — two constraints learned the hard way
+### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)
 
-1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
-   prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
-   shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
-   Secret deliberately.
-2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
-   `$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
-   therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
+1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
+   of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
+   userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
+   payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
+   is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
+   validate as an unknown field.
+2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
+   nothing.** The NoCloud seed's `instance-id` is exactly
+   `spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
+   (so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
+   product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
+   `runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
+   effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
+   to force a re-run. See the long comment in `30-virtualmachine.yaml`.
+3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
+   been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
+   envsubst therefore runs over the cloud-init payload, and the escape contract in
+   `clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
+   against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
+   indexed expansion is a hard error that fails the whole Kustomization.
 
 ### Node prerequisites this VM forced onto the cluster
 
@@ -56,15 +101,31 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 - **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
   3-replica class made the import fail with insufficient storage.
 
-## UOS install method (researched)
+## UOS install method (researched 2026-09-14)
 
-- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
-  2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
-- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
-  `ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
-  **`https://<host>:11443`**.
-- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
-  independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
+- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
+  Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
+  (the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
+  Podman must be at `/usr/bin/podman`.
+- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
+  in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
+- **Is the URL a short-lived signed link?** *Probably not.* The links look like
+  `https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
+  **no signature query string**, and public community installers hard-code them. It is
+  account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
+  account-derived material out of plaintext config — but do **not** build logic around expiry.
+- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
+  Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
+  ecosystem actually does is pipe confirmations in: the widely used community installer
+  (`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
+  either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
+  `yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
+  fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
+  (no URL has been supplied, and UOS is already installed here), so treat it as "established by
+  convergent third-party practice", not as verified-by-us.
+- The previously shipped helper had two defects worth recording: it resolved the URL from
+  `fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
+  `unifi-os-server`, and it invoked the binary with **no `install` subcommand**.
 
 ### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
 
@@ -78,76 +139,143 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 | 3478 | UDP | **STUN** (device ⇄ controller) |
 | 10001 | UDP | **L2 device discovery** |
 
+**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
+the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.
+
 ## L4 exposure design (the hard part)
 
 The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
 or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
 
 - **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
-  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
-  The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
-  flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
+  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
+  (`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
+  `default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
+  the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
+  existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
+  / `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
+  decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
 - `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
-  declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
-- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
-  — via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
-  Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
-  reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
-  masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
-  explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
-- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
-  LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
-  `minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
+  declared ports into the guest.
+- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
+  used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
+  Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
+  leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
+  and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
+  it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
+  why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
+- **Device-adoption story:** point devices' **inform host** at the VIP
+  (`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
+  `set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
+  subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
+  broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
+  zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
+  on other VLANs must route to the VIP and be inform-pointed anyway.
+- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
+  per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.
 
-**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
-`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
-(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
+**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
+the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
+`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
+zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
+apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
+flip, so this hostname is current, not a dev leftover.
+
+## The operator's one field — installing UOS without a shell
+
+`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
+the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
+`uos-install.service` reads it **on every boot**:
+
+```bash
+sops apps/vms/unifi/unifi-uos-installer.sops.yaml    # fill installer-url and/or console-password
+git commit && git push                               # merge to the branch prod's Flux tracks
+kubectl -n unifi delete vmi unifi                    # `running: true` recreates it; no virtctl needed
+kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
+  -c guest-console-log                               # watch the boot + the install, read-only
+```
+
+Design notes that matter if you change this:
+
+- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
+  cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
+  L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
+  resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
+  operator values in SOPS.
+- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
+  the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
+  firmware-UUID bump.
+- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
+  a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
+  guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
+  controller on next boot".
+- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
+  `apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
+  `clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
+  guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.
 
 ## Open questions / risks
 
-1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
-   account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
-   **interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
-   the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
-   **Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
-   invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
-2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
+1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
+   requires a shell, which requires `console-password` above plus one restart. First things to run
+   on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
+   `ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
+   own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
+   capture the state before rebooting a second time if you can.
+2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
+   install-method section: no official silent flag exists, `yes y | <installer> install` is the
+   convergent community practice, and it has not been executed on our guest. Treat the first real
+   install as something to watch on the console log, not as a fire-and-forget.
+3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
    import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
    Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
    fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
-3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
-4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
-   `192.168.1.53–.60` pool (`.53` is the shared Gateway).
-5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
+4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
+5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
+   prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
+   (`.23`) or the node addresses (`.10/.11/.12`).
+6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
    migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
-6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
+7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
    installer and for optional remote access.
-7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
+8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
    masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
+9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
+   `kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
+   `virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
+   the cluster's **v1.9.0**.
 
 ## Remaining bring-up
 
-Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
+Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
+is written for a from-scratch cluster; the prod addresses are the live ones.
 
-1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
+1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
 2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
-   Kustomization (not an app wave).
+   Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
 3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
 4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
-   `192.168.1.54`.
-5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
-   and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
-   ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
-6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
+   `192.168.2.21`.
+5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
+   `uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
+   fill `installer-url`, commit, and the guest installs itself on first boot.
+6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
+   console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
+7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
    complete the first-run wizard.
-7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
-8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
+8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
+   split-horizon) and from in-cluster (coredns-custom).
+9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
    connects (green). Test guest portal / throughput if used.
-9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
-10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
+10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
+11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
 
 **Quick health check today:**
 ```bash
 kubectl -n unifi get dv,vmi,svc
 kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
+# the two that would have caught the current failure:
+kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}'   # expect an ipAddress
+kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
+  -c compute -- ip neigh                                                 # expect 10.0.2.2 REACHABLE
 ```

apps/vms/unifi/kustomization.yaml

@@ -11,3 +11,11 @@ resources:
   - 30-virtualmachine.yaml
   - 40-networkpolicy.yaml
   - 50-service-l4.yaml
+  # SOPS-encrypted. Carries the operator's UOS installer URL (+ optional SSH key) and is mounted
+  # into the guest as the UOSDATA disk by 30-virtualmachine.yaml. Decrypted by the `apps-vms` Flux
+  # Kustomization, which clusters/dev and clusters/prod patch with the `decryption` stanza — the
+  # FIRST encrypted content under apps/vms, so that patch was added with this file. The plaintext
+  # template + the operator hand-off steps live in the `.example` sibling of the file below, which
+  # is deliberately NOT named here: the L0 no-plaintext-secrets check fails any kustomization whose
+  # text contains an example template's filename, on the assumption it is being applied.
+  - unifi-uos-installer.sops.yaml

apps/vms/unifi/unifi-uos-installer.sops.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: unifi-uos-installer
+    namespace: unifi
+type: Opaque
+stringData:
+    installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
+    console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
+            WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
+            TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
+            cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
+            kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-19T02:47:13Z"
+    mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/vms/unifi/unifi-uos-installer.sops.yaml.example

@@ -0,0 +1,71 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the two values that (a) let the UniFi VM install UniFi OS Server by itself, with
+# nobody typing commands inside the guest, and (b) give the operator a way INTO that guest at all.
+# KubeVirt mounts this Secret into the VM as a tiny read-only iso9660 disk labelled `UOSDATA` (see
+# the `uosdata` volume in 30-virtualmachine.yaml); each key below becomes a file of the same name on
+# that disk, and the in-guest `uos-install.service` reads them on EVERY boot.
+#
+# READ THIS FIRST — UOS IS ALREADY INSTALLED ON THE LIVE PROD GUEST (verified 2026-09-14, see the
+# README "What is actually wrong"). `installer-url` is therefore NOT the fix for today's outage; it
+# is the rebuild path and the "the install never happened" path. `install-uos` refuses to run the
+# installer at all while systemd knows a `uosserver.service`, so filling this in cannot clobber a
+# working controller. `console-password` IS useful today: it is the only way to get a shell.
+#
+# WHY THE URL IS BEHIND SOPS: it is obtained through the operator's own (free) Ubiquiti account
+# session at ui.com/download/software/unifi-os-server, it identifies a specific build, and it is not
+# ours to redistribute. Treat it as account-derived material, not as public configuration.
+# HONEST CAVEAT, checked 2026-09-14: the resulting `https://fw-download.ubnt.com/data/
+# unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` URL carries NO signature query string,
+# and public community install scripts hard-code these URLs — so it does not behave like a
+# short-lived signed link. It does change every release. Encrypting it costs nothing and keeps the
+# repo's "account-derived material is not config" line clean, but do NOT design around "it expires".
+#
+# HAND-OFF STEPS (operator) — this is the whole job:
+#  (a) Sign in at https://ui.com/download/software/unifi-os-server, pick the Linux (x64) build, and
+#      copy the download link (right-click -> Copy link address). Do NOT download it here; the guest
+#      fetches it itself.
+#  (b) Open the encrypted sibling in your editor:
+#         sops apps/vms/unifi/unifi-uos-installer.sops.yaml
+#  (c) Replace the two REPLACE-ME values. Either may be left as the placeholder — the guest skips a
+#      placeholder rather than failing.
+#  (d) Save/quit (sops re-encrypts in place), commit, push, merge to the branch prod tracks.
+#  (e) Restart the VM once so the new cloud-init instance and the new disk are picked up:
+#         kubectl -n unifi delete vmi unifi        # `running: true` recreates it; no virtctl needed
+#      Watch it come up WITHOUT virtctl:
+#         kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher \
+#           -o name | head -1) -c guest-console-log
+#  (f) UI: https://192.168.2.21:11443 (prod) or https://unifi.home.keithjackson.dev:11443.
+#
+# IF YOU LATER NEED A NEW URL (new release, dead link): repeat (b)-(e). No firmware-UUID bump is
+# needed for that — the disk is rebuilt from this Secret on every VMI start and the systemd unit
+# re-reads it every boot. The UUID bump in 30-virtualmachine.yaml is only needed when the cloud-init
+# payload ITSELF changes.
+#
+# The file is already listed under `resources:` in apps/vms/unifi/kustomization.yaml, and `apps-vms`
+# is patched with the SOPS `decryption` stanza in clusters/dev + clusters/prod.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: unifi-uos-installer
+  namespace: unifi
+type: Opaque
+stringData:
+  # KEY NAMES ARE LOAD-BEARING: each key becomes a FILE of that exact name on the UOSDATA disk, and
+  # /usr/local/sbin/install-uos reads them by path. Renaming a key breaks the install silently.
+  #
+  # The signed-in download link for UniFi OS Server, Linux x64. Anything that is not an https URL
+  # (this placeholder included) is treated by the guest as "not provided yet": the unit logs
+  # "UOS installer-url not set yet" and exits 0, so the VM stays healthy and simply does not install.
+  installer-url: REPLACE-ME
+  # Password set on the guest's `ubuntu` account on every boot, for `virtctl -n unifi console unifi`.
+  # THIS IS THE ONLY DOOR INTO THE GUEST. The stock Ubuntu 24.04 cloud image ships `ubuntu` locked,
+  # no key was ever installed, and an SSH key would not help: the VM's masquerade interface forwards
+  # only the ports declared in 30-virtualmachine.yaml and 22 is not among them (probed from inside
+  # virt-launcher, 2026-09-14 — the guest RSTs on 22 just as it does on 11443). Generate one with
+  #     openssl rand -base64 24
+  # Leave it as REPLACE-ME to keep the account locked; the script skips the placeholder by name.
+  console-password: REPLACE-ME

apps/wave-0/gitea.yaml

@@ -238,26 +238,6 @@ spec:
     - name: gitea
       port: 3000
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: gitea
-  namespace: gitea
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: gitea
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: gitea
-  namespaceSelector:
-    matchNames:
-    - gitea
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -272,11 +252,11 @@ spec:
   - name: gitea.slis
     rules:
     - alert: gitea-down
-      expr: up{job="gitea"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"} == 0 or absent(kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"})
       for: 5m
       labels:
         severity: critical
         service: gitea
       annotations:
         summary: gitea is down
-        description: 'gitea has been unreachable for 5m (SLI: up).'
+        description: 'gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/jackett.yaml

@@ -268,6 +268,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -284,26 +285,6 @@ spec:
     - name: jackett
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: jackett
-  namespace: jackett
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: jackett
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: jackett
-  namespaceSelector:
-    matchNames:
-    - jackett
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -318,11 +299,11 @@ spec:
   - name: jackett.slis
     rules:
     - alert: jackett-down
-      expr: up{job="jackett"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"} == 0 or absent(kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"})
       for: 5m
       labels:
         severity: critical
         service: jackett
       annotations:
         summary: jackett is down
-        description: 'jackett has been unreachable for 5m (SLI: up).'
+        description: 'jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/kustomization.yaml

@@ -14,8 +14,10 @@ resources:
 - nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
+- pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
+- plex-claim-secret.sops.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml

apps/wave-0/pihole-dns.yaml

@@ -0,0 +1,196 @@
+---
+# =================================================================================================
+# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole`
+# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS
+# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition
+# schema genuinely cannot express it: schemas/servicedefinition.schema.json is
+# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned
+# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at
+# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and
+# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this
+# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and
+# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog
+# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the
+# schema has no word for.
+#
+# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is
+# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service
+# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09
+# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on
+# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that
+# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0).
+#
+# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the
+# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config
+# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is
+# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it
+# can never be allowed to move.
+#
+# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53
+# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same
+# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the
+# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a
+# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently
+# and confusingly.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC
+# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself
+# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS,
+# and publishing a resolver's address in a public zone invites exactly the traffic the source-range
+# filter below exists to refuse. Internal name, internal answer, no public record — which is also
+# why this change touches nothing under platform/externaldns/.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to
+# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured,
+# not assumed (Cilium 1.20.1, prod, 2026-09-14):
+#
+#   * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the
+#     CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does
+#     NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services
+#     landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3,
+#     `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1.
+#   * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the
+#     packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three
+#     nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol.
+#   * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod
+#     rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three
+#     minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a
+#     re-election — and that re-election is random, so it may pick a backend-less node again.
+#
+# A resolver that works one time in three and stays broken after any reschedule is not a resolver.
+# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is
+# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement
+# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all
+# answering.
+#
+# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in
+# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the
+# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client
+# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two
+# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole
+# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts
+# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity —
+# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode:
+# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract
+# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
+#
+# -------------------------------------------------------------------------------------------------
+# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
+# -------------------------------------------------------------------------------------------------
+# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
+# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
+# datapath — it never reaches a backend and never reaches a network policy. This is what makes
+# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
+# range to a /24 that excludes the test client, after which the same query that had been answering
+# stopped being answered at all, and restoring it, after which it answered again.
+#
+# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
+# restriction once the source has been rewritten: measured on this cluster, the forwarded query
+# arrives from the ingress node's internal router address and Cilium resolves that source to the
+# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the
+# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction
+# lives here, where it is actually enforceable, and the policy below restricts the PORTS.
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-udp
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+  annotations:
+    lbipam.cilium.io/sharing-key: "pihole-dns"
+    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  loadBalancerSourceRanges:
+    - ${LAN_CIDR}
+  selector:
+    app.kubernetes.io/name: pihole
+  ports:
+    - { name: dns-udp, port: 53, targetPort: 53, protocol: UDP }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-tcp
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+  annotations:
+    lbipam.cilium.io/sharing-key: "pihole-dns"   # same key + same IP => shares the VIP with -udp
+    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  loadBalancerSourceRanges:
+    - ${LAN_CIDR}
+  selector:
+    app.kubernetes.io/name: pihole
+  ports:
+    - { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP }
+---
+# =================================================================================================
+# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports.
+# =================================================================================================
+# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered
+# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the
+# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything
+# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the
+# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule:
+# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that
+# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for
+# it and no policy for it.
+#
+# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is
+# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's
+# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium
+# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was
+# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced
+# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
+# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
+# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
+# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
+#
+# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
+# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
+# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none
+# needs PiHole on :53) on fewer ports, behind a filter those two do not have.
+#
+# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the
+# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to
+# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same
+# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not
+# an internet-facing open resolver. Say so out loud rather than implying the filter is total.
+#
+# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved
+# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the
+# node-to-node forwarding that IS the Cluster-policy datapath both need them.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-lan-dns-ingress
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+    - fromEntities:
+        - world         # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
+        - host          # this node's kubelet probes
+        - remote-node   # the node-to-node hop that forwards a query to wherever the pod runs
+      toPorts:
+        - ports:
+            - { port: "53", protocol: UDP }
+            - { port: "53", protocol: TCP }

apps/wave-0/pihole.yaml

@@ -81,6 +81,15 @@ spec:
         image: pihole/pihole
         ports:
         - containerPort: 80
+        env:
+        - name: FTLCONF_dns_listeningMode
+          value: ALL
+        - name: FTLCONF_dns_upstreams
+          value: 8.8.8.8;8.8.4.4
+        - name: FTLCONF_dns_rateLimit_count
+          value: '0'
+        - name: FTLCONF_misc_dnsmasq_lines
+          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
         volumeMounts:
         - name: config
           mountPath: /data/config
@@ -223,6 +232,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -239,26 +249,6 @@ spec:
     - name: pihole
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: pihole
-  namespace: pihole
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: pihole
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: pihole
-  namespaceSelector:
-    matchNames:
-    - pihole
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -273,11 +263,11 @@ spec:
   - name: pihole.slis
     rules:
     - alert: pihole-down
-      expr: up{job="pihole"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
       for: 5m
       labels:
         severity: critical
         service: pihole
       annotations:
         summary: pihole is down
-        description: 'pihole has been unreachable for 5m (SLI: up).'
+        description: 'pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/plex-claim-secret.sops.yaml

@@ -0,0 +1,77 @@
+# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
+# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
+#
+# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
+# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
+# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
+# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
+# not prose, so the upstream conditions below are described in words.
+#
+# THIS FILE NO LONGER HOLDS THE PLACEHOLDER. Corrected 2026-09-29: the server was CLAIMED on that
+# date, and the value here is the SPENT token from that claim. It is inert and safe to leave --
+# upstream gates the exchange on PlexOnlineToken being ABSENT from /config, and it is now present on
+# the plex-config PVC, so this value is a NO-OP on every start whatever it holds. Deleting the Secret
+# would NOT be safe (see below), which is why a spent token stays here rather than being blanked.
+#
+# The claim token is SHORT-LIVED -- minutes. Read the countdown that plex.tv/claim renders when it
+# issues one; do NOT budget against a number written in this tree. Earlier revisions of this comment
+# asserted a "~4-minute TTL" as established fact with no citation, and upstream plexinc/pms-docker
+# documents no expiry at all. That uncited number was load-bearing for the whole procedure design.
+#
+# This file exists to make the Secret — and therefore the env reference — ALWAYS RESOLVABLE IN THE
+# FLUX GRAPH, not to carry a working token. Re-claiming (only needed after a /config wipe) is
+# scripted: infra/plex/claim-ship.sh, measured at ~2m45s end to end.
+#
+# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
+# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
+# the same file extracted from the running `plexinc/pms-docker:latest` image):
+#   - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
+#     PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
+#     skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
+#     README: "If server is already logged in, this parameter is ignored."
+#   - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
+#     empty body. curl's exit status is never checked, the sed for the authentication-token element
+#     yields the empty string, and the success branch at line 120 simply does not fire. There is no
+#     else branch and no non-zero exit; the script has no `set -e` and ends by touching
+#     /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
+#     It does NOT crashloop.
+#   - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
+#     if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
+# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
+# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
+# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
+# by hand at claim time.
+#
+# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
+#   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+# Never decrypt this file into a shell, a log, or a shared terminal.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: plex-claim
+    namespace: plex
+    labels:
+        app.kubernetes.io/name: plex
+type: Opaque
+stringData:
+    PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
+            cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
+            THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
+            QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
+            RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-29T06:31:16Z"
+    mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/plex.yaml

@@ -132,6 +132,11 @@ spec:
           value: Etc/UTC
         - name: ALLOWED_NETWORKS
           value: 10.0.0.0/8,192.168.0.0/16
+        - name: PLEX_CLAIM
+          valueFrom:
+            secretKeyRef:
+              name: plex-claim
+              key: PLEX_CLAIM
         volumeMounts:
         - name: config
           mountPath: /config
@@ -244,26 +249,6 @@ spec:
     - name: plex
       port: 32400
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: plex
-  namespace: plex
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: plex
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: plex
-  namespaceSelector:
-    matchNames:
-    - plex
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -278,11 +263,11 @@ spec:
   - name: plex.slis
     rules:
     - alert: plex-down
-      expr: up{job="plex"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})
       for: 5m
       labels:
         severity: critical
         service: plex
       annotations:
         summary: plex is down
-        description: 'plex has been unreachable for 5m (SLI: up).'
+        description: 'plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/qbittorrent.yaml

@@ -344,6 +344,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -360,26 +361,6 @@ spec:
     - name: qbittorrent
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: qbittorrent
-  namespace: qbittorrent
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: qbittorrent
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: qbittorrent
-  namespaceSelector:
-    matchNames:
-    - qbittorrent
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -394,11 +375,11 @@ spec:
   - name: qbittorrent.slis
     rules:
     - alert: qbittorrent-down
-      expr: up{job="qbittorrent"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"} == 0 or absent(kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"})
       for: 5m
       labels:
         severity: critical
         service: qbittorrent
       annotations:
         summary: qbittorrent is down
-        description: 'qbittorrent has been unreachable for 5m (SLI: up).'
+        description: 'qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-0/tests/app-slis.test.yaml

@@ -0,0 +1,267 @@
+# promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
+# Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
+# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
+# the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
+# tests run against the exact expressions Flux applies, never a copy.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHY THESE EXIST. Until 2026-09-13 every one of these rules read `up{job="<app>"} == 0` while the
+# rendered ServiceMonitor named a Service port (`metrics`) that no rendered Service defined. The
+# monitor discovered ZERO targets, so no `up{job="<app>"}` series existed and the rule could not
+# fire — on prod, `count by (job) (up)` listed 24 jobs and not one of the twelve appeared.
+# `PrometheusTargetsMissing` could not see it either: a target that was never discovered has no `up`
+# series to be 0. The rules now read kube-state-metrics / kubevirt series that DO exist for these
+# workloads (verified on prod), and every rule below is pinned in BOTH directions:
+#   * QUIET on the healthy state -> it is not permanently firing;
+#   * FIRING when the workload has no ready replica -> it is not inert;
+#   * FIRING when the signal is ABSENT -> the failure mode that produced this whole defect class
+#     (a rule that goes quiet because its series vanished) is itself alerted on.
+#
+# CLOCK/`for` CONVENTION: sample interval 1m, series 21 samples long, evaluated at 10m — comfortably
+# past every rule's `for: 5m` and well inside Prometheus' 5m lookback.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # ===========================================================================================
+  # 1. HEALTHY WAVE-0 (what prod looks like right now) -> every rule SILENT.
+  #    This is the "not permanently firing" half. It is also the test that catches a mis-spelled
+  #    label: get `deployment=` wrong and the `== 0` half matches nothing while the `absent()` half
+  #    matches everything, so this test goes red instead of the rule going quietly wrong.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 healthy - every app has a ready replica, the demo VM runs - all quiet
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running", node="k3s-prod-1"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 2. EACH APP DOWN IN TURN (0 available replicas) -> ONLY that app's rule fires.
+  #    The per-app isolation matters: a rule that forgot its `namespace=`/`deployment=` matchers
+  #    would light up every app at once and be useless for routing.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 plex has no available replica - only plex-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: plex
+              namespace: plex
+              deployment: plex
+            exp_annotations:
+              summary: "plex is down"
+              description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+    # Before `for: 5m` has elapsed the rule must be PENDING, not firing — proof the 5m is real and
+    # a single scrape blip does not page anyone.
+    promql_expr_test:
+      - expr: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})'
+        eval_time: 2m
+        exp_samples:
+          - labels: 'kube_deployment_status_replicas_available{deployment="plex", namespace="plex"}'
+            value: 0
+
+  - interval: 1m
+    name: wave-0 gitea/jackett/pihole/qbittorrent each down - each own rule fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
+        values: '0+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: gitea
+              namespace: gitea
+              deployment: gitea
+            exp_annotations:
+              summary: "gitea is down"
+              description: "gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: jackett-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: jackett
+              namespace: jackett
+              deployment: jackett
+            exp_annotations:
+              summary: "jackett is down"
+              description: "jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: pihole-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: pihole
+              namespace: pihole
+              deployment: pihole
+            exp_annotations:
+              summary: "pihole is down"
+              description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: qbittorrent-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: qbittorrent
+              namespace: qbittorrent
+              deployment: qbittorrent
+            exp_annotations:
+              summary: "qbittorrent is down"
+              description: "qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 3. THE SIGNAL VANISHES -> the rule FIRES. This is the defect class itself, asserted on.
+  #    The old `up{job=...} == 0` rule had exactly this shape of hole and was silent for it. Here
+  #    the Deployment series for plex is simply not in the input at all, and plex-down must still
+  #    fire (via the `absent()` half), with the labels the selector's equality matchers carry.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 plex Deployment series ABSENT (never scraped / workload gone) - plex-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: plex-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: plex
+              namespace: plex
+              deployment: plex
+            exp_annotations:
+              summary: "plex is down"
+              description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: gitea-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 4. THE VM: a VMI carries no replica count, so liveness is the EXISTENCE of a running-phase
+  #    series. Two ways to be down, both asserted: the VMI is gone entirely, and the VMI exists in
+  #    some OTHER phase. The second is the one a naive `kubevirt_vmi_info == 1` would miss, and it
+  #    is why the phase value is matched — LOWERCASE `running`, the value KubeVirt actually writes
+  #    (a capital-R matcher selects nothing and would be just as inert).
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-0 demo VM has no running-phase VMI - vm-demo-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: vm-demo
+              namespace: vm-demo
+              name: vm-demo
+              phase: running
+            exp_annotations:
+              summary: "vm-demo is down"
+              description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."
+
+  - interval: 1m
+    name: wave-0 demo VM exists but is Failed, not running - vm-demo-down still fires
+    input_series:
+      - series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="failed"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: vm-demo-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: vm-demo
+              namespace: vm-demo
+              name: vm-demo
+              phase: running
+            exp_annotations:
+              summary: "vm-demo is down"
+              description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."

apps/wave-0/vm-demo.yaml

@@ -115,26 +115,6 @@ spec:
   - toEntities:
     - world
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: vm-demo
-  namespace: vm-demo
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: vm-demo
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: vm-demo
-  namespaceSelector:
-    matchNames:
-    - vm-demo
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -149,11 +129,11 @@ spec:
   - name: vm-demo.slis
     rules:
     - alert: vm-demo-down
-      expr: up{job="vm-demo"} == 0
+      expr: absent(kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"})
       for: 5m
       labels:
         severity: critical
         service: vm-demo
       annotations:
         summary: vm-demo is down
-        description: 'vm-demo has been unreachable for 5m (SLI: up).'
+        description: 'vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info).'

apps/wave-1/gitea-actions-runner-secret.sops.yaml

@@ -4,7 +4,13 @@
 # against Gitea. Flux decrypts it via the sops-age key at reconcile (apps Kustomization
 # decryption is wired in clusters/dev/kustomization.yaml). Rotate by resetting the token
 # in Gitea (admin -> actions -> runners) and re-encrypting a fresh value here.
-# Recover/inspect: infra/devcontainer/dev.sh run sops -d apps/gitea-actions-runner-secret.sops.yaml
+# ROTATE: ./infra/gitea/runner-token-rotate.sh -- it mints a fresh instance-level token from the
+# Gitea API and writes it here via `sops set`, without the value ever being printed.
+# Do NOT `sops -d` this file into a terminal, a log, or an agent transcript. To check only that
+# it still decrypts, redirect: `sops -d <file> > /dev/null`.
+# NOTE: a fresh token alone may not re-register the runner. act_runner registers ONCE and keeps
+# its identity in /data/.runner on the act-runner-data PVC; if Gitea no longer holds a matching
+# row it skips registration silently. Clearing that PVC is what forces a real re-registration.
 apiVersion: v1
 kind: Secret
 metadata:
@@ -14,7 +20,7 @@ metadata:
         app.kubernetes.io/name: act-runner
 type: Opaque
 stringData:
-    token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
+    token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
 sops:
     kms: []
     gcp_kms: []
@@ -30,8 +36,8 @@ sops:
             MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
             UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-08-26T04:48:14Z"
-    mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
+    lastmodified: "2026-09-29T05:48:35Z"
+    mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
     pgp: []
     encrypted_regex: ^(data|stringData)$
     version: 3.9.4

apps/wave-1/gitea-actions.yaml

@@ -99,70 +99,79 @@ spec:
       # CI holds no cluster credentials — do not mount a SA token into the runner or
       # (via the shared netns) the job containers (R-SEC-02, docs/14).
       automountServiceAccountToken: false
+      # ---- Docker-in-Docker: the job executor daemon (privileged) -------------------
+      #
+      # NATIVE SIDECAR (k3s v1.36; sidecar containers are GA). It is an initContainer with
+      # `restartPolicy: Always`, which means: it starts BEFORE the regular containers, keeps
+      # running alongside them, and is terminated after them. kubelet holds `runner` until
+      # this container's startupProbe passes, so the ordering is enforced by the kubelet
+      # instead of hand-rolled shell in the runner.
+      #
+      # This REPLACES two bounded wait-loops that used to live in the runner's `command:`.
+      # Deleting them also deletes an envsubst hazard: clusters/base/apps-wave-1.yaml runs
+      # postBuild substitution over this ENTIRE manifest, comments included, and those loops
+      # depended on bare dollar-names surviving it untouched. There is now no shell here at
+      # all, so there is nothing to escape or mis-escape.
+      #
+      # WHY THE PROBE IS `exec docker ... info` AND NOT `tcpSocket`. A tcpSocket probe would
+      # re-open the exact gap the old Stage 1 existed to close: dockerd writes the TLS certs
+      # and only THEN begins listening, so the port can accept a connection while the certs
+      # are absent or incomplete -- and act_runner hard-exits on a missing
+      # /certs/client/ca.pem. A full TLS-verified `docker info` proves BOTH that the certs
+      # are usable and that the daemon answers.
+      #
+      # CORRECTION (measured 2026-09-29, after this conversion shipped): the commit that
+      # introduced this claimed the conversion would REMOVE the one-per-start dockerd line
+      # `http: TLS handshake error from 127.0.0.1: EOF`. IT DOES NOT. Exactly one still
+      # appears, immediately after "API listen on [::]:2376" -- it is now this probe's FIRST
+      # attempt, landing microseconds after dockerd begins listening and cut off by
+      # timeoutSeconds: 1. The next poll succeeds. It is cosmetic, exactly as cosmetic as it
+      # was before, and deliberately NOT chased: raising timeoutSeconds to hide one benign
+      # startup line would slow every subsequent poll. Left documented rather than silenced.
+      #
+      # periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
+      initContainers:
+        - name: dind
+          restartPolicy: Always   # <- this field is what makes it a SIDECAR
+          image: docker:27-dind
+          securityContext:
+            privileged: true
+          env:
+            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
+            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
+            - name: DOCKER_TLS_CERTDIR
+              value: /certs
+          volumeMounts:
+            - name: certs
+              mountPath: /certs
+            - name: dind-storage
+              mountPath: /var/lib/docker
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              memory: 2Gi
+          startupProbe:
+            exec:
+              command:
+                - docker
+                - --host=tcp://127.0.0.1:2376
+                - --tlsverify
+                - --tlscacert=/certs/client/ca.pem
+                - --tlscert=/certs/client/cert.pem
+                - --tlskey=/certs/client/key.pem
+                - info
+            periodSeconds: 1
+            failureThreshold: 60
       containers:
         # ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
         - name: runner
           image: gitea/act_runner:0.2.11
-          # TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
-          # entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
-          # boot and runs `act_runner daemon`.
-          #
-          # STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
-          #   and hard-exits if it is missing (a startup race against the sidecar).
-          # STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
-          #   necessary but NOT sufficient. dockerd writes the certs and only THEN begins
-          #   listening on :2376, so a runner that proceeds on the file alone still hard-exits
-          #   with "cannot ping the docker daemon at tcp://localhost:2376" — observed
-          #   crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
-          # Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
-          # surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
-          #
-          # ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
-          #   braced form or the double-dollar form; both are wrong here, for different reasons.
-          #   clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
-          #   Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
-          #   Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
-          #   and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
-          #     - A bare  $name  is NOT a reference to envsubst: it passes through untouched, and
-          #       that holds even when the name IS set in cluster-vars. So $i and $j reach the
-          #       cluster verbatim and both 60s bounds below are real, enforced bounds.
-          #     - $((i + 1))  is likewise untouched: "(" cannot begin a variable name.
-          #     - A BRACED dollar-name IS a reference and gets replaced — with the empty string
-          #       when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
-          #       which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
-          #     - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
-          #       form — still wrong for us, because on any path where envsubst does NOT run
-          #       (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
-          #       reads that leading double-dollar as its own PID, so a bound would compare a
-          #       string like 1234{i} against 60. The bare form is the only spelling that is
-          #       correct on BOTH paths.
-          #     - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
-          #       name") that fails this whole Kustomization — and this Kustomization gates every
-          #       merge in the repo. So never write one anywhere in this file, INCLUDING in a
-          #       comment. That is why this note names the three forms in words rather than
-          #       showing them: an earlier draft of it spelled them out literally and broke
-          #       `flux envsubst` on the file. (Same trap the indexed-expansion warning in
-          #       clusters/base/apps-vms.yaml describes.)
-          command:
-            - /bin/sh
-            - -c
-            - |
-              # Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
-              i=0
-              until [ -f /certs/client/ca.pem ]; do
-                i=$((i + 1))
-                [ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
-                echo "waiting for dind TLS certs ($i)"; sleep 1
-              done
-              # Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
-              # Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
-              j=0
-              until nc -z -w1 127.0.0.1 2376; do
-                j=$((j + 1))
-                [ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
-                echo "waiting for dind daemon on :2376 ($j)"; sleep 1
-              done
-              exec /sbin/tini -- /opt/act/run.sh
+          # No `command:` override. The image's own entrypoint (/sbin/tini -- /opt/act/run.sh)
+          # registers from env on first boot and runs `act_runner daemon`. The two wait-loops
+          # that used to live here are gone: kubelet now holds this container until the dind
+          # sidecar's startupProbe passes (see initContainers above).
           env:
             - name: GITEA_INSTANCE_URL
               value: https://gitea.${BASE_DOMAIN}
@@ -198,52 +207,6 @@ spec:
               memory: 128Mi
             limits:
               memory: 512Mi
-        # ---- Docker-in-Docker: the job executor daemon (privileged) ----
-        #
-        # FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
-        #   Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
-        #   k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
-        #   ordering that the loops above hand-roll — which also removes the envsubst hazard
-        #   from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
-        #   Shape to build:
-        #     1. Move this whole container into `initContainers:` as the FIRST entry.
-        #     2. Add `restartPolicy: Always` to it. That single field is what makes an
-        #        initContainer a sidecar: it starts before the regular containers, keeps
-        #        running alongside them, and is terminated after them.
-        #     3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
-        #        connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
-        #        --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
-        #        --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
-        #        it would re-open the same "port accepting but TLS/certs not ready" gap that
-        #        Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
-        #        preserves today's 60s cap.
-        #     4. kubelet then holds the `runner` container until that startupProbe passes, so
-        #        `runner` can drop its `command:` override completely and run the image's
-        #        default entrypoint.
-        #   Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
-        #   entire repository. If the conversion is wrong, nothing can merge — so it must land
-        #   alone, where the failure is unambiguous and not entangled with anything else, and
-        #   only after a real workflow run has been observed green on the converted pod.
-        - name: dind
-          image: docker:27-dind
-          securityContext:
-            privileged: true
-          env:
-            # Generate + publish TLS certs into the shared emptyDir; dockerd then listens
-            # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376.
-            - name: DOCKER_TLS_CERTDIR
-              value: /certs
-          volumeMounts:
-            - name: certs
-              mountPath: /certs
-            - name: dind-storage
-              mountPath: /var/lib/docker
-          resources:
-            requests:
-              cpu: 100m
-              memory: 256Mi
-            limits:
-              memory: 2Gi
       volumes:
         - name: data
           persistentVolumeClaim:

apps/wave-1/keycloak.yaml

@@ -184,26 +184,6 @@ spec:
     - name: keycloak
       port: 8080
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: keycloak
-  namespace: keycloak
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: keycloak
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: keycloak
-  namespaceSelector:
-    matchNames:
-    - keycloak
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -218,11 +198,11 @@ spec:
   - name: keycloak.slis
     rules:
     - alert: keycloak-down
-      expr: up{job="keycloak"} == 0
+      expr: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"})
       for: 5m
       labels:
         severity: critical
         service: keycloak
       annotations:
         summary: keycloak is down
-        description: 'keycloak has been unreachable for 5m (SLI: up).'
+        description: 'keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'

apps/wave-1/nextcloud.yaml

@@ -290,26 +290,6 @@ spec:
     - name: nextcloud
       port: 80
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: nextcloud
-  namespace: nextcloud
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: nextcloud
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: nextcloud
-  namespaceSelector:
-    matchNames:
-    - nextcloud
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -324,11 +304,11 @@ spec:
   - name: nextcloud.slis
     rules:
     - alert: nextcloud-down
-      expr: up{job="nextcloud"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"} == 0 or absent(kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"})
       for: 5m
       labels:
         severity: critical
         service: nextcloud
       annotations:
         summary: nextcloud is down
-        description: 'nextcloud has been unreachable for 5m (SLI: up).'
+        description: 'nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/radarr.yaml

@@ -348,6 +348,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -364,26 +365,6 @@ spec:
     - name: radarr
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: radarr
-  namespace: radarr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: radarr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: radarr
-  namespaceSelector:
-    matchNames:
-    - radarr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -398,11 +379,11 @@ spec:
   - name: radarr.slis
     rules:
     - alert: radarr-down
-      expr: up{job="radarr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"})
       for: 5m
       labels:
         severity: critical
         service: radarr
       annotations:
         summary: radarr is down
-        description: 'radarr has been unreachable for 5m (SLI: up).'
+        description: 'radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/semaphore.yaml

@@ -227,6 +227,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: oidc
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -243,26 +244,6 @@ spec:
     - name: semaphore
       port: 3000
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: semaphore
-  namespace: semaphore
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: semaphore
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: semaphore
-  namespaceSelector:
-    matchNames:
-    - semaphore
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -277,11 +258,11 @@ spec:
   - name: semaphore.slis
     rules:
     - alert: semaphore-down
-      expr: up{job="semaphore"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"} == 0 or absent(kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"})
       for: 5m
       labels:
         severity: critical
         service: semaphore
       annotations:
         summary: semaphore is down
-        description: 'semaphore has been unreachable for 5m (SLI: up).'
+        description: 'semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/sonarr.yaml

@@ -348,6 +348,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -364,26 +365,6 @@ spec:
     - name: sonarr
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: sonarr
-  namespace: sonarr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: sonarr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: sonarr
-  namespaceSelector:
-    matchNames:
-    - sonarr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -398,11 +379,11 @@ spec:
   - name: sonarr.slis
     rules:
     - alert: sonarr-down
-      expr: up{job="sonarr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
       for: 5m
       labels:
         severity: critical
         service: sonarr
       annotations:
         summary: sonarr is down
-        description: 'sonarr has been unreachable for 5m (SLI: up).'
+        description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-1/tests/app-slis.test.yaml

@@ -0,0 +1,245 @@
+# promtool rule-unit tests for the wave-1 per-app `<app>-down` SLI rules.
+# Manifests under test: every PrometheusRule in apps/wave-1/ — nextcloud, radarr, semaphore, sonarr
+# (container/Deployment) and keycloak (StatefulSet, see below). Discovered by the SUITE convention;
+# the flattened manifests are the RENDERED ones Flux applies.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-1/tests/app-slis.test.yaml
+#
+# See apps/wave-0/tests/app-slis.test.yaml for the full history: these rules all read
+# `up{job="<app>"} == 0` until 2026-09-13 against a job no scrape target produced, and were
+# structurally inert on prod. Each is now pinned QUIET on the healthy state, FIRING on no-ready-
+# replica, and FIRING when the signal is ABSENT.
+#
+# KEYCLOAK IS THE INTERESTING ONE. Its server is a StatefulSet run by the Keycloak operator, NOT the
+# Deployment this ServiceDefinition would otherwise imply — so its rule reads
+# kube_statefulset_status_replicas_ready (catalog/keycloak.yaml declares
+# `observability.livenessWorkload: {kind: StatefulSet, name: keycloak}`). Had it been left to the
+# default, the `absent()` half would have matched forever and the alert would have been PERMANENTLY
+# FIRING, which is the other failure mode this repo has shipped to prod. Verified on prod
+# 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}
+# = 2, and the only Deployment in that namespace is keycloak-operator.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # ===========================================================================================
+  # 1. HEALTHY WAVE-1 -> every rule SILENT ("not permanently firing").
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 healthy - every app ready, keycloak StatefulSet at 2 replicas - all quiet
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '2+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '1+0x20'
+      # The namespace's OTHER workloads are present on purpose: keycloak-operator is a Deployment in
+      # the keycloak namespace, and keycloak-db is a StatefulSet in it. A rule that matched the
+      # namespace only (not the object) would be satisfied — or triggered — by these instead of by
+      # the IdP itself. That is the exact masking KeycloakDown had to be fixed for.
+      - series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
+        values: '3+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 2. THE IdP ITSELF IS DOWN while its operator and DB are fine -> ONLY keycloak-down fires.
+  #    This is the anti-masking assertion: the neighbours in that namespace stay ready.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 keycloak StatefulSet has no ready replica (operator + DB fine) - keycloak-down fires
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
+        values: '1+0x20'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
+        values: '3+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '1+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: keycloak
+              namespace: keycloak
+              statefulset: keycloak
+            exp_annotations:
+              summary: "keycloak is down"
+              description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 3. EACH DEPLOYMENT-BACKED APP DOWN -> its own rule fires, keycloak stays quiet.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 nextcloud/radarr/semaphore/sonarr have no available replica - each own rule fires
+    input_series:
+      - series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
+        values: '2+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
+        values: '0+0x20'
+      - series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: nextcloud
+              namespace: nextcloud
+              deployment: nextcloud
+            exp_annotations:
+              summary: "nextcloud is down"
+              description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: radarr
+              namespace: radarr
+              deployment: radarr
+            exp_annotations:
+              summary: "radarr is down"
+              description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: semaphore
+              namespace: semaphore
+              deployment: semaphore
+            exp_annotations:
+              summary: "semaphore is down"
+              description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: sonarr
+              namespace: sonarr
+              deployment: sonarr
+            exp_annotations:
+              summary: "sonarr is down"
+              description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 4. THE SIGNAL VANISHES -> the rules FIRE. The defect class itself, asserted on: a rule that can
+  #    only fire on `== 0` goes SILENT when kube-state-metrics or the workload object disappears,
+  #    which is indistinguishable from healthy. Here NOTHING in wave-1 is scraped.
+  # ===========================================================================================
+  - interval: 1m
+    name: wave-1 all liveness series ABSENT (kube-state-metrics gone) - every rule fires
+    input_series:
+      # One unrelated series so the test has an input at all; no wave-1 liveness series exists.
+      - series: 'up{job="kube-state-metrics"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: keycloak-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: keycloak
+              namespace: keycloak
+              statefulset: keycloak
+            exp_annotations:
+              summary: "keycloak is down"
+              description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+      - eval_time: 10m
+        alertname: nextcloud-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: nextcloud
+              namespace: nextcloud
+              deployment: nextcloud
+            exp_annotations:
+              summary: "nextcloud is down"
+              description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: radarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: radarr
+              namespace: radarr
+              deployment: radarr
+            exp_annotations:
+              summary: "radarr is down"
+              description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: semaphore-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: semaphore
+              namespace: semaphore
+              deployment: semaphore
+            exp_annotations:
+              summary: "semaphore is down"
+              description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+      - eval_time: 10m
+        alertname: sonarr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: sonarr
+              namespace: sonarr
+              deployment: sonarr
+            exp_annotations:
+              summary: "sonarr is down"
+              description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."

apps/wave-2/seerr.yaml

@@ -233,26 +233,6 @@ spec:
     - name: seerr
       port: 5055
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: seerr
-  namespace: seerr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: seerr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: seerr
-  namespaceSelector:
-    matchNames:
-    - seerr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -267,11 +247,11 @@ spec:
   - name: seerr.slis
     rules:
     - alert: seerr-down
-      expr: up{job="seerr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})
       for: 5m
       labels:
         severity: critical
         service: seerr
       annotations:
         summary: seerr is down
-        description: 'seerr has been unreachable for 5m (SLI: up).'
+        description: 'seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'

apps/wave-2/tests/app-slis.test.yaml

@@ -0,0 +1,69 @@
+# promtool rule-unit tests for the wave-2 per-app `<app>-down` SLI rule (seerr).
+# Manifest under test: apps/wave-2/seerr.yaml. Discovered by the SUITE convention; the flattened
+# manifest is the RENDERED one Flux applies.
+#
+# Run with: platform/monitoring/tests/run.sh apps/wave-2/tests/app-slis.test.yaml
+#
+# seerr (Jellyseerr/Overseerr) exports NO Prometheus metrics of its own, so its `up` SLI could never
+# have been met by scraping the app — see apps/wave-0/tests/app-slis.test.yaml for the full history
+# of the inert `up{job="seerr"} == 0` rule this replaces. Pinned in both directions plus the
+# signal-vanishes case.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  - interval: 1m
+    name: wave-2 healthy - seerr has an available replica - quiet
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts: []
+
+  - interval: 1m
+    name: wave-2 seerr has no available replica - seerr-down fires
+    input_series:
+      - series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
+        values: '0+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: seerr
+              namespace: seerr
+              deployment: seerr
+            exp_annotations:
+              summary: "seerr is down"
+              description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+    # Under `for: 5m` a 2m-old condition must NOT yet be a firing alert, but the expression must
+    # already be selecting — the distinction an inert rule cannot make.
+    promql_expr_test:
+      - expr: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})'
+        eval_time: 2m
+        exp_samples:
+          - labels: 'kube_deployment_status_replicas_available{deployment="seerr", namespace="seerr"}'
+            value: 0
+
+  - interval: 1m
+    name: wave-2 seerr liveness series ABSENT - seerr-down still fires
+    input_series:
+      - series: 'up{job="kube-state-metrics"}'
+        values: '1+0x20'
+    alert_rule_test:
+      - eval_time: 10m
+        alertname: seerr-down
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              service: seerr
+              namespace: seerr
+              deployment: seerr
+            exp_annotations:
+              summary: "seerr is down"
+              description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."

catalog/keycloak.yaml

@@ -31,5 +31,15 @@ observability:
   metrics: true
   slis:
     - up                      # IdP down/unhealthy is a critical alert (docs/10; R-OBS-01)
+  # WHAT "up" IS for Keycloak. The Keycloak OPERATOR runs the server as a StatefulSet named
+  # `keycloak` (pods keycloak-0/1), not as the Deployment this ServiceDefinition would otherwise
+  # imply — the operator owns the workload (platform/keycloak/keycloak-instance.yaml) and this SD
+  # is the catalog's declaration of intent for it. Without this override the rendered SLI would
+  # read a Deployment series that does not exist and its `absent()` half would fire FOREVER.
+  # Verified on prod 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak",
+  # statefulset="keycloak"} = 2, and the only Deployment in that namespace is keycloak-operator.
+  livenessWorkload:
+    kind: StatefulSet
+    name: keycloak
 backup:
   policy: daily-offcluster    # DR-critical: rebuilds from DB + git realm config (docs/15, docs/04; R-STATE-03)

catalog/pihole.yaml

@@ -13,6 +13,93 @@ storage:
     size: 2Gi
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
+# ------------------------------------------------------------------------------------------------
+# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
+# ------------------------------------------------------------------------------------------------
+# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
+# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
+# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
+# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
+# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
+# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
+# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
+# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
+#
+# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
+# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
+# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
+# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
+# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
+# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
+# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
+# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
+# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
+# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
+# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
+# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
+#
+# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
+# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
+# configured through the UI is lost on the next reschedule — which is exactly what happened to the
+# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
+# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
+# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
+# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
+env:
+  # Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
+  # pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
+  # required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
+  # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
+  # enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
+  # nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
+  - name: FTLCONF_dns_listeningMode
+    value: "ALL"
+  # Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
+  # router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
+  # answers, and forwarding to a device that may forward back here invites a loop.
+  - name: FTLCONF_dns_upstreams
+    value: "8.8.8.8;8.8.4.4"
+  # Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
+  # because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
+  # externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
+  # announcement lease and the pihole pod land on different nodes, which is most of the time — and
+  # Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
+  # INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
+  # apparent client's budget, and a normal evening of browsing from a handful of machines would trip
+  # a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
+  # off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
+  # limit becomes meaningful again — and should be restored — if Cilium is ever switched to
+  # `loadBalancer.mode: dsr`, which preserves the client address.
+  - name: FTLCONF_dns_rateLimit_count
+    value: "0"
+  # THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
+  # every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
+  # so the first line is a wildcard for the whole zone and each later line is a specific override.
+  # PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
+  # line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
+  # changes nothing.
+  #
+  # A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
+  # `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
+  # ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
+  # platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
+  # behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
+  # repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
+  # is the annotation that means "this Service publishes its own hostname at its own address":
+  # apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
+  # platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
+  # resolver address, are the complete set of exceptions today.
+  #
+  # AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
+  # (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
+  # NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
+  #
+  # Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
+  # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
+  # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
+  # while resolution for the zone does not work. Confirmed both ways against the live image.
+  - name: FTLCONF_misc_dnsmasq_lines
+    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
 ingressRoutes:
   - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)

catalog/plex.yaml

@@ -32,11 +32,46 @@ env:
   # (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
   - name: ALLOWED_NETWORKS
     value: 10.0.0.0/8,192.168.0.0/16
-  # PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
-  # shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
-  # is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
-  # single API POST) against the running server; the claim then persists on the /config PVC and
-  # survives pod rolls (including the roll this env change causes), so it never belongs in git.
+  # PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
+  # (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
+  # workload). Plex cannot create libraries over the mounted bulk share while the server is
+  # UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
+  # half of that reasoning survived contact with the upstream source.
+  #
+  # A SHORT TTL IS THE OPERATIONAL CONSTRAINT, not a reason to abstain. A https://plex.tv/claim token
+  # expires in MINUTES — read the countdown the claim page renders; do not budget against a number
+  # written in this tree. (This comment previously asserted "~4 minutes" as established fact. It was
+  # UNCITED, upstream plexinc/pms-docker documents no expiry at all, and that number was load-bearing
+  # for the whole procedure design — including a since-disproven claim that the delivery pipeline was
+  # too slow to fit inside it. Measured 2026-09-29: the full loop is ~2m45s and fits.)
+  #
+  # The live token is injected by the operator inside that window, with an explicit `flux reconcile`
+  # to collapse the wait. Scripted end to end as infra/plex/claim-ship.sh; runbooks/plex-claim.md is
+  # the procedure. SERVER CLAIMED 2026-09-29 — this env is now a permanent no-op (see below), and the
+  # committed value is the spent token from that claim rather than the original placeholder.
+  #
+  # ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
+  # (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+  # the running `:latest` image), NOT assumed:
+  #   - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
+  #     PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
+  #     no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
+  #     logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
+  #   - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
+  #     status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
+  #     there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
+  #     SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
+  #     default 0, which continues startup even when a cont-init script fails.
+  # So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
+  # placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
+  # so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
+  # listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
+  # is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
+  - name: PLEX_CLAIM
+    valueFrom:
+      secretKeyRef:
+        name: plex-claim
+        key: PLEX_CLAIM
   # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
   # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
   # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
@@ -56,10 +91,12 @@ storage:
                               # container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
                               # root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
                               # (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
-                              # makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
-                              # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
-                              # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
-                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
+                              # makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
+                              # above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
+                              # exchange is then skipped on every later start. (The older note here claimed a
+                              # placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
+                              # otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
+                              # failure, and that is why one is committed.)
   # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
   # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
   # downloads/ movies/ photos/ tv/).

catalog/render.py

@@ -80,7 +80,40 @@ BULK_NFS_SERVER = "${BULK_NFS_SERVER}"
 BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retrans=2"]
 GATEWAY_NAME = "shared"
 GATEWAY_NS = "gateway"
+# ExternalDNS per-resource scoping (R-DNS-03; see _externaldns_route_annotations for the full WHY
+# and for the v0.21.0 source citation). The KEY is external-dns's own `ControllerKey`
+# (annotationPrefix + "controller"); external-dns skips any source object whose value is PRESENT
+# and != `ControllerValue` ("dns-controller"), so the VALUE only has to be some other string.
+# `none` is chosen because it reads as intent in `kubectl get httproute -o yaml` rather than as a
+# typo — "no DNS controller owns this route" — and because the one value that must never be
+# emitted here is external-dns's own, which a literal `none` cannot drift into.
+EXTERNALDNS_CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
+EXTERNALDNS_CONTROLLER_DISOWN = "none"
+# The value external-dns claims for itself. Held here ONLY so the renderer and the L0 guard can
+# assert the disown value is not it; never emitted.
+EXTERNALDNS_CONTROLLER_CLAIMED = "dns-controller"
 KP_RELEASE = "kube-prometheus-stack"  # ServiceMonitor/PrometheusRule discovery label
+MONITORING_NS = "monitoring"          # where Prometheus runs; the only admitted scrape source
+
+# The kube-state-metrics series a workload's readiness IS (r_prometheusrule). kube-state-metrics
+# is scraped by the bundled kube-prometheus-stack ServiceMonitor, i.e. these exist for EVERY
+# workload in the cluster with no per-service exporter — which is the whole point of using them
+# for `slis: [up]`. Each entry is (metric, label naming the object). VERIFIED on prod 2026-09-13:
+# kube_deployment_status_replicas_available returned a series for all ten deployed app
+# Deployments, and kube_statefulset_status_replicas_ready{namespace="keycloak",
+# statefulset="keycloak"} returned 2.
+LIVENESS_METRICS = {
+    "Deployment": ("kube_deployment_status_replicas_available", "deployment"),
+    "StatefulSet": ("kube_statefulset_status_replicas_ready", "statefulset"),
+    "DaemonSet": ("kube_daemonset_status_number_available", "daemonset"),
+}
+# A KubeVirt VMI is not a replica count: kubevirt_vmi_info carries the phase as a LABEL, so
+# liveness is the EXISTENCE of a running-phase series. The value is lowercase `running` — a
+# capital-R matcher selects nothing and is just as inert (the same trap that made
+# kubevirt_vmi_phase_count{phase="Failed"} a no-op; verified on prod, where the one live VMI
+# reports phase="running").
+VMI_METRIC = "kubevirt_vmi_info"
+VMI_RUNNING_PHASE = "running"
 
 # egress classes and auth modes fixed by docs/03 / docs/13 + the Kyverno enums.
 EGRESS_CLASSES = ("internet", "vpn")
@@ -135,6 +168,26 @@ def apply_defaults(sd: dict) -> dict:
     obs = dict(s.get("observability") or {})
     obs.setdefault("metrics", True)
     obs.setdefault("slis", [])
+    # metricsEndpoint is NEVER defaulted (schema 1.6.0). `metrics: true` says the platform
+    # collects signals FOR this service; it cannot say the workload SERVES a Prometheus
+    # endpoint, because for most of this catalog it does not (see r_servicemonitor). When the
+    # endpoint IS declared, its name/path/interval carry defaults so the common case is
+    # `metricsEndpoint: {port: 9100}`.
+    ep = obs.get("metricsEndpoint")
+    if ep:
+        obs["metricsEndpoint"] = {
+            **{"name": "metrics", "path": "/metrics", "interval": "30s"},
+            **ep,
+        }
+    # livenessWorkload: WHICH Kubernetes object's readiness IS this service being up. Defaulted
+    # from `kind` — a container renders a Deployment named after the service, a vm renders a
+    # VirtualMachine whose VMI carries the same name — and overridable for the case where the
+    # object that actually runs on the cluster is not the one this SD renders (keycloak: the
+    # Keycloak operator runs a StatefulSet, so a Deployment-shaped rule would `absent()` forever).
+    lw = dict(obs.get("livenessWorkload") or {})
+    lw.setdefault("kind", "VirtualMachineInstance" if s["kind"] == "vm" else "Deployment")
+    lw.setdefault("name", s["name"])
+    obs["livenessWorkload"] = lw
     s["observability"] = obs
 
     # storage entry class default is longhorn (replicated tier); readOnly default is False
@@ -488,7 +541,12 @@ def r_deployment(s: dict) -> dict:
 
 def r_service(s: dict):
     ports = _service_ports(s)
-    if not ports:
+    # A declared metrics endpoint becomes a NAMED Service port, because that name is the only
+    # thing a ServiceMonitor endpoint can select on. Appended BELOW rather than inside
+    # _service_ports so it never leaks into r_gateway_ingress_policy's north-south allow list —
+    # the metrics port is for the monitoring namespace, not for the Gateway.
+    ep = s["observability"].get("metricsEndpoint")
+    if not ports and not ep:
         return None
     # forward-auth (Option B): the shared Gateway must reach the oauth2-proxy sidecar, so front it
     # with a `authproxy` :4180 Service port (the HTTPRoute backendRef targets this). The app UI/API
@@ -498,6 +556,10 @@ def r_service(s: dict):
         ports = ports + [
             {"name": "authproxy", "port": OAUTH2_PROXY_PORT, "targetPort": OAUTH2_PROXY_PORT}
         ]
+    if ep:
+        ports = ports + [
+            {"name": ep["name"], "port": ep["port"], "targetPort": ep["port"]}
+        ]
     return {
         "apiVersion": "v1",
         "kind": "Service",
@@ -786,12 +848,15 @@ def r_in_cluster_client_policies(s: dict) -> list:
 
 
 def _route_meta(s: dict, route: dict) -> dict:
+    """HTTPRoute metadata: auth wiring hints PLUS the ExternalDNS scoping derived from
+    `exposure` (_externaldns_route_annotations — R-DNS-03: an `internal` service renders no
+    public DNS record, and now cannot)."""
     meta = {
         "name": f"{s['name']}-{route['port']}",
         "namespace": s["name"],
         "labels": {"app.kubernetes.io/name": s["name"]},
     }
-    ann = _auth_annotations(s)
+    ann = {**_auth_annotations(s), **_externaldns_route_annotations(s)}
     if ann:
         meta["annotations"] = ann
     return meta
@@ -989,8 +1054,92 @@ def _auth_annotations(s: dict) -> dict:
     return {}
 
 
+def _externaldns_route_annotations(s: dict) -> dict:
+    """ExternalDNS scoping, rendered FROM `exposure` (renderer-contract §exposure; R-DNS-03).
+
+    `internal` renders NO public DNS record. That sentence was in the contract from the start and
+    nothing enforced it: ExternalDNS watched EVERY HTTPRoute the shared Gateway carried, so an
+    `internal` service's route published `<name>.${BASE_DOMAIN}` into the PUBLIC authoritative
+    Cloudflare zone, answering with the Gateway VIP — an RFC1918 address. Six of them did
+    (jackett, pihole, qbittorrent, radarr, semaphore, sonarr), which published the internal
+    subnet, the Gateway VIP and a full service inventory to anyone who asked, and made every one
+    of those names a DNS-rebind primitive that the SOHO router strips on the way back in.
+
+    THE MECHANISM, verified against the DEPLOYED version (chart >=1.21.0 <1.22.0, app v0.21.0 —
+    `registry.k8s.io/external-dns/external-dns:v0.21.0`, read off the live prod Deployment), not
+    from memory. external-dns v0.21.0 `source/annotations/processors.go`:
+
+        func IsControllerMismatch(entity metav1.ObjectMetaAccessor, rType string) bool {
+            value, ok := entity.GetObjectMeta().GetAnnotations()[ControllerKey]
+            if ok && value != ControllerValue { ...; return true }
+            return false
+        }
+
+    with `ControllerKey = "external-dns.alpha.kubernetes.io/controller"` and
+    `ControllerValue = "dns-controller"`. `source/gateway.go` calls it per route inside
+    `Endpoints()`, BEFORE any hostname is resolved:
+
+        if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
+
+    so an HTTPRoute carrying any other value is not merely unpublished — it never becomes an
+    endpoint at all, for any hostname it names. `source/service.go` applies the identical check,
+    so the same annotation scopes the `service` source too if a rendered Service ever grows an
+    external-dns hostname.
+
+    WHY THIS IS STRUCTURAL AND NOT INCIDENTAL. The annotation is derived HERE, from `exposure`,
+    with no per-service list anywhere: every HTTPRoute this renderer emits for an `internal`
+    service carries it, and ci/render-golden.sh proves `apps/` reproduces byte-for-byte from
+    `catalog/`, so an `internal` service cannot EXIST in the applied tree without it. Flipping
+    `exposure: internal` to `public` is the only edit that can publish a name, which is exactly
+    the operator-facing declaration R-DNS-03 is written in terms of. The L0 check
+    conformance/tests/l0-static/internal-services-publish-no-public-dns re-proves all of that on
+    every gate run, including that this function reads `exposure` rather than naming services.
+
+    WHY NOT A CLUSTER-WIDE OPT-IN FILTER (`--label-filter`, chart `labelFilter`). It exists, and
+    v0.21.0's own `--help` names both sources in use as supporting it ("currently supported by
+    source types crd, gateway-httproute, ... service and ambassador-host"), so a "publish only
+    what is labelled" posture is available and is STRONGER by default. It is not taken here
+    because its blast radius is not this defect's: seven published names come from hand-authored
+    platform/apps manifests that carry no `exposure` field and that this renderer never sees
+    (auth/oauth2-proxy, crafty + crafty-mc, grafana, headlamp, the homepage apex, unifi). An
+    opt-in filter DELETES all seven the moment it is set, and whether each should stay published
+    is an operator decision, not a renderer one. Per-resource exclusion has exactly the blast
+    radius of the reported defect. The filter remains the available hardening once those seven
+    are adjudicated — see platform/externaldns/helmrelease.yaml.
+    """
+    if s["exposure"] == "internal":
+        return {EXTERNALDNS_CONTROLLER_KEY: EXTERNALDNS_CONTROLLER_DISOWN}
+    return {}
+
+
 def r_servicemonitor(s: dict):
-    if not s["observability"]["metrics"]:
+    """ServiceMonitor — rendered ONLY when the SD says WHERE metrics are served.
+
+    THIS FUNCTION USED TO RENDER A MONITOR UNCONDITIONALLY FROM `metrics: true`, hardcoding
+    `endpoints: [{port: metrics}]`. No Service this renderer emits had ever defined a port of
+    that name, so all twelve monitors discovered ZERO targets: no `up{job="<app>"}` series
+    existed anywhere, `up{job="<app>"} == 0` could not fire, and `PrometheusTargetsMissing`
+    could not see the hole either, because a target that was never discovered has no `up` series
+    to be 0. Verified on prod 2026-09-13: `count by (job) (up)` listed 24 jobs and not one of
+    the twelve appeared; `up{job="plex"}` and `up{job="gitea"}` both returned nothing.
+
+    THE ROOT CAUSE WAS NOT A TYPO — it was that `metrics: true` cannot state a fact the workload
+    does not have. Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and
+    Nextcloud serve no Prometheus endpoint at all without a separate exporter; Gitea can serve
+    one but only once its app.ini enables it. So the port name was never the missing piece. A
+    monitor is now rendered only from `observability.metricsEndpoint`, which names the port the
+    workload actually serves — and that same declaration renders the Service port (r_service) and
+    the monitoring-namespace ingress allow (r_monitoring_ingress_policy), so the three halves of
+    a working scrape cannot be authored apart.
+
+    `metrics: true` still means something without it: logs ship to Loki via the node agent, and
+    liveness comes from kube-state-metrics (r_prometheusrule) — neither needs a per-service CR.
+    """
+    obs = s["observability"]
+    if not obs["metrics"]:
+        return None
+    ep = obs.get("metricsEndpoint")
+    if not ep:
         return None
     return {
         "apiVersion": "monitoring.coreos.com/v1",
@@ -1003,7 +1152,108 @@ def r_servicemonitor(s: dict):
         "spec": {
             "selector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
             "namespaceSelector": {"matchNames": [s["name"]]},
-            "endpoints": [{"port": "metrics", "interval": "30s"}],
+            "endpoints": [
+                {"port": ep["name"], "path": ep["path"], "interval": ep["interval"]}
+            ],
+        },
+    }
+
+
+def r_monitoring_ingress_policy(s: dict):
+    """Let the monitoring namespace reach a declared metrics port (R-NET-02, R-OBS-01).
+
+    renderer-contract.md has always said the default-deny baseline carries "a standing allow ...
+    for observability scraping from the monitoring namespace" — and no renderer output ever did.
+    Every app namespace ships `ingress: []`, so a scrape of an app metrics port would be DROPPED
+    by policy even with the Service port and the monitor endpoint in agreement. That is the same
+    trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
+    for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.
+
+    Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
+    monitoring namespace: an allow for a port nothing serves would be a standing hole for no
+    benefit.
+    """
+    ep = s["observability"].get("metricsEndpoint")
+    if not ep or not s["observability"]["metrics"]:
+        return None
+    return {
+        "apiVersion": "cilium.io/v2",
+        "kind": "CiliumNetworkPolicy",
+        "metadata": {
+            "name": f"{s['name']}-allow-monitoring-scrape",
+            "namespace": s["name"],
+            "labels": {"app.kubernetes.io/name": s["name"]},
+        },
+        "spec": {
+            "endpointSelector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
+            "ingress": [
+                {
+                    "fromEndpoints": [
+                        {"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
+                    ],
+                    "toPorts": [
+                        {"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
+                    ],
+                }
+            ],
+        },
+    }
+
+
+def _up_sli_rule(s: dict) -> dict:
+    """`slis: [up]` -> a `<name>-down` alert against a signal THAT EXISTS on this cluster.
+
+    IT USED TO READ `up{job="<name>"} == 0`. That is the textbook expression and it was inert
+    for all twelve services in the catalog, because `up` only exists for a target Prometheus
+    actually DISCOVERED and the rendered ServiceMonitor discovered none (see r_servicemonitor).
+    Worse than a wrong threshold: an inert rule and a correct rule are indistinguishable on a
+    healthy cluster, and the generic safety net cannot see the difference either — a target that
+    was never discovered has no `up` series for PrometheusTargetsMissing to find at 0.
+
+    Rewriting the SLI against the app's own /metrics was NOT an option for most of this catalog:
+    Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export no
+    Prometheus metrics without a separate exporter sidecar per app. So the `up` SLI is expressed
+    against kube-state-metrics, which IS scraped and DOES carry a series for every workload — the
+    same idiom KeycloakDown in platform/monitoring/alerts.yaml has used since it was fixed.
+
+    The `or absent(...)` half is deliberate and load-bearing: it makes the DISAPPEARANCE of the
+    signal fire the alert instead of silencing it. A rule that can only fire on `== 0` is a rule
+    that goes quiet when the workload object (or kube-state-metrics) vanishes, which is precisely
+    how the old expression failed. It is also why this rule cannot be permanently firing where the
+    workload exists: the series is present and non-zero on a healthy cluster (verified on prod).
+    """
+    lw = s["observability"]["livenessWorkload"]
+    kind, obj = lw["kind"], lw["name"]
+    ns = s["name"]
+    if kind == "VirtualMachineInstance":
+        sel = f'{VMI_METRIC}{{namespace="{ns}", name="{obj}", phase="{VMI_RUNNING_PHASE}"}}'
+        expr = f"absent({sel})"
+        desc = (
+            f"{s['name']}: no KubeVirt VMI in phase {VMI_RUNNING_PHASE} for 5m "
+            f"(SLI: up, via {VMI_METRIC})."
+        )
+    else:
+        try:
+            metric, label = LIVENESS_METRICS[kind]
+        except KeyError:
+            raise ValueError(
+                f"{s['name']}: observability.livenessWorkload.kind {kind!r} has no liveness "
+                f"metric; known kinds: {sorted(LIVENESS_METRICS)} + VirtualMachineInstance"
+            ) from None
+        sel = f'{metric}{{namespace="{ns}", {label}="{obj}"}}'
+        expr = f"{sel} == 0 or absent({sel})"
+        desc = (
+            f"{s['name']}: its {kind} has no available/ready replica for 5m, or the signal is "
+            f"gone entirely (SLI: up, via {metric})."
+        )
+    return {
+        "alert": f"{s['name']}-down".replace("_", "-"),
+        "expr": expr,
+        "for": "5m",
+        "labels": {"severity": "critical", "service": s["name"]},
+        "annotations": {
+            "summary": f"{s['name']} is down",
+            "description": desc,
         },
     }
 
@@ -1016,18 +1266,7 @@ def r_prometheusrule(s: dict):
     rules = []
     for sli in slis:
         if sli == "up":
-            rules.append(
-                {
-                    "alert": f"{s['name']}-down".replace("_", "-"),
-                    "expr": f'up{{job="{s["name"]}"}} == 0',
-                    "for": "5m",
-                    "labels": {"severity": "critical", "service": s["name"]},
-                    "annotations": {
-                        "summary": f"{s['name']} is down",
-                        "description": f"{s['name']} has been unreachable for 5m (SLI: up).",
-                    },
-                }
-            )
+            rules.append(_up_sli_rule(s))
         else:
             rules.append(
                 {
@@ -1386,6 +1625,16 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
             )
         )
 
+    mon_allow = r_monitoring_ingress_policy(s)
+    if mon_allow:
+        docs.append(
+            (
+                "CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): "
+                f"{mon_allow['metadata']['name']}",
+                mon_allow,
+            )
+        )
+
     ie = r_internet_egress(s)
     docs.append(
         (
@@ -1503,6 +1752,25 @@ EXTRA_UNITS: dict = {
             "gitea-actions-runner-secret.sops.yaml",
         ],
     },
+    # PiHole's DNS exposure: the udp/53 + tcp/53 LoadBalancer Services on a pinned VIP, plus the
+    # narrow LAN ingress CiliumNetworkPolicy that admits them (apps/wave-0/pihole-dns.yaml).
+    # `above: False` co-locates it in pihole's OWN wave: these are the pihole pod's own Services and
+    # its own policy, applied in the same wave as the Deployment they select, exactly as the
+    # rendered Service and CNPs in pihole.yaml are. It is NOT a runtime prerequisite of anything, so
+    # there is nothing for a higher wave to gate on.
+    #
+    # It is a non-catalog unit because the ServiceDefinition schema cannot express a LoadBalancer
+    # Service, a pinned lbipam VIP, a sharing key or UDP exposure at all (additionalProperties is
+    # false, and `ingressRoutes` only renders HTTPRoutes on the L7 Gateway) — the same reason
+    # apps/vms/unifi/50-service-l4.yaml and platform/crafty/service-l4.yaml are hand-written. The
+    # split-horizon ANSWERS are in the catalog, as FTLCONF env on catalog/pihole.yaml.
+    "pihole-dns": {
+        "anchor": "pihole",
+        "above": False,
+        "files": [
+            "pihole-dns.yaml",
+        ],
+    },
 }
 
 
@@ -1576,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
     convention:
       - `<svc>.yaml`                        -> wave(svc)
       - `<svc>-bootstrap-secret.sops.yaml`  -> wave(svc)   (consumed by svc's env, same apply)
+      - `<svc>-claim-secret.sops.yaml`      -> wave(svc)   (a one-shot upstream-service claim token
+        consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
+        bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
+        in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
       - `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc)  (forward-auth sidecar OIDC creds)
       - `<svc>-db.yaml`                     -> wave(svc-db) (a strictly-lower wave; CNPG DB)
       - `<svc>-db-app.sops.yaml`            -> wave(svc-db) (CNPG initdb owner secret)
@@ -1602,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
         w = waves[name]
         add(w, f"{name}.yaml")
         add(w, f"{name}-bootstrap-secret.sops.yaml")
+        # A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
+        # app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
+        # so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
+        # Placed only when the file exists, so services without one are unaffected.
+        add(w, f"{name}-claim-secret.sops.yaml")
         # forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
         # Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
         # so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
@@ -1902,6 +2179,96 @@ def _selftest() -> int:
         "a non-exposed service must not render a gateway-ingress allow (least privilege)",
     )
 
+    # 4b-dns) exposure -> ExternalDNS SCOPING on the HTTPRoute (R-DNS-03). `internal` renders no
+    #     public DNS record and must be INCAPABLE of rendering one: its route carries
+    #     external-dns's own controller annotation set to something other than `dns-controller`,
+    #     which v0.21.0's gateway-httproute source (source/gateway.go ->
+    #     annotations.IsControllerMismatch) uses to skip the route before any hostname is
+    #     resolved. `public` must NOT carry it, or the five public names would vanish instead.
+    #     Both directions are asserted here because only asserting the internal half would pass
+    #     just as happily if the renderer stamped EVERY route and silently unpublished the zone.
+    dns_int_docs = render(
+        {
+            "name": "di",
+            "source": {"image": "a:1"},
+            "exposure": "internal",
+            "ingressRoutes": [{"host": "di.home.keithjackson.dev", "port": 80}],
+            "auth": {"mode": "none"},
+        }
+    )
+    dns_int_routes = [d[1] for d in dns_int_docs if d[1]["kind"] == "HTTPRoute"]
+    check(
+        len(dns_int_routes) == 1,
+        "an internal exposed service must still render its HTTPRoute (LAN reachability is "
+        "unchanged — only the PUBLIC record is withdrawn)",
+    )
+    if dns_int_routes:
+        iann = dns_int_routes[0]["metadata"].get("annotations", {})
+        check(
+            iann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN,
+            "an internal service's HTTPRoute must carry the ExternalDNS disown annotation",
+        )
+        check(
+            iann.get(EXTERNALDNS_CONTROLLER_KEY) != EXTERNALDNS_CONTROLLER_CLAIMED,
+            "the disown value must never be external-dns's own `dns-controller` (that CLAIMS "
+            "the route rather than releasing it — the one value that inverts the guarantee)",
+        )
+    dns_pub_docs = render(
+        {
+            "name": "dp",
+            "source": {"image": "a:1"},
+            "exposure": "public",
+            "ingressRoutes": [{"host": "dp.home.keithjackson.dev", "port": 80}],
+            "auth": {"mode": "native"},
+        }
+    )
+    dns_pub_routes = [d[1] for d in dns_pub_docs if d[1]["kind"] == "HTTPRoute"]
+    check(len(dns_pub_routes) == 1, "a public service must render its HTTPRoute")
+    if dns_pub_routes:
+        check(
+            EXTERNALDNS_CONTROLLER_KEY
+            not in dns_pub_routes[0]["metadata"].get("annotations", {}),
+            "a public service's HTTPRoute must NOT carry the ExternalDNS disown annotation",
+        )
+    # The DEFAULT posture is `internal` (renderer-contract "Default posture"), so a
+    # ServiceDefinition that simply omits `exposure` must also be unpublishable. Omitting the
+    # field is the likeliest way a new service arrives, and it is the case a per-service
+    # allow-list would miss.
+    dns_dflt_docs = render(
+        {
+            "name": "dd",
+            "source": {"image": "a:1"},
+            "ingressRoutes": [{"host": "dd.home.keithjackson.dev", "port": 80}],
+            "auth": {"mode": "none"},
+        }
+    )
+    dns_dflt_route = next(d[1] for d in dns_dflt_docs if d[1]["kind"] == "HTTPRoute")
+    check(
+        dns_dflt_route["metadata"]["annotations"].get(EXTERNALDNS_CONTROLLER_KEY)
+        == EXTERNALDNS_CONTROLLER_DISOWN,
+        "a ServiceDefinition that OMITS `exposure` defaults to internal and must also be "
+        "unpublishable (default posture, renderer-contract)",
+    )
+    # The scoping annotation must COEXIST with the auth hints rather than replace them: both are
+    # merged into one `annotations` map, and a forward-auth internal app is the overlap case.
+    dns_fa_docs = render(
+        {
+            "name": "df",
+            "source": {"image": "a:1"},
+            "exposure": "internal",
+            "ingressRoutes": [{"host": "df.home.keithjackson.dev", "port": 80}],
+            "auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
+        }
+    )
+    dns_fa_ann = next(
+        d[1] for d in dns_fa_docs if d[1]["kind"] == "HTTPRoute"
+    )["metadata"]["annotations"]
+    check(
+        dns_fa_ann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN
+        and dns_fa_ann.get(f"{LBL}/auth-mode") == "forward-auth",
+        "DNS scoping and auth-mode hints must COEXIST on one internal forward-auth route",
+    )
+
     # 4c) inClusterClients -> one ADDITIVE, least-privilege ingress allow per entry that admits a
     #     PLATFORM controller's NAMESPACE (not the reserved `ingress` identity, not `all`) to ONE
     #     declared port. This is what keeps a cluster whose GitOps source is an in-cluster Service
@@ -2190,7 +2557,8 @@ def _selftest() -> int:
     # 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
     #     EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
     present = {
-        "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
+        "a.yaml", "a-claim-secret.sops.yaml",
+        "svc.yaml", "svc-bootstrap-secret.sops.yaml",
         "svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
         "svc-db-backup-credentials-prod.sops.yaml",
         "gitea.yaml",
@@ -2201,6 +2569,10 @@ def _selftest() -> int:
     plan = plan_apps_waves(plan_sds, present)
     flat = {f: wv for wv, fs in plan.items() for f in fs}
     check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
+    check(flat.get("a-claim-secret.sops.yaml") == 0,
+          "a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
+          "as its app — an env secretKeyRef has no `optional`, so a later wave means "
+          "CreateContainerConfigError")
     check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
           and flat.get("svc-db-backup-credentials.sops.yaml") == 0
           and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
@@ -2315,6 +2687,154 @@ def _selftest() -> int:
         "the escaped `$${...}` form must render (and reach the shell as `${...}`)",
     )
 
+    # 9) observability (R-OBS-01): `metrics: true` alone renders NO ServiceMonitor, and the `up`
+    #    SLI reads a signal that exists. This is the regression test for the defect that made all
+    #    twelve `<app>-down` rules structurally inert on prod (see r_servicemonitor).
+    obs_sd = {
+        "name": "obsdemo",
+        "source": {"image": "a:1"},
+        "ingressRoutes": [{"host": "obsdemo.${BASE_DOMAIN}", "port": 8080}],
+        "auth": {"mode": "none"},
+        "observability": {"metrics": True, "slis": ["up"]},
+    }
+    obs_docs = render(obs_sd)
+    check(
+        not [d for d in obs_docs if d[1]["kind"] == "ServiceMonitor"],
+        "metrics: true with NO metricsEndpoint must render no ServiceMonitor — a monitor naming a "
+        "port the Service does not define discovers zero targets and makes the SLI inert",
+    )
+    pr = next(d[1] for d in obs_docs if d[1]["kind"] == "PrometheusRule")
+    up_rule = pr["spec"]["groups"][0]["rules"][0]
+    check(up_rule["alert"] == "obsdemo-down", "slis: [up] must render <name>-down")
+    check(
+        "up{job=" not in up_rule["expr"],
+        "the up SLI must NOT read up{job=...}: no repo-rendered scrape target produces that job, "
+        "so the rule can never fire (verified inert on prod for all twelve apps)",
+    )
+    check(
+        up_rule["expr"]
+        == 'kube_deployment_status_replicas_available{namespace="obsdemo", deployment="obsdemo"} '
+        '== 0 or absent(kube_deployment_status_replicas_available{namespace="obsdemo", '
+        'deployment="obsdemo"})',
+        "container up SLI must read kube_deployment_status_replicas_available with an absent() "
+        f"half; got: {up_rule['expr']}",
+    )
+
+    # 9a) livenessWorkload override: the object that RUNS is not always the one this SD renders.
+    sts_docs = render(
+        dict(
+            obs_sd,
+            name="obssts",
+            observability={
+                "metrics": True,
+                "slis": ["up"],
+                "livenessWorkload": {"kind": "StatefulSet", "name": "obssts"},
+            },
+        )
+    )
+    sts_expr = next(
+        d[1] for d in sts_docs if d[1]["kind"] == "PrometheusRule"
+    )["spec"]["groups"][0]["rules"][0]["expr"]
+    check(
+        "kube_statefulset_status_replicas_ready" in sts_expr,
+        f"livenessWorkload kind StatefulSet must read the statefulset series; got: {sts_expr}",
+    )
+
+    # 9b) vm kind: a VMI has no replica count, so liveness is a running-phase series EXISTING.
+    vm_pr = next(
+        d[1]
+        for d in render(
+            {
+                "name": "obsvm",
+                "kind": "vm",
+                "source": {"vmTemplate": "quay.io/x/y:1"},
+                "storage": [{"name": "root", "size": "6Gi"}],
+                "observability": {"metrics": True, "slis": ["up"]},
+            }
+        )
+        if d[1]["kind"] == "PrometheusRule"
+    )
+    check(
+        vm_pr["spec"]["groups"][0]["rules"][0]["expr"]
+        == 'absent(kubevirt_vmi_info{namespace="obsvm", name="obsvm", phase="running"})',
+        "vm up SLI must read kubevirt_vmi_info with the LOWERCASE running phase: "
+        f"{vm_pr['spec']['groups'][0]['rules'][0]['expr']}",
+    )
+
+    # 9c) metricsEndpoint renders ALL THREE halves of a working scrape, and they AGREE. Any one of
+    #     them missing is an independent way for the scrape to silently discover/deliver nothing.
+    ep_docs = render(
+        dict(
+            obs_sd,
+            name="obsep",
+            observability={
+                "metrics": True,
+                "slis": ["up"],
+                "metricsEndpoint": {"port": 9100},
+            },
+        )
+    )
+    ep_svc = next(d[1] for d in ep_docs if d[1]["kind"] == "Service")
+    ep_sm = next(d[1] for d in ep_docs if d[1]["kind"] == "ServiceMonitor")
+    svc_port_names = {p["name"] for p in ep_svc["spec"]["ports"]}
+    ep_ports = {e["port"] for e in ep_sm["spec"]["endpoints"]}
+    check(
+        ep_ports <= svc_port_names,
+        f"ServiceMonitor endpoint port(s) {sorted(ep_ports)} must all exist on the Service "
+        f"{sorted(svc_port_names)} — this exact disagreement is what made twelve monitors inert",
+    )
+    check(
+        {"name": "metrics", "port": 9100, "targetPort": 9100} in ep_svc["spec"]["ports"],
+        "metricsEndpoint must render the named Service port it declares",
+    )
+    mon_cnp = [
+        d[1]
+        for d in ep_docs
+        if d[1]["kind"] == "CiliumNetworkPolicy"
+        and d[1]["metadata"]["name"] == "obsep-allow-monitoring-scrape"
+    ]
+    check(
+        len(mon_cnp) == 1,
+        "a declared metricsEndpoint must render the monitoring-namespace ingress allow — the "
+        "namespace default-deny has `ingress: []` and would DROP the scrape without it",
+    )
+    if mon_cnp:
+        ing = mon_cnp[0]["spec"]["ingress"][0]
+        check(
+            ing["fromEndpoints"][0]["matchLabels"]["k8s:io.kubernetes.pod.namespace"]
+            == MONITORING_NS
+            and ing["toPorts"][0]["ports"] == [{"port": "9100", "protocol": "TCP"}],
+            "the scrape allow must admit ONLY the monitoring namespace to ONLY the metrics port",
+        )
+    gw_cnp = next(
+        d[1]
+        for d in ep_docs
+        if d[1]["kind"] == "CiliumNetworkPolicy"
+        and d[1]["metadata"]["name"] == "obsep-allow-gateway-ingress"
+    )
+    gw_ports = {p["port"] for p in gw_cnp["spec"]["ingress"][0]["toPorts"][0]["ports"]}
+    check(
+        "9100" not in gw_ports,
+        "the metrics port must NOT be opened to the Gateway (north-south); it is for the "
+        f"monitoring namespace only. Gateway allows: {sorted(gw_ports)}",
+    )
+    # A metricsEndpoint the renderer cannot express must be REFUSED, never rendered inert.
+    try:
+        render(
+            dict(
+                obs_sd,
+                name="obsbad",
+                observability={
+                    "metrics": True,
+                    "slis": ["up"],
+                    "livenessWorkload": {"kind": "CronJob", "name": "obsbad"},
+                },
+            )
+        )
+        check(False, "an unknown livenessWorkload.kind must raise, not render a bogus expression")
+    except ValueError as exc:
+        check("CronJob" in str(exc), "the refusal must name the unsupported kind")
+
     if failures:
         for f in failures:
             sys.stderr.write(f"SELFTEST FAIL: {f}\n")

ci/README.md

@@ -1,5 +1,19 @@
 # `ci/` — portable merge gate (runnable today, without Gitea)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 **Task T-0.2 · R-TEST-02 (reproducible, single-command) · Owner WS-GITOPS (CI).**
 
 The same merge gate the Gitea Actions workflow runs

ci/conformance.sh

@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
   cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi
 
+# ---------------------------------------------------------------------------------------
+# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
+#
+# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
+# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
+# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
+# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
+# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
+#
+# MEASURED 2026-09-29, before this guard existed:
+#   CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
+#   "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
+#   "[ ok ] L2/L3 conformance passed for touched subset"
+# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
+# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
+# it because nobody had ever set the variable.
+#
+# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
+# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
+# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
+# makes the stage itself honour that instead of reporting a pass.
+JUNIT="${CONF_ART}/report.junit.xml"
+if [ -f "$JUNIT" ]; then
+  total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
+  skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
+  if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
+    err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
+    err "  (scaffold mode -- run.sh could not reach a cluster and exited 0)."
+    err "  This stage will NOT report that as a pass. Either the cluster is unreachable from"
+    err "  this runner, or CLUSTER_AVAILABLE is set when it should not be."
+    err "  NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
+    err "  docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
+    err "  by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
+    err "  runner has no credential for (automountServiceAccountToken: false, by design)."
+    summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
+    exit 1
+  fi
+  log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
+fi
+
 ok "L2/L3 conformance passed for touched subset"
 summary "- conformance-l2-l3: **passed**"

ci/l1.sh

@@ -30,11 +30,13 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
 # (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
 # touched subset, then COLLECT its artifacts into the CI artifact dirs.
 # --include-scope: union the touched subset with every L1 test DECLARING
-# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
-# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
-# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
-# "someone must remember" is the failure mode this whole change removes; see
-# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
+# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
+# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
+# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
+# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
+# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
+# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
+# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
 : "${WHOLE_REPO_SCOPE:=whole-repo}"
 l1_rc=0
 "$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \

ci/tests/touched-requirements.test.sh

@@ -136,6 +136,65 @@ printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
 assert_present "R-ENV-02"     "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
 assert_present "R-GITOPS-01"  "$OUT4" "clusters/* attributes R-GITOPS-01"
 
+# ---------------------------------------------------------------------------------------
+# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
+#
+# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
+# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
+# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
+# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
+# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
+# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
+#
+# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
+# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
+#
+# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
+# from the PATH and not from the content scan.
+
+# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
+# under test here, not just the presence of both arms.
+git -C "$TMP" checkout -q -b feature-apps-vms main
+mkdir -p "$TMP/apps/vms/demo"
+printf 'kind: VirtualMachine\nmetadata:\n  name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
+OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
+assert_present "R-PLAT-03"  "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
+assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
+assert_absent  "R-PLAT-02"  "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
+
+# A plain ServiceDefinition under apps/ gets the container-workload set.
+git -C "$TMP" checkout -q -b feature-apps main
+mkdir -p "$TMP/apps/wave-1"
+printf 'kind: ServiceDefinition\nmetadata:\n  name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
+OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
+assert_present "R-PLAT-02"  "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
+assert_present "R-NET-01"   "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
+assert_present "R-NET-06"   "$OUT_APPS" "apps/* attributes the egress-class requirements"
+assert_present "R-IDP-02"   "$OUT_APPS" "apps/* attributes the auth-mode requirements"
+
+# homepage / headlamp ARE the command-centre surface.
+git -C "$TMP" checkout -q -b feature-c2 main
+mkdir -p "$TMP/platform/headlamp"
+printf 'kind: Deployment\nmetadata:\n  name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
+OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
+assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
+assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
+
+# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
+git -C "$TMP" checkout -q -b feature-assembly main
+mkdir -p "$TMP/platform"
+printf 'resources:\n  - cilium\n  - longhorn\n' > "$TMP/platform/kustomization.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
+OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
+assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
+
 # ---------------------------------------------------------------------------------------
 # THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
 #

ci/touched-requirements.sh

@@ -177,6 +177,23 @@ path_requirements() {
   case "$1" in
     catalog/*)                 echo "R-GITOPS-02" ;;
     schemas/*)                 echo "R-GITOPS-02" ;;
+    # apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
+    # manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
+    # adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
+    # checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
+    # class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
+    # violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
+    # catch it at apply time" is post-merge, and the gate exists to catch it before.
+    #
+    # ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
+    # apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
+    # subject to default-deny (R-NET-01).
+    apps/vms/*)                echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
+    # Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
+    # workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
+    # (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
+    # default-deny (R-NET-01).
+    apps/*)                    echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
     platform/kyverno/*)        echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
     platform/cilium/*)         echo "R-NET-01 R-NET-02 R-NET-05" ;;
     platform/longhorn/*)       echo "R-STATE-01 R-STATE-02" ;;
@@ -208,6 +225,39 @@ path_requirements() {
     platform/keycloak/*)       echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
     platform/oauth2-proxy/*)   echo "R-IDP-02 R-IDP-06" ;;
     platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
+    # The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
+    # rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
+    # platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
+    # itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
+    # docs/12-command-center-ui.md), so editing them must select the C2 tests.
+    platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
+    # KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
+    platform/kubevirt/*)       echo "R-PLAT-03 R-OBS-01" ;;
+    # The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
+    # TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
+    platform/gateway/*)        echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
+    # CoreDNS is cluster DNS resolution (R-DNS-01).
+    platform/coredns/*)        echo "R-DNS-01" ;;
+    # crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
+    # its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
+    platform/crafty/*)         echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
+    # Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
+    platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
+    # runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
+    # resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
+    # is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
+    # runbook change is an observability change.
+    runbooks/*)                echo "R-OBS-04" ;;
+    # platform-config/ is the env-specific configuration Flux substitutes into and applies.
+    platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
+    # platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
+    # Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
+    # file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
+    # change. That is the most consequential single-line edit in the repo and it attributed nothing.
+    platform/kustomization.yaml) echo "R-GITOPS-01" ;;
+    # .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
+    # here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
+    .sops.yaml)                echo "R-SEC-01 R-GITOPS-01" ;;
     conformance/*)             echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
     # infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
     # nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but

clusters/README.md

@@ -1,5 +1,19 @@
 # clusters/
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Flux entrypoints and the environment-overlay model (**R-ENV-02**, **R-GITOPS-01/02/03**).
 
 Governing spec: [`docs/05-gitops-and-delivery.md`](../../home-network-k3s/docs/05-gitops-and-delivery.md),

clusters/base/cluster-vars.yaml

@@ -99,7 +99,9 @@ data:
   # (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
   # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
   # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
-  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH), velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
+  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
+  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
   # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
   # UNSUBSTITUTED path would ship the literal string into the cluster.
   #
@@ -159,6 +161,30 @@ data:
   # HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
   UNIFI_L4_VIP: "192.168.1.54"               # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
   CRAFTY_L4_VIP: "192.168.1.55"              # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
+  # PiHole's DNS listener: udp/53 + tcp/53 for LAN CLIENTS (apps/wave-0/pihole-dns.yaml). It is a
+  # THIRD pinned L4 VIP for the same reason as the two above — the `shared` Gateway is an L7 HTTPS
+  # terminator and cannot express DNS at all (udp/53 especially). It must NOT be ${GATEWAY_VIP}:
+  # LAN clients are pointed at this address as their resolver, and it has to stay put across
+  # reschedules, so it is pinned by lbipam.cilium.io/ips rather than taken from the pool head.
+  #
+  # WHY A SEPARATE ADDRESS AND NOT THE GATEWAY VIP: every `<svc>.${BASE_DOMAIN}` name resolves TO
+  # the Gateway VIP. Serving DNS from the same address would be legal but conflates "the thing that
+  # answers where services live" with "a service", and it would put LAN resolution behind the
+  # Gateway Service's own lifecycle. Keep them separate.
+  #
+  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
+  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
+  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+
+  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
+  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
+  # from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
+  # CIDR, not to Cilium's `world` entity the way the unifi/crafty game-port allows are. It is the
+  # same /24 the node IPs and ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} come out of.
+  #
+  # dev  -> 192.168.1.0/24  (Proxmox LAN; nodes .50/.51/.52)
+  # prod -> 192.168.2.0/24  (prod LAN; nodes .10/.11/.12)
+  LAN_CIDR: "192.168.1.0/24"                 # ingress scope for the PiHole DNS listener
 
   # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
   # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media

clusters/dev/kustomization.yaml

@@ -122,6 +122,23 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
+  # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
+  # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
+  # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
+  # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
+  # as load-bearing as the file itself. The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-vms
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   - patch: |-
       apiVersion: source.toolkit.fluxcd.io/v1
       kind: GitRepository

clusters/prod/kustomization.yaml

@@ -113,6 +113,23 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
+  # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
+  # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
+  # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
+  # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
+  # as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-vms
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
   #
   # ############################################################################################
@@ -349,8 +366,16 @@ patches:
       # Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
       # size cap before 30d if series volume is high, keeping the disk from filling either way.
       - op: replace
+      # LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for
+      # what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL,
+      # not the head's in-flight segments, and not compaction scratch, and a large compaction can need
+      # roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on
+      # /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every
+      # absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB
+      # at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history,
+      # grow the PVC rather than the fraction of it.
         path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
-        value: "45GiB"
+        value: "36GiB"
       - op: replace
         path: /data/MONITORING_LOKI_PVC_SIZE
         value: "50Gi"
@@ -440,6 +465,23 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.2.22"
+      # PiHole's DNS listener VIP — udp/53 + tcp/53 for LAN clients (apps/wave-0/pihole-dns.yaml).
+      # 192.168.2.23 is the NEXT FREE address in the prod pool 192.168.2.20-.30, verified live
+      # 2026-09-14: the CiliumLoadBalancerIPPool `default-pool` reported IPsTotal 11 / IPsUsed 3 /
+      # IPsAvailable 8, and the only three allocations are .20 (cilium-gateway-shared), .21
+      # (unifi-l4-tcp/-udp) and .22 (crafty-l4-tcp/-udp). ARP-probed from a LAN host in the same
+      # broadcast domain (arping 192.168.2.23, 3 probes, 0 responses), so nothing off-cluster holds
+      # it either, and it is inside the pool block the operator already asserted sits outside the
+      # SOHO router's DHCP scope.
+      - op: replace
+        path: /data/PIHOLE_DNS_VIP
+        value: "192.168.2.23"
+      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
+      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
+      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
+      - op: replace
+        path: /data/LAN_CIDR
+        value: "192.168.2.0/24"
       # ------------------------------------------------------------------------------------------
       # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
       # ------------------------------------------------------------------------------------------

conformance/README.md

@@ -1,5 +1,19 @@
 # conformance/ — the home-k3s conformance harness (WS-TEST)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 This subtree is the **definition of "done"** for the whole platform. Nothing is accepted
 on a demo, a screenshot, or "it reconciled": every requirement is proven by an
 **automated, reproducible, behavior-based test** tagged with the requirement ID(s) it

conformance/incluster/README.md

@@ -1,5 +1,19 @@
 # In-cluster conformance Job (L2/L3)
 
+> **STATUS NOTE (2026-09-29) — read before trusting the "Phase 0" framing below.**
+> This document speaks as though no cluster exists and the platform is scaffolding. **A
+> three-node k3s cluster has existed for weeks** and Flux has been reconciling against it.
+> Gitea is live, the Actions runner `incluster-dind` is online, and `main` is a protected
+> branch. Where the text says "Phase 0", "stub", or "no cluster exists", read it as
+> describing a **live gap** rather than an expected stage — most importantly, L2/L3
+> conformance still does not run in CI (`vars.CLUSTER_AVAILABLE` is unset), which is a real
+> coverage hole, not a phase.
+>
+> This mismatch is not cosmetic. A month of "the gate enforces this" claims rested on
+> documentation rather than measurement, and `main` was unprotected with zero runners
+> registered the whole time. **Query the live API before asserting platform state.**
+
+
 Per **ADR-0011** and **docs/14 "Where each layer runs"**: L0/L1 run in CI without a
 cluster, but **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job**
 on the dev/test cluster that **CI triggers and waits for**. CI holds **no** cluster

conformance/lib/record_run.py

@@ -489,6 +489,36 @@ def pinned_chainsaw_version(conf_root: Path) -> str:
     return "unknown"
 
 
+def context_env() -> str:
+    """The env a run TARGETED, derived from the live kube context — never guessed.
+
+    WHY THIS IS NOT A DEFAULT STRING. `--env` used to fall back to the literal "dev"
+    (`os.environ.get("CONF_ENV", "dev")`), and CONF_ENV is set nowhere in this repo, so EVERY
+    recorded proof was stamped `env: dev` regardless of the cluster it ran against. That was
+    harmless while dev was the only cluster. Dev was decommissioned 2026-09-12 and prod is now
+    the only one, so on 2026-09-13 a genuine prod DR-restore proof was recorded as `env: dev` —
+    a proof that lies about where it was proven is worse than no proof, because the ledger's
+    whole purpose is to say what was demonstrated WHERE. Caught before it was committed.
+
+    Derived from `kubectl config current-context` (`k3s-prod` -> `prod`), which cannot drift
+    from the cluster the run actually talked to. Undeterminable is "unknown", NEVER a cluster
+    name: an unreadable context must not silently attribute a proof to an environment.
+    """
+    try:
+        p = subprocess.run(["kubectl", "config", "current-context"],
+                           capture_output=True, text=True, timeout=15)
+    except (OSError, subprocess.TimeoutExpired):
+        return "unknown"
+    if p.returncode != 0:
+        return "unknown"
+    ctx = (p.stdout or "").strip()
+    if not ctx:
+        return "unknown"
+    # contexts in this repo are named k3s-<env>; anything else is recorded verbatim so an
+    # unexpected context is visible in the ledger rather than being coerced into a known name.
+    return ctx[len("k3s-"):] if ctx.startswith("k3s-") else ctx
+
+
 def server_kube_version() -> str:
     """Server gitVersion via `kubectl version` (read-only), else "unknown"."""
     for args in (["kubectl", "version", "-o", "json"],
@@ -883,7 +913,9 @@ def main() -> int:
                     help=f"append-only ledger (default {DEFAULT_LEDGER})")
     ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
     ap.add_argument("--repo", type=Path, default=REPO_ROOT)
-    ap.add_argument("--env", default=os.environ.get("CONF_ENV", "dev"))
+    ap.add_argument("--env", default=os.environ.get("CONF_ENV") or context_env(),
+                    help="env the run targeted; derived from the kube context when "
+                         "unset (never defaulted to a cluster name)")
     ap.add_argument("--chainsaw-version", default=None,
                     help="override the versions.env pin (recorded verbatim)")
     ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),

conformance/run.sh

@@ -201,7 +201,7 @@ if [ "${RECORD}" = 1 ] && [ -z "${CONF_RECORD_CHILD:-}" ]; then
   # CONF_LEDGER exists so the harness's own tests can exercise this path without writing
   # to the real ledger; operators should never set it.
   "${PYTHON}" "${CONF_ROOT}/lib/record_run.py" \
-    --run-dir "${RUN_DIR}" --env "${CONF_ENV:-dev}" \
+    --run-dir "${RUN_DIR}" ${CONF_ENV:+--env "$CONF_ENV"} \
     --ledger "${CONF_LEDGER:-${CONF_ROOT}/runs/ledger.jsonl}" || true
   log "RECORD: run rc=${RUN_RC}. Commit conformance/runs/${RUN_ID}/ WITH its run.log and"
   log "RECORD: conformance/runs/ledger.jsonl — an uncommitted ledger proves nothing."
@@ -239,7 +239,7 @@ gen_status() {
     --registry "${CONF_ROOT}/requirements/registry.yaml" \
     --tests "${CONF_ROOT}/tests" \
     --out "${STATUS_OUT}" \
-    --env "${CONF_ENV:-dev}" \
+    ${CONF_ENV:+--env "$CONF_ENV"} \
     "$@"
 }
 

conformance/runs/20260914T021505Z-24b8111/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/dr-restore" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="dr-restore" classname="" time="150.698290" />
+  </testsuite>
+</testsuites>

conformance/runs/20260914T021505Z-24b8111/run.log

@@ -0,0 +1,672 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-03').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — dr-restore
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/dr-restore]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- dr-restore (/work/conformance/tests/l3-fault-injection/dr-restore)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/dr-restore
+=== PAUSE chainsaw/dr-restore
+=== CONT  chainsaw/dr-restore
+    l.go:52: | 02:15:05 | dr-restore | @chainsaw                                 | CREATE    | OK    | v1/Namespace @ chainsaw-glorious-sheep
+    l.go:52: | 02:15:05 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | BEGIN |
+    l.go:52: | 02:15:05 | dr-restore | assert-backup-target-is-off-cluster       | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        
+        # ---- helpers -------------------------------------------------------
+        # Classify a kubectl failure: an RBAC denial is a HARNESS DEFECT and must
+        # fail loudly; an absent API is a genuine precondition gap and may skip.
+        die_or_skip() {
+          case "$2" in
+            *orbidden*|*Unauthorized*|*cannot\ list*|*cannot\ get*)
+              echo "FAIL: conformance runner lacks RBAC for $1 -- fix conformance/incluster/rbac.yaml" >&2
+              echo "$2" >&2
+              exit 1 ;;
+            *)
+              echo "SKIP: $1 not available on this cluster -- $2"
+              echo DR_RESTORE_SKIPPED_OK
+              exit 0 ;;
+          esac
+        }
+        
+        ip2int() {
+          _q="$(echo "$1" | tr '.' ' ')"
+          set -- $_q
+          [ $# -eq 4 ] || { echo -1; return 0; }
+          echo $(( ($1 * 16777216) + ($2 * 65536) + ($3 * 256) + $4 ))
+        }
+        
+        # in_cidr <ip> <a.b.c.d/len> -> 0 when the ip is inside the prefix
+        in_cidr() {
+          case "$2" in *"/"*) ;; *) return 1 ;; esac
+          _net="${2%/*}"; _len="${2#*/}"
+          case "$_len" in ''|*[!0-9]*) return 1 ;; esac
+          [ "$_len" -ge 0 ] && [ "$_len" -le 32 ] || return 1
+          _i="$(ip2int "$1")"; _n="$(ip2int "$_net")"
+          [ "$_i" -ge 0 ] && [ "$_n" -ge 0 ] || return 1
+          [ "$_len" -eq 0 ] && return 0
+          _d=1; _s=0
+          while [ "$_s" -lt $((32 - _len)) ]; do _d=$((_d * 2)); _s=$((_s + 1)); done
+          [ $(( _i / _d )) -eq $(( _n / _d )) ]
+        }
+        
+        # ---- Gate 1: the replicated storage tier the DR-critical data lives on --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent -- no DR-critical data tier on this cluster yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Gate 2: the Velero API + a configured BackupStorageLocation --------
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' \
+          2>&1)" || die_or_skip "velero.io/backupstoragelocations" "$BSL_LINES"
+        if [ -z "$(echo "$BSL_LINES" | tr -d '[:space:]')" ]; then
+          echo "SKIP: Velero API present but NO BackupStorageLocation is configured -- no off-cluster backup target yet"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        echo "backup storage locations:"
+        echo "$BSL_LINES"
+        
+        VNS="$(echo "$BSL_LINES" | head -1 | cut -d' ' -f1)"
+        
+        # ---- Gate 3: Velero server + node-agent (kopia FSB) actually ready -------
+        # File System Backup is the volume-DATA path (platform/velero/helmrelease.yaml:
+        # deployNodeAgent=true, uploaderType=kopia, snapshotsEnabled=false). Without a
+        # ready node-agent no volume data can be captured, so the drill is not runnable.
+        VREADY=0; NAREADY=0
+        for i in $(seq 1 12); do
+          VREADY="$(kubectl -n "$VNS" get deploy velero -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)"
+          NAREADY="$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || true)"
+          echo "poll $i: velero readyReplicas=${VREADY:-0} node-agent numberReady=${NAREADY:-0}"
+          if [ "${VREADY:-0}" -ge 1 ] && [ "${NAREADY:-0}" -ge 1 ]; then break; fi
+          sleep 10
+        done
+        if [ "${VREADY:-0}" -lt 1 ] || [ "${NAREADY:-0}" -lt 1 ]; then
+          echo "SKIP: Velero server and/or its node-agent DaemonSet are not ready in ns '$VNS' -- backup/restore path not available"
+          echo DR_RESTORE_SKIPPED_OK
+          exit 0
+        fi
+        
+        # ---- Pick the BSL this drill will use: prefer the default, and it MUST be
+        #      Available. A CONFIGURED-BUT-UNAVAILABLE target is NOT a precondition
+        #      gap -- it is a live R-STATE-03 defect, so it FAILS (after a bounded
+        #      wait for first-validation on a cold start), it does not skip. --------
+        BSL_NS=""; BSL_NAME=""; BSL_URL=""; BSL_BUCKET=""
+        for i in $(seq 1 12); do
+          BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+            -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.provider}{" "}{.spec.default}{" "}{.spec.config.s3Url}{" "}{.spec.objectStorage.bucket}{"\n"}{end}' 2>/dev/null || true)"
+          CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true ' | head -1 || true)"
+          [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+          if [ -n "$CAND" ]; then
+            BSL_NS="$(echo "$CAND" | cut -d' ' -f1)"
+            BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+            BSL_URL="$(echo "$CAND" | cut -d' ' -f6)"
+            BSL_BUCKET="$(echo "$CAND" | cut -d' ' -f7)"
+            break
+          fi
+          echo "poll $i: no Available BackupStorageLocation yet"
+          sleep 10
+        done
+        if [ -z "$BSL_NAME" ]; then
+          echo "FAIL: a BackupStorageLocation is configured but NONE reached phase 'Available'." >&2
+          echo "      The off-cluster backup target is unreachable -- R-STATE-03 is BROKEN, not merely untested." >&2
+          echo "$BSL_LINES" >&2
+          exit 1
+        fi
+        echo "using BSL ${BSL_NS}/${BSL_NAME}  bucket=${BSL_BUCKET}  endpoint=${BSL_URL}"
+        
+        # ---- OFF-CLUSTER PROOF (1): an endpoint we can evaluate ------------------
+        if [ -z "$BSL_URL" ]; then
+          echo "FAIL: BackupStorageLocation ${BSL_NS}/${BSL_NAME} exposes no spec.config.s3Url;" >&2
+          echo "      the OFF-CLUSTER property of the backup target cannot be established, so this" >&2
+          echo "      test refuses to claim it. (Extend this check for a non-S3 provider.)" >&2
+          exit 1
+        fi
+        HOST="${BSL_URL#*://}"      # strip scheme
+        HOST="${HOST%%/*}"          # strip path
+        HOST="${HOST%%:*}"          # strip port
+        echo "backup endpoint host: $HOST"
+        
+        case "$HOST" in
+          localhost|*.svc|*.svc.cluster.local|*.cluster.local)
+            echo "FAIL: backup endpoint '$HOST' is a CLUSTER-INTERNAL name -- the backup target is IN the cluster." >&2
+            echo "      An in-cluster backup store cannot satisfy R-STATE-03 (off-cluster backup)." >&2
+            exit 1 ;;
+        esac
+        
+        # Resolve to an IPv4 address. If it cannot be resolved we CANNOT establish the
+        # off-cluster property -- that is a failure, never a silent pass.
+        case "$HOST" in
+          *[!0-9.]*)
+            # Endpoint given as a name. Try the resolvers that may exist in the
+            # runner image, in order; if none is available or none resolves, the
+            # test FAILS below rather than guessing.
+            EPIP="$(getent ahostsv4 "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(getent hosts "$HOST" 2>/dev/null | head -1 | cut -d' ' -f1 || true)"
+            [ -n "$EPIP" ] || EPIP="$(nslookup "$HOST" 2>/dev/null | grep -A2 '^Name:' | grep '^Address' | head -1 | sed 's/.*: *//' || true)" ;;
+          *) EPIP="$HOST" ;;
+        esac
+        if [ -z "$EPIP" ]; then
+          echo "FAIL: could not resolve backup endpoint host '$HOST' to an IPv4 address;" >&2
+          echo "      the OFF-CLUSTER property cannot be asserted, so this test does not pass." >&2
+          exit 1
+        fi
+        echo "backup endpoint resolves to: $EPIP"
+        
+        # ---- OFF-CLUSTER PROOF (2): the endpoint is outside every cluster address --
+        case "$EPIP" in 127.*|0.0.0.0) echo "FAIL: backup endpoint $EPIP is loopback -- not off-cluster" >&2; exit 1 ;; esac
+        
+        NODE_IPS="$(kubectl get nodes -o jsonpath='{range .items[*]}{range .status.addresses[*]}{.address}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for a in $NODE_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a NODE address of this cluster -- the backup is not off-cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any node address (checked $(echo "$NODE_IPS" | grep -c . || true) node addresses)"
+        
+        SVC_IPS="$(kubectl get svc -A -o jsonpath='{range .items[*]}{.spec.clusterIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $SVC_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a Service ClusterIP -- the backup target lives INSIDE the cluster." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Service ClusterIP (checked $(echo "$SVC_IPS" | grep -c . || true))"
+        
+        POD_IPS="$(kubectl get pods -A -o jsonpath='{range .items[*]}{.status.podIP}{"\n"}{end}' 2>/dev/null || true)"
+        for a in $POD_IPS; do
+          if [ "$a" = "$EPIP" ]; then
+            echo "FAIL: backup endpoint $EPIP is a POD IP -- the backup target is an in-cluster pod." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is not any Pod IP (checked $(echo "$POD_IPS" | grep -c . || true))"
+        
+        POD_CIDRS="$(kubectl get nodes -o jsonpath='{range .items[*]}{.spec.podCIDR}{"\n"}{range .spec.podCIDRs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        for c in $POD_CIDRS; do
+          case "$c" in *:*) continue ;; esac      # IPv6 prefixes are out of scope here
+          if in_cidr "$EPIP" "$c"; then
+            echo "FAIL: backup endpoint $EPIP falls inside node podCIDR $c -- it is in the cluster pod network." >&2
+            exit 1
+          fi
+        done
+        echo "endpoint is outside every node podCIDR (checked: $(echo "$POD_CIDRS" | tr '\n' ' '))"
+        
+        # Best-effort (needs networking.k8s.io/ServiceCIDR, which the least-privilege
+        # runner may not read): the checks above are the authoritative ones.
+        SVC_CIDRS="$(kubectl get servicecidrs.networking.k8s.io -o jsonpath='{range .items[*]}{range .spec.cidrs[*]}{@}{"\n"}{end}{end}' 2>/dev/null || true)"
+        if [ -n "$SVC_CIDRS" ]; then
+          for c in $SVC_CIDRS; do
+            case "$c" in *:*) continue ;; esac
+            if in_cidr "$EPIP" "$c"; then
+              echo "FAIL: backup endpoint $EPIP falls inside the Service CIDR $c -- it is in the cluster." >&2
+              exit 1
+            fi
+          done
+          echo "endpoint is outside every ServiceCIDR ($(echo "$SVC_CIDRS" | tr '\n' ' '))"
+        else
+          echo "(ServiceCIDR objects unreadable -- skipping that best-effort check; ClusterIP/PodIP/podCIDR checks stand)"
+        fi
+        
+        echo "OFF_CLUSTER_TARGET_OK  bsl=${BSL_NS}/${BSL_NAME} endpoint=${EPIP} bucket=${BSL_BUCKET}"
+    l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | LOG   |
+        === STDOUT
+        backup storage locations:
+        velero default Available aws true https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com k3s-prod-backup
+        poll 1: velero readyReplicas=1 node-agent numberReady=3
+        using BSL velero/default  bucket=k3s-prod-backup  endpoint=https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
+        backup endpoint host: 7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com
+        backup endpoint resolves to: 172.64.66.1
+        endpoint is not any node address (checked 6 node addresses)
+        endpoint is not any Service ClusterIP (checked 82)
+        endpoint is not any Pod IP (checked 197)
+        endpoint is outside every node podCIDR (checked: 10.42.0.0/24 10.42.0.0/24 10.42.1.0/24 10.42.1.0/24 10.42.2.0/24 10.42.2.0/24 )
+        endpoint is outside every ServiceCIDR (10.43.0.0/16 )
+        OFF_CLUSTER_TARGET_OK  bsl=velero/default endpoint=172.64.66.1 bucket=k3s-prod-backup
+    l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster       | SCRIPT    | DONE  |
+    l.go:52: | 02:15:08 | dr-restore | assert-backup-target-is-off-cluster       | TRY       | END   |
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | TRY       | BEGIN |
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        SCHED="$(kubectl get schedules.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" phase="}{.status.phase}{" cron="}{.spec.schedule}{" lastBackup="}{.status.lastBackup}{"\n"}{end}' 2>/dev/null || true)"
+        if [ -z "$(echo "$SCHED" | tr -d '[:space:]')" ]; then
+          # Distinguish "Velero absent" (skip, in lock-step with step 1) from
+          # "Velero present but nothing scheduled" (a real R-STATE-03 defect).
+          if kubectl get backupstoragelocations.velero.io -A >/dev/null 2>&1; then
+            if [ -z "$(kubectl get backupstoragelocations.velero.io -A -o name 2>/dev/null || true)" ]; then
+              echo "SKIP: no BackupStorageLocation configured -- scheduling not applicable yet"
+              echo DR_SCHEDULE_SKIPPED_OK
+              exit 0
+            fi
+            echo "FAIL: Velero is configured with a backup target but declares NO Schedule --" >&2
+            echo "      R-STATE-03 requires SCHEDULED off-cluster backups, not manual ones." >&2
+            exit 1
+          fi
+          echo "SKIP: Velero API absent -- backup scheduling not applicable on this cluster yet"
+          echo DR_SCHEDULE_SKIPPED_OK
+          exit 0
+        fi
+        echo "velero schedules:"
+        echo "$SCHED"
+        # A Schedule that exists but is paused/disabled backs nothing up.
+        if echo "$SCHED" | grep -q 'phase=Enabled'; then
+          echo BACKUPS_ARE_SCHEDULED_OK
+        else
+          echo "FAIL: no Velero Schedule is in phase 'Enabled' -- scheduled off-cluster backups are not running." >&2
+          exit 1
+        fi
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | LOG   |
+        === STDOUT
+        velero schedules:
+        velero/velero-daily phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T04:01:06Z
+        velero/velero-daily-resources-only phase=Enabled cron=0 3 * * * lastBackup=2026-09-13T03:00:03Z
+        BACKUPS_ARE_SCHEDULED_OK
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | SCRIPT    | DONE  |
+    l.go:52: | 02:15:08 | dr-restore | assert-backups-are-scheduled              | TRY       | END   |
+    l.go:52: | 02:15:08 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | BEGIN |
+    l.go:52: | 02:15:08 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -eu
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"          # scratch source namespace (destroyed by design)
+        DST="${NS}-dr-dst"          # clean restore target (must not pre-exist)
+        BK="${NS}-dr-backup"
+        RS="${NS}-dr-restore"
+        OWNER_LABEL="conformance.home.k3s/created-by=${NS}"
+        
+        # ---- Re-derive the preconditions (steps do not share shell state). Every
+        #      skip below mirrors step 1 exactly, so the whole test skips or runs
+        #      as a unit -- it can never half-run. -----------------------------
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent"; echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        BSL_LINES="$(kubectl get backupstoragelocations.velero.io -A \
+          -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.phase}{" "}{.spec.default}{"\n"}{end}' 2>/dev/null || true)"
+        CAND="$(echo "$BSL_LINES" | grep ' Available ' | grep ' true$' | head -1 || true)"
+        [ -n "$CAND" ] || CAND="$(echo "$BSL_LINES" | grep ' Available ' | head -1 || true)"
+        if [ -z "$CAND" ]; then
+          echo "SKIP: no Available BackupStorageLocation (see the off-cluster step for the authoritative verdict)"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        VNS="$(echo "$CAND" | cut -d' ' -f1)"
+        BSL_NAME="$(echo "$CAND" | cut -d' ' -f2)"
+        if [ "$(kubectl -n "$VNS" get daemonset node-agent -o jsonpath='{.status.numberReady}' 2>/dev/null || echo 0)" -lt 1 ]; then
+          echo "SKIP: Velero node-agent (kopia FSB) not ready -- volume data cannot be backed up"
+          echo DR_RESTORE_SKIPPED_OK; exit 0
+        fi
+        echo "velero ns=$VNS bsl=$BSL_NAME src=$SRC dst=$DST"
+        
+        # A clean restore target means PREVIOUSLY NONEXISTENT, and the source name must
+        # be free. If either is lying around from an aborted run, remove it before we
+        # claim "clean" -- and fail loudly if it will not go, because a half-deleted
+        # namespace would silently invalidate the destroy proof below.
+        kubectl delete namespace "$DST" "$SRC" --ignore-not-found --wait=true --timeout=300s
+        
+        # ============================ GIVEN ==============================
+        # A DR-critical stateful workload with known data on the replicated tier.
+        CANARY="dr-canary-$(date +%s%N)"
+        kubectl create namespace "$SRC"
+        kubectl label namespace "$SRC" "$OWNER_LABEL" --overwrite
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: dr-data
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        kubectl -n "$SRC" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: dr-workload
+          labels:
+            app: dr-workload
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 7200"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: dr-data
+        EOF
+        
+        kubectl -n "$SRC" wait --for=condition=Ready pod/dr-workload --timeout=300s
+        printf '%s' "$CANARY" | kubectl -n "$SRC" exec -i dr-workload -- sh -c 'cat > /data/canary && sync'
+        echo "seeded canary: $(kubectl -n "$SRC" exec dr-workload -- cat /data/canary)"
+        
+        ORIG_PV="$(kubectl -n "$SRC" get pvc dr-data -o jsonpath='{.spec.volumeName}')"
+        [ -n "$ORIG_PV" ] || { echo "FAIL: source PVC never bound to a PV" >&2; exit 1; }
+        echo "original in-cluster PersistentVolume: $ORIG_PV"
+        
+        # ======================= WHEN (part 1): back up off-cluster =======
+        # Created as a Velero Backup CR -- exactly what `velero backup create` does.
+        # defaultVolumesToFsBackup:true selects the kopia File System Backup path, so
+        # the VOLUME DATA (not just the resource JSON) is uploaded to the object store.
+        # ttl is short: if the finally-step DeleteBackupRequest is ever lost, Velero's
+        # own GC still removes this drill's objects from the bucket.
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Backup
+        metadata:
+          name: ${BK}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          includedNamespaces: [${SRC}]
+          storageLocation: ${BSL_NAME}
+          snapshotVolumes: false
+          defaultVolumesToFsBackup: true
+          ttl: 2h0m0s
+        EOF
+        
+        BPHASE=""
+        for i in $(seq 1 90); do
+          BPHASE="$(kubectl -n "$VNS" get backups.velero.io "$BK" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: backup phase=${BPHASE:-<pending>}"
+          case "${BPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${BPHASE:-}" != "Completed" ]; then
+          echo "FAIL: off-cluster backup did not complete (phase=${BPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get backups.velero.io "$BK" -o yaml >&2 || true
+          exit 1
+        fi
+        echo "backup $BK Completed to ${BSL_NAME}"
+        
+        # ---- OFF-CLUSTER PROOF (3): the VOLUME DATA really landed in the remote
+        #      kopia repository -- a PodVolumeBackup Completed with a snapshot ID.
+        PVB="$(kubectl -n "$VNS" get podvolumebackups.velero.io \
+                 -l velero.io/backup-name="$BK" \
+                 -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.phase}{" "}{.status.snapshotID}{"\n"}{end}' 2>/dev/null || true)"
+        echo "pod volume backups:"
+        echo "$PVB"
+        if [ -z "$(echo "$PVB" | tr -d '[:space:]')" ]; then
+          echo "FAIL: the backup captured NO pod volume data (no PodVolumeBackup)." >&2
+          echo "      Resource-only backups cannot restore DR-critical DATA." >&2
+          exit 1
+        fi
+        # A line is "<name> <phase> <snapshotID>"; require Completed AND a
+        # non-empty snapshot ID (an empty ID means nothing reached the repository).
+        GOOD_PVB="$(echo "$PVB" | grep -cE ' Completed [^[:space:]]+$' || true)"
+        if [ "${GOOD_PVB:-0}" -lt 1 ]; then
+          echo "FAIL: no PodVolumeBackup reached phase Completed with a snapshot ID --" >&2
+          echo "      the volume data did not reach the off-cluster repository." >&2
+          exit 1
+        fi
+        echo "volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)"
+        
+        # ============ WHEN (part 2): FAULT INJECTION -- annihilate the source ====
+        # Destroy EVERY in-cluster copy: the pod, the PVC, the namespace, and the
+        # underlying PersistentVolume (and with it the Longhorn replicas). After this
+        # there is no in-cluster source the restore could quietly read from.
+        kubectl delete namespace "$SRC" --wait=true --timeout=420s
+        for i in $(seq 1 30); do
+          if ! kubectl get namespace "$SRC" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for namespace $SRC to disappear"
+          sleep 10
+        done
+        if kubectl get namespace "$SRC" >/dev/null 2>&1; then
+          echo "FAIL: source namespace $SRC did not terminate; the destroy step is not convincing" >&2
+          exit 1
+        fi
+        echo "source namespace destroyed"
+        
+        # The longhorn StorageClass reclaims Delete, so the PV should already be gone;
+        # force it if a Retain policy (or a stuck finalizer) left it behind, because a
+        # surviving PV would make the restore prove nothing about off-cluster recovery.
+        kubectl delete pv "$ORIG_PV" --ignore-not-found --wait=false || true
+        for i in $(seq 1 30); do
+          if ! kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then break; fi
+          echo "poll $i: waiting for PersistentVolume $ORIG_PV to disappear"
+          sleep 10
+        done
+        if kubectl get pv "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: the original PersistentVolume $ORIG_PV SURVIVED the destroy step." >&2
+          echo "      An in-cluster copy of the data still exists, so a successful restore" >&2
+          echo "      would NOT prove recovery from the off-cluster backup. Refusing to pass." >&2
+          exit 1
+        fi
+        echo "original PersistentVolume $ORIG_PV is GONE -- no in-cluster copy of the data remains"
+        
+        # Best-effort corroboration (the least-privilege runner may not read longhorn.io):
+        # the Longhorn Volume CR and its replicas should be gone too.
+        if kubectl -n longhorn-system get volumes.longhorn.io "$ORIG_PV" >/dev/null 2>&1; then
+          echo "FAIL: Longhorn Volume $ORIG_PV still exists after PV deletion -- replicas survive in-cluster" >&2
+          exit 1
+        fi
+        echo "(longhorn Volume CR $ORIG_PV not present -- either deleted or not readable by this identity)"
+        
+        # ============================ RESTORE ============================
+        # Into a CLEAN, previously-nonexistent namespace (Velero creates it), which is
+        # the catalog's "restored into a clean state".
+        kubectl apply -f - <<EOF
+        apiVersion: velero.io/v1
+        kind: Restore
+        metadata:
+          name: ${RS}
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+          includedNamespaces: [${SRC}]
+          namespaceMapping:
+            ${SRC}: ${DST}
+          existingResourcePolicy: none
+        EOF
+        
+        RPHASE=""
+        for i in $(seq 1 90); do
+          RPHASE="$(kubectl -n "$VNS" get restores.velero.io "$RS" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
+          echo "poll $i: restore phase=${RPHASE:-<pending>}"
+          case "${RPHASE:-}" in
+            Completed|Failed|PartiallyFailed|FailedValidation) break ;;
+          esac
+          sleep 10
+        done
+        if [ "${RPHASE:-}" != "Completed" ]; then
+          echo "FAIL: restore did not complete (phase=${RPHASE:-<timeout>})" >&2
+          kubectl -n "$VNS" get restores.velero.io "$RS" -o yaml >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        echo "restore $RS Completed into $DST"
+        
+        # ============================= THEN ==============================
+        # The service returns WITH ITS DATA. With kopia FSB the restored pod carries a
+        # `restore-wait` init container that blocks until the node-agent has written the
+        # volume contents back, so Ready here already implies the data landed.
+        if ! kubectl -n "$DST" wait --for=condition=Ready pod/dr-workload --timeout=600s; then
+          echo "FAIL: restored workload never became Ready in the clean namespace $DST" >&2
+          kubectl -n "$DST" describe pod dr-workload >&2 || true
+          kubectl -n "$VNS" get podvolumerestores.velero.io -l velero.io/restore-name="$RS" -o wide >&2 || true
+          exit 1
+        fi
+        
+        NEW_PV="$(kubectl -n "$DST" get pvc dr-data -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        echo "restored PVC bound to PersistentVolume: ${NEW_PV:-<none>}"
+        [ -n "$NEW_PV" ] || { echo "FAIL: restored PVC did not bind a PV" >&2; exit 1; }
+        if [ "$NEW_PV" = "$ORIG_PV" ]; then
+          echo "FAIL: the restored PVC rebound the ORIGINAL PersistentVolume ($ORIG_PV)." >&2
+          echo "      That is an in-cluster survival, not an off-cluster restore." >&2
+          exit 1
+        fi
+        echo "restored onto a NEW volume ($NEW_PV != $ORIG_PV) -- the data came from outside the cluster"
+        
+        GOT="$(kubectl -n "$DST" exec dr-workload -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        if [ "$GOT" != "$CANARY" ]; then
+          echo "FAIL: restored data does not match what was backed up -- DR restore lost/corrupted data" >&2
+          exit 1
+        fi
+        
+        echo DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        velero ns=velero bsl=default src=chainsaw-glorious-sheep-dr-src dst=chainsaw-glorious-sheep-dr-dst
+        namespace/chainsaw-glorious-sheep-dr-src created
+        namespace/chainsaw-glorious-sheep-dr-src labeled
+        persistentvolumeclaim/dr-data created
+        pod/dr-workload created
+        pod/dr-workload condition met
+        seeded canary: dr-canary-1789352108935754215
+        original in-cluster PersistentVolume: pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9
+        backup.velero.io/chainsaw-glorious-sheep-dr-backup created
+        poll 1: backup phase=InProgress
+        poll 2: backup phase=InProgress
+        poll 3: backup phase=InProgress
+        poll 4: backup phase=Completed
+        backup chainsaw-glorious-sheep-dr-backup Completed to default
+        pod volume backups:
+        chainsaw-glorious-sheep-dr-backup-2gjrq Completed 979eed437233b45c7f85ba80074ab56b
+        volume data uploaded off-cluster (PodVolumeBackup Completed with a kopia snapshotID)
+        namespace "chainsaw-glorious-sheep-dr-src" deleted
+        source namespace destroyed
+        original PersistentVolume pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 is GONE -- no in-cluster copy of the data remains
+        (longhorn Volume CR pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9 not present -- either deleted or not readable by this identity)
+        restore.velero.io/chainsaw-glorious-sheep-dr-restore created
+        poll 1: restore phase=InProgress
+        poll 2: restore phase=InProgress
+        poll 3: restore phase=InProgress
+        poll 4: restore phase=Completed
+        restore chainsaw-glorious-sheep-dr-restore Completed into chainsaw-glorious-sheep-dr-dst
+        pod/dr-workload condition met
+        restored PVC bound to PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
+        restored onto a NEW volume (pvc-f0c061e1-c080-49f2-a381-23da5c886df2 != pvc-4107d59d-a34f-4cba-bd2a-c5b2446156d9) -- the data came from outside the cluster
+        recovered='dr-canary-1789352108935754215' expected='dr-canary-1789352108935754215'
+        DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK
+        === STDERR
+        Defaulted container "app" out of: app, restore-wait (init)
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | TRY       | END   |
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | BEGIN |
+    l.go:52: | 02:17:18 | dr-restore | data-restores-from-the-off-cluster-backup | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -u
+        NS="$NAMESPACE"
+        SRC="${NS}-dr-src"; DST="${NS}-dr-dst"; BK="${NS}-dr-backup"; RS="${NS}-dr-restore"
+        
+        VNS="$(kubectl get backupstoragelocations.velero.io -A -o jsonpath='{.items[0].metadata.namespace}' 2>/dev/null || true)"
+        
+        # Velero objects first: a DeleteBackupRequest makes Velero remove the backup
+        # DATA from the off-cluster bucket, not just the CR (self-cleaning contract).
+        if [ -n "$VNS" ]; then
+          kubectl -n "$VNS" delete restores.velero.io "$RS" --ignore-not-found --wait=false || true
+          if kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1; then
+            kubectl apply -f - <<EOF || true
+        apiVersion: velero.io/v1
+        kind: DeleteBackupRequest
+        metadata:
+          name: ${BK}-delete
+          namespace: ${VNS}
+          labels:
+            conformance.home.k3s/created-by: ${NS}
+        spec:
+          backupName: ${BK}
+        EOF
+            for i in $(seq 1 18); do
+              kubectl -n "$VNS" get backups.velero.io "$BK" >/dev/null 2>&1 || break
+              sleep 10
+            done
+            # Backstop if Velero could not process the request (the 2h ttl also GCs it).
+            kubectl -n "$VNS" delete backups.velero.io "$BK" --ignore-not-found --wait=false || true
+            kubectl -n "$VNS" delete deletebackuprequests.velero.io "${BK}-delete" --ignore-not-found --wait=false || true
+          fi
+          echo "velero objects for $NS cleaned up (ns $VNS)"
+        fi
+        
+        # Scratch namespaces (both, in case the run aborted at any point).
+        for n in "$DST" "$SRC"; do
+          kubectl delete namespace "$n" --ignore-not-found --wait=false || true
+        done
+        # Anything left labelled as ours (defensive; the two names above are the only
+        # namespaces this test ever creates).
+        kubectl delete namespace -l "conformance.home.k3s/created-by=${NS}" --ignore-not-found --wait=false || true
+        
+        # Released PVs from either namespace's PVCs would otherwise leak storage.
+        for pv in $(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.claimRef.namespace}{"\n"}{end}' 2>/dev/null | grep -E " (${SRC}|${DST})$" | cut -d' ' -f1 || true); do
+          kubectl delete pv "$pv" --ignore-not-found --wait=false || true
+          echo "released leftover PersistentVolume: $pv"
+        done
+        
+        echo "dr-restore teardown complete (best-effort; every object was created by this test)"
+    l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | LOG   |
+        === STDOUT
+        restore.velero.io "chainsaw-glorious-sheep-dr-restore" deleted from velero namespace
+        deletebackuprequest.velero.io/chainsaw-glorious-sheep-dr-backup-delete created
+        velero objects for chainsaw-glorious-sheep cleaned up (ns velero)
+        namespace "chainsaw-glorious-sheep-dr-dst" deleted
+        namespace "chainsaw-glorious-sheep-dr-dst" deleted
+        persistentvolume "pvc-f0c061e1-c080-49f2-a381-23da5c886df2" deleted
+        released leftover PersistentVolume: pvc-f0c061e1-c080-49f2-a381-23da5c886df2
+        dr-restore teardown complete (best-effort; every object was created by this test)
+    l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | SCRIPT    | DONE  |
+    l.go:52: | 02:17:30 | dr-restore | data-restores-from-the-off-cluster-backup | FINALLY   | END   |
+    l.go:52: | 02:17:30 | dr-restore | @chainsaw                                 | CLEANUP   | BEGIN |
+    l.go:52: | 02:17:30 | dr-restore | @chainsaw                                 | DELETE    | OK    | v1/Namespace @ chainsaw-glorious-sheep
+    l.go:52: | 02:17:36 | dr-restore | @chainsaw                                 | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/dr-restore (150.70s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260914T021902Z-bd17b65/report.junit.xml

@@ -0,0 +1,13 @@
+<?xml version="1.0" ?>
+<testsuites name="home-k3s-conformance" tests="1" failures="0" errors="0" skipped="0">
+  <testsuite name="home-k3s-conformance L0/L1 (offline)" tests="1" failures="0" errors="0" skipped="0">
+    <testcase classname="L0:R-ENV-02,R-STATE-03" name="placeholder-guard-coupling">
+      <properties>
+        <property name="requirements" value="R-ENV-02,R-STATE-03"/>
+        <property name="layer" value="L0"/>
+        <property name="path" value="/work/conformance/tests/l0-static/placeholder-guard-coupling"/>
+        <property name="scope" value=""/>
+      </properties>
+    </testcase>
+  </testsuite>
+</testsuites>

conformance/runs/20260914T021902Z-bd17b65/run.log

@@ -0,0 +1,7 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] Running 1 offline L0/L1 check(s) (no cluster; docs/14).
+[offline] L0  PASS placeholder-guard-coupling (R-ENV-02,R-STATE-03)
+[offline] wrote JUnit (1 case(s), 0 failure(s), 0 skip(s), 0 unwired) -> /work/conformance/_artifacts/report.junit.xml
+[run.sh] Done (offline l0). JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/20260919T051633Z-6eaaaae/run.log

@@ -0,0 +1,6 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.md and /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+ERROR: cluster reachable but 'chainsaw' (v0.2.12) not on PATH.
+Install the pinned binary or run via the in-cluster Job (incluster/).

conformance/runs/20260919T052225Z-9339d5a/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="node-loss" classname="" time="115.855688" />
+  </testsuite>
+</testsuites>

conformance/runs/20260919T052225Z-9339d5a/run.log

@@ -0,0 +1,232 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/node-loss
+=== PAUSE chainsaw/node-loss
+=== CONT  chainsaw/node-loss
+    l.go:52: | 05:22:26 | node-loss | @chainsaw                               | CREATE    | OK    | v1/Namespace @ chainsaw-advanced-boxer
+    l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | TRY       | BEGIN |
+    l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -euo pipefail
+        NS="$NAMESPACE"
+        
+        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
+        READY="$(kubectl get nodes \
+          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
+          | grep -c '^True$' || true)"
+        echo "ready nodes: ${READY}"
+        if [ "${READY:-0}" -lt 3 ]; then
+          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        CANARY="node-loss-canary-$(date +%s%N)"
+        
+        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
+        #    writer pod that writes the canary and flushes it to the replicated volume. --
+        kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: survivor
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        apply_pod() {
+          kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: $1
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 3600"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: survivor
+        EOF
+        }
+        
+        apply_pod writer
+        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
+        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
+        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
+        
+        # The victim node is the one the attached volume currently lives on.
+        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
+        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
+        echo "victim node (currently hosts the attached volume): $VICTIM"
+        
+        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
+        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
+        # runner is least-privilege and may not have longhorn.io read rights) — the
+        # data-availability assertion below is the authoritative proof, not this.
+        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        if [ -n "$PVNAME" ]; then
+          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
+            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
+        fi
+        
+        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
+        #    Label it with THIS test's unique namespace so `finally` can always find and
+        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
+        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
+        #    pods on the node are never touched.)
+        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
+        kubectl cordon "$VICTIM"
+        echo "cordoned + labelled victim node: $VICTIM"
+        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
+        echo "evicted writer off the victim node; volume detached from $VICTIM"
+        
+        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
+        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
+        #    the volume there from its replicas on the other nodes. --
+        apply_pod reader
+        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
+        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
+        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
+        if [ "$READER_NODE" = "$VICTIM" ]; then
+          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
+          exit 1
+        fi
+        
+        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
+        echo "volume available + data intact from a surviving node"
+        
+        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
+        # (bounded) for robustness to settle back to available (healthy|degraded — both
+        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
+        # Non-fatal for the same least-privilege reason as the pre-fault log.
+        if [ -n "$PVNAME" ]; then
+          for i in $(seq 1 30); do
+            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
+            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
+            echo "post-fault robustness[$i]=$ROB"
+            case "$ROB" in healthy|degraded) break ;; esac
+            sleep 10
+          done
+        fi
+        
+        echo NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        ready nodes: 3
+        persistentvolumeclaim/survivor created
+        pod/writer created
+        pod/writer condition met
+        wrote canary: node-loss-canary-1789795346371286043
+        victim node (currently hosts the attached volume): k3s-prod-2
+        pre-fault robustness=healthy replicas=3
+        node/k3s-prod-2 labeled
+        node/k3s-prod-2 cordoned
+        cordoned + labelled victim node: k3s-prod-2
+        pod "writer" deleted from chainsaw-advanced-boxer namespace
+        evicted writer off the victim node; volume detached from k3s-prod-2
+        pod/reader created
+        pod/reader condition met
+        reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
+        recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
+        volume available + data intact from a surviving node
+        post-fault robustness[1]=healthy
+        NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | TRY       | END   |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | FINALLY   | BEGIN |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -uo pipefail
+        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
+          kubectl uncordon "${n#node/}" || true
+          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
+          echo "restored node to service: ${n#node/}"
+        done
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        node/k3s-prod-2 uncordoned
+        node/k3s-prod-2 unlabeled
+        restored node to service: k3s-prod-2
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
+        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
+        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        pod "reader" deleted from chainsaw-advanced-boxer namespace
+        persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
+        cleanup best-effort; chainsaw namespace teardown finalizes
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | FINALLY   | END   |
+    l.go:52: | 05:23:38 | node-loss | @chainsaw                               | CLEANUP   | BEGIN |
+    l.go:52: | 05:23:38 | node-loss | @chainsaw                               | DELETE    | OK    | v1/Namespace @ chainsaw-advanced-boxer
+    l.go:52: | 05:24:21 | node-loss | @chainsaw                               | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/node-loss (115.86s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/ledger.jsonl

@@ -3,3 +3,6 @@
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L1", "note": "L1 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-NET-05", "R-NET-06", "R-NET-07"], "run_id": "20260913T014321Z-19fc8bc", "sentinel": null, "test": "egress-class-render", "ts_utc": "2026-09-13T01:43:21Z"}
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "19fc8bcb27eab42a0007b28a831519d3e325c7b6-dirty", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014437Z-19fc8bc", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:44:37Z"}
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
+{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
+{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}
+{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9339d5a79e2301d399e4e20722b164583c53c55d", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260919T052225Z-9339d5a", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-19T05:22:25Z"}

conformance/tests/l0-static/alert-metrics-have-scrape-targets/chainsaw-test.yaml

@@ -36,18 +36,27 @@
 #       (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
 #           check above can never see this failure: where the repo authors a Service and a
 #           ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
-#           actually defines. All 12 apps/* `<app>-down` SLI rules fail this today (the monitors name
-#           port `metrics`; no app Service defines one), so they are registered in
-#           KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series do not exist;
+#           actually defines. All 12 apps/* `<app>-down` SLI rules used to fail this (the monitors
+#           named port `metrics`; no app Service defined one) and were registered in
+#           KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series did not exist.
+#           THAT REGISTER IS NOW EMPTY BECAUSE THE TWELVE RULES WERE FIXED, not waived: they read
+#           kube-state-metrics readiness of their own workload object instead (2026-09-13), the dead
+#           ServiceMonitors are gone, and so is this axis' only subject. The monitor side therefore
+#           moved into its own check — tests/l0-static/servicemonitor-ports-resolve — which needs no
+#           alert rule to reach a monitor, resolves by selector across the whole tree instead of by
+#           same-file adjacency, and proves its resolver on fixtures so it cannot pass merely for
+#           lack of subjects;
 #   AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
 #           so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
 #           invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
 #
 # WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
-# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would make the gate red until
-# exporters exist for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends —
-# none of which speak Prometheus natively — so the gate would be disabled, and a disabled gate
-# protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
+# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would have made the gate red
+# until exporters existed for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends
+# — none of which speak Prometheus natively — so the gate would have been disabled, and a disabled
+# gate protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
+# (The ratchet did its job: the twelve were closed a day later without an exporter, by reading a
+# signal that already existed. The register is empty now, and that is the outcome it was for.)
 # So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
 # metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
 # register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -182,14 +182,42 @@ SCRAPE_OWNERS: list[dict] = [
     },
     {
         "owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)",
-        "prefixes": ("kubelet_",),
+        # container_* is cAdvisor, which the kubelet ServiceMonitor scrapes on the SAME target as
+        # kubelet_* via its /metrics/cadvisor path (this entry's title always said so; the prefix
+        # was added 2026-09-14 when ContainerIOStallHigh became the first rule to query it).
+        # Verified on prod: container_pressure_io_stalled_seconds_total, 405 series across the
+        # three nodes, job="kubelet", metrics_path="/metrics/cadvisor".
+        "prefixes": ("kubelet_", "container_"),
         "file": KPS,
         # Chart default kubelet.enabled:true. There is no explicit enable to require, so the only
         # honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were.
         "require": [r"kubeProxy:"],   # sentinel: the exporter-toggle block is still authored here
         "forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
         "why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three "
-               "pvc-capacity-alerts rules",
+               "pvc-capacity-alerts rules; container_pressure_io_stalled_seconds_total is the "
+               "only signal on this cluster that can name WHICH workload an I/O stall is hitting",
+    },
+    {
+        "owner": "kube-prometheus-stack: the kube-apiserver ServiceMonitor (chart default, NOT "
+                 "disabled the way kubeEtcd/kubeProxy are)",
+        # DELIBERATELY EXACT NAMES, NOT AN `etcd_` PREFIX. On this cluster the only etcd-shaped
+        # metrics that exist are the APISERVER'S CLIENT-SIDE view of its own local member. etcd's
+        # server-side disk metrics (etcd_disk_wal_fsync_duration_seconds,
+        # etcd_disk_backend_commit_duration_seconds) are NOT scraped — kubeEtcd.enabled:false for
+        # k3s' embedded etcd — and verified absent on prod 2026-09-14: a query for
+        # {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing at all. A blanket
+        # `etcd_` claim would let a future rule on those names pass this check while being inert,
+        # which is precisely the hole this test exists to close. Widen only with a real scrape.
+        "exact": (
+            "etcd_request_duration_seconds_bucket",
+            "etcd_request_duration_seconds_count",
+        ),
+        "file": KPS,
+        "require": [r"kubeProxy:"],   # sentinel: the exporter-toggle block is still authored here
+        "forbid": [r"\n\s*kubeApiServer:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
+        "why": "etcd_request_duration_seconds_* is the ONLY per-node view of the etcd storage path "
+               "this cluster has, and NodeEtcdRequestsSlow is the only alert that can see the "
+               "condition that took k3s-prod-1 NotReady on 2026-09-07 before the node drops",
     },
     {
         "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules",
@@ -228,7 +256,13 @@ SCRAPE_OWNERS: list[dict] = [
         "file": KPS,
         "require": [
             r"metricLabelsAllowlist:",
-            r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
+              # The label must be IN the allowlist, not BE the whole allowlist. This pattern used to
+              # end in a literal `]`, asserting the list held exactly this one label — so adding a
+              # second one (`egress-uplink`, for the steering-staleness alerts) made this check FAIL
+              # while the property it guards stayed perfectly satisfied. A guard that breaks when you
+              # correctly extend the thing it guards teaches people to edit the guard, which is how it
+              # stops guarding. `[,\\]]` accepts either the end of the list or a further entry.
+              r"nodes=\[homelab\.keithjackson\.dev/egress-gateway[,\]]",
         ],
         "forbid": [],
         "why": "KSM exposes NO custom labels by default. Without the allowlist entry "
@@ -333,38 +367,36 @@ KNOWN_UNWIRED_METRICS: dict[str, str] = {
 }
 
 # =============================================================================================
-# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only. Each is a `up{job="X"}` selector whose monitor
-# names an endpoint port that the Service it selects does not define, so the monitor discovers
-# ZERO targets, no `up{job="X"}` series exists, and `up{job="X"} == 0` can never fire. This is the
-# nastiest variant of the class: `up` exists in abundance, so nothing about the metric name looks
-# wrong, and `PrometheusTargetsMissing` cannot see it either — a target that was never discovered
-# has no up series to be 0.
+# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only, and NOW EMPTY.
 #
-# All twelve are the apps/* ServiceDefinition SLI pattern: the rendered ServiceMonitor names
-# `port: metrics`, and no app Service defines a port of that name (verified: zero occurrences of
-# `name: metrics` across all twelve manifests). VERIFIED on prod 2026-09-13: `count by (job) (up)`
-# listed 20 jobs and none of these twelve appeared; all twelve `<app>-down` rules returned no
-# series. Closing this needs a Prometheus exporter per app — Plex, Pi-hole, Jackett, qBittorrent,
-# Sonarr, Radarr and Seerr do not speak Prometheus natively — so it is its own piece of work, and
-# for keycloak specifically the job labels that DO exist are `keycloak-service`/`keycloak-discovery`
-# (platform/keycloak/servicemonitor.yaml), so that one is a rename away rather than an exporter.
+# Each entry was a `up{job="X"}` selector whose monitor named an endpoint port the Service it
+# selects did not define, so the monitor discovered ZERO targets, no `up{job="X"}` series existed,
+# and `up{job="X"} == 0` could never fire. The nastiest variant of the class: `up` exists in
+# abundance, so nothing about the metric name looks wrong, and `PrometheusTargetsMissing` cannot see
+# it either — a target that was never discovered has no up series to be 0.
+#
+# IT HELD ALL TWELVE apps/* SLI RULES, AND IT IS EMPTY BECAUSE THEY WERE FIXED, not because the gap
+# was waived (2026-09-13, branch fix/app-sli-rules-discover-no-targets). The register's own note
+# said closing it needed "a Prometheus exporter per app". That was the wrong conclusion from a true
+# premise: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud
+# genuinely export no Prometheus metrics, so instead the `up` SLI was rewritten against a signal that
+# DOES exist for every workload — kube-state-metrics readiness of the service's own workload object
+# (`kube_deployment_status_replicas_available` / `kube_statefulset_status_replicas_ready`), and
+# `kubevirt_vmi_info{phase="running"}` for the VM. No exporter, no new scrape target; each rule is
+# unit-tested firing AND quiet (apps/wave-*/tests/app-slis.test.yaml) and each expression was
+# evaluated live on prod, where eleven of the twelve return no series on a healthy cluster and the
+# twelfth (vm-demo) is not deployed at all.
+#
+# The rendered ServiceMonitors went with them: a monitor for a workload that serves no metrics is
+# dead configuration that reads as coverage. So assertion (4) below now has NOTHING to audit under
+# apps/ — which is why the monitor-side axis moved into its own check,
+# tests/l0-static/servicemonitor-ports-resolve. THAT is where a ServiceMonitor naming a port no
+# Service defines fails, with no alert rule required, resolving by selector across the whole tree
+# rather than by same-file adjacency, and with its resolver proven on fixtures so it cannot pass for
+# lack of subjects. Keeping this register non-empty as "documentation" of a closed gap is exactly the
+# fiction the ratchet forbids: the stale-entry assertion below would fail, and rightly.
 # =============================================================================================
-KNOWN_UNWIRED_JOBS: dict[str, str] = {
-    "gitea": "apps/wave-0/gitea.yaml — Service defines only port-3000",
-    "jackett": "apps/wave-0/jackett.yaml — no exporter",
-    "pihole": "apps/wave-0/pihole.yaml — needs pihole-exporter",
-    "plex": "apps/wave-0/plex.yaml — no native Prometheus endpoint",
-    "qbittorrent": "apps/wave-0/qbittorrent.yaml — needs qbittorrent-exporter",
-    "vm-demo": "apps/wave-0/vm-demo.yaml — a KubeVirt demo VM, no exporter",
-    "keycloak": "apps/wave-1/keycloak.yaml — the live jobs are keycloak-service / "
-                "keycloak-discovery (platform/keycloak/servicemonitor.yaml); this SLI rule names "
-                "neither, so it is a rename away from being live rather than an exporter away",
-    "nextcloud": "apps/wave-1/nextcloud.yaml — needs nextcloud-exporter (serverinfo token)",
-    "radarr": "apps/wave-1/radarr.yaml — needs exportarr",
-    "semaphore": "apps/wave-1/semaphore.yaml — no exporter",
-    "sonarr": "apps/wave-1/sonarr.yaml — needs exportarr",
-    "seerr": "apps/wave-2/seerr.yaml — no exporter",
-}
+KNOWN_UNWIRED_JOBS: dict[str, str] = {}
 
 # =============================================================================================
 # (0) PARSE EVERY ALERT RULE
@@ -422,9 +454,14 @@ def is_golden_fixture(p: Path) -> bool:
     return "conformance" in parts and "golden" in parts
 
 
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 rule_files = sorted(
     p for p in ROOT.rglob("*.yaml")
-    if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
+    if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
 )
 check(len(rule_files) > 0,
       "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
@@ -593,7 +630,7 @@ def yaml_docs(path: Path) -> list[dict]:
 # job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
 same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
 for p in sorted(ROOT.rglob("*.yaml")):
-    if ".git" in p.parts:
+    if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts):
         continue
     txt = read(p)
     if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:

conformance/tests/l0-static/alert-rules-unit-tested/check.sh

@@ -147,25 +147,20 @@ def check(cond, msg):
 # =============================================================================
 # THE DECLARED REGISTER (assertion 7). Shrink-only; see the header for why this is a ratchet.
 #
-# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. The twelve apps/* entries
-# are single `<app>-down` liveness rules generated from one ServiceDefinition template;
-# platform/kubevirt/alerts.yaml (4 rules) is the substantial one and is where the next tests
-# belong. A file listed here that later gains coverage, or stops carrying a PrometheusRule,
-# FAILS as stale.
+# UNTESTED_FILES — PrometheusRule files no test file covers AT ALL. A file listed here that later
+# gains coverage, or stops carrying a PrometheusRule, FAILS as stale.
+#
+# THE TWELVE apps/* ENTRIES ARE GONE (2026-09-13). They were the single `<app>-down` liveness rules
+# generated from one ServiceDefinition template, and the register's description of them as one
+# template was exactly why leaving them untested was so expensive: ONE wrong expression was wrong
+# twelve times, and it was — `up{job="<app>"} == 0` against a job no scrape target produced, inert on
+# prod for all twelve. They are now covered by apps/wave-0/tests/app-slis.test.yaml,
+# apps/wave-1/tests/app-slis.test.yaml and apps/wave-2/tests/app-slis.test.yaml, each rule pinned
+# BOTH ways (quiet on the healthy state, firing on no-ready-replica, and firing when the signal
+# VANISHES — the failure mode that produced the class). platform/kubevirt/alerts.yaml (4 rules) is
+# now the only entry left and is where the next tests belong.
 # =============================================================================
 UNTESTED_FILES = {
-    "apps/wave-0/gitea.yaml",
-    "apps/wave-0/jackett.yaml",
-    "apps/wave-0/pihole.yaml",
-    "apps/wave-0/plex.yaml",
-    "apps/wave-0/qbittorrent.yaml",
-    "apps/wave-0/vm-demo.yaml",
-    "apps/wave-1/keycloak.yaml",
-    "apps/wave-1/nextcloud.yaml",
-    "apps/wave-1/radarr.yaml",
-    "apps/wave-1/semaphore.yaml",
-    "apps/wave-1/sonarr.yaml",
-    "apps/wave-2/seerr.yaml",
     "platform/kubevirt/alerts.yaml",
 }
 
@@ -173,8 +168,21 @@ UNTESTED_FILES = {
 # own rule nothing asserts on. This is the half that stops a new alert being slipped into an
 # already-"tested" file untested: anything not listed here must appear in a covering test.
 # The monitoring block is large because backups.test.yaml (the repo's first suite) deliberately
-# covers the Velero/backup rules only; pvc-capacity-alerts.yaml is flattened into the same
-# rules file by the suite convention but has no assertions of its own yet.
+# covers the Velero/backup rules only. It SHRANK by seven on 2026-09-13, when
+# platform/monitoring/tests/longhorn.test.yaml took on the whole platform-health.longhorn group
+# plus the three kubelet-based PVC-fill rules it has to be measured against (the per-volume
+# Longhorn "capacity" rule was deleted as a permanently-firing duplicate of them, so proving the
+# redundancy meant asserting on the rules that survive). Only
+# PersistentVolumeFillingUpPredicted is still unasserted in pvc-capacity-alerts.yaml.
+# It SHRANK BY TWO MORE on 2026-09-14: platform/monitoring/tests/node-io-cause.test.yaml pins the
+# eight new platform-health.node-io-cause rules, and its healthy fixture asserts NodeIOPressureHigh
+# and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
+# that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
+# detectors are no longer unasserted.
+# It SHRANK BY ONE on 2026-09-19: platform/monitoring/tests/vpn-egress.test.yaml pins the two new
+# egress-steering rules, and asserts VPNTunnelDown QUIET in all four fixtures — that silence IS the
+# claim those rules exist for (tun0 up and passing traffic while vpn-class egress is dropped), so it
+# had to become an assertion rather than a sentence in a comment.
 UNTESTED_ALERTS = {
     ("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
     ("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
@@ -186,30 +194,20 @@ UNTESTED_ALERTS = {
     ("platform/monitoring/alerts.yaml", "NodeDiskFillingUp"),
     ("platform/monitoring/alerts.yaml", "NodeCPUPressureHigh"),
     ("platform/monitoring/alerts.yaml", "NodeCPULoadHigh"),
-    ("platform/monitoring/alerts.yaml", "NodeIOPressureHigh"),
-    ("platform/monitoring/alerts.yaml", "NodeIOPressureCritical"),
-    ("platform/monitoring/alerts.yaml", "LonghornVolumeDegraded"),
-    ("platform/monitoring/alerts.yaml", "LonghornVolumeFaulted"),
-    ("platform/monitoring/alerts.yaml", "LonghornVolumeCapacityHigh"),
-    ("platform/monitoring/alerts.yaml", "LonghornNodeStorageHigh"),
     ("platform/monitoring/alerts.yaml", "FluxReconciliationFailure"),
     ("platform/monitoring/alerts.yaml", "FluxResourceSuspended"),
     ("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
     ("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
     ("platform/monitoring/alerts.yaml", "CertificateNotReady"),
-    ("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
     ("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
     ("platform/monitoring/alerts.yaml", "KeycloakDown"),
     ("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
     ("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerFailing"),
     ("platform/monitoring/alerts.yaml", "KeycloakClientSecretReconcilerStale"),
-    ("platform/monitoring/alerts.yaml", "MonitoringVolumeFillingUp"),
     ("platform/monitoring/alerts.yaml", "PrometheusTargetsMissing"),
     ("platform/monitoring/alerts.yaml", "ImageUpdateAutomationFailing"),
     ("platform/monitoring/alerts.yaml", "ImagePolicyStalled"),
     ("platform/monitoring/alerts.yaml", "UpdateHealthGateFailed"),
-    ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUp"),
-    ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeAlmostFull"),
     ("platform/monitoring/pvc-capacity-alerts.yaml", "PersistentVolumeFillingUpPredicted"),
 }
 
@@ -285,11 +283,16 @@ FETCH_RE = re.compile(
 #     versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
 #     version, the network and the machine are decided silently at test time, and a missing
 #     toolchain becomes a green run instead of an exit 3.
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
 INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
 for p in sorted(ROOT.rglob("*.sh")):
     rel = str(p.relative_to(ROOT))
-    if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
+    if rel.startswith((".git/", ".claude/", "conformance/runs/", SELF_DIR)):
         continue
     try:
         text = p.read_text()
@@ -324,7 +327,7 @@ def tracked_yaml() -> list[Path]:
         return [ROOT / p for p in out.stdout.split("\0") if p]
     # No usable index (the gate container mounts a worktree whose .git points outside the
     # mount). Fall back to a filesystem walk rather than auditing nothing.
-    return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
+    return sorted(p for p in ROOT.rglob("*.y*ml") if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts))
 
 
 KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
@@ -478,7 +481,34 @@ for rel in sorted(alert_files):
           f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
           f"shrink-only: delete the line so the next missing test is caught")
     text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
-    covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
+    # "COVERED" MUST MEAN ASSERTED, NOT MENTIONED (tightened 2026-09-19).
+    # This used to be a bare word match over the whole test file, so an alert named anywhere —
+    # including in a PROSE COMMENT — counted as tested. Hit for real while adding
+    # platform/monitoring/tests/vpn-egress.test.yaml: its header comment explains that VPNTunnelDown
+    # stays quiet during an egress-steering failure, and that sentence alone was enough to make this
+    # check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Deleting
+    # the register entry would then have recorded coverage that did not exist — the exact "looks fixed"
+    # outcome this check exists to prevent, produced BY the check.
+    # Now an alert counts only when a test file names it in an `alertname:` position, with comment
+    # lines stripped first so a commented-out assertion does not count either.
+    # MEASURED BLAST RADIUS BEFORE TIGHTENING: of 61 alerts across the PrometheusRule manifests, 32 are
+    # asserted in one of the two idioms below, 29 are not mentioned at all (already declared in the
+    # registers), and ZERO are prose-only — so this change failed nothing on the day it landed. That
+    # mattered because the registers are shrink-only: had any alert been prose-only, tightening would
+    # have needed a register entry it is not allowed to add. (The first measurement said 31/30/0 because
+    # it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companions.)
+    text_no_comments = "\n".join(
+        l for l in text.split("\n") if not l.lstrip().startswith("#")
+    )
+    # BOTH promtool idioms count, because both are real assertions:
+    #   alert_rule_test:   - alertname: CNPGPrimariesCoLocated
+    #   promql_expr_test:  - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
+    # Matching only the first produced a FALSE NEGATIVE on CNPGPrimariesCoLocated, which
+    # platform/cnpg/alerts.promtool-test.yaml pins across FOUR scenarios entirely through the second
+    # form. Had that been believed, the "fix" would have been a redundant duplicate test for an alert
+    # already better covered than most.
+    covered = [n for n in names
+               if re.search(rf'(?:alertname:\s*|alertname="){re.escape(n)}\b', text_no_comments)]
     covered_total += len(covered)
     for n in names:
         if n in covered:

conformance/tests/l0-static/alert-runbooks-resolvable/check.sh

@@ -126,9 +126,14 @@ def read(p: Path) -> str:
 SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
 RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
 
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 rule_files = sorted(
     p for p in ROOT.rglob("*.yaml")
-    if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
+    if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and "kind: PrometheusRule" in read(p)
 )
 check(len(rule_files) > 0,
       "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "

conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml

@@ -0,0 +1,136 @@
+# =============================================================================
+# L0 STATIC — no service declared `internal` can produce a PUBLIC DNS record
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. docs/09 R-DNS-03 says `internal` services are unreachable from the
+# public internet, that external names resolve to the PUBLIC IP, and that Pi-hole provides the
+# internal split-horizon answers. schemas/renderer-contract.md §exposure says an `internal`
+# service renders "no public DNS record". NOTHING ENFORCED EITHER SENTENCE. ExternalDNS
+# (platform/externaldns/helmrelease.yaml, `sources: [gateway-httproute, service]`,
+# `policy: sync`, provider cloudflare) considered EVERY HTTPRoute on the shared Gateway, so six
+# `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — each
+# published `<name>.home.keithjackson.dev` into the PUBLIC authoritative Cloudflare zone,
+# answering with the Gateway VIP 192.168.2.20. Verified at the authority on 2026-09-14; the
+# pre-state is committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt.
+#
+# Three things were wrong at once, and none of them was visible from inside the cluster:
+#   * the internal subnet, the Gateway VIP and a complete service inventory were publicly
+#     enumerable by anyone who could guess or brute-force a label;
+#   * a public name answering with an RFC1918 address IS a DNS-rebind primitive, which is why the
+#     SOHO router at 192.168.2.1 strips those answers and LAN clients need per-client dnsmasq
+#     overrides to reach their own services;
+#   * the guarantee was only ever prose. `exposure` was the operator's one declaration of intent
+#     and it had NO effect on DNS whatsoever, so "internal" meant nothing an attacker had to
+#     respect.
+#
+# WHY THE FIX NEEDS A GUARD AND NOT JUST A COMMIT. The mechanism is an annotation on a rendered
+# HTTPRoute. An annotation is exactly the kind of thing that survives review by looking harmless
+# and then disappears in a refactor of `_route_meta`, a hand-edit of apps/, or a new service added
+# by copying a public one. Nothing about its ABSENCE is loud: the service keeps working on the LAN,
+# the Kustomization stays Ready, and the only symptom is a public record nobody is looking at. The
+# 2026-09-13 ServiceMonitor incident in this same tree is the precedent — twelve monitors pointed
+# at a port that did not exist for weeks, silently, because the only thing asserting the wiring was
+# prose. This test makes the DNS guarantee a gate condition instead.
+#
+# GIVEN every ServiceDefinition in catalog/ and every HTTPRoute and external-dns-annotated
+#       Service this repo authors,
+# WHEN  each catalog service's rendered routes are resolved in the applied tree (apps/),
+# THEN  (1) every route of an `exposure: internal` service carries ExternalDNS's own controller
+#           annotation set to something OTHER than `dns-controller`, which external-dns v0.21.0
+#           uses to skip the source object outright (source/gateway.go ->
+#           annotations.IsControllerMismatch, checked before any hostname is resolved), so the
+#           route cannot become an endpoint for ANY name it declares;
+#       (2) NO route of an `exposure: public` service carries it — asserted explicitly, because a
+#           renderer that stamped every route would satisfy (1) perfectly while silently
+#           withdrawing the five public services' records, and `policy: sync` would delete them;
+#       (3) the same holds for the `service` source: an internal service's rendered Service may
+#           not carry an external-dns hostname annotation without the disown annotation beside it.
+#           Two sources are configured, so closing only the route half is half a guarantee;
+#       (4) the annotation is DERIVED FROM `exposure` in catalog/render.py and from nothing else —
+#           the emitting function must read `exposure`, must not name any catalog service, and
+#           must not be able to emit external-dns's own `dns-controller` value (the one value that
+#           CLAIMS a route rather than releasing it, inverting the guarantee while looking
+#           plausible). A per-service list is how twelve ServiceMonitors acquired one identical
+#           wrong port name in a single edit;
+#       (5) THE RENDERER IS PROVEN END-TO-END on synthetic ServiceDefinitions, by invoking the
+#           SHIPPED catalog/render.py as a subprocess: an `internal` fixture must yield the
+#           annotation, a `public` fixture must not, and a fixture that OMITS `exposure` must also
+#           yield it (the contract's default posture is `internal`, and omitting the field is the
+#           likeliest way a new service arrives). Without this the check would still pass if the
+#           renderer stopped working and someone hand-maintained apps/;
+#       (6) every HTTPRoute or annotated Service in the tree that the catalog does NOT render is
+#           DECLARED in KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why it is out of
+#           scope. Those are hand-authored platform/apps manifests with no `exposure` field for a
+#           renderer to read, and they are the seven names that REMAIN publicly advertised with an
+#           RFC1918 answer after this change. The register is shrink-only: an entry that stops
+#           matching a real object FAILS as stale, so the set of unadjudicated publishers cannot
+#           quietly grow, and adding a new one is a review event;
+#   AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
+#           so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
+#           ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
+#           ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
+#
+# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh's path map keys apps/* to NOTHING, and
+# apps/ is exactly where the rendered routes live. A requirement-scoped selection would therefore
+# filter this check out of the change most likely to break it — a hand-edit of a rendered route.
+# Same reasoning as its whole-repo siblings.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
+# the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it directly,
+# and this Chainsaw Test wraps the SAME script.
+#
+# NOT proven (honest gaps; this is static, and it runs with no cluster and no network):
+#   - That the record is actually GONE from Cloudflare. That is a live read against an external
+#     authority and belongs in L2 beside conformance/tests/l2-behavioral/dns-exposure, which owns
+#     the positive direction (a public name DOES appear). The negative direction was proven
+#     MANUALLY on prod against the deployed v0.21.0 before this landed — a throwaway HTTPRoute
+#     carrying the annotation produced no record at all, and adding the annotation to a route that
+#     already had one produced `action=DELETE` for both its A record and its `a-` TXT registry
+#     entry within one ~60s reconcile — but a manual proof is not a gate.
+#   - That an `internal` name is unreachable from the internet. No DNS record is necessary for
+#     reachability; a port-forward plus a hardcoded IP still reaches the Gateway. This check owns
+#     the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
+#   - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
+#     each publishes so the claim is reviewable, which is all a static check can offer.
+#   - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
+#     rather than a routable address; that is an open operator decision recorded in
+#     platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: internal-services-publish-no-public-dns
+  labels:
+    conformance.home.k3s/R-DNS-02: "true"
+    conformance.home.k3s/R-DNS-03: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — this invariant quantifies over every ServiceDefinition in catalog/, every
+    # HTTPRoute in apps/wave-* and platform/* and runbooks/, and every external-dns-annotated
+    # Service (apps/vms/unifi, platform/crafty). No single requirement path-maps to that set, and
+    # ci/touched-requirements.sh maps apps/* — where the rendered routes live — to no requirement
+    # at all, so a requirement-scoped selection would skip this check on precisely the changes
+    # that can break it.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "true"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      A service the operator declared `internal` cannot produce a public DNS record: the
+      guarantee is derived from `exposure` by catalog/render.py, present on every rendered route
+      in the applied tree, absent from every public one, proven end-to-end on synthetic fixtures,
+      and the hand-authored manifests the catalog does not render are enumerated in a shrink-only
+      register so the unadjudicated set cannot grow silently.
+spec:
+  steps:
+    - name: assert-no-internal-service-can-publish-a-public-dns-record
+      description: >-
+        Resolve every catalog service's rendered HTTPRoutes in apps/, assert the ExternalDNS
+        disown annotation on every internal one and on no public one, extend the same rule to the
+        `service` source, assert catalog/render.py derives the annotation from `exposure` rather
+        than from a service list, prove the shipped renderer on internal/public/default-posture
+        fixtures, and ratchet the register of hand-authored publishers the catalog does not render.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh

@@ -0,0 +1,458 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — no service declared `internal` can produce a PUBLIC DNS record.
+# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
+#
+# docs/09 R-DNS-03 and schemas/renderer-contract.md §exposure both say an `internal` service
+# renders no public DNS record. Nothing enforced it: ExternalDNS watched every HTTPRoute on the
+# shared Gateway, so six internal services published `<name>.home.keithjackson.dev` into the
+# public Cloudflare zone answering with the RFC1918 Gateway VIP (verified at the authority
+# 2026-09-14; pre-state committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt). The fix
+# derives ExternalDNS's controller annotation from `exposure` in catalog/render.py; this check is
+# what keeps it derived, present, and scoped.
+#
+# ASSERTIONS
+#   (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
+#   (1) every rendered route of an `internal` service carries the disown annotation;
+#   (2) NO rendered route of a `public` service carries it — a renderer stamping everything would
+#       satisfy (1) and silently delete the five public records under policy: sync;
+#   (3) the `service` source half: an internal service's Service may not carry an external-dns
+#       hostname annotation without the disown annotation beside it;
+#   (4) catalog/render.py DERIVES the annotation from `exposure` — the emitting function reads that
+#       field, names no catalog service, and cannot emit external-dns's own `dns-controller`;
+#   (5) the SHIPPED renderer is proven end-to-end on synthetic ServiceDefinitions (internal,
+#       public, and `exposure` OMITTED — the contract's default posture is internal);
+#   (6) every HTTPRoute / annotated Service the catalog does NOT render is declared in
+#       KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL;
+#   (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by
+#       ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written
+#       ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears).
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
+#   - That the record is gone from Cloudflare. Live read; belongs beside
+#     conformance/tests/l2-behavioral/dns-exposure, which owns the positive direction. The negative
+#     direction was proven manually on prod against the deployed v0.21.0 (a route carrying the
+#     annotation produced no record; adding it to a route that had one produced action=DELETE for
+#     both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
+#   - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
+#     This check owns the NAME axis only.
+#   - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
+#     routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
+# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if
+# catalog/ or apps/ is absent — i.e. the subject tree has vanished, which is not a pass.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if the subject trees are not there at all.
+if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ] || [ ! -f "${ROOT}/catalog/render.py" ]; then
+  echo "check SKIPPED: catalog/, apps/ or catalog/render.py absent — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: an `internal` ServiceDefinition cannot produce a public DNS record.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than
+line-based parsing: an HTTPRoute's annotations and a ServiceDefinition's exposure are nested
+structures, and a line-based reader of them is precisely the approximate parser that silently
+stops matching — the failure class this check exists to prevent.
+"""
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+# external-dns's own constants (v0.21.0 source/annotations/annotations.go):
+#   ControllerKey   = annotationPrefix + "controller"   -> the key below
+#   ControllerValue = "dns-controller"                  -> the value that CLAIMS an object
+# source/annotations/processors.go IsControllerMismatch skips a source object when the key is
+# PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed.
+CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
+CONTROLLER_CLAIMED = "dns-controller"
+HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname"
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+def docs_of(p: Path):
+    """Every YAML document in a file, tolerating the non-manifest files in the tree."""
+    try:
+        return [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
+    except yaml.YAMLError:
+        return []
+
+
+# =============================================================================================
+# REGISTER — KNOWN_NON_CATALOG_PUBLISHERS. Shrink-only.
+#
+# These are HAND-AUTHORED HTTPRoutes and Services that put a name into the same public Cloudflare
+# zone but that catalog/render.py does not render, so there is no `exposure` field for the renderer
+# to derive anything from and this check cannot adjudicate them. That is a real limit, not a pass —
+# so each one is DECLARED, with the name it publishes and why it is out of this check's scope, and
+# the register can only shrink. An entry naming an object that no longer exists FAILS as stale, so
+# a deleted publisher cannot leave a lie behind; a NEW publisher fails until it is declared, so the
+# unadjudicated set cannot grow without a review event. Same ratchet reasoning as
+# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
+#
+# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
+# the honest state after this change, and it is deliberate: converting them is an operator
+# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
+# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
+# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
+#
+# Keyed by "<relpath>:<namespace>/<name>".
+# =============================================================================================
+KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
+    "platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
+        "publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
+        "platform component, not a catalog service; it must be reachable by every browser that "
+        "any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
+        "whether the five public services are — an operator decision, not a renderer one.",
+    "platform/keycloak/httproute.yaml:keycloak/keycloak":
+        "publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
+        "OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
+        "the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
+        "hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
+        "is meant to be public either way; it is listed because the PUBLISHING object is the "
+        "hand-authored platform route, which carries no exposure field.",
+    "platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
+        "publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
+        "certainly should not be publicly resolvable, but it is not a catalog service and "
+        "withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
+        "internal services (Pi-hole must answer first).",
+    "platform/headlamp/httproute.yaml:headlamp/headlamp":
+        "publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
+        "Keycloak OIDC; same adjudication as grafana.",
+    "platform/homepage/httproute.yaml:homepage/homepage":
+        "publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
+        "listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
+        "declaration.",
+    "runbooks/serving/httproute.yaml:runbooks/runbooks":
+        "declares NO hostnames and inherits the https-apex listener's ${BASE_DOMAIN}, so it "
+        "contributes to the apex record rather than a name of its own (the apex TXT registry "
+        "entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
+        "header for why gating runbooks on Keycloak is a loop with no exit.",
+    "platform/crafty/httproute.yaml:crafty/crafty":
+        "publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
+        "oauth2-proxy sidecar). Platform component, no exposure field.",
+    "platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
+        "the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
+        "${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
+        "Gateway cannot express. A game server is the one case where a public name may be "
+        "genuinely wanted; the address it answers with is still RFC1918.",
+    "apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
+        "the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
+        "(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
+        "under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
+        "vm-demo, which declares no ingressRoutes and publishes nothing.",
+}
+
+# =============================================================================================
+# (0) INVENTORY — the catalog, and every HTTPRoute / annotated Service in the tree.
+# =============================================================================================
+CATALOG = ROOT / "catalog"
+sds: dict[str, dict] = {}
+for p in sorted(CATALOG.glob("*.yaml")):
+    for d in docs_of(p):
+        if "name" in d and "source" in d:
+            sds[d["name"]] = d
+
+check(len(sds) > 0, "inventory: catalog/ yielded no ServiceDefinition — parse contract broken")
+
+# The default posture is `internal` (renderer-contract "Default posture"), so an ABSENT field is
+# internal, never "unclassified". Getting this wrong in the check would be the same bug as in the
+# renderer: a new service that omits exposure would go unaudited.
+exposure_of = {n: (sd.get("exposure") or "internal") for n, sd in sds.items()}
+# (namespace, routename) -> service, for the routes the renderer emits: one per ingressRoute,
+# named "<service>-<port>" in namespace "<service>" (catalog/render.py _route_meta).
+expected_routes: dict[tuple[str, str], str] = {}
+for n, sd in sds.items():
+    for r in sd.get("ingressRoutes") or []:
+        expected_routes[(n, f"{n}-{r['port']}")] = n
+
+internal_with_routes = sorted({s for (_, _), s in expected_routes.items()
+                               if exposure_of[s] == "internal"})
+public_with_routes = sorted({s for (_, _), s in expected_routes.items()
+                             if exposure_of[s] == "public"})
+check(len(internal_with_routes) > 0,
+      "inventory: no `internal` catalog service declares an ingressRoute — the subject set of "
+      "this check is empty, which means either the catalog changed shape or the parse is wrong")
+check(len(public_with_routes) > 0,
+      "inventory: no `public` catalog service declares an ingressRoute — assertion (2) would be "
+      "vacuous, so the parse is presumed wrong rather than the repo")
+
+SCAN_DIRS = ["apps", "platform", "runbooks"]
+routes: list[tuple[str, dict]] = []          # (relpath, HTTPRoute doc)
+dns_services: list[tuple[str, dict]] = []    # (relpath, Service doc carrying an external-dns ann)
+for sub in SCAN_DIRS:
+    base = ROOT / sub
+    if not base.is_dir():
+        continue
+    for p in sorted(base.rglob("*.yaml")):
+        rel = p.relative_to(ROOT).as_posix()
+        for d in docs_of(p):
+            kind = d.get("kind")
+            md = d.get("metadata") or {}
+            ann = md.get("annotations") or {}
+            if kind == "HTTPRoute":
+                routes.append((rel, d))
+            elif kind == "Service" and any(
+                a.startswith("external-dns.alpha.kubernetes.io/") for a in ann
+            ):
+                dns_services.append((rel, d))
+
+check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse "
+                       "contract broken (the tree is known to author several)")
+
+
+def ident(d: dict) -> tuple[str, str]:
+    md = d.get("metadata") or {}
+    return (md.get("namespace") or "", md.get("name") or "")
+
+
+def annotations_of(d: dict) -> dict:
+    return (d.get("metadata") or {}).get("annotations") or {}
+
+
+def is_disowned(d: dict) -> bool:
+    """True iff external-dns v0.21.0 would SKIP this object: the controller annotation is present
+    and its value is not external-dns's own. Absent == published (the default), which is the whole
+    point — the annotation is the only thing standing between a route and the public zone."""
+    v = annotations_of(d).get(CONTROLLER_KEY)
+    return v is not None and v != CONTROLLER_CLAIMED
+
+
+# =============================================================================================
+# (1) + (2) every internal service's rendered route is disowned; no public one is.
+# =============================================================================================
+by_ident: dict[tuple[str, str], list[tuple[str, dict]]] = {}
+for rel, d in routes:
+    by_ident.setdefault(ident(d), []).append((rel, d))
+
+for (ns, rname), svc in sorted(expected_routes.items()):
+    found = by_ident.get((ns, rname), [])
+    check(len(found) >= 1,
+          f"({svc}) the catalog declares an ingressRoute rendering HTTPRoute {ns}/{rname}, but no "
+          f"such HTTPRoute exists in the applied tree — apps/ is out of sync with catalog/ "
+          f"(ci/render-golden.sh owns the byte-level half of this)")
+    for rel, d in found:
+        if exposure_of[svc] == "internal":
+            v = annotations_of(d).get(CONTROLLER_KEY)
+            check(
+                v is not None,
+                f"({svc}) R-DNS-03 VIOLATED: {rel} HTTPRoute {ns}/{rname} belongs to an "
+                f"`exposure: internal` service but carries NO `{CONTROLLER_KEY}` annotation, so "
+                f"external-dns publishes {ns}.<zone> into the PUBLIC Cloudflare zone answering "
+                f"with the RFC1918 Gateway VIP. An internal service must render no public record.",
+            )
+            check(
+                v != CONTROLLER_CLAIMED,
+                f"({svc}) {rel} HTTPRoute {ns}/{rname} sets `{CONTROLLER_KEY}: "
+                f"{CONTROLLER_CLAIMED}` — that is external-dns's OWN value, which CLAIMS the route "
+                f"rather than releasing it. It reads like the guard and is its exact inverse.",
+            )
+        else:
+            check(
+                CONTROLLER_KEY not in annotations_of(d),
+                f"({svc}) {rel} HTTPRoute {ns}/{rname} belongs to an `exposure: "
+                f"{exposure_of[svc]}` service yet carries `{CONTROLLER_KEY}`. Under "
+                f"`policy: sync` that DELETES its live public record — the five public services "
+                f"must keep publishing.",
+            )
+
+# =============================================================================================
+# (3) the `service` source half — an internal service's Service may not publish either.
+# =============================================================================================
+for rel, d in dns_services:
+    ns, name = ident(d)
+    svc = ns if ns in sds else None
+    if svc and exposure_of[svc] == "internal":
+        check(
+            is_disowned(d),
+            f"({svc}) {rel} Service {ns}/{name} carries an external-dns annotation "
+            f"({sorted(a for a in annotations_of(d) if a.startswith('external-dns'))}) for an "
+            f"`exposure: internal` service without a disowning `{CONTROLLER_KEY}`. Both "
+            f"`gateway-httproute` AND `service` are configured sources, so closing only the route "
+            f"half is half a guarantee.",
+        )
+
+# =============================================================================================
+# (4) the GENERATOR derives it from `exposure` — not from a list of service names.
+# =============================================================================================
+render_py = ROOT / "catalog" / "render.py"
+src = read(render_py)
+check(bool(src), "catalog/render.py unreadable")
+
+emitter = re.search(
+    r"def _externaldns_route_annotations\(.*?\n(?=\n?def |\n?# =====)", src, re.S
+)
+check(emitter is not None,
+      "catalog/render.py must define `_externaldns_route_annotations` — the single place the "
+      "ExternalDNS scoping is derived. If it was renamed, rename it here too: this check has to "
+      "read the DERIVATION, not just the output, or a hardcoded per-service list would pass.")
+if emitter:
+    body = emitter.group(0)
+    # The derivation must read `exposure`. A body that does not mention it cannot be deriving
+    # anything from the operator's declaration.
+    check('"exposure"' in body or "'exposure'" in body,
+          "`_externaldns_route_annotations` must read the `exposure` field — the guarantee is "
+          "that the OPERATOR'S DECLARATION decides, so `exposure` has to appear in the decision")
+    # and it must not name any catalog service: a per-service list is how twelve ServiceMonitors
+    # acquired one identical wrong port name in a single edit. Only the executable statements are
+    # inspected — the docstring legitimately names the affected services as evidence.
+    code = re.sub(r'"""(?:.|\n)*?"""', "", body, count=1)
+    named = sorted(n for n in sds if re.search(rf"""['"]{re.escape(n)}['"]""", code))
+    check(not named,
+          f"`_externaldns_route_annotations` names catalog service(s) {named} in its CODE. The "
+          f"annotation must be derived from `exposure` alone; a per-service list silently omits "
+          f"the next service anyone adds.")
+    check(CONTROLLER_CLAIMED not in code,
+          f"`_externaldns_route_annotations` must never be able to emit `{CONTROLLER_CLAIMED}` — "
+          f"external-dns's own value CLAIMS the object instead of releasing it")
+
+# =============================================================================================
+# (5) THE SHIPPED RENDERER IS PROVEN on synthetic ServiceDefinitions. Three fixtures, because
+#     "internal is annotated" alone would also pass if the renderer annotated everything, and
+#     because the contract's DEFAULT posture is internal — the case a new service hits by
+#     omitting the field entirely, and the case a per-service list would miss.
+# =============================================================================================
+FIXTURES = [
+    ("fixture-internal", {"exposure": "internal"}, True,
+     "an `internal` fixture must render the disown annotation"),
+    ("fixture-public", {"exposure": "public"}, False,
+     "a `public` fixture must render NO disown annotation"),
+    ("fixture-default", {}, True,
+     "a fixture OMITTING `exposure` defaults to internal (renderer-contract default posture) and "
+     "must also render the disown annotation"),
+]
+with tempfile.TemporaryDirectory() as td:
+    for fname, extra, want, msg in FIXTURES:
+        sd = {
+            "name": fname,
+            "kind": "container",
+            "source": {"image": "example/app:1.0.0"},
+            "egress": "internet",
+            "dependencies": [],
+            "ingressRoutes": [{"host": f"{fname}.home.keithjackson.dev", "port": 8080}],
+            # `native` avoids the oauth2-proxy sidecar path and the schema's groups requirement,
+            # keeping the fixture about DNS scoping and nothing else. A public service may not use
+            # auth.mode none (schema invariant since 1.1.0), so both fixtures use the same mode.
+            "auth": {"mode": "native"},
+            "update": {"channel": "stable", "cadence": "0 4 * * 0",
+                       "autoRollbackOnFailedHealth": True},
+            "observability": {"metrics": True, "slis": ["up"]},
+            "backup": {"policy": "reacquirable"},
+        }
+        sd.update(extra)
+        f = Path(td) / f"{fname}.yaml"
+        f.write_text(yaml.safe_dump(sd, sort_keys=False))
+        proc = subprocess.run(
+            [sys.executable, str(render_py), str(f), "--no-image-automation"],
+            capture_output=True, text=True, cwd=str(ROOT),
+        )
+        if proc.returncode != 0:
+            check(False, f"fixture {fname}: the shipped renderer failed: "
+                         f"{proc.stderr.strip()[:400]}")
+            continue
+        rendered = [d for d in yaml.safe_load_all(proc.stdout)
+                    if isinstance(d, dict) and d.get("kind") == "HTTPRoute"]
+        check(len(rendered) == 1, f"fixture {fname}: expected exactly one rendered HTTPRoute")
+        if rendered:
+            got = is_disowned(rendered[0])
+            check(got is want, f"fixture {fname}: {msg} (rendered annotations: "
+                               f"{annotations_of(rendered[0])})")
+
+# =============================================================================================
+# (6) REGISTER RATCHET — every publisher the catalog does not render is declared, and every
+#     declaration still names a real object.
+# =============================================================================================
+catalog_rendered = set(expected_routes)
+seen_register_keys: set[str] = set()
+for rel, d in routes + dns_services:
+    ns, name = ident(d)
+    if (ns, name) in catalog_rendered:
+        continue
+    key = f"{rel}:{ns}/{name}"
+    seen_register_keys.add(key)
+    check(
+        key in KNOWN_NON_CATALOG_PUBLISHERS,
+        f"UNDECLARED publisher {key}: a {d.get('kind')} the catalog does not render, so it has no "
+        f"`exposure` field this check can adjudicate, and it can put a name into the public "
+        f"Cloudflare zone. Add it to KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why "
+        f"it is out of scope — or render it from a ServiceDefinition instead.",
+    )
+stale = sorted(set(KNOWN_NON_CATALOG_PUBLISHERS) - seen_register_keys)
+check(not stale,
+      f"STALE register entries {stale}: KNOWN_NON_CATALOG_PUBLISHERS is shrink-only, and these "
+      f"name no object in the tree any more. Delete them — a register that keeps entries for "
+      f"deleted objects stops being evidence of anything.")
+
+# =============================================================================================
+# (7) LOAD-BEARING — the scope is DECLARED here (so ci/l0.sh Stage 3b's --include-scope union
+#     always selects this test) and NO stage is hand-written into ci/l0.sh (the
+#     whole-repo-invariants-wired meta-test fails the gate if one is).
+# =============================================================================================
+TEST_DIR = ROOT / "conformance/tests/l0-static/internal-services-publish-no-public-dns"
+test_yaml = read(TEST_DIR / "chainsaw-test.yaml")
+check(bool(test_yaml), "this test's own chainsaw-test.yaml is unreadable")
+check(
+    re.search(r'^\s*conformance\.home\.k3s/scope:\s*"?whole-repo"?\s*$', test_yaml, re.M)
+    is not None,
+    "chainsaw-test.yaml must DECLARE `conformance.home.k3s/scope: \"whole-repo\"` on a real "
+    "mapping line — that declaration is the only thing that unions this check into every gate "
+    "run (ci/l0.sh Stage 3b). A scope named in prose is documentation, not a declaration.",
+)
+l0_sh = read(ROOT / "ci/l0.sh")
+check(
+    "internal-services-publish-no-public-dns" not in l0_sh,
+    "ci/l0.sh must NOT name this test: a whole-repo invariant is selected by its DECLARED scope, "
+    "and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.",
+)
+
+# =============================================================================================
+if FAILURES:
+    for f in FAILURES:
+        print(f"check FAIL: {f}", file=sys.stderr)
+    print(f"check FAILED: {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: {CHECKS} assertions passed "
+      f"({len(sds)} ServiceDefinitions; {len(internal_with_routes)} internal + "
+      f"{len(public_with_routes)} public services with routes; {len(routes)} HTTPRoutes and "
+      f"{len(dns_services)} external-dns-annotated Services inventoried; "
+      f"{len(KNOWN_NON_CATALOG_PUBLISHERS)} declared non-catalog publishers)")
+PY

conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/chainsaw-test.yaml

@@ -0,0 +1,95 @@
+# =============================================================================
+# L0 STATIC — an L2-announced LoadBalancer VIP must be servable from ANY announcing node
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE BUG CLASS THIS CLOSES, AND THE OUTAGE THAT PAID FOR IT.
+# Cilium puts a LoadBalancer VIP on the LAN by electing ONE node per Service to answer ARP for
+# it. That election is a leader lease over the nodes the CiliumL2AnnouncementPolicy's
+# nodeSelector matches, and it is NOT backend-aware — Cilium does not restrict the candidates
+# to nodes running a backend, and has no field that would. Combine that with
+# `externalTrafficPolicy: Local`, which lets a node deliver only to a backend on ITSELF, and
+# the VIP becomes a lottery: an elected announcer holding no backend answers ARP for the
+# address and then has nowhere to send the packet, so the client gets ICMP port unreachable.
+# On three nodes with one replica that is a 1-in-3 chance of working, drawn independently per
+# Service — and it never recovers on its own, because the lease holder keeps renewing after the
+# backend moves away.
+#
+# MEASURED on prod 2026-09-14: unifi-l4-tcp/-udp (VM on k3s-prod-1, both leases on k3s-prod-2)
+# left 192.168.2.21:8443/:8080/:11443 dead from the LAN for up to 8 days, and crafty-l4-tcp
+# (pod on k3s-prod-1, lease on k3s-prod-2) left the Minecraft Java port 192.168.2.22:25565
+# dead — while its twin crafty-l4-udp, same pod, happened to draw k3s-prod-1 and worked. One
+# Service broken and its identical twin working is the tell. The mechanism was then confirmed
+# in both directions on the announcing node with `cilium-dbg service list`: under `Local` the
+# frontend is ABSENT from that node's service table; under `Cluster` it appears with an active
+# backend; flipped back, it vanishes again.
+#
+# WHY A REVIEWER CANNOT CATCH THIS. `Local` reads as the safe, source-IP-preserving choice, and
+# both shipped manifests said so in a comment that also claimed it "pins the VIP to the node
+# running the pod" — the precise thing that is false here. The defect is invisible in review
+# and silent at apply time; it only appears as an outage, days later, after an unrelated
+# reschedule moves a pod. apps/wave-0/pihole-dns.yaml hit it first and is the reference
+# implementation; this check keeps the rest of the tree on that side of the line.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
+#       that would announce its VIP,
+# THEN  (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
+#           node is guaranteed to hold a backend — true only for a DaemonSet-backed Service,
+#           and that exemption is computed from the tree rather than hand-maintained;
+#       (2) the premise is still live (some policy actually announces LoadBalancer VIPs), so
+#           the check cannot pass vacuously if L2 announcement is ever removed;
+#       (3) the announcer is still backend-BLIND — asserted structurally, so that if a future
+#           Cilium gains a backend-aware field the exemption logic is revisited deliberately
+#           instead of going stale in silence;
+#   AND (4) the check is NON-VACUOUS: a real checkout, a non-empty LoadBalancer corpus, and a
+#           floor on the assertion count.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
+# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
+# SAME script.
+#
+# NOT proven — the runtime half, which needs a cluster: that the VIP is actually reachable,
+# that a backend exists, or that an application is listening behind it. That gap is not
+# academic. On 2026-09-14 all four Services above ALSO had no listening backend at all (UniFi
+# OS had never been installed in the VM; no Minecraft server had ever been created in Crafty),
+# so fixing this configuration defect did NOT by itself make those ports answer. A static
+# check cannot see an absent application; only an L2 behavioural probe can. Nor does this
+# prove `Cluster` is costless — Cilium runs in SNAT mode here, so a cross-node connection
+# reaches the backend bearing the ingress node's address, and whether that is acceptable is a
+# per-service judgement recorded in each manifest.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: l2-announced-vips-serve-from-any-node
+  labels:
+    conformance.home.k3s/R-DNS-01: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-DNS-01"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE, and necessarily so. ci/touched-requirements.sh's path map has NO arm for
+    # `apps/*` and none for `platform/crafty/*`, and nothing in it maps to R-DNS-01 at all. So the
+    # change most likely to break this invariant — authoring or editing an L4 LoadBalancer Service
+    # under apps/ or platform/, exactly where both halves of the 2026-09-14 outage lived — would
+    # attribute NO requirement and be gated by nothing. The invariant also spans three trees at
+    # once (the Services under apps/ and platform/, and the announcement policy under
+    # platform-config/), which no single requirement tag can select.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      No LoadBalancer Service whose VIP is put on the LAN by Cilium L2 announcement may use
+      externalTrafficPolicy Local, because the announcing node is elected without regard to
+      backend placement — so an announcer holding no backend blackholes the VIP and never
+      self-heals. The sole exemption is a DaemonSet-backed Service, where every announcing
+      node is guaranteed a local backend.
+spec:
+  steps:
+    - name: assert-no-l2-announced-vip-is-etp-local
+      description: >-
+        Pair every tracked type=LoadBalancer Service with the CiliumL2AnnouncementPolicy that
+        would announce its VIP, and assert none combines that announcement with
+        externalTrafficPolicy Local unless a DaemonSet guarantees a backend on every node.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/check.sh

@@ -0,0 +1,242 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — l2-announced-vips-serve-from-any-node
+# Serves: R-DNS-01 ("Services MUST be reachable at `<service>.home.keithjackson.dev`."
+#         — spec docs/00-vision-and-scope.md, section "DNS & ingress").
+#
+# THE BUG CLASS THIS CLOSES — it is not hypothetical, it cost an 8-day outage.
+# -----------------------------------------------------------------------------
+# A type=LoadBalancer Service on this cluster gets its VIP from Cilium LB-IPAM and is put on
+# the LAN by a CiliumL2AnnouncementPolicy, which elects ONE node per Service to answer ARP for
+# that VIP. The election is a plain leader lease among the nodes the policy's nodeSelector
+# matches. IT IS NOT BACKEND-AWARE: Cilium does not restrict the candidate set to nodes that
+# actually run a backend, and it has no field that would let it.
+#
+# With `externalTrafficPolicy: Local`, a node may only deliver to a backend on ITSELF. Compose
+# the two and the VIP is a lottery: if the elected announcer holds no backend, it answers ARP
+# for the address and then has nowhere to send the packet, so the client gets ICMP port
+# unreachable. On three nodes with one replica that is a 1-in-3 chance of working, drawn
+# independently PER SERVICE — and it does not self-heal, because the lease holder keeps
+# renewing after the backend moves away. Deleting the Lease only redraws the lottery.
+#
+# MEASURED on prod 2026-09-14, which is why this check exists:
+#   * unifi-l4-tcp / unifi-l4-udp — VM on k3s-prod-1, both leases on k3s-prod-2. 192.168.2.21
+#     :8443, :8080 and :11443 dead from the LAN for up to 8 days (the leases were 8 days old).
+#   * crafty-l4-tcp — pod on k3s-prod-1, lease on k3s-prod-2: 192.168.2.22:25565 (Minecraft
+#     Java) dead. crafty-l4-udp — same pod, lease happened to land on k3s-prod-1, so Bedrock
+#     worked. One Service broken and its twin working, from the same pod, is the whole tell.
+#   * The causal mechanism was confirmed both directions on the announcing node with
+#     `cilium-dbg service list`: under `Local` the frontend 192.168.2.22:25565 is ABSENT from
+#     that node's service table entirely; flipped to `Cluster` it appears as
+#     `1 => 10.0.0.11:25565 (active)`. Flipped back, it vanishes again.
+#
+# `Cluster` is the fix: every node can serve the VIP by forwarding to wherever the backend is,
+# so the announcement and the backend no longer have to coincide. apps/wave-0/pihole-dns.yaml
+# is the reference implementation (and carries the cost analysis); this check keeps the whole
+# tree on that side of the line, because the defect is INVISIBLE in review — `Local` reads as
+# the safe, source-IP-preserving choice, and the manifests that shipped it said so explicitly.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
+#       that would announce its VIP,
+# THEN  (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
+#           node is guaranteed to hold a backend — which is true only for a DaemonSet-backed
+#           Service, and that exemption is computed from the tree, not hand-maintained;
+#       (2) the premise is still live: at least one CiliumL2AnnouncementPolicy actually
+#           announces LoadBalancer VIPs (`loadBalancerIPs: true`). If that ever becomes false
+#           this check is vacuous and must be re-derived rather than silently passing;
+#       (3) the announcer's candidate set really is backend-blind — asserted structurally, by
+#           requiring that the policy carries no field pinning it to backend-holding nodes;
+#   AND (4) the check is NON-VACUOUS: a real checkout is present, the LoadBalancer corpus is
+#           non-empty, and a floor on both the corpus and the assertion count keeps a pass
+#           from degenerating into a no-op when a refactor moves these files.
+#
+# WHY THE EXEMPTION IS NARROW, AND WHY IT IS SAFE. `Local` is correct precisely when the
+# announcing node always has a local backend. Under `default-l2`'s nodeSelector
+# (kubernetes.io/os=linux, i.e. every node) that holds only for a DaemonSet. A Deployment with
+# replicas == node count does NOT qualify: nothing pins one replica per node, a rollout or a
+# cordon breaks it, and the failure mode is this same silent outage. So the exemption keys on
+# DaemonSet backing only.
+#
+# WHAT THIS DOES NOT PROVE (honest gaps):
+#   - That the VIP is actually reachable. This is a static check; it proves the configuration
+#     cannot produce the backend-blind-announcer failure, not that a backend exists, that an
+#     application is listening behind it, or that the LAN can route to it. On 2026-09-14 all
+#     four Services above ALSO had no listening backend (UniFi OS was never installed in the
+#     VM; no Minecraft server had been created in Crafty), and no static check can see that.
+#     The end-to-end claim belongs to an L2 behavioural test against a live cluster.
+#   - That `Cluster` is COSTLESS. It is not: Cilium runs in SNAT mode here, so a cross-node
+#     connection reaches the backend bearing the ingress node's address. Whether that cost is
+#     acceptable is a per-service judgement recorded in each manifest, not something this
+#     check can decide.
+#   - Anything about Services created in-cluster rather than authored here (the Gateway's own
+#     LoadBalancer Service is generated by Cilium from the Gateway resource and is not in the
+#     tree, so it is outside this static scan).
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+import re, subprocess, sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def tracked() -> list[str]:
+    out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
+                         capture_output=True, text=True, check=True).stdout
+    return [p for p in out.split("\0") if p]
+
+
+def docs_of(rel: str):
+    try:
+        return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
+                if isinstance(d, dict)]
+    except Exception:
+        return []
+
+
+FILES = tracked()
+YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
+check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
+
+# --------------------------------------------------------------------------- the announcers
+# A CiliumL2AnnouncementPolicy with loadBalancerIPs: true puts LoadBalancer VIPs on the LAN.
+# `serviceSelector` absent means EVERY LoadBalancer Service is announced (Cilium's default).
+announcers: list[dict] = []
+for rel in YAML_FILES:
+    for d in docs_of(rel):
+        if d.get("kind") == "CiliumL2AnnouncementPolicy":
+            spec = d.get("spec") or {}
+            announcers.append({
+                "file": rel,
+                "name": (d.get("metadata") or {}).get("name"),
+                "lb_ips": bool(spec.get("loadBalancerIPs")),
+                "service_selector": spec.get("serviceSelector"),
+                "spec": spec,
+            })
+
+check(len(announcers) > 0,
+      "no CiliumL2AnnouncementPolicy is tracked — this check's premise (VIPs are put on the "
+      "LAN by L2 announcement) no longer holds and the invariant must be re-derived (R-DNS-01)")
+
+lb_announcers = [a for a in announcers if a["lb_ips"]]
+# (2) the premise is live.
+check(len(lb_announcers) > 0,
+      "no CiliumL2AnnouncementPolicy sets loadBalancerIPs: true — nothing announces LoadBalancer "
+      "VIPs, so this check would pass vacuously; re-derive it (R-DNS-01)")
+
+# (3) the candidate set is backend-blind. Cilium has no field that restricts L2 announcement to
+#     nodes holding a backend; assert none has appeared, so that if a future Cilium adds one the
+#     exemption logic here is revisited deliberately instead of silently going stale.
+BACKEND_AWARE_FIELDS = {"backendSelector", "endpointSelector", "localBackendsOnly"}
+for a in lb_announcers:
+    present = BACKEND_AWARE_FIELDS & set(a["spec"].keys())
+    check(not present,
+          f"{a['file']}: CiliumL2AnnouncementPolicy/{a['name']} carries {sorted(present)}, which "
+          "may make announcement backend-aware — the `Local` exemption below must be re-derived "
+          "before this check can keep passing (R-DNS-01)")
+
+# --------------------------------------------------------------------------- DaemonSet backing
+# Service.spec.selector is matched against DaemonSet pod-template labels in the same namespace.
+# A DaemonSet puts a backend on every node, so `Local` cannot strand an announcing node.
+daemonset_labels: list[tuple[str, dict]] = []   # (namespace, template labels)
+for rel in YAML_FILES:
+    for d in docs_of(rel):
+        if d.get("kind") == "DaemonSet":
+            md = d.get("metadata") or {}
+            tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {})
+            daemonset_labels.append((md.get("namespace"), tmpl.get("labels") or {}))
+
+
+def daemonset_backed(ns, selector) -> bool:
+    if not selector:
+        return False
+    for ds_ns, labels in daemonset_labels:
+        if ds_ns == ns and all(labels.get(k) == v for k, v in selector.items()):
+            return True
+    return False
+
+
+# --------------------------------------------------------------------------- (1) the invariant
+lb_services: list[dict] = []
+for rel in YAML_FILES:
+    for d in docs_of(rel):
+        if d.get("kind") != "Service":
+            continue
+        spec = d.get("spec") or {}
+        if spec.get("type") != "LoadBalancer":
+            continue
+        md = d.get("metadata") or {}
+        lb_services.append({
+            "file": rel,
+            "name": md.get("name"),
+            "ns": md.get("namespace"),
+            "etp": spec.get("externalTrafficPolicy"),
+            "selector": spec.get("selector") or {},
+            "pinned": (md.get("annotations") or {}).get("lbipam.cilium.io/ips"),
+        })
+
+# Non-vacuity floor: these Services are the reason the check exists; if the corpus empties out,
+# something moved and the check must be re-pointed rather than passing on nothing.
+check(len(lb_services) >= 3,
+      f"only {len(lb_services)} tracked type=LoadBalancer Service(s) found — the corpus this "
+      "check quantifies over has moved or emptied; re-point it (R-DNS-01)")
+
+exempt: list[str] = []
+for s in lb_services:
+    where = f"{s['file']}: Service/{s['ns']}/{s['name']}"
+    if s["etp"] != "Local":
+        CHECKS += 1                      # counted: every Service is genuinely inspected
+        continue
+    if daemonset_backed(s["ns"], s["selector"]):
+        exempt.append(f"{s['ns']}/{s['name']}")
+        CHECKS += 1
+        continue
+    pin = f" pinned at {s['pinned']}" if s["pinned"] else ""
+    check(False,
+          f"{where}{pin} combines type=LoadBalancer with externalTrafficPolicy: Local while "
+          f"CiliumL2AnnouncementPolicy/{lb_announcers[0]['name'] if lb_announcers else '?'} "
+          "announces its VIP from nodes chosen WITHOUT regard to backend placement. When the "
+          "elected announcer holds no backend the VIP answers ICMP port unreachable and does "
+          "not self-heal — this is the 8-day unifi/crafty outage of 2026-09-14. Use "
+          "externalTrafficPolicy: Cluster (see apps/wave-0/pihole-dns.yaml for the pattern and "
+          "the client-IP cost it trades away), or back the Service with a DaemonSet so every "
+          "announcing node holds a backend (R-DNS-01)")
+
+# --------------------------------------------------------------------------- (4) non-vacuity
+check(CHECKS >= 8,
+      f"only {CHECKS} assertions ran — too few for this invariant to be meaningful (R-DNS-01)")
+
+# --------------------------------------------------------------------------------- verdict
+if FAILURES:
+    print(f"CHECK FAIL (l2-announced-vips-serve-from-any-node): {len(FAILURES)} of {CHECKS} "
+          "assertions failed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+note = f", {len(exempt)} DaemonSet-backed exemption(s): {sorted(exempt)}" if exempt else ""
+print(f"check OK: l2-announced-vips-serve-from-any-node — {CHECKS} assertions over "
+      f"{len(lb_services)} LoadBalancer Service(s) and {len(lb_announcers)} announcing "
+      f"policy(ies){note}")
+PY

conformance/tests/l0-static/placeholder-guard-coupling/check.sh

@@ -238,7 +238,7 @@ def bail(msg):
 
 DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
 SELF_DIR = ROOT / "conformance" / "tests" / "l0-static" / "placeholder-guard-coupling"
-SKIP_DIRS = {".git", "_artifacts", "node_modules", ".venv"}
+SKIP_DIRS = {".git", ".claude", "_artifacts", "node_modules", ".venv"}
 
 # ------------------------------------------------------------------ discover the guard FILE
 # Hardcoding a path is what let a file that never landed disable this whole check.

conformance/tests/l0-static/servicemonitor-ports-resolve/chainsaw-test.yaml

@@ -0,0 +1,105 @@
+# =============================================================================
+# L0 STATIC — every ServiceMonitor endpoint names a scrape target that RESOLVES
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES, AND WHY ITS SIBLING COULD NOT.
+# alert-metrics-have-scrape-targets audits scrape wiring FROM THE ALERT SIDE: it walks alert rules
+# and asks whether something scrapes the metric each one reads. Its assertion (4) is the `up{job=X}`
+# axis, and it is reachable ONLY through an alert that selects that job, only where the Service and
+# the ServiceMonitor sit in the SAME FILE. Its own header names the hole. This test is the other
+# direction: it walks the MONITORS, and asks of each one whether the target it names can exist —
+# with no alert rule required anywhere.
+#
+# That difference is not academic. On 2026-09-13 all twelve apps/* ServiceMonitors named
+# `endpoints[].port: metrics` and NO app Service defined a port of that name, so each discovered
+# ZERO targets (verified on prod: `count by (job) (up)` listed 24 jobs, none of the twelve; both
+# `up{job="plex"}` and `up{job="gitea"}` returned nothing). That was caught only because each app
+# also shipped an `up{job=...}` alert. THE FIX TO THOSE ALERTS DELETED THE ONLY THING THAT MADE THE
+# BROKEN MONITORS VISIBLE — they no longer read `up{job=...}` at all — so a monitor that discovers
+# nothing would from that moment have been invisible again, and the twelve register entries that
+# recorded the gap correctly became stale and were removed. A monitor scraping nothing is a defect
+# on its own terms: it is dead configuration that reads as live coverage, and every dashboard,
+# recording rule or future alert built on its series inherits the emptiness.
+#
+# GIVEN every ServiceMonitor and PodMonitor this repo authors,
+# WHEN  each endpoint's scrape target is resolved against every Service this repo authors,
+# THEN  (1) every endpoint names a target AT ALL (`port` or `targetPort`) — an endpoint naming
+#           neither scrapes nothing and is accepted silently by the operator;
+#       (2) every endpoint that names a `port` names a port that at least one Service its selector
+#           matches ACTUALLY DEFINES — the exact disagreement that made twelve monitors inert.
+#           Resolution is by SELECTOR ACROSS THE WHOLE TREE, not by same-file adjacency, so moving
+#           a Service into its own file cannot defeat it;
+#       (3) a monitor whose selector matches no Service in this repo is UNAUDITABLE (its Service is
+#           created by a Helm chart or an operator, so the port name lives outside the tree) and
+#           must be DECLARED in KNOWN_EXTERNAL_MONITORS, naming who creates the Service. That
+#           register is shrink-only: a monitor that later gains an in-repo Service, or that is
+#           deleted, FAILS as stale, so the unauditable set cannot quietly grow;
+#       (4) THE RESOLVER ITSELF IS PROVEN, on synthetic fixtures inside the check — one agreeing
+#           pair it must accept, one disagreeing pair it must reject. This is not decoration: after
+#           the fix there are zero in-repo monitors left for (2) to judge, and a guard whose only
+#           evidence is "nothing failed" is indistinguishable from a guard that stopped working.
+#           The fixtures are the exact shape catalog/render.py emits from
+#           `observability.metricsEndpoint`;
+#       (5) the GENERATOR cannot reintroduce the class: catalog/render.py must not hardcode an
+#           endpoint port name — it must take it from the ServiceDefinition, the same declaration
+#           that renders the Service port. A hardcoded literal is how twelve services acquired an
+#           identical wrong port name in one edit;
+#   AND (6) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
+#           so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
+#           ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
+#           ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
+#
+# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh maps apps/* to NOTHING, so a change that
+# adds a ServiceMonitor under apps/ attributes no requirement and would filter this check out of
+# exactly the change that breaks it. Same reasoning as its sibling.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
+# the sibling check use); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
+# Chainsaw Test wraps the SAME script.
+#
+# NOT proven (honest gaps; this is static):
+#   - That a resolved target is UP, answers, or emits the series an alert wants. The sibling check
+#     owns the metric-name axis and admits it cannot prove the SERIES NAME either; both are live
+#     reads and belong to a future L1/L3 check against /api/v1/label/__name__/values.
+#   - That a NetworkPolicy permits the scrape. The default-deny baseline drops it unless an allow is
+#     rendered, which is now part of what `observability.metricsEndpoint` emits
+#     (r_monitoring_ingress_policy) — but this check does not verify the allow, only the ports.
+#   - For a monitor in KNOWN_EXTERNAL_MONITORS, nothing about the chart's port names. The register
+#     records WHO owns them so the claim is reviewable, which is all a static check can offer.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: servicemonitor-ports-resolve
+  labels:
+    conformance.home.k3s/R-OBS-01: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-OBS-01"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — this invariant quantifies over every ServiceMonitor/PodMonitor and every
+    # Service in the tree (apps/wave-*, apps/vms, platform/keycloak, platform/kubevirt,
+    # platform/oauth2-proxy, platform/monitoring, platform/crafty, platform/homepage,
+    # runbooks/serving). No single requirement path-maps to that set, and ci/touched-requirements.sh
+    # maps apps/* to no requirement at all — so a requirement-scoped selection would skip this
+    # check on precisely the changes that can break it.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every ServiceMonitor/PodMonitor endpoint this repo authors names a scrape target that
+      resolves — a port some matching Service actually defines — the monitors whose Services come
+      from outside the tree are enumerated in a shrink-only register, and the resolver that decides
+      this is itself proven on fixtures, so a monitor can never again be authored, reviewed and
+      merged against a port nothing serves.
+spec:
+  steps:
+    - name: resolve-every-monitor-endpoint-against-every-service-this-repo-authors
+      description: >-
+        Inventory every ServiceMonitor/PodMonitor and Service in the tree, resolve each endpoint's
+        port against the Services its selector matches, ratchet the register of monitors whose
+        Service is chart-owned, prove the resolver on synthetic agreeing/disagreeing fixtures, and
+        assert the renderer does not hardcode an endpoint port name.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh

@@ -0,0 +1,421 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — every ServiceMonitor/PodMonitor endpoint names a scrape target that RESOLVES.
+# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
+#
+# alert-metrics-have-scrape-targets audits scrape wiring from the ALERT side, and its `up{job=X}`
+# axis is reachable only through an alert that selects that job, only for a Service and a
+# ServiceMonitor in the SAME FILE. That is how it caught the twelve inert apps/* monitors on
+# 2026-09-13 (`endpoints[].port: metrics`, and no app Service defined such a port; on prod
+# `count by (job) (up)` listed 24 jobs and none of the twelve appeared). But FIXING those alerts
+# removed their `up{job=...}` selectors, which removed the only path by which the broken monitors
+# were visible at all. This check walks the MONITORS instead, needs no alert rule, and resolves by
+# SELECTOR across the whole tree — so a monitor that discovers nothing fails on its own terms.
+#
+# ASSERTIONS
+#   (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
+#   (1) every endpoint names a scrape target at all — `port` or `targetPort`;
+#   (2) every endpoint naming a `port` names a port at least one Service its selector matches
+#       actually DEFINES;
+#   (3) a monitor whose selector matches no in-repo Service is unauditable (chart/operator-owned
+#       Service) and must be declared in KNOWN_EXTERNAL_MONITORS — shrink-only, stale entries FAIL;
+#   (4) the RESOLVER is proven on synthetic fixtures (one agreeing pair, one disagreeing pair),
+#       because after the fix there are zero in-repo monitors for (2) to judge and a guard whose
+#       only evidence is "nothing failed" cannot be told apart from one that stopped working;
+#   (5) catalog/render.py does not HARDCODE an endpoint port name — it derives it from the
+#       ServiceDefinition, the same declaration that renders the Service port. A hardcoded literal
+#       is how twelve services acquired one identical wrong port name in a single edit;
+#   (6) the check is LOAD-BEARING: whole-repo scope declared here, honoured by ci/l0.sh's
+#       --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written ci/l0.sh
+#       stage (whole-repo-invariants-wired fails the gate if one appears).
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC):
+#   - Nothing about liveness: that a resolved target is UP, answers, or emits the series a rule
+#     wants. The sibling check owns the metric-name axis and admits it cannot prove the series NAME
+#     either. Both need a live read of /api/v1/label/__name__/values (a future L1/L3 check).
+#   - Nothing about NetworkPolicy. An app namespace's default-deny baseline is `ingress: []`, so a
+#     scrape is dropped unless an allow is rendered; that allow is now part of what
+#     `observability.metricsEndpoint` emits (catalog/render.py r_monitoring_ingress_policy), but this
+#     check verifies ports, not policy.
+#   - Nothing about a chart's port names, for a monitor in KNOWN_EXTERNAL_MONITORS. The register
+#     records WHO owns them so the claim is reviewable; that is all a static check can offer.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns into
+# rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if the repo
+# contains no ServiceMonitor or PodMonitor at all — genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
+if ! grep -rlE '^[[:space:]]*kind:[[:space:]]*(Service|Pod)Monitor[[:space:]]*$' \
+       "${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then
+  echo "check SKIPPED: no ServiceMonitor/PodMonitor found under platform/ or apps/ — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: every ServiceMonitor/PodMonitor endpoint's scrape target resolves.
+
+PyYAML (the same dependency as catalog/render.py and the sibling check) rather than line-based
+parsing: a monitor's selector, namespaceSelector and endpoint list are nested structures, and a
+line-based reader of them is precisely the approximate parser that silently stops matching — the
+failure class this check exists to prevent.
+"""
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+# =============================================================================================
+# REGISTER — KNOWN_EXTERNAL_MONITORS. Shrink-only.
+#
+# A monitor whose selector matches NO Service in this repo cannot be audited here: the Service is
+# created by a Helm chart or an operator, so the port name lives outside the tree. That is a real
+# limit, not a pass — so each one is DECLARED, with WHO creates the Service it selects, and the
+# register can only shrink. If such a monitor later gains an in-repo Service its entry goes stale
+# and FAILS; if it is deleted, likewise. The same ratchet reasoning as the two registers in
+# alert-metrics-have-scrape-targets: failing outright would make the gate red for chart-owned
+# monitors that are perfectly correct, and a gate that is red for correct code gets disabled.
+#
+# Keyed by "<relpath>:<monitor name>".
+# =============================================================================================
+KNOWN_EXTERNAL_MONITORS: dict[str, str] = {
+    "platform/keycloak/servicemonitor.yaml:keycloak":
+        "selects the operator-created `keycloak-service` (labels app=keycloak) — the Keycloak "
+        "operator owns that Service. Scrapes by targetPort 9000 (the Quarkus management "
+        "interface), so it names no Service port at all and assertion (2) has nothing to resolve.",
+    "platform/kubevirt/servicemonitor.yaml:kubevirt":
+        "selects the KubeVirt operator's own `kubevirt-prometheus-metrics` Service (label "
+        "prometheus.kubevirt.io=true), created by the KubeVirt CR, not by this repo. Its `metrics` "
+        "port is defined there. Live on prod: job kubevirt-prometheus-metrics, 11 targets up.",
+    "platform/oauth2-proxy/servicemonitor.yaml:oauth2-proxy":
+        "selects the oauth2-proxy Helm chart's Service (label app=oauth2-proxy); the chart defines "
+        "its `metrics` port. Live on prod: job oauth2-proxy, 2 targets up.",
+    "platform/monitoring/flux-podmonitor.yaml:flux-system":
+        "a PodMonitor, so it resolves against POD ports, not Service ports — the Flux controller "
+        "Deployments (and their `http-prom` container port) are installed by the Flux bootstrap "
+        "manifests, not authored here. Live on prod: job flux-system/flux-system, 6 targets up.",
+}
+
+MONITOR_KINDS = ("ServiceMonitor", "PodMonitor")
+
+
+def is_test_fixture(p: Path) -> bool:
+    """conformance/tests/**/{golden,fixtures}/** is assertion DATA, not deployed manifests.
+
+    The l1-rendering goldens are expected-output copies of rendered apps; the l2/l3 fixtures are
+    throwaway objects a behavioural test applies itself. Auditing either would be auditing a copy —
+    the same "reference, not declaration" distinction ci/touched-requirements.sh draws for fixture
+    paths, and the same exclusion the sibling check makes for goldens. The templates they come from
+    ARE audited, through apps/ and catalog/.
+    """
+    parts = p.parts
+    return "conformance" in parts and ("golden" in parts or "fixtures" in parts)
+
+
+# ============================================================ (0) INVENTORY
+# services: list of (relpath, namespace, name, labels, {named ports})
+# monitors: list of (relpath, kind, namespace, name, selector, namespaceSelector, [endpoints])
+services: list[tuple] = []
+monitors: list[tuple] = []
+unparseable: list[str] = []
+
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
+for p in sorted(ROOT.rglob("*.yaml")):
+    if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) or is_test_fixture(p) or ".sops." in p.name:
+        continue
+    txt = read(p)
+    if not txt:
+        continue
+    if not re.search(r"^\s*kind:\s*(Service|ServiceMonitor|PodMonitor)\s*$", txt, re.M):
+        continue
+    rel = str(p.relative_to(ROOT))
+    try:
+        docs = [d for d in yaml.safe_load_all(txt) if isinstance(d, dict)]
+    except yaml.YAMLError as exc:
+        unparseable.append(f"{rel} ({exc})")
+        continue
+    for d in docs:
+        kind = d.get("kind")
+        md = d.get("metadata") or {}
+        spec = d.get("spec") or {}
+        if kind == "Service":
+            names = {str(port["name"]) for port in (spec.get("ports") or [])
+                     if isinstance(port, dict) and port.get("name")}
+            services.append((rel, md.get("namespace"), md.get("name"),
+                             dict(md.get("labels") or {}), names))
+        elif kind in MONITOR_KINDS:
+            eps = spec.get("endpoints") or spec.get("podMetricsEndpoints") or []
+            monitors.append((rel, kind, md.get("namespace"), md.get("name"),
+                             dict((spec.get("selector") or {}).get("matchLabels") or {}),
+                             dict(spec.get("namespaceSelector") or {}),
+                             [e for e in eps if isinstance(e, dict)]))
+
+for u in unparseable:
+    check(False, f"{u} is not parseable YAML — every monitor and Service in it went unaudited")
+
+check(len(monitors) > 0,
+      "no ServiceMonitor/PodMonitor parsed out of the tree, yet the grep gate above found one. The "
+      "parse contract broke and this check would pass vacuously while every scrape target went "
+      "unaudited")
+check(len(services) >= 10,
+      f"only {len(services)} Service(s) parsed from the tree. This repo authors more than that "
+      f"(every apps/wave-* ServiceDefinition renders one), so the inventory has stopped being "
+      f"populated — assertion (2) would then resolve nothing and pass for the wrong reason")
+
+
+# ============================================================ THE RESOLVER
+def monitor_namespaces(mon_ns, ns_selector) -> set | None:
+    """Namespaces a monitor selects. None means ANY (`any: true`).
+
+    An empty namespaceSelector defaults to the monitor's own namespace — the operator's documented
+    behaviour, and the case every repo-authored monitor uses.
+    """
+    if ns_selector.get("any") is True:
+        return None
+    names = ns_selector.get("matchNames")
+    if names:
+        return {str(n) for n in names}
+    return {mon_ns} if mon_ns else set()
+
+
+def matching_services(selector: dict, namespaces: set | None) -> list[tuple]:
+    """Services whose labels are a SUPERSET of the selector and whose namespace is in range.
+
+    An EMPTY selector matches every Service in range (Kubernetes label-selector semantics). A
+    PodMonitor selector is compared against Service labels too — deliberately, so the resolver has
+    one code path; when that yields no match the monitor lands in the unauditable register with the
+    reason recorded, which is the honest outcome for a pod-side target this repo does not author.
+    """
+    out = []
+    for svc in services:
+        _, svc_ns, _, labels, _ = svc
+        if namespaces is not None and svc_ns not in namespaces:
+            continue
+        if all(labels.get(k) == v for k, v in selector.items()):
+            out.append(svc)
+    return out
+
+
+def resolve(mon: tuple) -> tuple[list[str], list[tuple]]:
+    """(failures, matched services) for one monitor. The unit assertions (1) and (2) live here so
+    the SELF-TEST at the bottom exercises the same code path the repo audit does."""
+    rel, kind, ns, name, selector, ns_sel, eps = mon
+    site = f"{rel}:{name}"
+    problems: list[str] = []
+    matched = matching_services(selector, monitor_namespaces(ns, ns_sel))
+
+    for i, ep in enumerate(eps):
+        if not ep.get("port") and not ep.get("targetPort"):
+            problems.append(
+                f"{site} endpoint[{i}] names NEITHER `port` NOR `targetPort`. The operator accepts "
+                f"that silently and the monitor scrapes nothing")
+    if not eps:
+        problems.append(
+            f"{site} has NO endpoints at all — it can never produce a target, and nothing about "
+            f"the object looks wrong")
+
+    named = [(i, str(ep["port"])) for i, ep in enumerate(eps) if ep.get("port")]
+    if named and matched:
+        defined: set = set()
+        for svc in matched:
+            defined |= svc[4]
+        for i, port in named:
+            if port not in defined:
+                where = ", ".join(f"{s[0]}:{s[2]}" for s in matched)
+                problems.append(
+                    f"{site} endpoint[{i}] scrapes port {port!r}, which NO Service its selector "
+                    f"matches defines. Matched Service(s): {where}; the port names they define are "
+                    f"{sorted(defined) or ['(none)']}. The monitor therefore discovers ZERO "
+                    f"targets: no `up{{job=\"{name}\"}}` series exists, anything built on its "
+                    f"series is empty, and PrometheusTargetsMissing cannot see it either because a "
+                    f"target that was never discovered has no up series to be 0. Give the Service a "
+                    f"named port the workload actually serves metrics on (for a ServiceDefinition: "
+                    f"`observability.metricsEndpoint`, which renders the Service port, this "
+                    f"endpoint and the monitoring ingress allow together), or scrape by targetPort")
+    return problems, matched
+
+
+# ============================================================ (1)(2)(3) THE REPO AUDIT
+auditable = 0
+for mon in monitors:
+    rel, kind, ns, name, selector, ns_sel, eps = mon
+    site = f"{rel}:{name}"
+    problems, matched = resolve(mon)
+    for msg in problems:
+        check(False, msg)
+    if not problems:
+        # An endpoint-shape assertion passed for this monitor; count it so the report is honest
+        # about how much of the run was real work.
+        check(True, f"{site}: every endpoint names a scrape target")
+    if matched:
+        auditable += 1
+        check(site not in KNOWN_EXTERNAL_MONITORS,
+              f"{site} is listed in KNOWN_EXTERNAL_MONITORS but its selector NOW matches "
+              f"{len(matched)} Service(s) this repo authors — the register is shrink-only: delete "
+              f"the entry so the monitor is audited for real")
+    else:
+        check(site in KNOWN_EXTERNAL_MONITORS,
+              f"{site} selects {selector or '(everything)'} in namespace(s) "
+              f"{sorted(monitor_namespaces(ns, ns_sel) or ['<any>'])} and NO Service in this repo "
+              f"matches, so its endpoint ports cannot be audited here. Either the selector is wrong "
+              f"(a monitor matching nothing discovers nothing), or the Service is chart/operator "
+              f"owned — in which case DECLARE it in KNOWN_EXTERNAL_MONITORS with who creates it")
+
+for site in sorted(KNOWN_EXTERNAL_MONITORS):
+    check(any(f"{m[0]}:{m[3]}" == site for m in monitors),
+          f"KNOWN_EXTERNAL_MONITORS declares {site}, which is not a monitor in this repo any more "
+          f"— a stale exemption hides the next real gap; delete the entry")
+
+
+# ============================================================ (4) THE RESOLVER IS PROVEN
+# After the twelve apps/* monitors were removed there is NO in-repo monitor left for assertion (2)
+# to judge (every survivor is chart-owned, and keycloak's scrapes by targetPort). A guard in that
+# state passes whether it works or not. So the resolver is run against two synthetic monitors over
+# two synthetic Services, in the exact shape catalog/render.py emits from
+# `observability.metricsEndpoint`, and BOTH outcomes are asserted.
+_real_services = services
+services = [
+    ("<selftest>", "fixture", "agreeing", {"app.kubernetes.io/name": "agreeing"},
+     {"port-8080", "metrics"}),
+    ("<selftest>", "fixture", "disagreeing", {"app.kubernetes.io/name": "disagreeing"},
+     {"port-8080"}),
+]
+good = ("<selftest>", "ServiceMonitor", "fixture", "agreeing",
+        {"app.kubernetes.io/name": "agreeing"}, {"matchNames": ["fixture"]},
+        [{"port": "metrics", "path": "/metrics", "interval": "30s"}])
+bad = ("<selftest>", "ServiceMonitor", "fixture", "disagreeing",
+       {"app.kubernetes.io/name": "disagreeing"}, {"matchNames": ["fixture"]},
+       [{"port": "metrics", "path": "/metrics", "interval": "30s"}])
+empty = ("<selftest>", "ServiceMonitor", "fixture", "agreeing",
+         {"app.kubernetes.io/name": "agreeing"}, {"matchNames": ["fixture"]},
+         [{"path": "/metrics"}])
+
+good_problems, good_matched = resolve(good)
+bad_problems, bad_matched = resolve(bad)
+empty_problems, _ = resolve(empty)
+services = _real_services
+
+check(good_matched and not good_problems,
+      f"SELF-TEST: the resolver rejected a CORRECT monitor (Service defines `metrics`, endpoint "
+      f"scrapes `metrics`): {good_problems or 'selector matched no fixture Service'}. The resolver "
+      f"is broken in the direction that makes the gate red for correct code")
+check(bad_matched and len(bad_problems) == 1 and "discovers ZERO targets" in bad_problems[0],
+      "SELF-TEST: the resolver ACCEPTED a monitor whose endpoint names a port its Service does not "
+      "define — the exact defect that left twelve apps/* monitors discovering nothing on prod. "
+      "Assertion (2) is inert; every pass above proves nothing")
+check(len(empty_problems) == 1 and "NEITHER" in empty_problems[0],
+      "SELF-TEST: the resolver ACCEPTED an endpoint naming neither `port` nor `targetPort`, so "
+      "assertion (1) is inert")
+
+
+# ============================================================ (5) THE GENERATOR CANNOT REGRESS
+RENDER = read(ROOT / "catalog" / "render.py")
+check(RENDER != "", "catalog/render.py is unreadable — the generator half cannot be audited")
+sm_fn = RENDER.split("def r_servicemonitor")[-1].split("\ndef ")[0] if RENDER else ""
+check("def r_servicemonitor" in RENDER,
+      "catalog/render.py no longer defines r_servicemonitor — this assertion is auditing nothing")
+check(not re.search(r'"port":\s*"[a-z][a-z0-9-]*"', sm_fn),
+      "catalog/render.py's r_servicemonitor HARDCODES a literal endpoint port name. That is the "
+      "original defect's mechanism: one edit gave twelve services the same wrong port name "
+      "(`metrics`), which no rendered Service defined, and every `up`-based SLI built on them was "
+      "structurally inert on prod. The port name must come from the ServiceDefinition's "
+      "`observability.metricsEndpoint`, the same declaration that renders the Service port")
+check("metricsEndpoint" in sm_fn,
+      "catalog/render.py's r_servicemonitor no longer reads `metricsEndpoint`, so it is back to "
+      "rendering a monitor from an intent (`metrics: true`) that cannot say where metrics are "
+      "served — for most of this catalog, nowhere")
+SCHEMA = read(ROOT / "schemas" / "servicedefinition.schema.json")
+check("metricsEndpoint" in SCHEMA,
+      "schemas/servicedefinition.schema.json does not define observability.metricsEndpoint, so the "
+      "field the renderer depends on is not part of the contract and `additionalProperties: false` "
+      "would reject any ServiceDefinition that declares one")
+
+
+# ============================================================ (6) LOAD-BEARING
+SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "servicemonitor-ports-resolve"
+            / "chainsaw-test.yaml")
+check(re.search(r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo", SELF, re.M) is not None,
+      "servicemonitor-ports-resolve/chainsaw-test.yaml does not declare "
+      "conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
+      "change happens to touch R-OBS-01. ci/touched-requirements.sh maps apps/* to NOTHING, so "
+      "adding a ServiceMonitor under apps/ would be filtered out of exactly the change that "
+      "reintroduces the defect")
+L0 = read(ROOT / "ci" / "l0.sh")
+GATE = read(ROOT / "ci" / "gate.sh")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
+      "purely requirement-scoped run again and the scope annotation above is inert decoration")
+check("servicemonitor-ports-resolve" not in L0,
+      "ci/l0.sh names this check in a hand-written stage. Whole-repo invariants are selected BY "
+      "DECLARING the scope; a per-test stage is the pattern tests/l0-static/"
+      "whole-repo-invariants-wired fails the gate for, because the next invariant's author has to "
+      "REMEMBER to add one and the omission is silent")
+check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None
+      and re.search(r"fail\).*BLOCK=1", GATE) is not None,
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+
+
+# ======================================================================== report
+print(f"Monitor inventory: {len(monitors)} ServiceMonitor/PodMonitor(s), {len(services)} "
+      f"Service(s) authored in this repo")
+for rel, kind, ns, name, selector, ns_sel, eps in monitors:
+    tgt = ", ".join(
+        str(e.get("port") or f"targetPort {e.get('targetPort')}" or "<none>") for e in eps
+    ) or "<no endpoints>"
+    mark = "EXTERNAL" if f"{rel}:{name}" in KNOWN_EXTERNAL_MONITORS else "resolved"
+    print(f"  {mark:<8} {kind:<14} {rel}:{name}  -> {tgt}")
+print(f"\n  resolvable against an in-repo Service: {auditable}/{len(monitors)}")
+print(f"  DECLARED GAP (shrink-only): KNOWN_EXTERNAL_MONITORS: {len(KNOWN_EXTERNAL_MONITORS)} "
+      f"monitor(s) whose Service is created by a chart/operator outside this tree")
+print("  RESOLVER SELF-TEST: agreeing pair accepted, disagreeing pair rejected, "
+      "target-less endpoint rejected")
+
+print()
+if not FAILURES:
+    print(f"check OK: servicemonitor-ports-resolve — {CHECKS} assertions (R-OBS-01)")
+    raise SystemExit(0)
+print(f"CHECK FAIL (servicemonitor-ports-resolve): {len(FAILURES)} failed, "
+      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+for f in FAILURES:
+    print(f"  FAIL {f}", file=sys.stderr)
+raise SystemExit(1)
+PY

conformance/tests/l0-static/whole-repo-invariants-wired/check.sh

@@ -39,20 +39,20 @@ if [ "$FAILED" -ne 0 ]; then
 fi
 
 grep -q 'CONF_SCOPE_ANNOTATION' "$COMMON_SH" \
-  && grep -q 'conf_dirs_for_scope' "$COMMON_SH"
-chk $? "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
+  && grep -q 'conf_dirs_for_scope' "$COMMON_SH" && arc=0 || arc=$?
+chk $arc "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
        "conformance/lib/common.sh no longer resolves the scope annotation — no test can be
        selected BY declaring a scope, so every whole-repo invariant is back to being
        selectable away by the requirement filter"
 
-grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH"
-chk $? "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
+grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH" && arc=0 || arc=$?
+chk $arc "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
        "conformance/run.sh does not union scope-declared tests (--include-scope /
        conf_dirs_for_scopes missing) — the annotation would be inert decoration"
 
 grep -Eq -- '--include-scope[[:space:]]+"?\$\{?WHOLE_REPO_SCOPE' "$L0_SH" \
-  || grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH"
-chk $? "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
+  || grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH" && arc=0 || arc=$?
+chk $arc "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
        "ci/l0.sh does not pass --include-scope ${SCOPE}: the L0 merge-gate stage is back to a
        purely requirement-scoped run, and every whole-repo invariant is filtered out of
        exactly the change that breaks it"
@@ -63,16 +63,16 @@ while IFS= read -r d; do [ -n "$d" ] && declared_dirs+=("$d"); done < <(
   CONF_ROOT="$CONF" bash -c 'source "${CONF_ROOT}/lib/common.sh"; conf_dirs_for_scope "$1"' _ "$SCOPE"
 )
 MIN_DECLARED="${MIN_DECLARED:-5}"
-[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ]
-chk $? "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
+[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ] && arc=0 || arc=$?
+chk $arc "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
        "only ${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED}) — either the
        annotation was stripped from the whole-repo checks, or the resolver stopped matching
        it. Every assertion below would then pass while checking nothing"
 
 for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
   rel="${d#"$ROOT"/}"
-  [ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ]
-  chk $? "${rel}: declares the scope AND carries an executable offline check" \
+  [ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ] && arc=0 || arc=$?
+  chk $arc "${rel}: declares the scope AND carries an executable offline check" \
          "${rel} declares scope='${SCOPE}' but carries neither check.sh nor render-check.sh — an
          unconditional run of a test with nothing to run is a guaranteed SKIP, i.e. a
          permanent not-green, not a guard"
@@ -91,25 +91,58 @@ BOGUS_REQ="r-nosuchreq-99"
 sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
 sel_with="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" --print-selection 2>/dev/null || true)"
 
-[ -z "$sel_without" ]
-chk $? "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
+[ -z "$sel_without" ] && arc=0 || arc=$?
+chk $arc "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
        "the control case is broken: --requirements ${BOGUS_REQ} still selected tests
        ($(printf '%s' "$sel_without" | tr '\n' ' ')), so the assertion below proves nothing
        about --include-scope"
 
+# PARTITIONED BY LAYER, because the union is layer-BOUNDED by design: `--layer l0` must select the
+# scope-declaring L0 tests and NOT the L1 ones (asserted separately below). This loop compared EVERY
+# scope-declaring dir against an `--layer l0` selection, which silently assumed only L0 tests would
+# ever declare the scope — and ci/l1.sh was written from the opposite premise, that a whole-repo
+# invariant could appear "at any layer". The moment the three L1 rendering tests declared it
+# (2026-09-18) this check failed, blaming the union for correctly excluding another layer.
 missing=""
 for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
+  case "$d" in */tests/l0-static/*) ;; *) continue ;; esac
   printf '%s\n' "$sel_with" | grep -qxF "$d" || missing="${missing} ${d#"$ROOT"/}"
 done
-[ -z "$missing" ]
-chk $? "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
+[ -z "$missing" ] && arc=0 || arc=$?
+chk $arc "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
        "the scope union does not actually select:${missing}. This is the blind spot itself: a
        change that attributes no requirement would not run these checks"
 
+# The SAME property one layer up. This is what makes the L1 declaration load-bearing rather than
+# decorative: ci/l1.sh has always passed --include-scope, but until an L1 test declared the scope
+# nothing proved the union reached L1 at all. The 2026-09-04 external-dns merge is the case in
+# point — it attributed no rendering requirement, selected ZERO L1 tests, gated GREEN, and left all
+# three rendering goldens stale.
+sel_l1="$(bash "$RUN_SH" --layer l1 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" \
+          --print-selection 2>/dev/null || true)"
+missing_l1=""
+declared_l1=0
+for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
+  case "$d" in */tests/l1-rendering/*) ;; *) continue ;; esac
+  declared_l1=$((declared_l1+1))
+  printf '%s\n' "$sel_l1" | grep -qxF "$d" || missing_l1="${missing_l1} ${d#"$ROOT"/}"
+done
+[ "$declared_l1" -gt 0 ] && [ -z "$missing_l1" ] && arc=0 || arc=$?
+chk $arc "every scope-declared L1 test is selected under --layer l1 DESPITE a requirement filter matching none" \
+       "scope-declared L1 tests: ${declared_l1}; not selected by --layer l1 --include-scope ${SCOPE}:
+       ${missing_l1:-<none>}. If the count is 0 the rendering goldens are back to being checked only
+       when a touched path happens to attribute R-PLAT-02/R-NET-01/R-NET-05/06/07"
+
+# And the L1 union must not leak the other way either.
+strays_l1="$(printf '%s\n' "$sel_l1" | grep -v '/tests/l1-rendering/' || true)"
+[ -z "$strays_l1" ] && arc=0 || arc=$?
+chk $arc "the scope union stays inside --layer l1 (no cross-layer leakage)" \
+       "--layer l1 --include-scope ${SCOPE} selected non-L1 test(s): $(printf '%s' "$strays_l1" | tr '\n' ' ')"
+
 # The union must not smuggle in another layer's tests when --layer bounds the run.
 strays="$(printf '%s\n' "$sel_with" | grep -v '/tests/l0-static/' || true)"
-[ -z "$strays" ]
-chk $? "the scope union stays inside --layer l0 (no cross-layer leakage)" \
+[ -z "$strays" ] && arc=0 || arc=$?
+chk $arc "the scope union stays inside --layer l0 (no cross-layer leakage)" \
        "--layer l0 --include-scope ${SCOPE} selected non-L0 test(s): $(printf '%s' "$strays" | tr '\n' ' ')"
 
 # ---- (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN -------------------------
@@ -143,23 +176,23 @@ run_offline() {  # run_offline <dir...> -> echoes rc
 rc_declared="$(run_offline "${WORK}/l0-static/synth-declared")"
 rc_undeclared="$(run_offline "${WORK}/l0-static/synth-undeclared")"
 
-[ "$rc_declared" = 3 ]
-chk $? "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
+[ "$rc_declared" = 3 ] && arc=0 || arc=$?
+chk $arc "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
        "a SKIPPED scope-declared check made the offline runner exit ${rc_declared}, not 3. A
        whole-repo invariant reporting 'nothing to audit' means the tree it audits has
        VANISHED; the deleted ci/l0.sh stages called that 'not a pass' and set the gate's
        rc=3. Exit 0 here turns that back into a silent green"
 
-[ "$rc_undeclared" = 0 ]
-chk $? "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
+[ "$rc_undeclared" = 0 ] && arc=0 || arc=$?
+chk $arc "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
        "a SKIPPED undeclared check made the runner exit ${rc_undeclared}, not 0 — the strict
        rule leaked onto ordinary scaffolds, which would make every unfinished test block"
 
 # ...and ci/l0.sh must map that rc 3 to the GATE's rc 3, never to a pass.
 l0_static_stage="$(sed -n '/Stage 3b: L0 STATIC/,/^case "\$rc" in/p' "$L0_SH")"
 printf '%s\n' "$l0_static_stage" | grep -Eq '^[[:space:]]*3\)' \
-  && printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3'
-chk $? "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
+  && printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3' && arc=0 || arc=$?
+chk $arc "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
        "ci/l0.sh's L0-static stage has no rc-3 arm setting the gate's rc=3, so a scope-declared
        check that SKIPPED would be reported as a PASS — the exact false green the deleted
        stages were written to prevent"
@@ -193,8 +226,8 @@ done
 
 # ---- (5) the gate still BLOCKS on an l0 failure -----------------------------------------
 # Carried over from the three deleted stages' own checks — asserted once, here.
-grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH"
-chk $? "ci/gate.sh sets the merge block when the l0 stage fails" \
+grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH" && arc=0 || arc=$?
+chk $arc "ci/gate.sh sets the merge block when the l0 stage fails" \
        "ci/gate.sh does not set the merge block on an l0 stage failure — every assertion above
        would hold while a red L0 stage merged anyway"
 

conformance/tests/l1-rendering/container-workload-render/chainsaw-test.yaml

@@ -23,6 +23,17 @@ metadata:
     conformance.home.k3s/requirements: "R-PLAT-02"
     conformance.home.k3s/layer: "L1"
     conformance.home.k3s/safety-critical: "false"
+    # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
+    # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
+    # rendered output breaks these goldens regardless of which requirement the touched path maps to.
+    # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
+    # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
+    # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
+    # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
+    # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
+    # GREEN, and left every golden stale; the staleness surfaced only on a later full run.
+    # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
+    conformance.home.k3s/scope: "whole-repo"
     conformance.home.k3s/intent: >-
       A kind: container ServiceDefinition renders a Deployment + Service (the PaaS path).
 spec:

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -196,6 +196,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -212,26 +213,6 @@ spec:
     - name: paas-demo
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: paas-demo
-  namespace: paas-demo
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: paas-demo
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: paas-demo
-  namespaceSelector:
-    matchNames:
-    - paas-demo
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # Flux ImageRepository (R-UPD)
 apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository

conformance/tests/l1-rendering/default-deny-baseline-render/chainsaw-test.yaml

@@ -27,6 +27,17 @@ metadata:
     conformance.home.k3s/requirements: "R-NET-01"
     conformance.home.k3s/layer: "L1"
     conformance.home.k3s/safety-critical: "false"
+    # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
+    # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
+    # rendered output breaks these goldens regardless of which requirement the touched path maps to.
+    # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
+    # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
+    # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
+    # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
+    # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
+    # GREEN, and left every golden stale; the staleness surfaced only on a later full run.
+    # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
+    conformance.home.k3s/scope: "whole-repo"
     conformance.home.k3s/intent: >-
       Rendering any service's namespace yields a default-deny baseline policy.
 spec:

conformance/tests/l1-rendering/default-deny-baseline-render/golden/expected.yaml

@@ -72,26 +72,6 @@ spec:
   - toEntities:
     - world
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: baseline-demo
-  namespace: baseline-demo
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: baseline-demo
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: baseline-demo
-  namespaceSelector:
-    matchNames:
-    - baseline-demo
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # Flux ImageRepository (R-UPD)
 apiVersion: image.toolkit.fluxcd.io/v1
 kind: ImageRepository

conformance/tests/l1-rendering/egress-class-render/chainsaw-test.yaml

@@ -32,6 +32,17 @@ metadata:
     conformance.home.k3s/requirements: "R-NET-05, R-NET-06, R-NET-07"
     conformance.home.k3s/layer: "L1"
     conformance.home.k3s/safety-critical: "false"
+    # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
+    # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
+    # rendered output breaks these goldens regardless of which requirement the touched path maps to.
+    # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
+    # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
+    # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
+    # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
+    # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
+    # GREEN, and left every golden stale; the staleness surfaced only on a later full run.
+    # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
+    conformance.home.k3s/scope: "whole-repo"
     conformance.home.k3s/intent: >-
       egress: vpn renders the egress-gateway + kill-switch path; internet does not.
 spec:

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -223,6 +223,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -239,26 +240,6 @@ spec:
     - name: sonarr
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: sonarr
-  namespace: sonarr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: sonarr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: sonarr
-  namespaceSelector:
-    matchNames:
-    - sonarr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -273,14 +254,14 @@ spec:
   - name: sonarr.slis
     rules:
     - alert: sonarr-down
-      expr: up{job="sonarr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
       for: 5m
       labels:
         severity: critical
         service: sonarr
       annotations:
         summary: sonarr is down
-        description: 'sonarr has been unreachable for 5m (SLI: up).'
+        description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
 ---
 # Flux ImageRepository (R-UPD)
 apiVersion: image.toolkit.fluxcd.io/v1

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -223,6 +223,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared
@@ -239,26 +240,6 @@ spec:
     - name: sonarr
       port: 4180
 ---
-# ServiceMonitor (R-OBS-01)
-apiVersion: monitoring.coreos.com/v1
-kind: ServiceMonitor
-metadata:
-  name: sonarr
-  namespace: sonarr
-  labels:
-    release: kube-prometheus-stack
-    app.kubernetes.io/name: sonarr
-spec:
-  selector:
-    matchLabels:
-      app.kubernetes.io/name: sonarr
-  namespaceSelector:
-    matchNames:
-    - sonarr
-  endpoints:
-  - port: metrics
-    interval: 30s
----
 # PrometheusRule SLI alerts (R-OBS-01)
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
@@ -273,14 +254,14 @@ spec:
   - name: sonarr.slis
     rules:
     - alert: sonarr-down
-      expr: up{job="sonarr"} == 0
+      expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
       for: 5m
       labels:
         severity: critical
         service: sonarr
       annotations:
         summary: sonarr is down
-        description: 'sonarr has been unreachable for 5m (SLI: up).'
+        description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
 ---
 # Flux ImageRepository (R-UPD)
 apiVersion: image.toolkit.fluxcd.io/v1

docs/plans/l2-l3-conformance-ci-wiring.md

@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
 | Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
 | Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
 
-**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
-is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
+**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
+LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
+
+> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
+> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
+> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
+> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
+> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
+> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
+> run" this stage attempts can never succeed; it can only skip and, before the guard added on
+> 2026-09-29, lie about it.
+>
+> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
+> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
+> not enable L2/L3, it only changes which way this stage fails.
 
 ---
 

infra/gitea/README.md

@@ -0,0 +1,140 @@
+# infra/gitea — platform settings that are NOT GitOps-managed
+
+Gitea's own repo settings (branch protection, required status checks, approvals) live
+in Gitea's database, not in a manifest Flux reconciles. They are therefore **drift-prone
+and invisible to the gate** — exactly how the state below went unnoticed.
+
+## The gap this closes
+
+Verified against the live API on 2026-09-29, against a `DECISIONS-LOG.md` entry
+(2026-08-31) asserting the opposite:
+
+| Claimed | Measured |
+|---|---|
+| `main` is a protected branch, gate is a required status check | `branch_protections` → `[]`; `protected:false`, `enable_status_check:false`, `required_approvals:0`, `user_can_push:true` |
+| Runner `incluster-dind` online | **zero** runners registered at repo, org and site level |
+
+So `main` accepted direct pushes, and the gate could not run at all. `CLAUDE.md` is the
+rule; this directory is how the rule gets enforced by the machine.
+
+## `branch-protection.sh`
+
+Blocks direct pushes to `main`. Run it, then run the push probe it prints — **verify,
+do not trust the API response**, since Gitea may let site admins bypass protection.
+
+    ./infra/gitea/branch-protection.sh
+
+It authenticates with the credential `git` already has stored for this host, and never
+prints the secret.
+
+## `runner-token-rotate.sh`
+
+Mints a fresh instance-level registration token from the Gitea API and writes it into
+`apps/wave-1/gitea-actions-runner-secret.sops.yaml` via `sops set`. **The token value is
+never printed.**
+
+    ./infra/gitea/runner-token-rotate.sh
+
+A fresh token is usually **not sufficient on its own**. `act_runner` registers exactly once
+and persists its identity in `/data/.runner` on the `act-runner-data` PVC. If Gitea no
+longer holds a matching runner row, the pod starts, sees `.runner`, **skips registration**,
+and polls with an identity Gitea does not recognise — so it never appears in the runner
+list. That is precisely the observed symptom: zero runners, pod running, no error anyone
+looks at. Clearing the PVC is what forces a real re-registration; the script prints the
+commands.
+
+## Sequence — DONE 2026-09-29
+
+All four steps are complete and each was verified, not assumed:
+
+| Step | State |
+|---|---|
+| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
+| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
+| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
+| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
+
+Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
+`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
+account agents authenticate as.
+
+### 5. Block the admin override — the step the first pass MISSED
+
+`enable_status_check` and `required_approvals` were set on 2026-09-29 and **still did not
+bite**, because `block_admin_merge_override` was `false`. Gitea lets repo admins override
+merge requirements without it, and `jakman85` holds `owner` — so the two settings that looked
+like the gate were overridable by precisely the account that merges.
+
+**The evidence was sitting in plain sight:** PRs #5 through #9 all merged with
+`required_approvals: 1` in force and **zero approvals recorded on any of them**. Checking the
+*branch* view (`/branches/main`) hid this — it does not expose the field. Read the rule object
+(`/branch_protections/main`) and enumerate its keys.
+
+> **Do not probe for field names you assume exist.** This Gitea (1.24.3) has no
+> `apply_to_admins`; asking `jq` for it returns `null`, which reads like "set to false" but
+> means "key absent". Enumerate with `to_entries` and act on what is really there.
+
+Now `block_admin_merge_override: true`. **That is still only a setting, not evidence** — it is
+proven when a green PR is REFUSED a merge until approved, and not before.
+
+### `require-gate.sh` / `verify-then-enforce.sh`
+
+`verify-then-enforce.sh` is the re-runnable form of the whole safe order. It merges nothing —
+merging is a human's reviewed act — but it forces the Flux reconcile and then **aborts** unless
+the runner pod is healthy, a runner is online, and a real workflow goes green on it, before
+enabling enforcement and probing the push. `require-gate.sh` is the enforcement step alone.
+
+Re-run `verify-then-enforce.sh` after any future change to the act-runner Deployment.
+
+### Why these scripts exist at all, and what is still missing
+
+Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
+conformance gate can read. Nothing detects drift from what this directory declares. That is
+the root cause of the entire 2026-09-29 session: branch protection was off and the runner was
+gone for a month while the log recorded both as working, and **no check existed that could
+have caught it.**
+
+These scripts encode the desired state and can re-assert it, but they are not a detector. The
+durable fix is a conformance check that reads the live Gitea API and fails when it diverges
+from what is declared here — turning a class of silent, invisible failure into a loud one.
+**That check does not exist yet and is the highest-value remaining work in this directory.**
+
+## Historical sequence (kept for context)
+
+### Original ordering notes
+
+1. **Get an Actions runner registered** — `runner-token-rotate.sh`, then clear the PVC so
+   the runner actually re-registers, then confirm it shows `online` in the API. Do not
+   skip the confirmation: the whole reason this went unnoticed is that a runner was
+   *recorded* as online without anyone checking.
+2. **Watch the gate actually report on a real PR**, and note the context name it produced.
+3. **Only then set `enable_status_check: true`** with that exact context.
+4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
+
+> **Do not do step 3 before step 1.** A required status check with no runner leaves every
+> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
+> sat `pending` for 30.6 minutes across 122 polls with no runner registered, and never
+> reported.
+
+### Measured, once the runner came up (2026-09-29)
+
+The first CI jobs this repo ever ran, on PR #1 and PR #2:
+
+| Context | Result | Time |
+|---|---|---|
+| `gate / gate (pull_request)` | success | 1m58s (PR #1), 1m34s (PR #2) |
+| `gate / gate (push)` | success | 1m32s (PR #2 only) |
+| `gate / conformance (pull_request)` | **success — "Has been skipped"** | — |
+
+**Require `gate / gate (pull_request)`. Do NOT require either `conformance` context —
+but not for the reason an earlier draft of this file gave.** A skipped Gitea Actions job
+does not hang a status check: Gitea reports it as **`success`** with the description
+"Has been skipped". So requiring it would not deadlock anything — it would pass
+*vacuously*, every time, forever, while nothing ran. Under HARD RULE 4 that is worse than
+a deadlock: a deadlock is loud, a permanently-green check that verifies nothing is
+exactly the fail-open shape this repo keeps finding.
+
+The `(push)` contexts appear only for branches matching `gate.yml`'s push trigger
+(`feature/**`, `fix/**`, `docs/**`, `chore/**`). `feat/**` is **not** in that list, which
+is why PR #1's `feat/` branch produced only `pull_request` contexts. Require only the
+`pull_request` context, or PRs from non-matching prefixes will never satisfy the rule.

infra/gitea/block-admin-override.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Stop repository admins from overriding the merge requirements on `main`.
+#
+# WHY. Gitea lets repo admins override branch-protection merge requirements unless
+# `block_admin_merge_override` is set. It was FALSE, and jakman85 holds `owner` -- so
+# required_approvals and the required status check were both overridable by the one account
+# that actually merges. Evidence: PRs #5-#9 all merged with ZERO approvals recorded against
+# `required_approvals: 1`.
+#
+# NOTE ON FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins` field. Querying it with jq
+# returns null, which reads like "set to false" but actually means "key absent". Enumerate the
+# real keys (`to_entries`) rather than probing names you assume exist.
+#
+# Recoverable: an owner can PATCH this back to false if it gets in the way.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
+
+echo "==> before: block_admin_merge_override = $(api "$B" | jq -r .block_admin_merge_override)"
+api -X PATCH -H 'Content-Type: application/json' \
+  -d '{"block_admin_merge_override": true}' "$B" >/dev/null
+AFTER=$(api "$B" | jq -r .block_admin_merge_override)
+echo "==> after:  block_admin_merge_override = $AFTER"
+[ "$AFTER" = true ] || { echo "PATCH did not take -- do not record this as enforced." >&2; exit 1; }
+
+api "$B" | jq -r '{enable_push,enable_status_check,status_check_contexts,required_approvals,block_admin_merge_override}'
+cat <<'NOTE'
+
+NOW PROVE IT -- the setting is not the evidence. On the next PR:
+  * let the gate go green, then try to merge WITHOUT approving it.
+  * Gitea must REFUSE. If it merges, this still is not doing its job.
+Do not record approvals as enforced until you have seen a merge refused.
+NOTE

infra/gitea/branch-protection.sh

@@ -0,0 +1,57 @@
+#!/usr/bin/env bash
+# Enable branch protection on main for home-network-k3s-impl.
+#
+# Sets enable_push:false  -> no direct pushes to main; changes land only by merging a PR.
+#
+# enable_status_check is deliberately FALSE. There is currently NO Gitea Actions runner
+# registered (verified 2026-09-29: zero at repo, org and site level), so requiring the
+# gate context would deadlock EVERY PR permanently -- including the PR that fixes it.
+# Turn it on only after a runner is live and you have seen the gate actually report on
+# a real PR, using the context name CI itself produced.
+#
+# required_approvals is 0 for the same class of reason: jakman85 is a site admin but NOT
+# a repo collaborator, so requiring an approval now could leave nobody able to approve.
+# Add the collaborator first, then raise this to 1.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev
+OWNER=Home-Network
+REPO=home-network-k3s-impl
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"
+P="$(sed -n 's/^password=//p' <<<"$creds")"
+[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
+echo "authenticating as '${U}' (secret not shown)"
+
+echo "==> creating branch protection on 'main'"
+jq -n '{
+  branch_name: "main",
+  enable_push: false,
+  enable_status_check: false,
+  status_check_contexts: [],
+  required_approvals: 0,
+  block_on_rejected_reviews: true,
+  block_on_outdated_branch: false,
+  dismiss_stale_approvals: true
+}' | curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
+  -X POST -H 'Content-Type: application/json' --data @- \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections" \
+| jq -r 'if .branch_name then "created" else "API said: \(.message // tostring)" end'
+
+echo "==> verifying (this is the part that matters)"
+curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
+| jq -r '{protected, user_can_push, required_approvals, enable_status_check, status_check_contexts}'
+
+cat <<'NOTE'
+
+Now prove it actually blocks you -- do not take the API's word for it. From a clean
+checkout on main, with nothing to lose:
+
+    git commit --allow-empty -m 'protection probe'
+    git push origin main          # EXPECT: rejected by the remote
+    git reset --hard HEAD~1       # tidy up either way
+
+If that push SUCCEEDS, Gitea is letting site admins bypass protection and this is not
+yet doing what you want.
+NOTE

infra/gitea/require-gate.sh

@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+# Items 1 + 2: make the gate a REQUIRED status check and require one approval.
+#
+# RUN THIS LAST. Prerequisites, in order:
+#   a) PR #7 (dind native sidecar) is MERGED, and
+#   b) you have SEEN a real workflow run go green on the converted pod.
+# The runner Deployment is the merge gate for the whole repo. Turning on a required check
+# while the runner is broken means no PR can merge -- including the revert. (Recoverable:
+# an owner can set enable_status_check back to false. But do not rely on that.)
+#
+# Requires exactly ONE context: `gate / gate (pull_request)`.
+#   * NOT `gate / conformance (*)` -- that job is `if: false`, and a skipped Gitea job reports
+#     SUCCESS ("Has been skipped"). Requiring it would pass VACUOUSLY forever while nothing
+#     ran, which under HARD RULE 4 is worse than a deadlock: a deadlock is loud.
+# block_admin_merge_override: true is LOAD-BEARING and was missing on the first pass. Gitea
+# lets repo admins override merge requirements without it, so required_approvals and the
+# required status check are both bypassable by exactly the account that does the merging.
+# Measured: PRs #5-#9 merged with ZERO approvals recorded against required_approvals: 1.
+#
+# FIELD NAMES: this Gitea (1.24.3) has NO `apply_to_admins`. Asking jq for it returns null,
+# which reads like "false" but means "key absent" -- enumerate real keys with `to_entries`
+# instead of probing names you assume exist. That mistake cost a wasted PATCH.
+#
+#   * NOT the `(push)` variant -- after PR #6 there are no per-branch push runs at all, so a
+#     PR would never produce that context.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+
+echo "==> updating branch protection on main"
+jq -n '{
+  enable_push: false,
+  enable_status_check: true,
+  status_check_contexts: ["gate / gate (pull_request)"],
+  required_approvals: 1,
+  block_on_rejected_reviews: true,
+  dismiss_stale_approvals: true,
+  block_on_outdated_branch: false,
+  block_admin_merge_override: true
+}' | api -X PATCH -H 'Content-Type: application/json' --data @- \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main" >/dev/null
+
+echo "==> verifying"
+api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
+ | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'
+
+cat <<'NOTE'
+
+NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
+  * Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
+  * PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
+    agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
+    and rely on the status check alone.
+
+Then prove it, rather than trusting the API:
+  git commit --allow-empty -m probe && git push origin main   # EXPECT: rejected
+  git reset --hard HEAD~1
+NOTE

infra/gitea/runner-token-rotate.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+# Mint a fresh Gitea Actions registration token and write it into the SOPS secret.
+#
+# The token value is NEVER printed. It goes from the Gitea API into `sops set` and
+# nowhere else. (It is briefly visible in this host's process list as a sops argument;
+# acceptable on a single-operator workstation, and far better than echoing it.)
+#
+# This ALONE may not bring the runner back -- see "IMPORTANT" at the end of the output.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev
+SECRET=apps/wave-1/gitea-actions-runner-secret.sops.yaml
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+cd "$REPO_ROOT"
+[ -f "$SECRET" ] || { echo "not in the repo root, or $SECRET is missing" >&2; exit 1; }
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"
+P="$(sed -n 's/^password=//p' <<<"$creds")"
+[ -n "$U" ] && [ -n "$P" ] || { echo "no stored git credential for $HOST" >&2; exit 1; }
+echo "==> minting an instance-level registration token as '${U}'"
+
+TOKEN="$(curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") \
+  "https://${HOST}/api/v1/admin/runners/registration-token" | jq -r '.token // empty')"
+[ -n "$TOKEN" ] || { echo "Gitea did not return a token" >&2; exit 1; }
+echo "==> got a ${#TOKEN}-character token (value not shown)"
+
+echo "==> writing it into ${SECRET} via sops"
+DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run \
+  sops set "$SECRET" '["stringData"]["token"]' "\"${TOKEN}\""
+unset TOKEN
+
+echo "==> done. Verify the file is still encrypted (no plaintext token):"
+grep -q 'ENC\[' "$SECRET" && echo "    OK - stringData is still SOPS-encrypted" \
+  || { echo "    REFUSING TO CONTINUE: $SECRET is not encrypted" >&2; exit 1; }
+
+cat <<'NOTE'
+
+Next: commit this on a branch, open a PR, and merge it so Flux applies the new Secret.
+
+IMPORTANT -- a new token is usually NOT sufficient on its own.
+act_runner registers exactly ONCE and then persists its identity in /data/.runner on the
+act-runner-data PVC. If Gitea no longer has a matching runner row (DB restore, the row was
+deleted, the instance was rebuilt), the pod starts, sees .runner, SKIPS registration, and
+polls with an identity Gitea does not recognise -- so it never appears in the runner list,
+which is exactly the observed symptom (zero runners, pod running).
+
+So after the Secret lands, force a genuine re-registration by clearing that state:
+
+    kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=0
+    kubectl --context k3s-prod -n gitea-actions delete pvc act-runner-data
+    kubectl --context k3s-prod -n gitea-actions scale deploy/act-runner --replicas=1
+
+(The PVC holds only runner registration state -- no build artifacts, nothing durable.
+Deleting it is the supported way to re-register. Flux recreates it from the manifest.)
+
+Then confirm it actually registered -- do not assume:
+
+    curl -s -u '<user>' https://gitea.home.keithjackson.dev/api/v1/admin/actions/runners | jq
+
+A runner must appear with status "online" and the labels ubuntu-latest / homelab.
+NOTE

infra/gitea/verify-then-enforce.sh

@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+# verify-then-enforce.sh — run this AFTER you have merged PRs 8, 5, 6, 7 in that order.
+#
+# It does NOT merge anything. Merging is your reviewed act (CLAUDE.md: a human approves and
+# merges). This script does the mechanical half: prove the converted runner actually works,
+# and only then turn on enforcement.
+#
+# WHY THIS ORDER. The act-runner Deployment IS the merge gate for this repository. Enabling a
+# required status check while the runner is broken means NO PR can merge -- including the
+# revert. So PR #7 lands while the gate is still advisory, is PROVEN green on a real run, and
+# only then does enforcement go on.
+#
+# Aborts at the first checkpoint that does not pass. Safe to re-run.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+REPO_ROOT=/var/home/keith/source/home-network-k3s-impl
+D="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "$REPO_ROOT"
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+A="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
+say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
+die() { printf '\n\033[31mABORT: %s\033[0m\n' "$1" >&2; exit 1; }
+
+say "PRECHECK — all four PRs must be merged first"
+for n in 8 5 6 7; do
+  m=$(api "$A/pulls/$n" | jq -r .merged)
+  printf '  PR #%s merged=%s\n' "$n" "$m"
+  [ "$m" = true ] || die "PR #$n is not merged yet. Merge 8, 5, 6, 7 in that order, then re-run."
+done
+
+say "STEP 1 — force Flux to apply the sidecar now, not on the 10m interval"
+./infra/devcontainer/dev.sh run bash -c \
+  'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-1 --with-source'
+
+say "STEP 2 — CHECKPOINT: the converted pod must come up healthy"
+./infra/devcontainer/dev.sh run bash -c '
+  kubectl --context k3s-prod -n gitea-actions rollout status deploy/act-runner --timeout=180s
+  echo "--- sidecar (initContainers) ---"
+  kubectl --context k3s-prod -n gitea-actions get deploy act-runner \
+    -o jsonpath="{range .spec.template.spec.initContainers[*]}{.name} restartPolicy={.restartPolicy}{\"\n\"}{end}"
+  kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner' \
+  || die "converted act-runner pod is not healthy -- REVERT PR #7 before anything else"
+
+say "STEP 3 — CHECKPOINT: a runner must be ONLINE again"
+n=0
+for i in $(seq 1 40); do
+  n=$(api "https://${HOST}/api/v1/admin/actions/runners" | jq -r '[(.runners//[])[]|select(.status=="online")]|length')
+  printf '\r  online runners: %s  (%ss)   ' "$n" "$((i*5))"
+  [ "${n:-0}" -ge 1 ] && { echo; break; }
+  sleep 5
+done
+[ "${n:-0}" -ge 1 ] || die "no runner online after the conversion -- REVERT PR #7"
+
+say "STEP 4 — CHECKPOINT: a REAL workflow must go green ON THE CONVERTED POD"
+MAIN_SHA=$(api "$A/branches/main" | jq -r .commit.id)
+echo "  watching the post-merge push run on main (${MAIN_SHA:0:8})..."
+st=""
+for i in $(seq 1 60); do
+  st=$(api "$A/commits/$MAIN_SHA/status" | jq -r .state)
+  printf '\r  main gate: %-8s (%ss)   ' "$st" "$((i*10))"
+  case "$st" in success) echo; break;; failure|error) die "gate FAILED on the converted pod -- REVERT PR #7";; esac
+  sleep 10
+done
+[ "$st" = success ] || die "gate never completed on the converted pod -- REVERT PR #7; do NOT enable enforcement"
+echo "  the converted sidecar ran a real job green. Enforcement is now safe."
+
+say "STEP 5 — require the gate + one approval"
+"$D/require-gate.sh"
+
+say "STEP 6 — prove a direct push is rejected (do not trust the API's word)"
+git fetch -q origin && git checkout -q main && git reset -q --hard origin/main
+git commit -q --allow-empty -m 'protection probe'
+if git push origin main 2>&1 | grep -q 'protected branch'; then
+  echo "  CONFIRMED: push to main rejected by the pre-receive hook."
+else
+  echo "  WARNING: the push was NOT rejected -- protection is not doing what it claims."
+fi
+git reset -q --hard origin/main
+
+say "FINAL STATE"
+api "$A/branches/main" | jq -r '{protected,user_can_push,enable_status_check,status_check_contexts,required_approvals}'

infra/plex/claim-ship.sh

@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+# plex-claim-ship.sh — run this IMMEDIATELY AFTER editing the SOPS secret with a fresh
+# plex.tv claim token. It drives the whole CI/CD loop end to end and prints a running
+# stopwatch so you can see the budget being spent.
+#
+#   branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verify
+#
+# It never sees, prints, or handles the token: the file is already encrypted by sops.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev
+OWNER=Home-Network
+REPO=home-network-k3s-impl
+SECRET=apps/wave-0/plex-claim-secret.sops.yaml
+T0=$(date +%s)
+el() { printf '\n[T+%02d:%02d] %s\n' $(( ($(date +%s)-T0)/60 )) $(( ($(date +%s)-T0)%60 )) "$1"; }
+
+cd "$(git rev-parse --show-toplevel)"
+
+# --- preflight: refuse to ship a file that is not what we think it is -------------
+git diff --quiet -- "$SECRET" && { echo "ERROR: $SECRET is unchanged. Edit it first:"; \
+  echo "  infra/devcontainer/dev.sh run sops $SECRET"; exit 1; }
+grep -q 'PLEX_CLAIM: ENC\[' "$SECRET" || { echo "ERROR: $SECRET is not encrypted. REFUSING."; exit 1; }
+[ -z "$(git status --porcelain | grep -v " $SECRET\$")" ] || echo "NOTE: other files are dirty; only $SECRET will be committed."
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+
+# Branch prefix is now cosmetic. It used to matter: gate.yml's push trigger covered
+# feature/** fix/** docs/** chore/** but not feat/**, and with act_runner at capacity 1 a
+# matching prefix fired a redundant run that serialised ahead of the PR run. Those
+# per-branch push triggers were deleted, so no prefix produces a duplicate run any more.
+BRANCH="feat/plex-claim-token-$(date +%H%M%S)"
+
+el "committing on $BRANCH"
+git checkout -q -b "$BRANCH"
+git add "$SECRET"
+git commit -q -m 'chore(plex): fresh plex.tv claim token
+
+Injected inside the plex.tv claim window per runbooks/plex-claim.md.'
+SHA=$(git rev-parse HEAD)
+
+el "pushing"
+git push -q -u origin "$BRANCH"
+
+el "opening PR"
+PR=$(jq -n --arg h "$BRANCH" --arg b main --arg t 'chore(plex): fresh plex.tv claim token' \
+       '{head:$h,base:$b,title:$t,body:"Token injection inside the claim window. Wiring already reviewed in #1."}' \
+     | api -X POST -H 'Content-Type: application/json' --data @- \
+         "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls" | jq -r '.number')
+[ -n "$PR" ] && [ "$PR" != "null" ] || { echo "PR creation failed"; exit 1; }
+echo "  PR #$PR"
+
+el "waiting for the gate (expect ~1m40s)"
+for i in $(seq 1 60); do
+  st=$(api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/commits/${SHA}/status" | jq -r '.state')
+  printf '\r  gate=%s  [T+%ss]   ' "$st" "$(( $(date +%s)-T0 ))"
+  case "$st" in success) echo; break;; failure|error) echo; echo "GATE FAILED — stopping."; exit 1;; esac
+  sleep 5
+done
+[ "$st" = "success" ] || { echo; echo "gate did not finish in time"; exit 1; }
+
+el "merging PR #$PR"
+api -X POST -H 'Content-Type: application/json' -d '{"Do":"merge"}' \
+  "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/pulls/${PR}/merge" >/dev/null
+git checkout -q main && git pull -q --ff-only origin main
+
+el "forcing the Flux reconcile (--with-source, or it applies the OLD revision)"
+./infra/devcontainer/dev.sh run bash -c \
+  'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
+
+# MANDATORY here. Only the Secret ciphertext changed, so the Deployment pod template is
+# byte-identical and NOTHING restarts on its own. PLEX_CLAIM is read once, at container
+# start. Skipping this is failure mode #2 in runbooks/plex-claim.md.
+el "rolling the pod (mandatory — a changed Secret restarts nothing)"
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex rollout restart deployment/plex && \
+   kubectl --context k3s-prod -n plex rollout status deployment/plex --timeout=120s'
+
+el "verdict"
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex logs deploy/plex --since=3m | grep -i -e claim -e token || true'
+cat <<'NOTE'
+
+  "Attempting to obtain server token..." + "Token obtained successfully"  -> CLAIMED
+  "Attempting..." with no success line                                    -> token expired; retry, nothing is broken
+  neither line                                                            -> already claimed, or env did not reach the container
+
+Confirm it persisted (prints the attribute NAME only, never its value):
+
+  infra/devcontainer/dev.sh run bash -c \
+    'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
+       "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
+NOTE

platform/coredns/coredns-custom.yaml

@@ -12,6 +12,25 @@
 # works. External/LAN clients keep their own answers (PiHole split-horizon, docs/09) — this ConfigMap
 # only affects CLUSTER DNS (CoreDNS), not LAN DNS.
 #
+# THE LAN HALF IS NOW REAL, AND WAS NOT UNTIL 2026-09-14. This header deferred LAN clients to
+# "PiHole split-horizon" from the day it was written, but PiHole served no DNS at all — its Service
+# was ClusterIP with ports 80 and 4180 only, no port 53 and no LoadBalancer — so the deferral
+# pointed at nothing and LAN clients depended on hand-maintained per-client dnsmasq files (one went
+# stale after the dev decommission and cost the operator sonarr.${BASE_DOMAIN}). PiHole now answers
+# the zone on udp/53 + tcp/53 at its own pinned VIP: exposure in apps/wave-0/pihole-dns.yaml, the
+# answers themselves as FTLCONF env in catalog/pihole.yaml. The two halves are deliberately
+# INDEPENDENT mechanisms over the same intent — this file for pod clients, dnsmasq `address=` lines
+# for LAN clients — and both resolve the whole zone to ${GATEWAY_VIP} by wildcard so a newly
+# onboarded service needs no DNS change on either side.
+#
+# ONE KNOWN DIVERGENCE, recorded rather than papered over: PiHole carries specific overrides for the
+# L4 services that live on their OWN VIPs (unifi -> ${UNIFI_L4_VIP}, crafty-mc -> ${CRAFTY_L4_VIP}),
+# because a LAN client asking for unifi.${BASE_DOMAIN} must reach the UniFi controller's own address.
+# The `template IN A` block below has no such overrides and answers those two names with
+# ${GATEWAY_VIP}, which the Gateway cannot route. No in-cluster workload resolves either name today,
+# so nothing is broken by it; fixing it means per-name `template` blocks or hosts entries here and
+# is a separate change from the LAN work.
+#
 # MECHANISM (k3s-specific, F-DNS / docs/09): k3s CoreDNS ships a Corefile with
 # `import /etc/coredns/custom/*.server`, and mounts the `coredns-custom` ConfigMap (kube-system) at
 # /etc/coredns/custom. Each `*.server` key becomes a top-level server block imported into the

platform/crafty/README.md

@@ -117,3 +117,29 @@ Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subjec
 3. **Server-create + game reach:** in the Crafty UI create a Java server bound to `25565`; start it;
    connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or `192.168.1.55:25565`)
    from the LAN and confirm the world loads. (Bedrock: a server on `19132/udp` reached at the VIP.)
+
+## Runtime state: the servers themselves are NOT in git
+
+As of 2026-09-14 one Java server exists — **"homelab"**, Paper 26.2, TCP 25565, 2 GiB heap, autostart
+on, UUID `13c7b85d-00bb-4a58-8c86-2433ca47891c`. It was created through Crafty's v2 API, and **Mojang's
+EULA (<https://aka.ms/MinecraftEULA>) was accepted on the operator's behalf** to let it start; see the
+DECISIONS-LOG entry of that date.
+
+Everything in this directory describes only the *capacity* to run Minecraft servers. The servers are
+application state on the `crafty-data` Longhorn PVC, in two places, and **Flux cannot recreate either**:
+
+- `/crafty/app/config/db/` (`config` subPath) — Crafty's SQLite row: name, port, jar, launch command,
+  heap sizes, `auto_start` / `crash_detection`.
+- `/crafty/servers/<uuid>/` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and
+  **the generated world**. The jar is re-downloadable; the world is not.
+
+So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565.
+Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH
+`auto_start` to true — see the next note. The world starts over. **Treat this PVC as backup-relevant.**
+
+### Trap: the create API silently ignores `autostart`
+
+`POST /api/v2/servers` accepts and validates `autostart` and `crashdetection`, but never stores them —
+the persisted columns are `auto_start` and `crash_detection`. After a create that asked for `true`,
+both read back `false`. Set them afterwards with `PATCH /api/v2/servers/<uuid>` and **read them back to
+confirm**, or the server will not come up after a pod restart.

platform/crafty/deployment.yaml

@@ -79,8 +79,19 @@ spec:
           resources:
             requests:
               cpu: 250m
-              memory: 1Gi
+              # Raised 1Gi -> 3Gi when the first Minecraft server ("homelab", Paper 26.2) was created:
+              # that JVM runs with -Xms2000M -Xmx2000M, so the container's steady-state floor is now the
+              # panel (~100Mi) PLUS a permanently-committed 2 GiB heap plus JVM overhead. Measured
+              # cgroup memory.current with the server idle-but-running: 2.86Gi. A 1Gi request understated
+              # that by ~3x, which both misleads the scheduler and makes this Burstable pod an early
+              # eviction candidate under node memory pressure (kubelet ranks by usage-over-request).
+              memory: 3Gi
             # Minecraft JVMs are memory-hungry; cap generously for a starter set of small servers.
+            # Headroom check for the 4Gi cap: 2 GiB heap + ~0.5 GiB JVM non-heap (metaspace, code cache,
+            # GC structures, thread stacks, mmap'd region files) + ~0.1 GiB panel = ~2.9Gi observed,
+            # leaving ~1.2Gi. A 3 GiB heap would have landed near 4Gi and risked an OOMKill, which is
+            # why the server was sized at a 2 GiB heap rather than larger. Raise BOTH this limit and the
+            # server's own memory in Crafty together if a bigger heap is ever wanted.
             limits:
               memory: 4Gi
           securityContext:

platform/crafty/service-l4.yaml

@@ -28,9 +28,67 @@
 # documentation) and to the game-port ingress CiliumNetworkPolicy (networkpolicy.yaml). K8s Services
 # enumerate ports individually (no ranges), so add one `{ name, port, targetPort, protocol }` line each.
 #
-# externalTrafficPolicy: Local preserves the LAN client source IP and pins the VIP to the node running
-# the crafty pod (also what L2 announcement does). external-dns (source: service) publishes the A
-# record from the -tcp Service.
+# external-dns (source: service) publishes the A record from the -tcp Service.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
+# -------------------------------------------------------------------------------------------------
+# This file used to say `Local`, justified as "preserves the LAN client source IP and pins the VIP to
+# the node running the crafty pod (also what L2 announcement does)". THE SECOND HALF IS FALSE, and it
+# is what took the Minecraft port off the LAN.
+#
+# L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder per
+# Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which matches ALL
+# linux nodes — and it does NOT filter that set by which nodes hold a backend. MEASURED on prod
+# 2026-09-14: the crafty pod ran on k3s-prod-1, `cilium-l2announce-crafty-crafty-l4-udp` was held by
+# k3s-prod-1 (so Bedrock/UDP happened to work) but `-tcp` was held by k3s-prod-2 — and with `Local` an
+# announcing node holding no backend has nowhere to send the packet and answers ICMP port unreachable.
+# 192.168.2.22:25565, the Java port, was dead from the LAN for up to 8 days. That the two protocols
+# landed on different nodes, one working and one not, is the whole tell: the election is per-Service
+# and random, not backend-aware. It also does NOT self-heal — the holder keeps RENEWING after the pod
+# moves, so only deleting the Lease forces a (still random) re-election. See
+# apps/wave-0/pihole-dns.yaml, which hit this first and is the reference implementation.
+#
+# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the pod is, which is what
+# makes this survive a reschedule and a rebuild.
+#
+# NO CHANGE IS NEEDED IN networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED. Under
+# `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's reserved
+# `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
+# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
+# crafty-allow-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node` (along
+# with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the crafty endpoint's datapath
+# policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
+# `reserved:remote-node` was already an Allow on 25565-25567/TCP and 19132-19134/UDP. Adding
+# `remote-node` to that rule would be a no-op, so it is deliberately NOT added here — do not "fix" it.
+#
+# THE CLIENT-IP COST, STATED PLAINLY AND ACCEPTED. Cilium runs in SNAT mode here (no
+# `loadBalancer.mode: dsr` in platform/cilium/helmrelease.yaml), so unlike UniFi — a KubeVirt VM whose
+# `masquerade` binding already destroyed the client address below the Service — crafty is a plain pod
+# and DOES lose a real client IP it was previously receiving. Every player will appear to the server
+# subprocesses as the ingress node's address. What that actually costs here:
+#   * PLAYER IDENTITY IS UNAFFECTED. Minecraft Java authenticates against Mojang/Microsoft and Bedrock
+#     against Xbox Live; a player is a UUID, not an address. Vanilla `/ban` and the whitelist are
+#     UUID-keyed (banned-players.json, whitelist.json) and keep working exactly as before.
+#   * `/ban-ip` (banned-ips.json) DEGRADES, and would degrade dangerously: with every player behind one
+#     apparent address, an IP ban would hit EVERY player at once. Use `/ban` (by player), not
+#     `/ban-ip`, on this cluster. No IP ban is configured today.
+#   * Per-IP rate limiting / connection throttling degrades the same way — the whole LAN counts as one
+#     client. Nothing in platform/crafty configures any, and Crafty's own web UI is a separate,
+#     Keycloak-gated path that does not ride these Services.
+# This is a real cost and it is worth paying: the alternative on offer is a Java port that is dead
+# outright, and the only fix that keeps BOTH the client IP and a working VIP is `loadBalancer.mode:
+# dsr` on the Cilium HelmRelease — a platform-wide change with an Ansible [parity] contract that rolls
+# the CNI across all three nodes, which belongs in its own change, not in an outage fix.
+#
+# NO loadBalancerSourceRanges HERE, DELIBERATELY — this is where this file departs from
+# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving up a
+# restriction it had and genuinely needed to be LAN-only. These Services never had a source
+# restriction at all (crafty-allow-ingress admits `world` on the game ports, and the block above is
+# explicit that player auth — not network position — is the control). `Cluster` therefore removes no
+# protection and there is nothing to compensate for. Pinning the game ports to ${LAN_CIDR} during an
+# outage fix would be a NEW restriction that would cut off any player reaching these ports from off
+# the LAN, trading this outage for a subtler one.
 apiVersion: v1
 kind: Service
 metadata:
@@ -45,7 +103,7 @@ metadata:
     external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     app.kubernetes.io/name: crafty
   ports:
@@ -65,7 +123,7 @@ metadata:
     lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     app.kubernetes.io/name: crafty
   ports:

platform/externaldns/helmrelease.yaml

@@ -26,10 +26,94 @@ spec:
     provider:
       name: cloudflare
     # Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent.
+    #
+    # BOTH SOURCES ARE UNFILTERED HERE, AND THAT IS DELIBERATE — see the WHICH RESOURCES block
+    # below. Scoping is done PER RESOURCE by the renderer, not cluster-wide by this list.
     sources:
       - gateway-httproute
       - service
     # ---------------------------------------------------------------------------------------------
+    # WHICH RESOURCES MAY PUBLISH — `internal` services no longer can (R-DNS-03, 2026-09-14)
+    #
+    # THE DEFECT. `sources` above admits EVERY HTTPRoute on the shared Gateway, and each one's
+    # address is the Gateway VIP. docs/09 R-DNS-03 says `internal` services are unreachable from the
+    # public internet and that Pi-hole serves the internal split-horizon answers; the renderer
+    # contract said an `internal` service renders "no public DNS record". Neither sentence was
+    # enforced by anything, so six `exposure: internal` services (jackett, pihole, qbittorrent,
+    # radarr, semaphore, sonarr) published `<name>.home.keithjackson.dev` into this PUBLIC
+    # authoritative zone answering 192.168.2.20. That exposed the internal subnet, the Gateway VIP
+    # and a complete service inventory to anyone who could guess a label, and made every one of
+    # those names a DNS-rebind primitive — which is exactly why the SOHO router at 192.168.2.1
+    # strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own
+    # services. Pre-state captured at the AUTHORITY and committed:
+    # .standup-dns/dns-20260914-pre-externaldns-scope.txt.
+    #
+    # THE MECHANISM — per-resource exclusion, emitted by catalog/render.py from `exposure`.
+    # An `internal` service's rendered HTTPRoute carries
+    #     external-dns.alpha.kubernetes.io/controller: none
+    # ExternalDNS v0.21.0 (the DEPLOYED app version; read off the live prod Deployment, image
+    # registry.k8s.io/external-dns/external-dns:v0.21.0) skips any source object whose `controller`
+    # annotation is present and != its own `dns-controller`:
+    #     source/annotations/processors.go  IsControllerMismatch()
+    #     source/gateway.go  Endpoints():  if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
+    #     source/service.go  Endpoints():  the identical check
+    # The gateway check runs BEFORE any hostname is resolved, so such a route cannot become an
+    # endpoint for any name it declares. PROVEN LIVE on prod before this landed, with a throwaway
+    # hostname and no existing record at risk: an annotated route published nothing at all, and
+    # ADDING the annotation to a route that already had a record produced
+    #   action=DELETE record=<name> type=A    and   action=DELETE record=a-<name> type=TXT
+    # one ~60s reconcile later, and moved nothing else in the zone.
+    #
+    # WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. The chart's `labelFilter`
+    # (--label-filter) is the stronger default posture — publish only what is labelled — and
+    # v0.21.0's own --help confirms BOTH configured sources support it ("currently supported by
+    # source types crd, gateway-httproute, ... service and ambassador-host"). It is NOT set here
+    # because its blast radius is not this defect's. Nine publishing objects in this repo are
+    # hand-authored platform/apps manifests with no `exposure` field for the renderer to read, and
+    # seven of them hold live records: auth (oauth2-proxy), crafty, crafty-mc, grafana, headlamp,
+    # the apex (homepage), unifi. Setting an opt-in filter DELETES all seven on the next reconcile,
+    # and whether each should be publicly resolvable at all is an OPERATOR decision per name, not a
+    # renderer one. They are enumerated in the shrink-only register in
+    # conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh, so the set
+    # cannot grow silently — adjudicate them there, then this filter becomes the right hardening.
+    #
+    # DELIBERATELY UNTOUCHED: `policy: sync`, `txtOwnerId`, `domainFilters`. Nothing above widens
+    # what this instance can see or own; per-resource exclusion only ever REMOVES an object from
+    # this instance's view, so it cannot make prod capable of deleting a record it does not own.
+    # ---------------------------------------------------------------------------------------------
+    # THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
+    #
+    # The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
+    # that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
+    # actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
+    # this change, for two independent reasons:
+    #
+    #   1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
+    #      (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
+    #      points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
+    #      certificate. Whether prod's public services belong behind it, behind a new forward, or
+    #      are not meant to be public at all is an operator decision. Breaking five services' DNS
+    #      on a guess is worse than leaving a known-wrong-but-stable state.
+    #   2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
+    #      real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
+    #      GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
+    #      gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
+    #      no per-service target override while every public service shares the `shared` Gateway.
+    #      Annotating the Gateway would move EVERY gateway-sourced record, which after this change
+    #      still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
+    #      routable address for admin UIs. That is strictly worse than the status quo.
+    #
+    # SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
+    # is answered, because a knob that cannot take effect invites someone to set it and expect one:
+    #   (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
+    #       port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
+    #   (b) HOW should they be separated from the non-public names that share the Gateway? Options,
+    #       roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
+    #       annotation, with only the public services' routes attached; or publishing the five names
+    #       from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
+    #       which restores per-service targeting without a second Gateway.
+    # Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
+    # ---------------------------------------------------------------------------------------------
     # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
     #
     # On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran

platform/longhorn/helmrelease.yaml

@@ -76,15 +76,104 @@ spec:
       # never blocks scheduling).
       replicaAutoBalance: "best-effort"
 
-      # --- rebuild I/O throttle for slow disks (etcd WAL fsync protection) ---
-      # The k3s VM disks sit on HDDs, so a full-replica rebuild's sequential I/O can saturate the
-      # spindle and starve etcd's WAL fsync — which crash-loops etcd and flaps the kube-vip VIP.
-      # Cap Longhorn to rebuilding ONE replica per node at a time so a node bounce / replica
-      # replenishment can never storm the HDDs. Deliberately conservative for this slow-disk cluster
-      # (Longhorn's chart default is higher). Pair with replica-replenishment-wait-interval=600
-      # (Longhorn default) so brief node bounces don't trigger replenishment at all.
+      # --- I/O THROTTLES: bound every Longhorn write source that competes with etcd's WAL fsync ---
+      # WHY THIS BLOCK IS BIGGER THAN THE ONE LINE IT REPLACES. On 2026-09-07 k3s-prod-1 went
+      # NotReady ("Kubelet stopped posting node status") with the host up and SSH-able: its embedded
+      # etcd could not finish a WAL fdatasync (took=3.56s/6.71s/4.95s/3.40s against a 1 s
+      # expectation), "apply request took too long took=8.30s", then "etcdserver: too many
+      # requests". /proc/pressure/io FULL avg300 read 11.36 there against 0.75 (node 2, NVMe) and
+      # 1.29 (node 3). Cause: EVERY prod node puts /data/k3s (etcd WAL + containerd) and
+      # /data/longhorn (every replica) on ONE partition, and node 1's was a 2011 consumer SATA SSD
+      # with 61,440 reallocated sectors (infra/PROD-PROVISIONING.md §2 and §0c).
+      #
+      # THE HONEST HEADLINE: the rebuild path was ALREADY throttled and it is not what this block
+      # fixes. concurrentReplicaRebuildPerNodeLimit was 1 during that outage and it HELD — the 18
+      # volumes rebuilt one at a time — and the node stalled anyway. Longhorn offers no further
+      # rebuild lever on the v1 data engine: replicaRebuildingBandwidthLimit is V2/SPDK ONLY
+      # (verified live 2026-09-13, the Setting reads a v2-only map with no v1 key) and there is no
+      # rebuild-concurrent-sync-limit Setting on 1.12.1 at all (NotFound). What is bounded below is
+      # therefore every OTHER Longhorn write source, each of which measured 5 per node.
+      #
+      # WHAT THIS BLOCK CANNOT DO. Node 1's drive HAS since been replaced (§0c), and THAT is the
+      # real mitigation: under the same load class — an 18-replica rebuild with node 1 as the write
+      # target — IO PSI full fell 11.36 -> 3.79 and the node stayed Ready. These settings only bound
+      # how much competing I/O Longhorn is ALLOWED to generate. etcd and Longhorn still share one
+      # ext4 filesystem, and no scheduler or cgroup control arbitrates the jbd2 journal that
+      # fdatasync waits on. 3.79 against the 5% NodeIOPressureHigh threshold is about 1.2 points of
+      # headroom, which is exactly why a SECOND concurrent heavy writer is still worth refusing.
+      #
+      # EVERY KEY BELOW IS PROVEN TO LAND, not assumed. `helm template longhorn/longhorn --version
+      # 1.12.1` (the version this chart range resolves to, per the HelmRelease's own
+      # lastAttemptedRevision) emits each one into the rendered longhorn-default-setting ConfigMap
+      # in kebab-case, and a live Setting of that name exists for each. This matters because the
+      # chart DROPS unknown defaultSettings keys with no error — see the orphanAutoDeletion note
+      # further down for the setting that was inert here for two minor versions because nobody
+      # checked the render.
+
+      # Rebuild ONE replica per node at a time. Unchanged, and load-bearing: without it a node
+      # bounce replenishes every degraded volume at once. The freeze/thaw to "0"/"1" during an
+      # operator bounce (runbooks/longhorn-degraded.md, docs/plans/cluster-upgrade-plan.md) patches
+      # the Setting directly — deliberate and temporary; this line is the value it returns to.
       concurrentReplicaRebuildPerNodeLimit: 1
 
+      # Seconds Longhorn waits before replenishing a replica on a node that went away. This is
+      # Longhorn's own default and the live Setting already reads 600 — PINNED, not changed, so no
+      # future chart or manager default can silently shorten the window in which a node can be
+      # recovered by hand without triggering a rebuild storm.
+      replicaReplenishmentWaitInterval: 600
+
+      # WAS 5, MEASURED LIVE ON PROD 2026-09-13. Restores read from the backup target and write
+      # whole volumes to the local disk — the heaviest sustained write Longhorn can produce. 5 means
+      # five parallel full-volume writes onto the partition etcd is fsyncing to.
+      # THE COST OF 1, STATED HONESTLY: today it is zero. Longhorn has NO backup target configured
+      # (the backup-target Setting does not even exist on the live cluster — it is still the
+      # WS-STORAGE Phase 4 TODO at the foot of this file), and velero's nightly run is NOT a
+      # Longhorn backup: its schedules set snapshotVolumes:false with defaultVolumesToFsBackup:true,
+      # i.e. kopia file-system backup through velero's own node-agent, which this Setting does not
+      # gate. CNPG base backups are barman/Postgres-level and are not gated either. The only
+      # consumer is an R-STATE-03 DR restore of ~24 volumes, where serialising is the correct trade
+      # against losing etcd mid-restore — and an operator who needs restore throughput can raise the
+      # Setting for the duration and watch NodeIOPressureHigh. REVISIT when a Longhorn backup target
+      # actually lands: from that day this knob starts costing real recovery time.
+      concurrentVolumeBackupRestorePerNodeLimit: 1
+
+      # WAS 5, MEASURED LIVE ON PROD 2026-09-13. Snapshot "heavy tasks" are purge and clone: full
+      # reads and rewrites of snapshot disk files.
+      # WHY 1 AND NOT 2 — this balance was judged, not copied. The work that generates purges here
+      # is rebuild cleanup: auto-cleanup-system-generated-snapshot is true, there are NO
+      # RecurringJobs on this cluster, and the 24 volumes hold 24 snapshots between them. Purge
+      # therefore arrives at the rate rebuilds retire snapshots, which is ALREADY serialised to 1
+      # per node — so a purge concurrency of 1 keeps pace with its own generator. 2 would only drain
+      # an existing backlog faster, at the price of a second heavy writer on the disk during
+      # precisely the rebuild that starved etcd once, with about 1.2 points of measured PSI headroom
+      # to spend.
+      # THE COST OF 1: space is reclaimed serially after a rebuild storm. That is a REAL cost here —
+      # a 37.7 GiB system snapshot from prod-1's rebuild once pinned a volume at 97.3% ALLOCATED —
+      # but it is the cost that is already instrumented (LonghornDiskAllocationHigh,
+      # LonghornDiskUnschedulable) and recoverable by hand, where a wedged etcd is neither.
+      snapshotHeavyTaskConcurrentLimit: 1
+
+      # Worker threads WITHIN one backup / one restore. Both sit at Longhorn's default of 2; PINNED
+      # rather than lowered — combined with the per-node limit of 1 above, one volume at a time with
+      # two threads is already conservative, and pinning stops an upstream bump from widening it.
+      backupConcurrentLimit: 2
+      restoreConcurrentLimit: 2
+
+      # Automatic engine upgrades roll every volume's engine after a Longhorn minor bump — a
+      # cluster-wide restart-and-resync storm on the same shared disk. 0 disables it, making engine
+      # upgrade a deliberate operator action, which matches this HelmRelease's own
+      # remediateLastFailure:false posture on minor upgrades. It is Longhorn's default; PINNED so
+      # both halves of that one decision live in one file.
+      concurrentAutomaticEngineUpgradePerNodeLimit: 0
+
+      # Never drain a node that holds the last healthy replica of a volume. This was ALREADY the
+      # live value on 2026-09-13 — but only because it is Longhorn's own default: it appeared in NO
+      # manifest in this repo and in NO rendered ConfigMap, so an upstream default change would have
+      # moved it silently. PINNED, value unchanged. runbooks/longhorn-degraded.md and
+      # docs/plans/cluster-upgrade-plan.md both reason about drain safety, so it has to be a
+      # declaration rather than a coincidence.
+      nodeDrainPolicy: "block-if-contains-last-replica"
+
       # --- capacity guardrails (docs/04 "reserve disk headroom") ---
       # Reserve 20% of the dedicated data disk so a node never fills to 100% (which wedges the
       # kubelet and Longhorn alike). Dev disks are small, so headroom matters.
@@ -99,13 +188,28 @@ spec:
       # --- misc dev-appropriate defaults ---
       # Keep replicas spread for durability rather than co-locating with the pod.
       defaultDataLocality: "disabled"
-      # Reclaim replica space of deleted/failed volumes promptly in dev.
-      orphanAutoDeletion: true
+      # `orphanAutoDeletion: true` USED TO BE HERE AND WAS INERT. Removed 2026-09-13, having been
+      # verified dead two independent ways against the DEPLOYED 1.12.1:
+      #   1. RENDER: `helm template longhorn/longhorn --version 1.12.1` with that key set emits NO
+      #      orphan key whatsoever into longhorn-default-setting. Longhorn 1.9 replaced
+      #      orphanAutoDeletion (a bool) with orphanResourceAutoDeletion (a semicolon-separated list
+      #      such as "replica-data;instance"), and the chart template drops unknown defaultSettings
+      #      keys with no error.
+      #   2. LIVE, from the other end: the rendered longhorn-default-setting ConfigMap in
+      #      longhorn-system carries no orphan key at all, `get setting orphan-auto-deletion` is
+      #      NotFound, and `get setting orphan-resource-auto-deletion` reads "" — Longhorn's default.
+      # DELIBERATELY NOT REPLACED with orphanResourceAutoDeletion: orphan deletion is extra write
+      # work on the one partition that is this cluster's constraint (see the I/O block above) and it
+      # buys only reclaimed space, of which prod has hundreds of GB free. Leaving it unset keeps
+      # Longhorn's default of not auto-deleting, which is the conservative side here — orphan
+      # cleanup stays manual, as runbooks/longhorn-capacity.md and runbooks/node-disk.md say.
+      # Left as a comment rather than deleted silently, because "config that looks applied and is
+      # not" is the failure this repo keeps re-learning, and two runbooks cite this key by name.
 
     # --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ----------
-    # platform/monitoring/alerts.yaml ships LonghornVolumeDegraded (critical),
-    # LonghornVolumeFaulted (critical), LonghornVolumeCapacityHigh and LonghornNodeStorageHigh —
-    # and NONE of them could ever fire, because the chart defaults
+    # platform/monitoring/alerts.yaml ships the platform-health.longhorn group (as of 2026-09-13:
+    # LonghornVolumeDegraded, LonghornVolumeFaulted, LonghornDiskAllocationHigh and
+    # LonghornDiskUnschedulable) — and NONE of them could ever fire, because the chart defaults
     # `metrics.serviceMonitor.enabled: false` and no ServiceMonitor existed in ns longhorn-system.
     # VERIFIED live on prod 2026-09-13: `count by (job) (up)` listed 19 jobs and none was Longhorn;
     # `count(longhorn_volume_robustness)` and `count({__name__=~"longhorn.*"})` both returned NOTHING.
@@ -140,12 +244,26 @@ spec:
     # colliding with an unowned object) with this file's networkpolicy-metrics.yaml already in place:
     #   count(up{job="longhorn-backend"})              => 3      (all three managers, up==1)
     #   count({__name__=~"longhorn.*"})                => 8717   (was EMPTY)
-    #   count(longhorn_volume_robustness)              => 44
-    #   count_values("v", longhorn_volume_robustness)  => 24 healthy(1), 72 unknown(0, detached)
-    # and the four alert rules went from "no series, cannot fire" to evaluating for real — one of
-    # them immediately found a true positive (LonghornVolumeCapacityHigh on the Prometheus TSDB
-    # volume at 97.1% ALLOCATED while its filesystem is only 20.4% used: Longhorn is not reclaiming
-    # freed blocks. Reported, NOT silenced by moving the threshold).
+    #   count(longhorn_volume_robustness)              => 96     (24 volumes x 4 `state` labels)
+    #   count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
+    # CORRECTION, re-measured 2026-09-13 later the same day: the line originally recorded here read
+    # "44 series … 24 healthy(1), 72 unknown(0, detached)". That was a MISREADING of a one-hot
+    # metric. `longhorn_volume_robustness` is NOT a 0-3 enum on longhorn 1.12.1; it carries a
+    # `state` label (healthy/degraded/faulted/unknown) and its value is only ever 0 or 1. The 72
+    # zeros were the other three state labels of the SAME 24 healthy volumes, not 72 detached
+    # volumes. The consequence was that enabling this scrape did NOT un-inert
+    # LonghornVolumeDegraded/Faulted: their `== 2` / `== 3` expressions still matched no sample.
+    # Fixed in platform/monitoring/alerts.yaml and pinned by
+    # platform/monitoring/tests/longhorn.test.yaml.
+    #
+    # Of the rules that WERE evaluating for real, one immediately found a true positive and one
+    # permanently-firing defect in the same reading: LonghornVolumeCapacityHigh went pending on the
+    # Prometheus TSDB volume at 97.3% ALLOCATED while its filesystem was 20.8% used. Longhorn does
+    # not reclaim freed blocks without discard/TRIM, and a 37.7 GiB system snapshot from prod-1's
+    # replica rebuild was pinning most of it. That rule has since been DELETED (not re-thresholded):
+    # it measured allocation and called it capacity, the filesystem question is already answered by
+    # kubelet_volume_stats_*, and the allocation question now lives per disk as
+    # LonghornDiskAllocationHigh / LonghornDiskUnschedulable.
     metrics:
       serviceMonitor:
         enabled: true

platform/longhorn/values.yaml

@@ -15,12 +15,30 @@
 #   replicaSoftAntiAffinity: false                   # HARD anti-affinity: one replica per node
 #   replicaZoneSoftAntiAffinity: true                # single zone (3 mini-PCs, no failure zones)
 #   replicaAutoBalance: best-effort                  # rebalance toward one-per-node when a node returns
-#   concurrentReplicaRebuildPerNodeLimit: 1          # throttle rebuild I/O so it can't starve etcd WAL fsync (HDD-backed)
+#
+#   # --- I/O throttles: keep Longhorn from starving etcd's WAL fsync. The etcd WAL and every
+#   # Longhorn replica share ONE partition on every prod node (PROD-PROVISIONING.md §2); see
+#   # helmrelease.yaml for the 2026-09-07 incident, the post-drive-replacement measurement, and the
+#   # cost of each value. The two marked "was 5" were measured at 5 on prod on 2026-09-13 while a
+#   # runbook claimed they were 1 — nothing in this repo declared them. ---
+#   concurrentReplicaRebuildPerNodeLimit: 1          # one replica rebuild per node at a time
+#   replicaReplenishmentWaitInterval: 600            # PINNED at Longhorn's default, not changed
+#   concurrentVolumeBackupRestorePerNodeLimit: 1     # was 5: five parallel full-volume restores
+#   snapshotHeavyTaskConcurrentLimit: 1              # was 5: snapshot purge/clone rewrites
+#   backupConcurrentLimit: 2                         # PINNED at Longhorn's default (threads per backup)
+#   restoreConcurrentLimit: 2                        # PINNED at Longhorn's default (threads per restore)
+#   concurrentAutomaticEngineUpgradePerNodeLimit: 0  # PINNED: engine upgrades are operator-driven
+#   nodeDrainPolicy: block-if-contains-last-replica  # PINNED: was live only as Longhorn's default
+#   # NOT AVAILABLE TO US, do not add: replicaRebuildingBandwidthLimit is v2-data-engine only, and
+#   # no rebuild-concurrent-sync-limit Setting exists on 1.12.1 at all.
+#
 #   storageReservedPercentageForDefaultDisk: "20"    # reserve disk headroom (docs/04 guardrail)
 #   storageMinimalAvailablePercentage: "25"          # stop scheduling below 25% free
 #   storageOverProvisioningPercentage: "100"         # dev: no thin over-provisioning (1:1)
 #   defaultDataLocality: disabled                    # keep replicas spread for durability
-#   orphanAutoDeletion: true                         # reclaim orphaned replica space promptly
+#   # orphanAutoDeletion — REMOVED 2026-09-13: the key does not exist in the 1.12 chart (renamed
+#   # orphanResourceAutoDeletion, now a list) and was being silently dropped from the render.
+#   # Deliberately NOT replaced; orphan cleanup stays manual. helmrelease.yaml carries the proof.
 #   backupTarget: <off-cluster>                      # TODO(WS-STORAGE, Phase 4): DR target (ADR-0003)
 #
 # metrics:

platform/monitoring/alerts.yaml

@@ -131,6 +131,478 @@ spec:
             description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} — the level at which k3s-prod-1 stalled etcd's WAL fsync and went NotReady on 2026-09-07 while the host stayed up. Expect etcd 'slow fdatasync' / 'apply request took too long' in `journalctl -u k3s`. Stop or throttle whatever is writing (Longhorn rebuild, backup, restore) before the node drops."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
 
+    # =========================================================================================
+    # I/O PRESSURE — CAUSE ATTRIBUTION (added 2026-09-14 after the second prod-1 I/O event)
+    # =========================================================================================
+    # WHY THIS GROUP EXISTS. `NodeIOPressureCritical` above is a good detector and a useless
+    # diagnosis. It fired on k3s-prod-1 at 02:40-02:45Z on 2026-09-14 with PSI(full) peaking at
+    # 20.98% — twice the critical threshold, four times the 2026-09-07 level — and the page said
+    # only "I/O PSI above 10%". Establishing that this was NOT a failing drive took eight ad-hoc
+    # PromQL queries after the fact, by which time the event had recovered unattended. Measured
+    # during that spike on prod-1:
+    #   writes            sda 3.5 MB/s, sdi 2.9 MB/s   (trivial)
+    #   reads             ~0 from local devices
+    #   device busy       rate(node_disk_io_time_seconds_total[5m])  sdi 0.21, sda 0.09  (idle)
+    #   write service     1.4-4.8 ms per op                          (healthy)
+    #   NFS retransmits   0
+    #   Longhorn degraded 0 of 24 volumes
+    #   NFS reads         114 ops/s on prod-1, 0 on prod-2/3         <- the one thing that moved
+    # Compare 2026-09-07, the incident the PSI rules were written for, re-queried from the same
+    # TSDB at offset 164h: PSI(full) 73.97% on prod-1 AND per-write service time on prod-1's dying
+    # 2011 SSD at 1,039 ms, against 1.1 ms and 4.3 ms on nodes 3 and 2. Two events, one alert,
+    # opposite causes, and nothing on the page to tell them apart.
+    #
+    # PSI(full) FOR I/O COUNTS NFS STALLS AS I/O STALLS. That single fact explains why tonight's
+    # local-device metrics all looked innocent, and it is why "the disks are fine" and "the node
+    # is not stalled" are different statements on this cluster. Five in-tree NFS PVs on
+    # 192.168.2.102 (/mnt/Storage/plex, /mnt/Storage/nextcloud and the arr/qbittorrent shares)
+    # carry the whole media library, so the block-layer view cannot explain a stall on its own.
+    #
+    # DESIGN. One rule per CAUSE, each naming its cause in summary/description, plus one rule
+    # (NodeIOPressureNotLocalDisk) that fires on the ABSENCE of every local-disk cause — that
+    # inference was the entire content of tonight's eight queries. Every threshold below is
+    # measured on prod over 24-48h, never guessed, and each rule records its headroom. The PSI
+    # thresholds themselves are NOT changed: tonight's 21% stall was real and quieting it would
+    # remove the only signal that reached the operator at all. See runbooks/node-io.md.
+    #
+    # ONE PROMQL TRAP, RECORDED BECAUSE THE FIRST DRAFT OF THREE OF THESE RULES FELL INTO IT.
+    # A per-operation latency rule wants both a ratio threshold and an op-rate guard, and the
+    # natural-reading form is wrong:
+    #     (rate(time) / rate(ops)) and rate(ops) > 10 > 0.020      # BROKEN
+    # `>` binds tighter than `and`, so that parses as `ratio and ((rate(ops) > 10) > 0.020)`: the
+    # right operand is just "devices doing more than 10 ops/s", 0.020 is compared against the OP
+    # RATE, and the threshold never touches the ratio at all. The rule then fires on every busy
+    # device, on a healthy cluster. The correct shape, used throughout below, binds the threshold to
+    # the ratio and makes the guard its own operand:
+    #     (rate(time) / rate(ops)) > 0.020 and rate(ops) > 10      # CORRECT
+    # platform/monitoring/tests/node-io-cause.test.yaml asserts BOTH forms on the healthy fixture —
+    # the broken one returning series, the shipped one returning none — so the mistake cannot
+    # silently come back.
+    - name: platform-health.node-io-cause
+      rules:
+        # ---- CAUSE 1: the device is queue-saturated ------------------------------------------
+        # 48h measured maxima per node: prod-1 0.610, prod-3 0.295, prod-2 0.217. At the
+        # 2026-09-07 window prod-1 peaked 0.788 (sdb) / 0.745 (sda). Tonight's spike: 0.271. So
+        # 0.85 for 10m sits 1.4x above the busiest thing this cluster does in normal operation and
+        # above BOTH incidents — which is the point: when this rule is silent, saturation is
+        # excluded. Its true positive is a Longhorn replica-rebuild storm (all 18 volumes rebuild
+        # after any reboot), which does pin a device for many minutes and is expected, not a fault.
+        - alert: NodeDiskDeviceSaturated
+          expr: rate(node_disk_io_time_seconds_total[5m]) > 0.85
+          for: 10m
+          labels:
+            severity: warning
+            io_cause: device-saturation
+          annotations:
+            summary: "CAUSE device saturation: {{ $labels.device }} on {{ $labels.instance }} is busy >85% of the time"
+            description: "Block device {{ $labels.device }} on {{ $labels.instance }} has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that."
+            diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        # ---- CAUSE 2: the device is slow per operation (the drive-is-dying signal) ------------
+        # THIS IS THE RULE THAT WOULD HAVE CALLED 2026-09-07 EARLY, and the one the stack did not
+        # have. Guarded at >10 writes/s so a device doing three writes a minute cannot manufacture
+        # a 200 ms "average" out of one unlucky operation — and the guard also self-scopes the rule
+        # to the devices that matter: only five device-instances on this cluster ever sustain 10
+        # writes/s (prod-1 sda+sdi, prod-2 nvme0n1, prod-3 sda+sdc), which are exactly the shared
+        # etcd-WAL/Longhorn partitions. Guarded maxima over 48h: 3.1 ms cluster-wide. At the
+        # 2026-09-07 window prod-1's sda measured 1,039 ms. 20 ms is 6.5x the healthy ceiling.
+        - alert: NodeDiskWriteLatencyHigh
+          expr: |
+            (
+              rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])
+            ) > 0.020
+              and rate(node_disk_writes_completed_total[5m]) > 10
+          for: 10m
+          labels:
+            severity: warning
+            io_cause: device-latency
+          annotations:
+            summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 20ms per write"
+            description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 20ms for 10m while the device was doing more than 10 writes/s. Healthy on this cluster is 1-3ms (48h maximum 3.1ms). This is the LATENCY class: the device itself is slow, not merely busy. Check SMART for reallocated and pending sectors — prod-1's original 2011 C400 had 61,440 reallocated and measured 1,039ms per write on 2026-09-07 before the node went NotReady. If SMART is clean, suspect a write storm queueing on the shared partition ahead of etcd's fsync."
+            diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: sudo smartctl -a /dev/{{ $labels.device }} | grep -Ei 'Reallocated|Pending|Uncorrectable|Wear'"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        - alert: NodeDiskWriteLatencyCritical
+          # 50 ms sustained 5m. etcd expects fdatasync inside 1s and logs 'slow fdatasync' well
+          # before that; a device averaging 50 ms per write cannot keep an etcd WAL healthy under
+          # load. At the 2026-09-07 window the measured value was 1,039 ms — this rule fires 20x
+          # over, and it names the device, which the PSI alert never could.
+          expr: |
+            (
+              rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])
+            ) > 0.050
+              and rate(node_disk_writes_completed_total[5m]) > 10
+          for: 5m
+          labels:
+            severity: critical
+            io_cause: device-latency
+          annotations:
+            summary: "CAUSE device latency CRITICAL: {{ $labels.device }} on {{ $labels.instance }} is taking over 50ms per write — etcd is at risk"
+            description: "Average write service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 50ms for 5m under real write load. On these nodes the etcd WAL and every Longhorn replica share one partition, so a device this slow starves etcd's fdatasync and the node leaves the cluster while the host stays up, pingable and SSH-able — exactly the 2026-09-07 failure, where this same figure measured 1,039ms. Treat as HARDWARE until SMART says otherwise. With one etcd member already impaired, losing a second ends quorum: do not reboot or drain another node."
+            diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took'"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        # ---- CAUSE 3: read service time ------------------------------------------------------
+        # Deliberately a far higher threshold than the write rules, for a measured reason: prod-1's
+        # sdg (a spinning media disk) legitimately sustains 156 ms average read service time over
+        # 5m windows while serving media, and ten of the fifteen block devices on that node are
+        # spinning disks. A 20 ms read rule would be permanently firing on a perfectly healthy
+        # cluster. 250 ms is 1.6x the measured HDD ceiling, so this rule means "a device is failing
+        # or resetting", never "a device is a hard disk". The SSD/NVMe devices read in 1.6-6.3 ms.
+        - alert: NodeDiskReadLatencyHigh
+          expr: |
+            (
+              rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m])
+            ) > 0.250
+              and rate(node_disk_reads_completed_total[5m]) > 10
+          for: 15m
+          labels:
+            severity: warning
+            io_cause: device-latency
+          annotations:
+            summary: "CAUSE device latency: {{ $labels.device }} on {{ $labels.instance }} is taking over 250ms per read"
+            description: "Average read service time on {{ $labels.device }} of {{ $labels.instance }} has exceeded 250ms for 15m at more than 10 reads/s. The spinning media disks on these nodes peak at 156ms and the SSDs at 6ms in normal operation, so this threshold means a failing or resetting device, not a slow one. Check SMART and dmesg for ATA resets and link errors."
+            diagnostic_query: "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        # ---- CAUSE 4: NFS transport ----------------------------------------------------------
+        # node_nfs_* comes from /proc/net/rpc/nfs via node-exporter's `nfs` collector, which is ON
+        # BY DEFAULT and is confirmed present on prod (2026-09-14: node_nfs_rpcs_total on all three
+        # nodes — prod-3 589,638, prod-1 131,973, prod-2 2,527; the DaemonSet is hostNetwork so the
+        # counters are the host's, not the container's). Retransmissions measured exactly 0 across
+        # 48h on all three nodes, including throughout tonight's spike — so this rule is quiet
+        # today and any non-zero value is unambiguous.
+        # WHAT IT CAN AND CANNOT SEE. A retransmission means the client gave up waiting and resent,
+        # i.e. the server or the path to 192.168.2.102 dropped or timed out the request. It does
+        # NOT see an export that is merely SLOW. That needs per-mount RPC round-trip time from the
+        # `mountstats` collector, which is switched on in kube-prometheus-stack.yaml in this same
+        # change and has NO rule here yet, deliberately: the series do not exist on prod until that
+        # rolls out, and a rule against a metric that does not exist is the inert-alert failure
+        # this repo has already shipped twice. runbooks/node-io.md records the gap and the follow-up.
+        - alert: NodeNFSRetransmissionRateHigh
+          expr: |
+            (
+              sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m]))
+                / sum by (instance) (rate(node_nfs_rpcs_total[5m]))
+            ) > 0.01
+              and sum by (instance) (rate(node_nfs_rpcs_total[5m])) > 1
+          for: 10m
+          labels:
+            severity: warning
+            io_cause: nfs-transport
+          annotations:
+            summary: "CAUSE NFS transport: {{ $labels.instance }} is retransmitting over 1% of its NFS RPCs"
+            description: "More than 1% of this node's NFS client RPCs are being retransmitted, sustained for 10m at more than 1 RPC/s. The measured baseline on all three nodes is exactly ZERO over 48h, so any value here is real: the NFS server 192.168.2.102, or the path to it, is dropping or timing out requests. Because I/O PSI counts NFS stalls as I/O stalls, this is the cause class that makes a node look disk-starved while every local device is idle. Investigate the server and the link — not this node's disks."
+            diagnostic_query: "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m])) — and the op rate driving it: sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m]))"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        # ---- THE DISCRIMINATOR: stalled, and it is not the local disks ------------------------
+        # This is the rule that collapses tonight's eight queries into one line on the page. It
+        # fires when the node is stalled at the critical PSI level AND all three local-block-device
+        # causes above are absent.
+        # `unless` RATHER THAN `and ... < threshold`, AND THAT CHOICE IS LOAD-BEARING. The
+        # right-hand side of an `unless` that produces no series removes nothing, so a node whose
+        # devices are too idle to have a measurable per-op latency STILL raises this alert. The
+        # `and ... <` formulation would instead go silent exactly when the local disks are quietest
+        # — an alert that disappears in the state it was written to detect, which is precisely the
+        # structurally-inert failure mode this repo shipped in VeleroNoRecentBackup.
+        #   tonight 2026-09-14: PSI 0.2098, busiest device 0.271, writes 1.6ms, reads 156ms -> FIRES
+        #   2026-09-07:         PSI 0.7397 but writes 1,039ms -> the write-latency clause suppresses
+        #                       it and NodeDiskWriteLatencyCritical names the device instead
+        #   healthy baseline:   PSI never exceeded 0.0142 on any node outside the two events -> quiet
+        - alert: NodeIOPressureNotLocalDisk
+          expr: |
+            (
+              max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m])) > 0.10
+            )
+            unless (
+              max by (instance) (rate(node_disk_io_time_seconds_total[5m])) > 0.50
+            )
+            unless (
+              max by (instance) (
+                (rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]))
+                  and rate(node_disk_writes_completed_total[5m]) > 10
+              ) > 0.020
+            )
+            unless (
+              max by (instance) (
+                (rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]))
+                  and rate(node_disk_reads_completed_total[5m]) > 10
+              ) > 0.250
+            )
+          for: 5m
+          labels:
+            severity: critical
+            io_cause: not-local-disk
+          annotations:
+            summary: "CAUSE not local disk: {{ $labels.instance }} is I/O-stalled while every local device is idle and healthy"
+            description: "I/O PSI (full) above 10% for 5m on {{ $labels.instance }} (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)."
+            diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        # ---- CAUSE 5: the etcd storage path --------------------------------------------------
+        # THE CONSEQUENCE THE OPERATOR ACTUALLY CARES ABOUT IS A NotReady NODE, and etcd is what
+        # fails first when its device stalls. etcd's OWN disk metrics are NOT AVAILABLE here and
+        # this rule does not pretend otherwise: etcd_disk_wal_fsync_duration_seconds and
+        # etcd_disk_backend_commit_duration_seconds DO NOT EXIST in this TSDB — verified on prod
+        # 2026-09-14, where {__name__=~"etcd_disk_.*|etcd_server_.*|etcd_mvcc_.*"} returns nothing
+        # at all, because kubeEtcd is deliberately disabled above for k3s' embedded etcd.
+        # WHAT DOES EXIST is the apiserver's CLIENT-SIDE view of its own local etcd member,
+        # etcd_request_duration_seconds on job="apiserver", one series set per node — and on k3s
+        # each apiserver talks to the etcd inside its own process, so `instance` attributes the
+        # stall to a node. Written as the FRACTION of etcd requests slower than one second, read
+        # straight off the le="1.0" bucket rather than through histogram_quantile, so there is no
+        # interpolation between the 0.1 and 1.0 boundaries to argue about.
+        # MEASURED: 0 across 48h on all three instances. At the 2026-09-07 window (offset 164h):
+        # 0.0054 on prod-1, 0.0205 on prod-2, 0.0165 on prod-3. A 0.005 threshold therefore fires
+        # for that incident with infinite headroom over a healthy baseline of exactly zero.
+        # HONEST LIMIT: this moves for ANY etcd slowness, not only disk — leader election, CPU
+        # starvation and a saturated network raise it too. It says "the etcd path is stalling on
+        # this node", which is the urgency; the device rules above say whether the cause is disk.
+        - alert: NodeEtcdRequestsSlow
+          expr: |
+            (
+              1 - (
+                sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m]))
+                  / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m]))
+              )
+            ) > 0.005
+              and sum by (instance) (rate(etcd_request_duration_seconds_count{job="apiserver"}[5m])) > 0.1
+          for: 5m
+          labels:
+            severity: critical
+            io_cause: etcd
+          annotations:
+            summary: "CAUSE etcd: over 0.5% of etcd requests from {{ $labels.instance }} are taking more than a second"
+            description: "The apiserver on {{ $labels.instance }} is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum."
+            diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        # ---- WHO is stalled ------------------------------------------------------------------
+        # Node PSI says the machine stalled; it cannot say which workload. cAdvisor's per-cgroup
+        # PSI can, and it is ALREADY SCRAPED by the kubelet ServiceMonitor — verified on prod
+        # 2026-09-14: container_pressure_io_stalled_seconds_total, 405 series across the three
+        # nodes. During tonight's spike it read sonarr 0.268 and qbittorrent 0.070 on prod-1
+        # against under 0.013 everywhere else. One query named the workload, and sonarr is an NFS
+        # media consumer, which is the whole story of the event.
+        # THRESHOLD FROM THE MEASURED DISTRIBUTION: over 48h, minutes spent above 0.15 were sonarr
+        # 6, longhorn instance-manager on prod-1 6, prometheus 4, keycloak-operator 3. With
+        # `for: 5m` the first two raise a page during a real event and the other two never do.
+        # `container!=""` drops the pod-level duplicate cgroup series so a stall is reported once.
+        - alert: ContainerIOStallHigh
+          expr: rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]) > 0.15
+          for: 5m
+          labels:
+            severity: warning
+            io_cause: workload
+          annotations:
+            summary: "WHO is stalled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) on {{ $labels.node }} is blocked on I/O"
+            description: "Container {{ $labels.container }} in {{ $labels.namespace }}/{{ $labels.pod }} on {{ $labels.node }} spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210."
+            diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        # ---- CAUSE 6: the NFS export is SLOW (the gap CAUSE 4 left open, closed 2026-09-15) ----
+        # WHAT THIS CLOSES. NodeNFSRetransmissionRateHigh above sees a DROPPED or TIMED-OUT RPC and
+        # nothing else; the comment on it records the honest gap in as many words — "an NFS export
+        # that is merely SLOW cannot be distinguished today" — because `mountstats` was switched on
+        # in the same change and its series did not exist yet. They exist now. VERIFIED on prod
+        # 2026-09-15: all eight families referenced below return series (426 series each for the
+        # five per-operation families, 6 each for the transport families, 1,842 node_mountstats_*
+        # in total, about 0.4% of a 451,650-series head).
+        #
+        # THE DISCRIMINATOR, AND IT IS THE WHOLE POINT OF THESE RULES. mountstats separates the two
+        # halves of a slow NFS operation that every other metric in this repo conflates:
+        #   response_time  the interval the SERVER had the request — 192.168.2.102's problem
+        #   queue_time     the interval the request sat on THIS NODE before it could be sent,
+        #                  waiting for an RPC slot — our problem
+        #   request_time   the sum, i.e. what the application actually waited (not alerted on
+        #                  directly; it cannot tell you which half moved, which is the question)
+        # So: high response with low queue means the SERVER is slow. High queue means WE are
+        # saturating the transport. And high response with low queue AND a LOW op rate of our own
+        # means the server is slow for someone ELSE — which is the operator's standing hypothesis
+        # (the Proxmox-hosted copy of this media stack still has operational control of the same
+        # files on 192.168.2.102) turned into something a rule can assert. That hypothesis was
+        # WEAKENED by the 2026-09-14 event, where prod-1 was itself doing 114 NFS reads/s during
+        # the spike, so the rules below are built to TEST it, not to assume it: NFSExportServerSlow
+        # and NFSExportServerSlowNotOurLoad partition the same slow-server state by whether our own
+        # op rate accounts for it, and exactly one of the two fires.
+        #
+        # MEASURED BASELINE, prod 2026-09-15 (every number below re-measured for this change):
+        #   server response time   4.25 - 4.83 ms/op over 10m, uniform across all six mounts
+        #                          history maximum 4.73 / 5.17 / 5.57 / 6.80 ms/op
+        #   client queue time      0.014 - 0.091 ms/op; history maximum 0.333 ms/op
+        #   our own op rate        0.019 - 0.124 ops/s; history maximum 0.163 ops/s (prod-1 plex)
+        #   major timeouts         0 on every mount
+        #   backlog / sending / pending queue depth   0 on every mount
+        #
+        # THREE HONEST LIMITS ON THAT BASELINE, recorded because they bound how much the thresholds
+        # below are worth:
+        #
+        # (a) THE HISTORY IS MINUTES, NOT DAYS. Every other threshold in this group is cut from 24
+        #     to 48 hours of prod data. These are not. The mountstats series begin when the
+        #     node-exporter DaemonSet rolled, and at measurement time each mount held 18 samples —
+        #     `count_over_time(node_mountstats_nfs_transport_backlog_queue_total[24h])` returned 18,
+        #     not ~2,880, and `time() - process_start_time_seconds{job="node-exporter"}` returned
+        #     558-561 s on all three nodes. So "history maximum" above means a maximum over about
+        #     ten minutes. Nothing below uses a range longer than 15m for that reason: a rule with a
+        #     24h range would also be unreliable for a day after any node-exporter restart.
+        #
+        # (b) IT IS AN IDLE BASELINE, and the loaded one is unknown. The busiest operation on every
+        #     one of the six mounts is STATFS keepalive, at 0.002-0.046 ops/s; there was no READ or
+        #     WRITE traffic at all while these numbers were taken. A 4.3 ms cached STATFS is not the
+        #     cost of a 1 MB READ off a spinning array, so the response-time thresholds below are
+        #     argued from the LINK AND THE ARRAY, with the measured figure used only as a floor.
+        #     They have never been checked against a real media event. runbooks/node-io.md carries
+        #     the query to re-cut them after the first one, and that is follow-up work, not a
+        #     silent assumption.
+        #
+        # (c) SIX MOUNTS, FOUR INDEPENDENT SETS OF COUNTERS. prod-1's /mnt/Storage/plex and
+        #     /mnt/Storage/plex/downloads report byte-identical per-operation counters (op rate
+        #     0.12443893333333335 on both, to every digit), and so do prod-3's two nextcloud
+        #     mounts, because mounts sharing one NFS superblock share the kernel's RPC statistics.
+        #     A single slow export on prod-1 therefore raises TWO alerts differing only in
+        #     `export`. Left alone deliberately: aggregating the label away would hide which mount
+        #     paths are affected, which is the one thing per-export alerting is for, and
+        #     Alertmanager groups them on the page anyway.
+        #
+        # WHY `unless` AND NOT `and ... < threshold` for the "slow but not queueing" clause: the
+        # same reason NodeIOPressureNotLocalDisk gives above. An `unless` whose right-hand side
+        # produces no series removes nothing, so a mount too quiet to have a measurable queue time
+        # still raises the alert; `and queue < 0.005` would instead go silent exactly when the
+        # client side is quietest, which is the state the rule exists to report.
+        # AND THE PRECEDENCE TRAP, again: the guard is its own operand
+        # (`(ratio) > T and rate(ops) > N`), never `(ratio) and rate(ops) > N > T`, which compares
+        # T against the OP RATE. Both forms are asserted in the test file.
+        - alert: NFSExportServerSlow
+          # 50 ms/op is 10x the measured 4.25-4.83 ms mean and 7.4x the 6.80 ms measured maximum,
+          # but the multiple is not the argument (see limit (b)) — the physics is. One outstanding
+          # 1 MB NFS read at 50 ms is 20 MB/s, well under what this gigabit path and this array
+          # deliver, so a mount whose MEAN operation costs 50 ms for 15 minutes is not keeping up.
+          # OP-RATE GUARD >20/s, and it is doing two jobs. It keeps the mean honest (6,000 ops in a
+          # 5m window, so no single unlucky operation can manufacture an average — the artefact the
+          # device rules guard at >10/s for), and it is the load side of the partition with
+          # NFSExportServerSlowNotOurLoad below: above 20 ops/s our own workload plausibly explains
+          # a slow server. 20/s is 123x the busiest measured op rate (0.163/s) and under a fifth of
+          # the 114 NFS reads/s prod-1 itself sustained at the 2026-09-14 02:40Z spike (that figure
+          # is from node_nfs_requests_total, a whole-node family, so the comparison is approximate).
+          expr: |
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.050
+              and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 20
+            unless (
+              (
+                sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
+                  / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+              ) > 0.005
+            )
+          for: 15m
+          labels:
+            severity: warning
+            io_cause: nfs-server-slow
+          annotations:
+            summary: "CAUSE NFS server slow: {{ $labels.export }} is taking over 50ms per operation for {{ $labels.instance }}, under load this node is generating"
+            description: "The NFS server is spending more than 50ms per RPC on {{ $labels.export }} as seen from {{ $labels.instance }}, sustained 15m, while this node is driving more than 20 ops/s at it and its own CLIENT-side queue time is under 5ms per op. Measured healthy is 4.25-4.83ms per op on every mount. The time is on the SERVER, not in our transport — but our own op rate is high enough to explain it, so this reads as self-inflicted: a library scan, an import, a torrent verifying, a Nextcloud reindex. Because I/O PSI counts an NFS wait as an I/O stall, this is a cause class that makes a node look disk-starved while every local device is idle. Do NOT touch local disks. Find the workload with ContainerIOStallHigh and slow it down. If instead our op rate were LOW, NFSExportServerSlowNotOurLoad would be firing and the reading would be the opposite one."
+            diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — split it by operation to see whether metadata or data is slow: sum by (export, operation) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (export, operation) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 0"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        - alert: NFSExportServerSlowNotOurLoad
+          # THE RULE THE OPERATOR ACTUALLY WANTS, and the reason this change exists: the same slow
+          # server, with our own demand too small to account for it. Same 50 ms/op threshold as
+          # NFSExportServerSlow so the two are one statement split by ONE variable — our op rate —
+          # and exactly one of them can fire: this one requires >1 op/s and excludes >20 op/s,
+          # the other requires >20 op/s. No gap between them, no overlap.
+          # >1 op/s is the statistical floor, not a load judgement: 300 operations in a 5m window,
+          # and 6x the busiest op rate ever measured on these mounts (0.163/s, where the top
+          # operation is STATFS keepalive). Below it the mount is idle and cannot stall a node.
+          # `unless` on BOTH exclusions, per the note above: if the op-rate or queue-time series is
+          # missing the alert still fires, which is the safe direction for a rule whose whole
+          # purpose is to report "the load is not ours".
+          expr: |
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.050
+              and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
+            unless (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 20
+            )
+            unless (
+              (
+                sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
+                  / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+              ) > 0.005
+            )
+          for: 15m
+          labels:
+            severity: warning
+            io_cause: nfs-server-slow-external
+          annotations:
+            summary: "CAUSE NFS server slow and the load is NOT OURS: {{ $labels.export }} takes over 50ms per operation while {{ $labels.instance }} asks for under 20 ops/s"
+            description: "{{ $labels.export }} is answering {{ $labels.instance }} in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
+            diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        - alert: NFSClientQueueTimeHigh
+          # THE OTHER HALF OF THE DISCRIMINATOR: the request had not even been sent yet. queue_time
+          # is the interval an RPC waits on this node for a transport slot, so it is OUR saturation,
+          # not the server's.
+          # 5 ms/op is 15x the 0.333 ms history maximum and 55-360x the 0.014-0.091 ms typical, but
+          # the anchor that matters is physical: 5 ms is longer than the server's own measured
+          # answer time (4.25-4.83 ms). A client that waits longer for a slot than the server takes
+          # to answer is unambiguously the bottleneck itself.
+          expr: |
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.005
+              and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
+          for: 10m
+          labels:
+            severity: warning
+            io_cause: nfs-client-backed-up
+          annotations:
+            summary: "CAUSE we are backed up, not the server: {{ $labels.instance }} queues over 5ms per operation before it can send to {{ $labels.export }}"
+            description: "On {{ $labels.instance }} NFS requests to {{ $labels.export }} are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
+            diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        - alert: NFSTransportBacklogDeep
+          # backlog_queue_total is a CUMULATIVE SUM OF QUEUE LENGTH, one sample added per send, so
+          # rate(backlog) / rate(sends) is the mean number of requests sitting in the backlog queue
+          # per send — a depth, not a rate. Measured 0 on all six mounts.
+          # WHY 8 AND NOT 1: this cluster's mounts use a DYNAMIC RPC slot table, and the high-water
+          # marks read 586 on prod-1, 162 on prod-3 and 2 on prod-2 — the client grows the table
+          # rather than queueing, which is why the backlog is 0 and why NFSClientQueueTimeHigh
+          # above, not this rule, is the primary client-saturation signal. A backlog forms only
+          # when even the grown table is full. 8 rather than 1 so a single sampling artefact
+          # cannot page, and it is still 8 deep against a measured floor of zero.
+          expr: |
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m]))
+            ) > 8
+              and sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) > 1
+          for: 10m
+          labels:
+            severity: warning
+            io_cause: nfs-client-backed-up
+          annotations:
+            summary: "CAUSE NFS transport backlog: {{ $labels.instance }} has over 8 requests queued behind a full slot table for {{ $labels.export }}"
+            description: "The RPC backlog queue for {{ $labels.export }} on {{ $labels.instance }} is averaging more than 8 waiting requests per send, for 10m. Measured healthy is exactly 0 on all six mounts. These mounts grow their slot table on demand (high-water 586 on prod-1, 162 on prod-3, 2 on prod-2), so a persistent backlog means even the grown table is full: this node is asking for far more concurrent NFS I/O than the transport can carry. Same remediation as NFSClientQueueTimeHigh — reduce concurrency on this node. This is a client-side ceiling, NOT a statement about the server."
+            diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) — and the two neighbouring depths, sending and pending, derived the same way; pending high with backlog low means the requests are on the wire and the server is holding them."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        - alert: NFSExportMajorTimeouts
+          # CRITICAL, AND THE ONLY THRESHOLD HERE THAT NEEDS NO MULTIPLE. A major timeout is the
+          # client having exhausted the FULL RPC retransmission backoff on one request without an
+          # answer — tens of seconds of a process blocked in uninterruptible sleep, and the point
+          # at which an NFS mount starts to look hung rather than slow. Measured 0 on every mount,
+          # so any non-zero value is unambiguous and "> 0" is the honest threshold.
+          # 15m range, not 24h, per limit (a) above. `for: 5m` is satisfiable because a single
+          # increment keeps increase() positive for the whole 15m window.
+          expr: sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0
+          for: 5m
+          labels:
+            severity: critical
+            io_cause: nfs-export-failing
+          annotations:
+            summary: "CAUSE NFS export failing: {{ $labels.export }} has timed out RPCs from {{ $labels.instance }} — the mount is hanging, not just slow"
+            description: "{{ $labels.instance }} has recorded NFS major timeouts against {{ $labels.export }} in the last 15m: requests that went unanswered through the entire retransmission backoff. Measured baseline is 0 on all six mounts, so this is never noise. Every process touching that mount is now blocked in uninterruptible sleep, which will show as I/O PSI on the node and as container stalls on the pods — and unlike the latency rules above this one means the export is effectively DOWN for those requests, not slow. Check 192.168.2.102 itself (is it up, is nfsd running, is its array degraded) and the path to it. Pair with NodeNFSRetransmissionRateHigh, which sees the retries that lead here. Do not reboot cluster nodes to clear a hung mount while volumes are not healthy."
+            diagnostic_query: "sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) — which operations: sum by (export, operation) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0 — and the retry that precedes a timeout: sum by (instance, export) (rate(node_mountstats_nfs_operations_transmissions_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+
+
 # =====================================================================================
 # Storage — Longhorn
 # =====================================================================================
@@ -147,9 +619,33 @@ spec:
   groups:
     - name: platform-health.longhorn
       rules:
-        # longhorn_volume_robustness: 0=unknown 1=healthy 2=degraded 3=faulted
+        # -------------------------------------------------------------------------------------
+        # `longhorn_volume_robustness` IS A STATE-LABELLED GAUGE, NOT A 0-3 ENUM (fixed 2026-09-13)
+        #
+        # Both rules below used to read `longhorn_volume_robustness == 2` / `== 3`, with the comment
+        # "0=unknown 1=healthy 2=degraded 3=faulted". That is the shape of an OLDER Longhorn. On
+        # longhorn 1.12.1 (the version this cluster runs) the metric is ONE-HOT over a `state`
+        # label and its value is only ever 0 or 1:
+        #     longhorn_volume_robustness{volume="…",state="healthy"}   1
+        #     longhorn_volume_robustness{volume="…",state="degraded"}  0
+        #     longhorn_volume_robustness{volume="…",state="faulted"}   0
+        #     longhorn_volume_robustness{volume="…",state="unknown"}   0
+        # MEASURED on prod 2026-09-13, scrape working, 24 volumes attached:
+        #     count(longhorn_volume_robustness)                  => 96   (24 volumes x 4 states)
+        #     count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
+        #     count(longhorn_volume_robustness == 2)             => (empty)
+        #     count(longhorn_volume_robustness == 3)             => (empty)
+        # So `== 2` and `== 3` matched NO sample. Enabling the Longhorn ServiceMonitor earlier today
+        # made the series exist; it did NOT make these two rules able to fire — a faulted volume
+        # would still have alerted nobody. The "44 series / 24 healthy(1), 72 unknown(0, detached)"
+        # reading recorded in platform/longhorn/helmrelease.yaml is the same misreading: those
+        # zeros are the OTHER state labels of healthy volumes, not detached volumes.
+        #
+        # Pinned in both directions by platform/monitoring/tests/longhorn.test.yaml, which keeps a
+        # regression case asserting the old `== 2` form matches nothing on this metric shape.
+        # -------------------------------------------------------------------------------------
         - alert: LonghornVolumeDegraded
-          expr: longhorn_volume_robustness == 2
+          expr: longhorn_volume_robustness{state="degraded"} == 1
           for: 10m
           labels:
             severity: warning
@@ -158,7 +654,7 @@ spec:
             description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
         - alert: LonghornVolumeFaulted
-          expr: longhorn_volume_robustness == 3
+          expr: longhorn_volume_robustness{state="faulted"} == 1
           for: 1m
           labels:
             severity: critical
@@ -166,23 +662,103 @@ spec:
             summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
             description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
-        - alert: LonghornVolumeCapacityHigh
-          expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85
+        # -------------------------------------------------------------------------------------
+        # WHERE `LonghornVolumeCapacityHigh` WENT (deleted 2026-09-13) — IT MEASURED ALLOCATION AND
+        # CALLED IT CAPACITY, AND ON ONE VOLUME IT COULD NEVER CLEAR.
+        #
+        # The deleted rule was
+        #     (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85   for 30m
+        # summarised "Longhorn volume near capacity", described as "over 85% used". Within minutes
+        # of the Longhorn scrape being enabled it went `pending` on the Prometheus TSDB volume.
+        # MEASURED on prod 2026-09-13:
+        #   pvc-fa59563f-…  (monitoring/prometheus-…-prometheus-db-…-0, 50Gi, class longhorn)
+        #     longhorn_volume_actual_size_bytes   52243681280   (48.66 GiB)
+        #     longhorn_volume_capacity_bytes      53687091200   (50 GiB)
+        #     ratio                               0.9731        <- what the alert saw
+        #     next-highest volume in the cluster  0.2818        (keycloak-db-1) — a lone outlier
+        #   the SAME filesystem, from the kubelet:
+        #     kubelet_volume_stats_used_bytes / …_capacity_bytes   0.2080   (20.8% used)
+        #
+        # `actual_size` is not "used". It is the size of the replica's disk files: the live head PLUS
+        # every snapshot still pinned. On this volume,
+        #     longhorn_snapshot_actual_size_bytes{user_created="false"}  40510484480   (37.7 GiB)
+        # ONE Longhorn system snapshot — the kind taken for a full replica rebuild, which is exactly
+        # what prod-1's drive replacement forced — holds 37.7 GiB of historical blocks; the live head
+        # is the remaining ~10.9 GiB, matching the kubelet's 11.2 GiB. Longhorn also never returns
+        # freed blocks to the replica file without discard/TRIM, so the number is monotonic: it was
+        # climbing at ~152 MB/h (deriv over 2h) while the filesystem sat flat at 20%. With
+        # `retention: 30d` and `retentionSize: 45GiB` deliberately targeting 90% of a 50Gi volume,
+        # the ratio is by DESIGN pinned near 1.0 and cannot fall back under 0.85. A rule that can
+        # only ever be true is the CNPGNoRecoverabilityPoint failure again — it teaches the operator
+        # to ignore the storage row, which is the row that matters on this cluster.
+        #
+        # NOT REPLACED PER VOLUME, BECAUSE THE HONEST QUESTION IS ALREADY ASKED — measured, not
+        # asserted:
+        #   * "is this PVC's filesystem filling?" -> kubelet_volume_stats_*. The monitoring
+        #     namespace is covered by MonitoringVolumeFillingUp (<15% free, 15m) later in THIS file;
+        #     every other namespace by PersistentVolumeFillingUp / PersistentVolumeAlmostFull in
+        #     platform/monitoring/pvc-capacity-alerts.yaml. COVERAGE MEASURED on prod 2026-09-13:
+        #     24 Longhorn volumes exist; 23 of them appear in `kubelet_volume_stats_inodes > 0` and
+        #     so fall inside one of those two scopes. The 24th is unifi/unifi-rootdisk, volumeMode
+        #     Block — no filesystem exists for the kubelet OR for Longhorn to report usage of, so
+        #     the deleted rule was not covering it either (its 0.2097 was raw-device allocation).
+        #     Block volumes are covered at the pool level, below.
+        #   * "is replica allocation exhausting the pool?" -> that IS an actual_size question, and
+        #     it is answered per DISK below, where allocation is a budget, instead of per volume
+        #     against its own nominal size, where it is a tautology.
+        # -------------------------------------------------------------------------------------
+        # THE ALLOCATION ALARM, AT THE LEVEL WHERE ALLOCATION IS A BUDGET.
+        #
+        # Renamed from LonghornNodeStorageHigh, re-keyed from longhorn_node_storage_* to
+        # longhorn_disk_*, re-thresholded from 0.85 to 0.70. Each for a reason:
+        #
+        #  * NAME AND WORDING. This is the number that legitimately counts blocks EVER allocated —
+        #    replica files plus their pinned snapshots — because that is what consumes the node's
+        #    data partition whether or not any guest filesystem thinks it is full. "Node storage
+        #    near capacity" invited the same misreading the deleted per-volume rule shipped;
+        #    "disk allocation high" says what it measures.
+        #  * PER DISK, not per node. longhorn_node_storage_usage_bytes SUMS a node's disks, so one
+        #    full disk could hide behind a half-empty second one. longhorn_disk_usage_bytes carries
+        #    a `disk` label. Identical today (one disk per node) and correct if a disk is added.
+        #  * 0.70, not 0.85, because 0.85 is PAST THE CLIFF. Longhorn's own
+        #    storage-minimal-available-percentage is 25 on this cluster (read live from
+        #    settings.longhorn.io), so a disk stops accepting new replicas at 75% used. The old
+        #    threshold could therefore only fire AFTER scheduling had already been refused — and
+        #    with hard node anti-affinity (replicaSoftAntiAffinity false, numberOfReplicas 3) a disk
+        #    that will not take a replica pins a degraded volume at 2 replicas indefinitely. 0.70
+        #    gives lead time on the way to Longhorn's own limit.
+        #
+        # MEASURED on prod 2026-09-13 — quiet, with room, on all three nodes:
+        #   node        capacity      usage         ratio   scheduled (nominal)
+        #   k3s-prod-1  445098295296  100326506496  0.2254  288836550656
+        #   k3s-prod-2  456002658304  105149128704  0.2306  288836550656
+        #   k3s-prod-3  445098295296  105464528896  0.2369  288836550656
+        # -------------------------------------------------------------------------------------
+        - alert: LonghornDiskAllocationHigh
+          expr: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70
           for: 30m
           labels:
             severity: warning
           annotations:
-            summary: "Longhorn volume near capacity: {{ $labels.volume }}"
-            description: "Volume {{ $labels.volume }} is over 85% used for 30m."
+            summary: "Longhorn disk allocation high: {{ $labels.node }} disk {{ $labels.disk }}"
+            description: "Longhorn disk {{ $labels.disk }} on node {{ $labels.node }} has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
-        - alert: LonghornNodeStorageHigh
-          expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
-          for: 30m
+        # The terminal form of the same condition, straight from Longhorn rather than inferred from a
+        # threshold: the disk itself reports whether it will accept another replica. More honest than
+        # re-deriving the scheduling formula in PromQL, which folds in storage-reserved-percentage
+        # 20, storage-over-provisioning-percentage 200 and storage-minimal-available-percentage 25 —
+        # three settings NO metric exports, so the rule would silently mis-measure if any changed.
+        # MEASURED on prod 2026-09-13: longhorn_disk_status{condition="schedulable"} == 1 on all
+        # three disks, so this is not a permanently-firing rule. `for: 15m` rides out the blip while
+        # a longhorn-manager restarts.
+        - alert: LonghornDiskUnschedulable
+          expr: longhorn_disk_status{condition="schedulable"} == 0
+          for: 15m
           labels:
             severity: warning
           annotations:
-            summary: "Longhorn node storage near capacity: {{ $labels.node }}"
-            description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m."
+            summary: "Longhorn will not place replicas on {{ $labels.node }} disk {{ $labels.disk }}"
+            description: "Longhorn marked disk {{ $labels.disk }} on node {{ $labels.node }} UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
 
 # =====================================================================================
@@ -328,6 +904,132 @@ spec:
             summary: "Egress VPN tunnel down"
             description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
+        # -----------------------------------------------------------------------------------------
+        # THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
+        #
+        # `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
+        # having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
+        # exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
+        # re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
+        # at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
+        # 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
+        # carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
+        # exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
+        #
+        # The datapath now self-heals: `roles/vpn_egress_gateway` publishes
+        # `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
+        # OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
+        # and that node write is what makes Cilium re-resolve. These two rules watch whether it
+        # ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
+        #
+        # WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
+        # `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
+        # observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
+        #   * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
+        #     `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
+        #     there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
+        #     metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
+        #     `cilium_*` and `hubble_*` query returns an empty vector today (which also means
+        #     `HubbleDeniedFlowSpike` above cannot fire yet).
+        #   * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
+        #     CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
+        #     map, not an exporter. No metric name is invented here to paper over that.
+        # So the observable is the label: the input to the re-steer, compared against the live tun0
+        # identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
+        # ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
+        # `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
+        #
+        # WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
+        # firing, the label write failing or never being deployed, the convergence timer being dead —
+        # i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
+        # "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
+        # until either the re-steer script exports the map's ifindex through a node-exporter textfile
+        # collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
+        # A probe was considered and rejected for now: there is no blackbox-exporter and no probe
+        # CronJob in this repo, so it means a NEW always-on component whose own failures become a new
+        # alerting surface — worth doing, but not as the thing that closes a hole today.
+        #
+        # WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
+        # VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
+        # label-to-number conversion, so the level-triggered equality test is not expressible. What IS
+        # expressible is the same invariant counted over a window: the published label must change at
+        # least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
+        # is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
+        # that held still, 2 for one that changed value inside the window); the `max by (<label>)`
+        # wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
+        # restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
+        # which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
+        #
+        # Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
+        # +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
+        # trade against the repo's alert-fatigue history (an unbounded critical on the one channel
+        # that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
+        # email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
+        - alert: VPNEgressSteeringStale
+          # FIRES when the live tun0 was re-created more times in the last hour than the gateway
+          # node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
+          # tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
+          # and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
+          # seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
+          # cold boot), so `for: 15m` cannot fire on a healthy reconnect.
+          expr: |
+            (
+                changes(max(node_network_iface_id{device="tun0"})[1h:1m])
+              >
+                (
+                  count(
+                    count_over_time(
+                      max by (label_homelab_keithjackson_dev_egress_uplink) (
+                        kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
+                      )[1h:1m]
+                    )
+                  ) - 1
+                )
+            )
+            and on()
+            (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "VPN egress steering is stale — vpn-class pods have no egress"
+            description: >-
+              The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
+              Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
+              being dropped, silently, while tun0 still looks healthy. Compare
+              `kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
+              `cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
+              (or setting any node label) forces a fresh resolution.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+        - alert: VPNEgressUplinkLabelMissing
+          # The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
+          # in kube-state-metrics, and if that label is missing its comparison degrades to an empty
+          # vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
+          # node with no published uplink means the re-steer mechanism is not in place at all
+          # (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
+          # kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
+          # the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
+          # rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
+          # that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
+          # on the way in, and so an Ansible run in progress does not either.
+          expr: |
+            (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+            and on()
+            absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Egress gateway node publishes no egress-uplink label"
+            description: >-
+              A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
+              `homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
+              nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
+              `VPNEgressSteeringStale` is inert. Check the label on the node, then the
+              `vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
+              metricLabelsAllowlist.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+        # -----------------------------------------------------------------------------------------
         # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
         # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
         # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`

platform/monitoring/kube-prometheus-stack.yaml

@@ -231,7 +231,7 @@ spec:
       # 3 extra series, no cardinality concern. Label name is sanitized by KSM to
       # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
       metricLabelsAllowlist:
-        - "nodes=[homelab.keithjackson.dev/egress-gateway]"
+        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
       # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
       # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
       # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
@@ -348,9 +348,74 @@ spec:
         limits:
           memory: 384Mi
     prometheus-node-exporter:
+      # --- NFS OBSERVABILITY (added 2026-09-14) -------------------------------------------------
+      # WHY. Five in-tree NFS PVs on 192.168.2.102 carry the whole media library and the legacy
+      # Nextcloud dataset, and I/O PSI counts an NFS stall as an I/O stall. That is why prod-1's
+      # 20.98% PSI spike at 02:40Z on 2026-09-14 looked inexplicable from the block layer: devices
+      # at 0.09-0.21 utilisation, 1.4-4.8 ms per write, zero NFS retransmissions. The one signal
+      # that moved was 114 NFS reads/s on prod-1 against 0 on the other two nodes.
+      #
+      # WHAT IS ALREADY THERE, SO NOBODY "ENABLES" IT TWICE. The `nfs` collector is ON BY DEFAULT
+      # and its series are confirmed present on prod (node_nfs_rpcs_total, node_nfs_requests_total
+      # by method/proto, node_nfs_rpc_retransmissions_total — all three nodes, 2026-09-14). It is
+      # host-wide, read from /proc/net/rpc/nfs, which works because this DaemonSet is hostNetwork.
+      # It gives op COUNTS and retransmissions. It gives NO LATENCY and no per-mount breakdown, so
+      # it can say "the server dropped requests" but not "the export is slow".
+      #
+      # WHAT THIS ARGUMENT ADDS. `mountstats` reads /proc/.../mountstats and emits per-mount,
+      # per-operation counters including cumulative RPC queue time and round-trip time, which
+      # divided by the request count is the per-operation NFS latency this platform has never had.
+      # That is the missing half of the I/O-cause question: with it, "PSI is high and the export is
+      # answering slowly" and "PSI is high because WE are asking for a lot" become distinguishable,
+      # and so does the case where neither holds and the load is coming from the Proxmox media
+      # stack that still has operational control of the same files.
+      #
+      # COST, MEASURED-ADJACENT NOT ASSUMED. mountstats is per-mount and per-operation, so it is
+      # the highest-cardinality collector in node-exporter. NFSv4.1 exposes ~70 operations and the
+      # collector emits ~8 counters per operation plus ~15 mount-level series, so roughly 575
+      # series per mounted export, and the mounts are per POD (kubelet mounts an in-tree NFS PV
+      # once per pod), so they churn on every restart. With five exports that is ~2.9k series
+      # against a head of 451,650 measured on prod 2026-09-14 — about 0.6%, inside the 3Gi
+      # Prometheus limit and the 45GiB retentionSize. Re-measure after rollout with
+      #   count({__name__=~"node_mountstats_.*"})
+      # and if it lands materially above ~5k series, drop the collector rather than growing the PVC.
+      #
+      # VERIFY BEFORE TRUSTING — THIS MAY LAND INERT, AND THAT MUST BE CHECKED NOT ASSUMED.
+      # node-exporter runs with --path.procfs=/host/proc and hostPID, so it reads ITS OWN mount
+      # namespace, not the host's. The kubelet-created NFS mounts should still be visible because
+      # the `root` volume is a hostPath of / with mountPropagation: HostToContainer, which
+      # propagates them in — they will appear with mount points prefixed /host/root/var/lib/kubelet/
+      # (the `export` label, e.g. 192.168.2.102:/mnt/Storage/plex, is the stable thing to aggregate
+      # by). The node-exporter image is distroless with no shell, so this could not be tested from
+      # outside the collector; note that node_filesystem_* currently reports only ext4 and tmpfs,
+      # which is explained by the kubelet mount-point exclusion above but is NOT proof either way.
+      # AFTER THIS ROLLS OUT, RUN:
+      #   count({__name__=~"node_mountstats_.*"})                       -> must be non-zero
+      #   count(node_mountstats_nfs_operations_requests_total) by (export)
+      # If it is empty, REVERT this argument — a collector that is on and produces nothing is the
+      # same lie as an alert rule on a metric nobody scrapes. No alert rule references
+      # node_mountstats_* yet, deliberately, for exactly that reason; the per-export latency rule
+      # is the follow-up once the series are confirmed, and runbooks/node-io.md records the gap.
+      #
+      # extraArgs REPLACES, IT DOES NOT APPEND — and the chart's DEFAULT for this key is not empty.
+      # prometheus-node-exporter's own values.yaml ships two filesystem excludes in `extraArgs`
+      # (confirmed against the chart, and against the live DaemonSet's argv on prod). Listing only
+      # `--collector.mountstats` here would silently DROP both, so the filesystem collector would
+      # start reporting every container overlay and every kubelet bind mount: a large cardinality
+      # jump and a behaviour change in NodeDiskFillingUp, which alerts per mountpoint. Both
+      # defaults are therefore restated verbatim below. If the chart's defaults change on an
+      # upgrade, this list must be re-reconciled against
+      #   helm show values prometheus-community/prometheus-node-exporter | grep -A3 '^extraArgs:'
+      extraArgs:
+        - --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|run/containerd/.+|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
+        - --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|erofs)$
+        - --collector.mountstats
       resources:
         requests:
           cpu: 20m
           memory: 32Mi
         limits:
-          memory: 64Mi
+          # mountstats parses a per-mount, per-operation table on every scrape. 64Mi was sized for
+          # the default collector set; raised with headroom for that parse rather than discovering
+          # the ceiling as an OOMKill on the exporter that every node alert depends on.
+          memory: 96Mi

platform/monitoring/node-pressure-dashboard.yaml

@@ -147,6 +147,103 @@ data:
           "targets": [
             {"expr": "rate(node_pressure_io_waiting_seconds_total[5m]) * 100", "legendFormat": "{{instance}}"}
           ]
+        },
+        {
+          "id": 11,
+          "type": "row",
+          "title": "I/O pressure — WHY: saturation / device latency / NFS / etcd / workload",
+          "collapsed": false,
+          "gridPos": {"h": 1, "w": 24, "x": 0, "y": 30},
+          "panels": []
+        },
+        {
+          "id": 12,
+          "type": "timeseries",
+          "title": "CAUSE 1 — device busy fraction, per device",
+          "description": "rate(node_disk_io_time_seconds_total[5m]). SATURATION: the device is queue-bound. Alert line at 0.85 (NodeDiskDeviceSaturated). Normal cluster maximum over 48h is 0.61. Tonight's 2026-09-14 spike read 0.27 and 2026-09-07 read 0.79 — so a flat line down here means saturation is NOT the cause, which is the single most useful thing this row tells you.",
+          "gridPos": {"h": 8, "w": 12, "x": 0, "y": 31},
+          "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "max": 1, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
+            {"color": "green", "value": null}, {"color": "red", "value": 0.85}
+          ]}}},
+          "targets": [
+            {"expr": "topk(8, rate(node_disk_io_time_seconds_total[5m]))", "legendFormat": "{{instance}} {{device}}"}
+          ]
+        },
+        {
+          "id": 13,
+          "type": "timeseries",
+          "title": "CAUSE 2 — write service time per op (devices above 10 writes/s)",
+          "description": "THE DRIVE-IS-DYING SIGNAL, and the one the stack lacked on 2026-09-07 when prod-1's 2011 SSD measured 1,039 ms per write. Alert lines at 20 ms (warning) and 50 ms (critical). Healthy on this cluster is 1-3 ms; the 48h maximum is 3.1 ms. Guarded at >10 writes/s, which also restricts the panel to the etcd-WAL/Longhorn partitions rather than idle media disks.",
+          "gridPos": {"h": 8, "w": 12, "x": 12, "y": 31},
+          "fieldConfig": {"defaults": {"unit": "s", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
+            {"color": "green", "value": null}, {"color": "yellow", "value": 0.020}, {"color": "red", "value": 0.050}
+          ]}}},
+          "targets": [
+            {"expr": "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10", "legendFormat": "{{instance}} {{device}}"}
+          ]
+        },
+        {
+          "id": 14,
+          "type": "timeseries",
+          "title": "CAUSE 3 — read service time per op (devices above 10 reads/s)",
+          "description": "Alert line at 250 ms (NodeDiskReadLatencyHigh). Deliberately far above the write thresholds: prod-1's sdg is a spinning media disk and legitimately sustains 156 ms while serving media, so anything under the line here is normal and not a fault. The SSD/NVMe devices sit at 1.6-6.3 ms.",
+          "gridPos": {"h": 8, "w": 12, "x": 0, "y": 39},
+          "fieldConfig": {"defaults": {"unit": "s", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
+            {"color": "green", "value": null}, {"color": "red", "value": 0.250}
+          ]}}},
+          "targets": [
+            {"expr": "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10", "legendFormat": "{{instance}} {{device}}"}
+          ]
+        },
+        {
+          "id": 15,
+          "type": "timeseries",
+          "title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
+          "description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
+          "gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
+          "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
+            {"color": "green", "value": null}, {"color": "red", "value": 0.01}
+          ]}}},
+          "targets": [
+            {"expr": "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m]))", "legendFormat": "{{instance}}"}
+          ]
+        },
+        {
+          "id": 16,
+          "type": "timeseries",
+          "title": "Is it OUR NFS traffic? — client op rate per node and method",
+          "description": "THE PANEL THAT ANSWERS THE QUESTION NodeIOPressureNotLocalDisk ASKS. If I/O PSI is high, the local devices are idle and THIS is high, our own workload is hammering the export (prod-1 peaked 114 reads/s during the 2026-09-14 spike, against 0 on the other two nodes). If PSI is high and this is near zero, the load is coming from OUTSIDE this cluster — the Proxmox media stack still has operational control of the same files — and nothing in-cluster will explain it.",
+          "gridPos": {"h": 8, "w": 12, "x": 0, "y": 47},
+          "fieldConfig": {"defaults": {"unit": "ops", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 10}}},
+          "targets": [
+            {"expr": "sum by (instance, method) (rate(node_nfs_requests_total{method=~\"Read|Write|Commit|Getattr|Lookup\"}[5m]))", "legendFormat": "{{instance}} {{method}}"}
+          ]
+        },
+        {
+          "id": 17,
+          "type": "timeseries",
+          "title": "CAUSE 5 — etcd requests slower than 1s, as a fraction, per node",
+          "description": "The apiserver's CLIENT-SIDE view of its own local embedded etcd member, read off the le=\"1.0\" bucket. Alert line at 0.5% (NodeEtcdRequestsSlow). etcd's own disk metrics do NOT exist on this cluster — etcd_disk_wal_fsync_duration_seconds and etcd_disk_backend_commit_duration_seconds are absent because kubeEtcd is disabled for k3s' embedded etcd — so this is the proxy, and it is the closest thing to the NotReady precursor. Measured 0 across 48h; at the 2026-09-07 outage window 0.5-2% on all three members. Rises for any etcd slowness, not only disk.",
+          "gridPos": {"h": 8, "w": 12, "x": 12, "y": 47},
+          "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
+            {"color": "green", "value": null}, {"color": "red", "value": 0.005}
+          ]}}},
+          "targets": [
+            {"expr": "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job=\"apiserver\",le=\"1.0\"}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job=\"apiserver\",le=\"+Inf\"}[5m])))", "legendFormat": "{{instance}}"}
+          ]
+        },
+        {
+          "id": 18,
+          "type": "timeseries",
+          "title": "WHO is stalled — per-container I/O PSI (full), top 10",
+          "description": "cAdvisor's per-cgroup PSI. Node PSI says the machine stalled; this says which workload. Alert line at 0.15 (ContainerIOStallHigh). On 2026-09-14 at 02:40Z this read sonarr 0.268 and qbittorrent 0.070 on k3s-prod-1 against under 0.013 everywhere else — one panel, one answer, and sonarr is an NFS media consumer. `container!=\"\"` drops cAdvisor's pod-level duplicate of the same cgroup.",
+          "gridPos": {"h": 8, "w": 24, "x": 0, "y": 55},
+          "fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
+            {"color": "green", "value": null}, {"color": "red", "value": 0.15}
+          ]}}},
+          "targets": [
+            {"expr": "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=\"\",namespace!=\"\"}[5m]))", "legendFormat": "{{node}} {{namespace}}/{{pod}} {{container}}"}
+          ]
         }
       ]
     }

platform/monitoring/pvc-capacity-alerts.yaml

@@ -10,12 +10,26 @@
 # the SSO database with nothing watching it.
 #
 # NOTE, because it changes what "covered" means: `platform-health.longhorn`'s
-# `LonghornVolumeCapacityHigh` looks like it might have covered these PVCs. It does not — the
-# longhorn_* series are ABSENT from Prometheus on BOTH clusters (verified read-only 2026-09-06:
-# `count(longhorn_volume_capacity_bytes)` is empty on dev and on prod), so that whole rule group is
-# currently inert. That is a separate defect (a missing Longhorn ServiceMonitor/metrics config) and
-# is NOT fixed here; these rules deliberately key on `kubelet_volume_stats_*`, which is present and
-# working on both clusters.
+# `LonghornVolumeCapacityHigh` looked like it might have covered these PVCs. It never did, for two
+# different reasons in sequence, and it is now GONE:
+#   * 2026-09-06 — the longhorn_* series were ABSENT from Prometheus on both clusters (verified
+#     read-only: `count(longhorn_volume_capacity_bytes)` empty on dev and prod), so the whole rule
+#     group was inert. That was a missing Longhorn ServiceMonitor, fixed 2026-09-13 in
+#     platform/longhorn/helmrelease.yaml.
+#   * 2026-09-13 — with the series flowing, the rule turned out to measure the WRONG THING. It
+#     divided `longhorn_volume_actual_size_bytes` (blocks ever allocated to the replica: live head
+#     plus pinned snapshots, never returned without discard/TRIM) by the volume's nominal size, and
+#     called the result "% used". On the Prometheus TSDB volume that read 0.9731 while the kubelet
+#     read the same filesystem 20.8% used — by design and permanently, because `retentionSize:
+#     45GiB` targets 90% of a 50Gi volume. It was DELETED rather than re-thresholded; see the
+#     comment block in platform/monitoring/alerts.yaml. The allocation question it half-asked now
+#     lives at the disk level as LonghornDiskAllocationHigh / LonghornDiskUnschedulable.
+# So the rules below are not a stopgap for an inert rule group — they are the ONLY correct answer to
+# "is this PVC's filesystem filling?", and they deliberately key on `kubelet_volume_stats_*`.
+# MEASURED on prod 2026-09-13: of 24 Longhorn volumes, 23 appear in
+# `kubelet_volume_stats_inodes > 0` and so fall in this file's scope or in
+# MonitoringVolumeFillingUp's; the 24th is unifi/unifi-rootdisk, volumeMode Block, which has no
+# filesystem for anyone to measure (see the inodes guard below).
 #
 # The rules live in a NEW FILE rather than in alerts.yaml on purpose: alerts.yaml is under concurrent
 # edit, and a second file needs no coordination. Wiring is identical either way — PrometheusRule is a

platform/monitoring/tests/longhorn.test.yaml

@@ -0,0 +1,465 @@
+# promtool rule-unit tests for the platform-health.longhorn group (platform/monitoring/alerts.yaml)
+# and for the kubelet-based PVC-fill rules that are the reason the per-volume Longhorn "capacity"
+# rule was DELETED rather than replaced (MonitoringVolumeFillingUp in the same file,
+# PersistentVolumeFillingUp / PersistentVolumeAlmostFull in pvc-capacity-alerts.yaml).
+#
+# Run with: platform/monitoring/tests/run.sh   (the merge gate runs the same harness via
+#           conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHAT THIS FILE IS FOR
+# On 2026-09-13 the Longhorn ServiceMonitor was enabled and longhorn_* series reached the TSDB for
+# the first time. Two things became visible within minutes:
+#   1. `LonghornVolumeCapacityHigh` went `pending` on the Prometheus TSDB volume at 0.9731 while the
+#      kubelet reported the same filesystem 20.8% used. The rule was named and documented as a
+#      CAPACITY alarm and measured ALLOCATION; on that volume the condition is by design and can
+#      never clear, so it was a PERMANENTLY-FIRING alert. It is deleted.
+#   2. `LonghornVolumeDegraded` / `LonghornVolumeFaulted` were STILL INERT even with the series
+#      present, because `longhorn_volume_robustness` on longhorn 1.12.1 is one-hot over a `state`
+#      label with values 0/1 — `== 2` and `== 3` match nothing. They are re-keyed.
+# Both failure modes are the ones this repo shipped to prod earlier the same week, so every rule
+# touched here is pinned in BOTH directions: silent on today's measured prod state, firing on a
+# constructed bad state, with the `for:` durations asserted as real.
+#
+# NUMBERS. Where a value is asserted exactly (promql_expr_test) the fixture uses round figures so
+# the arithmetic is exact in float64; where only "quiet" or "firing" is asserted it uses the
+# literal bytes measured on prod 2026-09-13. Both are noted at the point of use.
+#
+# CLOCK CONVENTION: promtool's test clock starts at the unix epoch and the sample interval is 1m, so
+# nothing is ever stale inside Prometheus' 5m lookback.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # ===========================================================================================
+  # 1. TODAY'S MEASURED PROD STATE -> the whole Longhorn group and both PVC-fill scopes are SILENT.
+  #    This is the "not permanently firing" half of the proof for every rule below, and it is the
+  #    test that would have caught the deleted rule: the SAME fixture carries the real
+  #    actual_size/capacity pair, and the promql_expr_test at the bottom shows the deleted
+  #    expression evaluating to 0.9731 on a volume whose filesystem is 20% used.
+  # ===========================================================================================
+  - interval: 1m
+    name: measured prod state 2026-09-13 - every Longhorn and PVC-fill rule quiet
+    input_series:
+      # --- volume robustness, one-hot over `state` (24 volumes on prod; two are enough here) ---
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="healthy"}'
+        values: '1+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="degraded"}'
+        values: '0+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="faulted"}'
+        values: '0+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f",state="unknown"}'
+        values: '0+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}'
+        values: '1+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}'
+        values: '0+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}'
+        values: '0+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}'
+        values: '0+0x120'
+      # --- the ALLOCATION figures the deleted rule divided, exactly as measured on prod ---
+      - series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
+        values: '52243681280+0x120'
+      - series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
+        values: '53687091200+0x120'
+      # --- per-disk allocation, literal prod bytes (ratios 0.2254 / 0.2306 / 0.2369) ---
+      - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
+        values: '445098295296+0x120'
+      - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
+        values: '100326506496+0x120'
+      - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}'
+        values: '456002658304+0x120'
+      - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19"}'
+        values: '105149128704+0x120'
+      - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
+        values: '445098295296+0x120'
+      - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
+        values: '105464528896+0x120'
+      # --- Longhorn's own scheduling verdict: all three disks accepting replicas ---
+      - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}'
+        values: '1+0x120'
+      - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",disk="default-disk-9c4b3f19",condition="schedulable"}'
+        values: '1+0x120'
+      - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}'
+        values: '1+0x120'
+      # `ready` is present on prod alongside `schedulable`; carried here so the rule's `condition`
+      # selector is load-bearing. Drop the selector and the rule would still be quiet on THIS
+      # fixture, so test 5 constructs the case where it matters.
+      - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="ready"}'
+        values: '1+0x120'
+      # --- the SAME filesystem as seen by the kubelet: 20% used, i.e. 80% free ---
+      # Round figures (10 GiB of 50 GiB) so the ratio is exact; prod measured 0.2080.
+      - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
+        values: '53687091200+0x120'
+      - series: 'kubelet_volume_stats_used_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
+        values: '10737418240+0x120'
+      - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
+        values: '42949672960+0x120'
+      - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
+        values: '3276800+0x120'
+      # and a non-monitoring Longhorn PVC, the other half of the partition (93% free on prod)
+      - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
+        values: '5368709120+0x120'
+      - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
+        values: '4993977548+0x120'
+      - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
+        values: '327680+0x120'
+    alert_rule_test:
+      - eval_time: 2h
+        alertname: LonghornVolumeDegraded
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: LonghornVolumeFaulted
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: LonghornDiskAllocationHigh
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: LonghornDiskUnschedulable
+        exp_alerts: []
+      # The rules that DO answer "is this filesystem filling?" are quiet too — so deleting the
+      # per-volume Longhorn rule does not trade a false alarm for a blind spot on this state.
+      - eval_time: 2h
+        alertname: MonitoringVolumeFillingUp
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: PersistentVolumeFillingUp
+        exp_alerts: []
+      - eval_time: 2h
+        alertname: PersistentVolumeAlmostFull
+        exp_alerts: []
+    promql_expr_test:
+      # THE DEFECT, AS ARITHMETIC. The deleted rule's expression, on the volume it fired on.
+      - expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes
+        eval_time: 1h
+        exp_samples:
+          - labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
+            value: 0.9731143951416016
+      # …and the honest question about the same filesystem, at the same instant. 0.20 vs 0.97 is the
+      # whole argument for the deletion, and it is now an assertion rather than a paragraph.
+      - expr: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes
+        eval_time: 1h
+        exp_samples:
+          - labels: '{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
+            value: 0.2
+      # THE INERTNESS, AS ARITHMETIC. The series exist (8 = 2 volumes x 4 states) …
+      - expr: count(longhorn_volume_robustness)
+        eval_time: 1h
+        exp_samples:
+          - labels: '{}'
+            value: 8
+      # … and the OLD expressions still match nothing on this metric shape. This is the regression
+      # guard: if someone "simplifies" the rules back to a numeric enum, these two go red.
+      - expr: count(longhorn_volume_robustness == 2) or vector(-1)
+        eval_time: 1h
+        exp_samples:
+          - labels: '{}'
+            value: -1
+      - expr: count(longhorn_volume_robustness == 3) or vector(-1)
+        eval_time: 1h
+        exp_samples:
+          - labels: '{}'
+            value: -1
+      # The NEW selector, on the same healthy fixture, does return a vector — so the re-keyed rules
+      # are quiet because the state is healthy, not because they select nothing.
+      - expr: count(longhorn_volume_robustness{state="degraded"})
+        eval_time: 1h
+        exp_samples:
+          - labels: '{}'
+            value: 2
+
+  # ===========================================================================================
+  # 2. A DEGRADED VOLUME -> LonghornVolumeDegraded fires, and only after `for: 10m`.
+  #    Against the pre-fix `== 2` expression this state produced NOTHING, which is the whole point.
+  # ===========================================================================================
+  - interval: 1m
+    name: one volume degraded (replica rebuilding) - Degraded fires after 10m, Faulted stays quiet
+    input_series:
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="healthy"}'
+        values: '0+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="degraded"}'
+        values: '1+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="faulted"}'
+        values: '0+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-2",pvc="keycloak-db-1",pvc_namespace="keycloak",volume="pvc-f30d3eb7",state="unknown"}'
+        values: '0+0x120'
+    alert_rule_test:
+      # `for: 10m` is real: nothing at 5m.
+      - eval_time: 5m
+        alertname: LonghornVolumeDegraded
+        exp_alerts: []
+      - eval_time: 30m
+        alertname: LonghornVolumeDegraded
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: longhorn-backend
+              namespace: longhorn-system
+              node: k3s-prod-2
+              pvc: keycloak-db-1
+              pvc_namespace: keycloak
+              volume: pvc-f30d3eb7
+              state: degraded
+            exp_annotations:
+              summary: "Longhorn volume degraded: pvc-f30d3eb7"
+              description: "Volume pvc-f30d3eb7 has been degraded (replica rebuild / missing replica) for 10m."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
+      # Degraded must not masquerade as faulted: the critical tier stays silent.
+      - eval_time: 30m
+        alertname: LonghornVolumeFaulted
+        exp_alerts: []
+    promql_expr_test:
+      # The pre-fix expression, on the state it was written for. Empty. This is the inert rule that
+      # reached prod, reproduced offline.
+      - expr: count(longhorn_volume_robustness == 2) or vector(-1)
+        eval_time: 30m
+        exp_samples:
+          - labels: '{}'
+            value: -1
+
+  # ===========================================================================================
+  # 3. A FAULTED VOLUME -> LonghornVolumeFaulted fires (critical), Degraded does not.
+  # ===========================================================================================
+  - interval: 1m
+    name: one volume faulted - Faulted fires after 1m, Degraded stays quiet
+    input_series:
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="healthy"}'
+        values: '0+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="degraded"}'
+        values: '0+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="faulted"}'
+        values: '1+0x120'
+      - series: 'longhorn_volume_robustness{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",pvc="nextcloud-db-1",pvc_namespace="nextcloud-db",volume="pvc-d6621428",state="unknown"}'
+        values: '0+0x120'
+    alert_rule_test:
+      # `for: 1m` is real: pending, not firing, at the first evaluation.
+      - eval_time: 0m
+        alertname: LonghornVolumeFaulted
+        exp_alerts: []
+      - eval_time: 10m
+        alertname: LonghornVolumeFaulted
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              job: longhorn-backend
+              namespace: longhorn-system
+              node: k3s-prod-3
+              pvc: nextcloud-db-1
+              pvc_namespace: nextcloud-db
+              volume: pvc-d6621428
+              state: faulted
+            exp_annotations:
+              summary: "Longhorn volume FAULTED: pvc-d6621428"
+              description: "Volume pvc-d6621428 is faulted — data at risk / offline. Immediate attention required."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
+      - eval_time: 10m
+        alertname: LonghornVolumeDegraded
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 4. REPLICA ALLOCATION EATING A DISK -> LonghornDiskAllocationHigh fires, per DISK.
+  #    Two disks on ONE node: 0.72 allocated and 0.23 allocated. Only the first alerts, and the
+  #    promql_expr_test shows why the rule had to move off longhorn_node_storage_*: summed over the
+  #    node the ratio is 0.475, under BOTH the new 0.70 and the old 0.85 threshold, so the node-level
+  #    form would have stayed silent while one disk was inside Longhorn's own 75% scheduling cliff.
+  # ===========================================================================================
+  - interval: 1m
+    name: one disk 72% allocated, its sibling 23% - the full disk alerts and the node average does not hide it
+    input_series:
+      - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
+        values: '400000000000+0x120'
+      - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
+        values: '288000000000+0x120'
+      - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
+        values: '400000000000+0x120'
+      - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
+        values: '92000000000+0x120'
+      # Longhorn has not (yet) given up on the disk — so this test proves the threshold rule fires
+      # on its own, ahead of the terminal signal tested in case 5.
+      - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new",condition="schedulable"}'
+        values: '1+0x120'
+    alert_rule_test:
+      # `for: 30m` is real: nothing at 20m.
+      - eval_time: 20m
+        alertname: LonghornDiskAllocationHigh
+        exp_alerts: []
+      - eval_time: 45m
+        alertname: LonghornDiskAllocationHigh
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: longhorn-backend
+              namespace: longhorn-system
+              node: k3s-prod-1
+              disk: default-disk-new
+            exp_annotations:
+              summary: "Longhorn disk allocation high: k3s-prod-1 disk default-disk-new"
+              description: "Longhorn disk default-disk-new on node k3s-prod-1 has over 70% of its bytes ALLOCATED to replica files and their pinned snapshots for 30m. This is allocation, not filesystem usage inside any one PVC — freed blocks are not returned without discard/TRIM, and a system snapshot from a replica rebuild keeps its historical blocks. At 75% Longhorn stops scheduling new replicas here (storage-minimal-available-percentage is 25), which with hard node anti-affinity pins any degraded volume at 2 replicas. Look for snapshots to purge (`kubectl -n longhorn-system get snapshots.longhorn.io`) before adding capacity."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
+      - eval_time: 45m
+        alertname: LonghornDiskUnschedulable
+        exp_alerts: []
+    promql_expr_test:
+      # Per disk: one at 0.72, one at 0.23.
+      - expr: sort(longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes)
+        eval_time: 45m
+        exp_samples:
+          - labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-second"}'
+            value: 0.23
+          - labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",disk="default-disk-new"}'
+            value: 0.72
+      # Summed over the node — the shape of the deleted expression — 0.475. Silent at 0.70, silent
+      # at the old 0.85. That is the blind spot the metric change closes.
+      - expr: sum by (node) (longhorn_disk_usage_bytes) / sum by (node) (longhorn_disk_capacity_bytes)
+        eval_time: 45m
+        exp_samples:
+          - labels: '{node="k3s-prod-1"}'
+            value: 0.475
+
+  # ===========================================================================================
+  # 5. LONGHORN ITSELF REFUSES THE DISK -> LonghornDiskUnschedulable fires.
+  #    The `condition="schedulable"` selector is load-bearing here: `ready` is 1 in this fixture, so
+  #    a rule that dropped the selector and asked `longhorn_disk_status == 0` would match the
+  #    schedulable series only by luck of which conditions happen to exist. Allocation is nowhere
+  #    near the threshold, so this rule is proven to stand on its own signal.
+  # ===========================================================================================
+  - interval: 1m
+    name: disk marked unschedulable while only 23% allocated - the terminal signal fires alone
+    input_series:
+      - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="schedulable"}'
+        values: '0+0x120'
+      - series: 'longhorn_disk_status{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d",condition="ready"}'
+        values: '1+0x120'
+      - series: 'longhorn_disk_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
+        values: '400000000000+0x120'
+      - series: 'longhorn_disk_usage_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-3",disk="default-disk-59b6657d"}'
+        values: '92000000000+0x120'
+    alert_rule_test:
+      # `for: 15m` is real: nothing at 10m, which is what rides out a longhorn-manager restart.
+      - eval_time: 10m
+        alertname: LonghornDiskUnschedulable
+        exp_alerts: []
+      - eval_time: 30m
+        alertname: LonghornDiskUnschedulable
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: longhorn-backend
+              namespace: longhorn-system
+              node: k3s-prod-3
+              disk: default-disk-59b6657d
+              condition: schedulable
+            exp_annotations:
+              summary: "Longhorn will not place replicas on k3s-prod-3 disk default-disk-59b6657d"
+              description: "Longhorn marked disk default-disk-59b6657d on node k3s-prod-3 UNSCHEDULABLE for 15m — it will accept no new or rebuilt replica. On a 3-node cluster with hard node anti-affinity that means any degraded volume stays at 2 replicas until this clears, so R-STATE-02's survive-a-node-loss promise is suspended meanwhile. Usually the disk is out of space (cross-read LonghornDiskAllocationHigh) or scheduling was disabled by hand on the Longhorn node."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
+      - eval_time: 30m
+        alertname: LonghornDiskAllocationHigh
+        exp_alerts: []
+
+  # ===========================================================================================
+  # 6. THE REDUNDANCY, PROVEN. The Prometheus TSDB filesystem genuinely filling -> the kubelet rule
+  #    fires. Same volume, same 0.9731 allocation figure as case 1 (carried on purpose): the
+  #    allocation number cannot tell these two situations apart, and the kubelet number can. So the
+  #    signal the deleted rule CLAIMED to provide is delivered — by MonitoringVolumeFillingUp for
+  #    the monitoring namespace and by the pvc-capacity rules for every other namespace.
+  # ===========================================================================================
+  - interval: 1m
+    name: filesystems really filling - the kubelet rules fire where the allocation ratio is unchanged
+    input_series:
+      # allocation IDENTICAL to the healthy case — 0.9731, indistinguishable
+      - series: 'longhorn_volume_actual_size_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
+        values: '52243681280+0x120'
+      - series: 'longhorn_volume_capacity_bytes{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
+        values: '53687091200+0x120'
+      # monitoring PVC: 10% free -> under MonitoringVolumeFillingUp's 15%
+      - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
+        values: '53687091200+0x120'
+      - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
+        values: '5368709120+0x120'
+      - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="monitoring",node="k3s-prod-1",persistentvolumeclaim="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"}'
+        values: '3276800+0x120'
+      # a non-monitoring Longhorn PVC: 10% free -> warning tier
+      - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
+        values: '5368709120+0x120'
+      - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
+        values: '536870912+0x120'
+      - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="keycloak",node="k3s-prod-2",persistentvolumeclaim="keycloak-db-1"}'
+        values: '327680+0x120'
+      # another one at 3% free -> critical tier as well as warning
+      - series: 'kubelet_volume_stats_capacity_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
+        values: '10737418240+0x120'
+      - series: 'kubelet_volume_stats_available_bytes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
+        values: '322122547+0x120'
+      - series: 'kubelet_volume_stats_inodes{job="kubelet",namespace="gitea",node="k3s-prod-3",persistentvolumeclaim="gitea-data"}'
+        values: '655360+0x120'
+    alert_rule_test:
+      # `for: 15m` on all three: nothing at 10m.
+      - eval_time: 10m
+        alertname: MonitoringVolumeFillingUp
+        exp_alerts: []
+      - eval_time: 30m
+        alertname: MonitoringVolumeFillingUp
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: kubelet
+              namespace: monitoring
+              node: k3s-prod-1
+              persistentvolumeclaim: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0
+            exp_annotations:
+              summary: "Monitoring PVC filling up: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0"
+              description: "PVC prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0 (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
+      - eval_time: 10m
+        alertname: PersistentVolumeFillingUp
+        exp_alerts: []
+      # Both non-monitoring PVCs are under 15% free; the monitoring one is NOT here, which is the
+      # exact-partition property pvc-capacity-alerts.yaml documents.
+      - eval_time: 30m
+        alertname: PersistentVolumeFillingUp
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              job: kubelet
+              namespace: gitea
+              node: k3s-prod-3
+              persistentvolumeclaim: gitea-data
+            exp_annotations:
+              summary: "PVC filling up: gitea/gitea-data"
+              description: "PVC gitea/gitea-data on node k3s-prod-3 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
+          - exp_labels:
+              severity: warning
+              job: kubelet
+              namespace: keycloak
+              node: k3s-prod-2
+              persistentvolumeclaim: keycloak-db-1
+            exp_annotations:
+              summary: "PVC filling up: keycloak/keycloak-db-1"
+              description: "PVC keycloak/keycloak-db-1 on node k3s-prod-2 is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
+      # Only the 3%-free one reaches the critical tier — the 10% one must not.
+      - eval_time: 30m
+        alertname: PersistentVolumeAlmostFull
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              job: kubelet
+              namespace: gitea
+              node: k3s-prod-3
+              persistentvolumeclaim: gitea-data
+            exp_annotations:
+              summary: "PVC almost full: gitea/gitea-data"
+              description: "PVC gitea/gitea-data on node k3s-prod-3 has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
+    promql_expr_test:
+      # The allocation ratio is bit-for-bit what it was on the HEALTHY fixture in case 1, while the
+      # filesystem went from 80% free to 10% free. One number moved; the other cannot.
+      - expr: longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes
+        eval_time: 30m
+        exp_samples:
+          - labels: '{job="longhorn-backend",namespace="longhorn-system",node="k3s-prod-1",pvc="prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0",pvc_namespace="monitoring",volume="pvc-fa59563f"}'
+            value: 0.9731143951416016

platform/monitoring/tests/nfs-export-latency.test.yaml

@@ -0,0 +1,672 @@
+# promtool rule-unit tests for the five NFS mountstats rules in the platform-health.node-io-cause
+# group (platform/monitoring/alerts.yaml): NFSExportServerSlow, NFSExportServerSlowNotOurLoad,
+# NFSClientQueueTimeHigh, NFSTransportBacklogDeep, NFSExportMajorTimeouts.
+#
+# Run with: platform/monitoring/tests/run.sh
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHY THIS FILE IS SEPARATE from node-io-cause.test.yaml, which covers the other eight rules of the
+# same group: every fixture in that file was cut from prod's TSDB at a moment when the mountstats
+# collector produced nothing, so not one of its fixtures carries a node_mountstats_* series. These
+# rules need their own, and mixing them in would make it ambiguous which fixture pins which rule.
+#
+# THE QUESTION THESE FIVE RULES ANSWER, and it is one question with four answers:
+#   SERVER SLOW, LOAD IS OURS      response_time high, queue_time low, our op rate high
+#   SERVER SLOW, LOAD IS NOT OURS  response_time high, queue_time low, our op rate LOW
+#   WE ARE BACKED UP               queue_time high (the request had not even been sent)
+#   THE EXPORT IS FAILING          major timeouts, i.e. the request was never answered at all
+# Fixtures 3-7 below are those four states, one apiece, and each asserts the OTHER rules stay quiet.
+# Fixture 4 is the one the operator asked for: it is the in-cluster half of the Proxmox hypothesis.
+#
+#   fixture                       source of the numbers
+#   ----------------------------  --------------------------------------------------------------
+#   1 healthy, measured           prod 2026-09-15, all six mounts, idle (see the two caveats below)
+#   2 healthy under real load     CONSTRUCTED, and it says so — the state that must not page
+#   3 server slow, load ours      CONSTRUCTED from the 2026-09-14 profile (prod-1 at 114 ops/s)
+#   4 server slow, load NOT ours  CONSTRUCTED — the Proxmox-hypothesis profile
+#   5 we are backed up            CONSTRUCTED — client queueing, backlog still empty
+#   6 transport backlog deep      CONSTRUCTED — even the grown slot table is full
+#   7 the export is failing       CONSTRUCTED — a major timeout, measured baseline exactly 0
+#
+# TWO CAVEATS ON FIXTURE 1, both recorded in the rule comments in alerts.yaml and repeated here
+# because they are what bound the value of these tests:
+#   (a) THE MEASURED HISTORY IS MINUTES. The mountstats series start when the node-exporter
+#       DaemonSet rolled; at measurement time each mount held 18 samples, and
+#       time() - process_start_time_seconds{job="node-exporter"} read 558-561 s on all three nodes.
+#       Fixture 1 is therefore a snapshot, not a distribution.
+#   (b) IT IS AN IDLE SNAPSHOT. The busiest operation on all six mounts is STATFS keepalive at
+#       0.002-0.046 ops/s; there was no READ or WRITE traffic while it was taken. A fixture built
+#       only from that would prove the rules are quiet when NOTHING IS HAPPENING, which is worth
+#       very little — an op-rate guard alone would satisfy it. THAT IS WHY FIXTURE 2 EXISTS: it is
+#       explicitly constructed, puts 60 ops/s of healthy-latency traffic on the mount, and is the
+#       fixture that actually pins the response-time and queue-time thresholds in the quiet
+#       direction. It also carries the two non-vacuity hazards below.
+#
+# NON-VACUITY, asserted rather than trusted (the same two hazards node-io-cause.test.yaml records):
+#   1. PROMQL PRECEDENCE. `(ratio) and rate(ops) > 1 > 0.050` parses as
+#      `(ratio) and ((rate(ops) > 1) > 0.050)` — the threshold is compared against the OP RATE and
+#      never touches the ratio, so the rule fires on any mount busier than 1 op/s. Fixture 2
+#      asserts the broken form RETURNS SERIES on a perfectly healthy 60 ops/s mount while the
+#      shipped form returns none. Fixture 1 cannot carry this assertion: at 0.125 ops/s the broken
+#      form is quiet too, for the wrong reason.
+#   2. THE OP-RATE AND SEND-RATE GUARDS ARE LOAD-BEARING. Fixture 2 carries a mount doing HALF an
+#      operation a second whose response time is a full second per op, whose queue time is a full
+#      second per op, and whose backlog queue is 20 deep — 20x, 200x and 2.5x the three thresholds
+#      at once — and asserts that all five rules stay quiet, because thirty operations a minute are
+#      not evidence of anything.
+#   3. THE SHARED-SUPERBLOCK DUPLICATION IS REAL AND IS PINNED. prod-1's /mnt/Storage/plex and
+#      /mnt/Storage/plex/downloads report byte-identical per-operation counters (mounts sharing one
+#      NFS superblock share the kernel's RPC statistics), and so do prod-3's two nextcloud mounts.
+#      Fixture 1 encodes that, and fixture 4 asserts that a single slow export on prod-1 therefore
+#      raises TWO alerts. If a future change aggregates the export label away, fixture 4 fails.
+#   4. THE DISCRIMINATOR ITSELF. Fixture 5 is a client that is queueing 30 ms per op while the
+#      server appears to be taking 60 ms per op — over the server-slow threshold. Both server-slow
+#      rules MUST stay quiet there, and that is asserted. This is the single most important
+#      assertion in the file: without it the `unless` clause could be deleted and nothing would
+#      notice.
+#
+# CLOCK CONVENTION (as in node-io-cause.test.yaml and longhorn.test.yaml): promtool's test clock
+# starts at the unix epoch with a 1m sample interval, so nothing is ever stale inside Prometheus'
+# 5m lookback. Counter fixtures are written as per-MINUTE increments, so `0+7.5x120` is a rate of
+# 7.5/60 = 0.125 per second. Every increment below is annotated with the per-second or per-op
+# figure it encodes.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. HEALTHY, MEASURED ON PROD 2026-09-15 — all five rules SILENT.
+  #    All six mounts, with the shared-superblock pairs encoded as the identical counters they
+  #    actually are. See caveats (a) and (b) in the header: this fixture is idle, so on its own it
+  #    proves little; fixture 2 is the one that pins the thresholds.
+  # =============================================================================================
+  - interval: 1m
+    name: measured healthy prod state 2026-09-15 - all five NFS mountstats rules quiet
+    input_series:
+      # --- prod-1: /mnt/Storage/plex and /mnt/Storage/plex/downloads. THE SAME NUMBERS TWICE, on
+      #     purpose: measured op rate was 0.12443893333333335 on BOTH, to every digit.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+7.5x120'            # 0.125 ops/s   (measured 0.1244)
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.031875x120'       # / 7.5 = 4.25 ms per op   (MEASURED)
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.000105x120'       # / 7.5 = 0.014 ms per op  (MEASURED)
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0x120'              # zero, on every mount, measured
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+9.48x120'           # 0.158 sends/s (measured 0.1579)
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'              # backlog depth 0, measured on all six mounts
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
+        values: '0+7.5x120'            # identical to the mount above — shared superblock
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
+        values: '0+0.031875x120'       # 4.25 ms per op
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
+        values: '0+0.000105x120'       # 0.014 ms per op
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
+        values: '0+9.48x120'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
+        values: '0+0x120'
+      # --- prod-2: one mount, /mnt/Storage/plex.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+1.26x120'           # 0.021 ops/s   (measured 0.02086)
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.0064008x120'      # / 1.26 = 5.08 ms per op  (MEASURED)
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.00010458x120'     # / 1.26 = 0.083 ms per op (MEASURED)
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+1.47x120'           # 0.0245 sends/s (measured 0.02456)
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'
+      # --- prod-3: /mnt/Storage/nextcloud and /mnt/Storage/nextcloud/nextcloud — the second shared
+      #     superblock pair, and split across TWO operations so the sum-across-operations in every
+      #     rule is exercised rather than assumed (the real mounts carry 71 operation rows each).
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
+        values: '0+0.63x120'           # half of 1.26/min
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="SEQUENCE"}'
+        values: '0+0.63x120'           # summed: 1.26/min = 0.021 ops/s (measured 0.020944)
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
+        values: '0+0.0030429x120'
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="SEQUENCE"}'
+        values: '0+0.0030429x120'      # summed / 1.26 = 4.83 ms per op (MEASURED)
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
+        values: '0+0.00005229x120'
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="SEQUENCE"}'
+        values: '0+0.00005229x120'     # summed / 1.26 = 0.083 ms per op (MEASURED)
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
+        values: '0+2.21x120'           # 0.0368 sends/s (measured 0.03684)
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
+        values: '0+0.63x120'
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="SEQUENCE"}'
+        values: '0+0.63x120'
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
+        values: '0+0.0030429x120'
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="SEQUENCE"}'
+        values: '0+0.0030429x120'
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
+        values: '0+0.00005229x120'
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="SEQUENCE"}'
+        values: '0+0.00005229x120'
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud"}'
+        values: '0+2.21x120'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud"}'
+        values: '0+0x120'
+      # --- prod-3: /mnt/Storage/plex, its own superblock and its own numbers.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+1.152x120'          # 0.0192 ops/s (measured 0.01920)
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.00544896x120'     # / 1.152 = 4.73 ms per op (MEASURED)
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.000104832x120'    # / 1.152 = 0.091 ms per op (MEASURED, the cluster maximum)
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+2.21x120'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+    promql_expr_test:
+      # THE MEASURED BASELINE IS PINNED, as a band rather than an exact float (promtool compares
+      # sample values bit-for-bit and a counter-increment fixture does not land on a round number).
+      # If someone "tidies" these increments the claim in the rule comments stops being true here.
+      - expr: |
+          count(
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.00424
+              and (
+                sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                  / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+              ) < 0.00426
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 2          # prod-1's two mounts, at the measured 4.25 ms per op
+      # Every one of the six mounts is between 4 and 7 ms per op — the whole measured range.
+      - expr: |
+          count(
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.004
+              and (
+                sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                  / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+              ) < 0.007
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 6
+      # And every one is under 0.1 ms of CLIENT queue time — the half of the discriminator that
+      # says "we are not the bottleneck".
+      - expr: |
+          count(
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) < 0.0001
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 6
+      # THE SHARED-SUPERBLOCK DUPLICATION, pinned: six exports, four distinct sets of counters.
+      # `count(count by (value))` is not expressible, so this asserts the pairs are equal by
+      # subtracting them and counting the zeroes.
+      - expr: |
+          count(
+            (
+              sum by (instance) (rate(node_mountstats_nfs_operations_requests_total{export="192.168.2.102:/mnt/Storage/plex"}[5m]))
+                - sum by (instance) (rate(node_mountstats_nfs_operations_requests_total{export="192.168.2.102:/mnt/Storage/plex/downloads"}[5m]))
+            ) == 0
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 1          # prod-1: the two mounts report the identical op rate
+
+  # =============================================================================================
+  # 2. HEALTHY UNDER REAL LOAD — CONSTRUCTED, and the fixture that actually pins the thresholds
+  #    in the quiet direction. 60 ops/s at healthy per-op latency must NOT page: that is a library
+  #    scan or a stream, and it is the state a badly-chosen threshold would turn into a permanent
+  #    alert. It also carries both non-vacuity hazards.
+  # =============================================================================================
+  - interval: 1m
+    name: constructed healthy under load - 60 ops per second at healthy latency stays quiet
+    input_series:
+      # --- prod-1 plex, genuinely busy and genuinely fine.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+3600x120'           # 60 ops/s — 3x the >20/s split, 60x the >1/s guard
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+16.2x120'           # / 3600 = 4.5 ms per op — healthy, and 11x under threshold
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0.072x120'          # / 3600 = 0.02 ms per op — 250x under threshold
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+3600x120'           # 60 sends/s
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'              # the slot table absorbed it; no backlog
+      # --- THE GUARDS' REASON FOR EXISTING, all three at once: prod-3's plex mount doing HALF an
+      #     operation a second, each taking a full SECOND on the server, a full SECOND in the queue,
+      #     and sitting 20 deep in the backlog. That is 20x, 200x and 2.5x the three thresholds —
+      #     and thirty operations a minute are not evidence of anything, so nothing may fire.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+30x120'             # 0.5 ops/s -> BELOW the >1/s guard
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+30x120'             # 1.0 SECOND per op, and deliberately ignored
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+30x120'             # 1.0 SECOND per op, and deliberately ignored
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+30x120'             # 0.5 sends/s -> BELOW the >1/s send guard
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+600x120'            # / 30 sends = depth 20, and deliberately ignored
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+    promql_expr_test:
+      # HAZARD 1 — PROMQL PRECEDENCE. The BROKEN form returns a series for the perfectly healthy
+      # 60 ops/s mount, because 60 > 0.050 is true and the threshold never reaches the ratio.
+      - expr: |
+          count(
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            )
+              and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1 > 0.050
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+      # The SHIPPED form, same fixture, returns NOTHING. This pair is the regression guard.
+      - expr: |
+          count(
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.050
+              and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
+          )
+        eval_time: 60m
+        exp_samples: []
+      # HAZARD 2 — the trickle mount really is 1 s/op and 20 deep. If a future edit "fixes" this
+      # fixture into something harmless, the guards stop being tested at all.
+      - expr: |
+          sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total{instance="192.168.2.12:9100"}[5m]))
+            / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total{instance="192.168.2.12:9100"}[5m]))
+        eval_time: 60m
+        exp_samples:
+          - labels: '{export="192.168.2.102:/mnt/Storage/plex", instance="192.168.2.12:9100"}'
+            value: 1
+      - expr: |
+          sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total{instance="192.168.2.12:9100"}[5m]))
+            / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total{instance="192.168.2.12:9100"}[5m]))
+        eval_time: 60m
+        exp_samples:
+          - labels: '{export="192.168.2.102:/mnt/Storage/plex", instance="192.168.2.12:9100"}'
+            value: 20
+
+  # =============================================================================================
+  # 3. SERVER SLOW, AND THE LOAD IS OURS — NFSExportServerSlow fires, alone.
+  #    The 2026-09-14 profile: prod-1 hammering the export itself (114 NFS reads/s measured then).
+  # =============================================================================================
+  - interval: 1m
+    name: server slow under our own load - NFSExportServerSlow fires and names the export
+    input_series:
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+7200x120'           # 120 ops/s -> above the >20/s "the load is ours" split
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+576x120'            # / 7200 = 80 ms per op -> 1.6x the 50 ms threshold
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0.36x120'           # / 7200 = 0.05 ms per op -> the client is NOT the problem
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+7200x120'           # 120 sends/s
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'              # no backlog: the transport is keeping up
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-server-slow
+              instance: 192.168.2.10:9100
+              export: 192.168.2.102:/mnt/Storage/plex
+            exp_annotations:
+              summary: "CAUSE NFS server slow: 192.168.2.102:/mnt/Storage/plex is taking over 50ms per operation for 192.168.2.10:9100, under load this node is generating"
+              description: "The NFS server is spending more than 50ms per RPC on 192.168.2.102:/mnt/Storage/plex as seen from 192.168.2.10:9100, sustained 15m, while this node is driving more than 20 ops/s at it and its own CLIENT-side queue time is under 5ms per op. Measured healthy is 4.25-4.83ms per op on every mount. The time is on the SERVER, not in our transport — but our own op rate is high enough to explain it, so this reads as self-inflicted: a library scan, an import, a torrent verifying, a Nextcloud reindex. Because I/O PSI counts an NFS wait as an I/O stall, this is a cause class that makes a node look disk-starved while every local device is idle. Do NOT touch local disks. Find the workload with ContainerIOStallHigh and slow it down. If instead our op rate were LOW, NFSExportServerSlowNotOurLoad would be firing and the reading would be the opposite one."
+              diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — split it by operation to see whether metadata or data is slow: sum by (export, operation) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (export, operation) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 0"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # THE PARTITION HOLDS: the not-our-load variant is excluded by our own 120 ops/s.
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. SERVER SLOW AND THE LOAD IS NOT OURS — THE HEADLINE FIXTURE.
+  #    This is the profile that makes the operator's Proxmox hypothesis testable: the export is
+  #    slow, our queue time is flat, and we are asking it for 5 ops/s. It also asserts the
+  #    shared-superblock duplication — one slow export on prod-1 raises TWO alerts.
+  #    prod-2 sees the SAME slow server on the same export path and does NOT alert, because at
+  #    0.021 ops/s it is below the statistical guard. That is correct and it is asserted.
+  # =============================================================================================
+  - interval: 1m
+    name: server slow while our own demand is small - NFSExportServerSlowNotOurLoad fires on both prod-1 mounts
+    input_series:
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+300x120'            # 5 ops/s -> above the >1/s guard, below the >20/s split
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+36x120'             # / 300 = 120 ms per op -> 25x the measured healthy 4.8 ms
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0.015x120'          # / 300 = 0.05 ms per op -> we are NOT backed up
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+300x120'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'
+      # The shared superblock: identical counters, as measured.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
+        values: '0+300x120'
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
+        values: '0+36x120'
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
+        values: '0+0.015x120'
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
+        values: '0+300x120'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
+        values: '0+0x120'
+      # prod-2 mounts the same export path and sees the same slow server — at 0.021 ops/s, which
+      # the guard correctly refuses to average.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+1.26x120'           # 0.021 ops/s -> below the >1/s guard
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.1512x120'         # / 1.26 = 120 ms per op — just as slow, still not alerted
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.0000630x120'      # 0.05 ms per op
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+1.26x120'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-server-slow-external
+              instance: 192.168.2.10:9100
+              export: 192.168.2.102:/mnt/Storage/plex
+            exp_annotations:
+              summary: "CAUSE NFS server slow and the load is NOT OURS: 192.168.2.102:/mnt/Storage/plex takes over 50ms per operation while 192.168.2.10:9100 asks for under 20 ops/s"
+              description: "192.168.2.102:/mnt/Storage/plex is answering 192.168.2.10:9100 in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
+              diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-server-slow-external
+              instance: 192.168.2.10:9100
+              export: 192.168.2.102:/mnt/Storage/plex/downloads
+            exp_annotations:
+              summary: "CAUSE NFS server slow and the load is NOT OURS: 192.168.2.102:/mnt/Storage/plex/downloads takes over 50ms per operation while 192.168.2.10:9100 asks for under 20 ops/s"
+              description: "192.168.2.102:/mnt/Storage/plex/downloads is answering 192.168.2.10:9100 in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
+              diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # The our-load variant does NOT fire: 5 ops/s is below its >20/s requirement. Exactly one of
+      # the two rules speaks, and it is the one that names the right cause.
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+
+  # =============================================================================================
+  # 5. WE ARE BACKED UP — NFSClientQueueTimeHigh fires AND BOTH SERVER-SLOW RULES STAY QUIET.
+  #    THE MOST IMPORTANT ASSERTION IN THIS FILE. The server here LOOKS slow — 60 ms per op, over
+  #    the 50 ms threshold — because this node is queueing 30 ms per request before it can send.
+  #    If the `unless` clause were dropped from the two server rules, this fixture would page three
+  #    times and blame the wrong machine. The backlog queue is EMPTY here (the dynamic slot table
+  #    grew instead, as prod-1's high-water of 586 slots shows it does), which is also what keeps
+  #    NFSTransportBacklogDeep from being a mere restatement of this rule.
+  # =============================================================================================
+  - interval: 1m
+    name: client transport backed up - queue-time rule fires and both server-slow rules stay quiet
+    input_series:
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+12000x120'          # 200 ops/s
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+720x120'            # / 12000 = 60 ms per op -> OVER the 50 ms server threshold
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+360x120'            # / 12000 = 30 ms per op -> 6x the 5 ms client threshold
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+12000x120'          # 200 sends/s
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'              # still EMPTY — the slot table absorbed the concurrency
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-client-backed-up
+              instance: 192.168.2.10:9100
+              export: 192.168.2.102:/mnt/Storage/plex
+            exp_annotations:
+              summary: "CAUSE we are backed up, not the server: 192.168.2.10:9100 queues over 5ms per operation before it can send to 192.168.2.102:/mnt/Storage/plex"
+              description: "On 192.168.2.10:9100 NFS requests to 192.168.2.102:/mnt/Storage/plex are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
+              diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # THE `unless` CLAUSE, PINNED. 60 ms per op is over the server threshold and 200 ops/s is over
+      # the our-load split, so NFSExportServerSlow would fire on every clause EXCEPT the queue-time
+      # exclusion. Delete that clause and this assertion fails.
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts: []
+      # And the backlog rule is NOT just this rule in other clothes.
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+    promql_expr_test:
+      # The server half really is over its threshold here — this fixture is only meaningful if it is.
+      - expr: |
+          count(
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.050
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+
+  # =============================================================================================
+  # 6. TRANSPORT BACKLOG DEEP — even the grown slot table is full.
+  #    The server is FAST here (8 ms per op), so this is unambiguously a client-side ceiling, and
+  #    both server-slow rules must stay quiet.
+  # =============================================================================================
+  - interval: 1m
+    name: transport backlog deep - backlog rule fires while the server is fast
+    input_series:
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+6000x120'           # 100 ops/s
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+48x120'             # / 6000 = 8 ms per op -> the SERVER is fine
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+240x120'            # / 6000 = 40 ms per op -> all the time is in OUR queue
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+6000x120'           # 100 sends/s -> above the >1/s send guard
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+72000x120'          # / 6000 sends = depth 12 -> 1.5x the threshold of 8
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-client-backed-up
+              instance: 192.168.2.10:9100
+              export: 192.168.2.102:/mnt/Storage/plex
+            exp_annotations:
+              summary: "CAUSE NFS transport backlog: 192.168.2.10:9100 has over 8 requests queued behind a full slot table for 192.168.2.102:/mnt/Storage/plex"
+              description: "The RPC backlog queue for 192.168.2.102:/mnt/Storage/plex on 192.168.2.10:9100 is averaging more than 8 waiting requests per send, for 10m. Measured healthy is exactly 0 on all six mounts. These mounts grow their slot table on demand (high-water 586 on prod-1, 162 on prod-3, 2 on prod-2), so a persistent backlog means even the grown table is full: this node is asking for far more concurrent NFS I/O than the transport can carry. Same remediation as NFSClientQueueTimeHigh — reduce concurrency on this node. This is a client-side ceiling, NOT a statement about the server."
+              diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) — and the two neighbouring depths, sending and pending, derived the same way; pending high with backlog low means the requests are on the wire and the server is holding them."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # The client is queueing too — the two client-side rules agree, which is expected.
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-client-backed-up
+              instance: 192.168.2.10:9100
+              export: 192.168.2.102:/mnt/Storage/plex
+            exp_annotations:
+              summary: "CAUSE we are backed up, not the server: 192.168.2.10:9100 queues over 5ms per operation before it can send to 192.168.2.102:/mnt/Storage/plex"
+              description: "On 192.168.2.10:9100 NFS requests to 192.168.2.102:/mnt/Storage/plex are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
+              diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # The server is NOT accused. 8 ms per op is a sixth of the threshold.
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+
+  # =============================================================================================
+  # 7. THE EXPORT IS FAILING — a major timeout, on prod-3's nextcloud mount.
+  #    Measured baseline is exactly 0 on all six mounts, so "> 0" is the whole threshold. The
+  #    per-op latency series stay at healthy idle values here on purpose: response_time does not
+  #    accumulate for a request that is never answered, so a hanging mount can look fast right up
+  #    to the timeout. That is precisely why this rule cannot be replaced by a latency rule.
+  # =============================================================================================
+  - interval: 1m
+    name: export failing - major timeouts raise a critical while the latency rules see nothing
+    input_series:
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
+        values: '0x50 1x90'            # flat zero for 50m, then ONE timeout at t=51m
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
+        values: '0+1.26x140'           # 0.021 ops/s — the measured idle rate
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
+        values: '0+0.0060858x140'      # / 1.26 = 4.83 ms per op — the answered requests look fine
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
+        values: '0+0.00010458x140'     # / 1.26 = 0.083 ms per op
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
+        values: '0+2.21x140'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
+        values: '0+0x140'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              io_cause: nfs-export-failing
+              instance: 192.168.2.12:9100
+              export: 192.168.2.102:/mnt/Storage/nextcloud
+            exp_annotations:
+              summary: "CAUSE NFS export failing: 192.168.2.102:/mnt/Storage/nextcloud has timed out RPCs from 192.168.2.12:9100 — the mount is hanging, not just slow"
+              description: "192.168.2.12:9100 has recorded NFS major timeouts against 192.168.2.102:/mnt/Storage/nextcloud in the last 15m: requests that went unanswered through the entire retransmission backoff. Measured baseline is 0 on all six mounts, so this is never noise. Every process touching that mount is now blocked in uninterruptible sleep, which will show as I/O PSI on the node and as container stalls on the pods — and unlike the latency rules above this one means the export is effectively DOWN for those requests, not slow. Check 192.168.2.102 itself (is it up, is nfsd running, is its array degraded) and the path to it. Pair with NodeNFSRetransmissionRateHigh, which sees the retries that lead here. Do not reboot cluster nodes to clear a hung mount while volumes are not healthy."
+              diagnostic_query: "sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) — which operations: sum by (export, operation) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0 — and the retry that precedes a timeout: sum by (instance, export) (rate(node_mountstats_nfs_operations_transmissions_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # BEFORE the timeout the rule is silent — this is the half that proves it is not permanently
+      # firing on a counter that merely EXISTS at zero.
+      - eval_time: 40m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+      # Nothing else has anything to say: the latency and queue rules see a healthy idle mount.
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts: []

platform/monitoring/tests/node-io-cause.test.yaml

@@ -0,0 +1,694 @@
+# promtool rule-unit tests for the platform-health.node-io-cause group (platform/monitoring/alerts.yaml)
+#
+# Run with: platform/monitoring/tests/run.sh
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHAT THIS FILE IS FOR
+# `NodeIOPressureCritical` has now fired twice on k3s-prod-1 for two OPPOSITE causes and said the
+# same thing both times. This group adds one rule per cause. Every rule in it is therefore pinned
+# in both directions against REAL MEASURED PROFILES, not invented ones:
+#
+#   fixture                     source of the numbers
+#   --------------------------  ----------------------------------------------------------------
+#   healthy baseline            prod, 2026-09-14 ~03:05Z, and 24-48h maxima from the same TSDB
+#   tonight                     prod, 2026-09-14 02:40-02:45Z (PSI 20.98%, devices idle)
+#   2026-09-07                  prod, re-queried from the same TSDB at offset 164h
+#                               (PSI 73.97%, 1,039 ms per write on prod-1's dying 2011 SSD)
+#   rebuild storm               a device pinned at 0.97 with HEALTHY per-op latency
+#   nfs transport               a retransmitting export, plus a low-traffic node
+#   read latency                a failing device, plus a device below the op-rate guard
+#
+# THE ACCEPTANCE TEST the operator asked for is the three middle fixtures read together:
+#   * tonight     -> NodeIOPressureNotLocalDisk FIRES, every device rule stays quiet,
+#                    ContainerIOStallHigh names sonarr
+#   * 2026-09-07  -> NodeDiskWriteLatencyHigh/Critical FIRE and name the device,
+#                    NodeEtcdRequestsSlow FIRES, and NodeIOPressureNotLocalDisk STAYS QUIET
+#   * healthy     -> all eight rules silent
+# One alert set, three verdicts. That is the gap this change closes.
+#
+# NON-VACUITY. Three hazards are asserted directly with promql_expr_test rather than trusted:
+#   1. PROMQL PRECEDENCE. `(ratio) and rate(ops) > 10 > T` does NOT mean what it reads like:
+#      `>` binds tighter than `and`, so it parses as `(ratio) and ((rate(ops) > 10) > T)` and the
+#      threshold T never touches the ratio at all — the rule then fires on ANY busy device. That
+#      was the first draft of three of these rules. The healthy fixture asserts the broken form
+#      returning series and the shipped form returning none, so the mistake cannot come back.
+#   2. THE OP-RATE GUARDS ARE LOAD-BEARING, NOT DECORATION. Each latency fixture carries a device
+#      whose per-op latency is a full SECOND at a trickle of operations — the arithmetic artefact
+#      the guard exists to suppress — and asserts it does not fire.
+#   3. `container!=""` IS LOAD-BEARING. cAdvisor emits a pod-level cgroup series alongside the
+#      container one. The healthy fixture gives the pod-level series a value that WOULD fire and
+#      asserts the shipped selector excludes it (double counting a stall is how one event becomes
+#      two pages).
+#
+# CLOCK CONVENTION (as in longhorn.test.yaml): promtool's test clock starts at the unix epoch with
+# a 1m sample interval, so nothing is ever stale inside Prometheus' 5m lookback. Counter fixtures
+# are written as per-MINUTE increments, so a `0+12.588x120` series is a rate of 12.588/60 = 0.2098
+# per second. Every increment below is annotated with the per-second figure it encodes.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. HEALTHY BASELINE — every rule in the group is SILENT.
+  #    This is the "not permanently firing" half of the proof for all eight rules, and it also
+  #    carries the three non-vacuity assertions described in the header.
+  # =============================================================================================
+  - interval: 1m
+    name: measured healthy prod state 2026-09-14 - all eight node-io-cause rules quiet
+    input_series:
+      # --- node I/O PSI (full). Measured 0.0120 / 0.0136 / 0.0061 per second. -----------------
+      - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+0.72x120'          # 0.0120/s
+      - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.11:9100"}'
+        values: '0+0.816x120'         # 0.0136/s
+      - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.12:9100"}'
+        values: '0+0.366x120'         # 0.0061/s
+      # --- device busy fraction. 48h maxima were 0.610 / 0.217 / 0.295; tonight's spike 0.271. --
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+5.4x120'           # 0.09
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}'
+        values: '0+12.6x120'          # 0.21
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
+        values: '0+3x120'             # 0.05  (a spinning media disk)
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",device="nvme0n1"}'
+        values: '0+1.2x120'           # 0.02
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",device="sda"}'
+        values: '0+1.8x120'           # 0.03
+      # --- per-op WRITE service time. Guarded maxima across 48h were 1-3 ms. ------------------
+      - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+3000x120'          # 50 writes/s  -> above the >10/s guard
+      - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+4.8x120'           # 0.08 s/s / 50 ops/s = 1.6 ms per write
+      - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}'
+        values: '0+1200x120'          # 20 writes/s  -> above the guard
+      - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}'
+        values: '0+1.92x120'          # 1.6 ms per write
+      # THE GUARD'S REASON FOR EXISTING: a media disk doing 3 writes a MINUTE, each taking a full
+      # second. 1,000 ms per write is 20x NodeDiskWriteLatencyCritical's threshold and must not
+      # fire, because three samples are not evidence of anything.
+      - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
+        values: '0+3x120'             # 0.05 writes/s -> BELOW the >10/s guard
+      - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
+        values: '0+3x120'             # 1.0 SECOND per write, and deliberately ignored
+      # --- per-op READ service time. The media HDD legitimately measures 156 ms. --------------
+      - series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
+        values: '0+1200x120'          # 20 reads/s -> above the guard
+      - series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
+        values: '0+187.2x120'         # 3.12 s/s / 20 ops/s = 156 ms per read, MEASURED on prod
+      - series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+900x120'           # 15 reads/s
+      - series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+1.962x120'         # 2.18 ms per read
+      # --- NFS client. Retransmissions measured EXACTLY ZERO across 48h on all three nodes. ----
+      - series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+7200x120'          # 120 RPC/s
+      - series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+0x120'             # zero
+      # --- etcd request duration, apiserver client side. Healthy = no request over 1s. ---------
+      - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="1.0"}'
+        values: '0+600x120'           # 10/s
+      - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="+Inf"}'
+        values: '0+600x120'           # 10/s  -> fraction over 1s = 0
+      - series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.10:6443",operation="get"}'
+        values: '0+600x120'           # 10/s  -> above the >0.1/s traffic guard
+      # --- per-container I/O PSI from cAdvisor. --------------------------------------------------
+      - series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="sonarr",pod="sonarr-6999bc759d-qlzqx",container="sonarr"}'
+        values: '0+0.3x120'           # 0.005
+      - series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="longhorn-system",pod="instance-manager-bb75a71171b0251af787ab065f38964e",container="instance-manager"}'
+        values: '0+0.24x120'          # 0.004
+      # cAdvisor's POD-level cgroup series has no `container` label. Given a value that WOULD fire,
+      # so the `container!=""` selector is proven to be what stops the same stall paging twice.
+      - series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="sonarr",pod="sonarr-6999bc759d-qlzqx"}'
+        values: '0+18x120'            # 0.30 — five times over the 0.15 threshold
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NodeDiskDeviceSaturated
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeDiskWriteLatencyHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeDiskWriteLatencyCritical
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeDiskReadLatencyHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeNFSRetransmissionRateHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeIOPressureNotLocalDisk
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeEtcdRequestsSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: ContainerIOStallHigh
+        exp_alerts: []
+      # The two PSI rules this group exists to explain are also quiet on this fixture, which is
+      # what makes it the healthy baseline and not just a quiet one.
+      - eval_time: 60m
+        alertname: NodeIOPressureHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeIOPressureCritical
+        exp_alerts: []
+    promql_expr_test:
+      # --- NON-VACUITY 1: the guard admits exactly the two devices it is meant to. -------------
+      - expr: count(rate(node_disk_writes_completed_total[5m]) > 10)
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 2
+      # --- NON-VACUITY 2: PROMQL PRECEDENCE. The broken form returns those same two devices
+      #     (i.e. it fires on "busy", ignoring the 20 ms threshold entirely); the SHIPPED form
+      #     returns nothing. If a future edit reintroduces the trailing-comparison shape, the
+      #     second assertion below starts failing.
+      - expr: |
+          count(
+            (rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]))
+              and rate(node_disk_writes_completed_total[5m]) > 10
+              > 0.020
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 2
+      - expr: |
+          count(
+            (rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m])) > 0.020
+              and rate(node_disk_writes_completed_total[5m]) > 10
+          )
+        eval_time: 60m
+        exp_samples: []
+      # --- NON-VACUITY 3: `container!=""` is what stops the pod-level duplicate paging. --------
+      - expr: count(rate(container_pressure_io_stalled_seconds_total[5m]) > 0.15)
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+      - expr: count(rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]) > 0.15)
+        eval_time: 60m
+        exp_samples: []
+
+  # =============================================================================================
+  # 2. TONIGHT — 2026-09-14 02:40-02:45Z on k3s-prod-1.
+  #    PSI(full) 20.98% with every local device idle and fast. The page must say "not local disk"
+  #    and name the stalled workload. This is the fixture the whole change exists for.
+  # =============================================================================================
+  - interval: 1m
+    name: 2026-09-14 02-40Z prod-1 - NotLocalDisk fires and names NFS, device rules stay quiet
+    input_series:
+      - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+12.588x120'        # 0.2098  <- MEASURED peak
+      - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.11:9100"}'
+        values: '0+0.852x120'         # 0.0142  <- prod-2 peaked under 1.5%
+      - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.12:9100"}'
+        values: '0+0.396x120'         # 0.0066
+      # Local devices: MEASURED at 0.21 (sdi) and 0.09 (sda). Nowhere near saturated.
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}'
+        values: '0+12.6x120'          # 0.21
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+5.4x120'           # 0.09
+      # Writes: MEASURED 1.6 ms per op on sdi. 3.5 MB/s and 2.9 MB/s of traffic — trivial.
+      - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}'
+        values: '0+1200x120'          # 20 writes/s
+      - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdi"}'
+        values: '0+1.92x120'          # 1.6 ms per write
+      - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+3000x120'          # 50 writes/s
+      - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+4.8x120'           # 1.6 ms per write
+      # Reads from local devices were ~0; the media disk keeps its normal 156 ms.
+      - series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
+        values: '0+1200x120'          # 20 reads/s
+      - series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
+        values: '0+187.2x120'         # 156 ms per read -> under the 250 ms threshold
+      # NFS: retransmissions MEASURED at zero throughout, while the op rate was 114 reads/s. This
+      # pair is the whole diagnosis — the transport is fine, we are simply waiting on the export.
+      - series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+6840x120'          # 114 RPC/s
+      - series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+0x120'
+      # etcd was healthy throughout; the node never went NotReady (kube_node_status_condition
+      # Ready held at 1 over the whole 2h window).
+      - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="1.0"}'
+        values: '0+600x120'
+      - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="+Inf"}'
+        values: '0+600x120'
+      - series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.10:6443",operation="get"}'
+        values: '0+600x120'
+      # The workload attribution, MEASURED: sonarr 0.268, qbittorrent 0.070, everything else under
+      # 0.013. sonarr is an NFS media consumer, which is the answer.
+      - series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="sonarr",pod="sonarr-6999bc759d-qlzqx",container="sonarr"}'
+        values: '0+16.08x120'         # 0.268
+      - series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="qbittorrent",pod="qbittorrent-56d6cfcf9c-k6rd8",container="qbittorrent"}'
+        values: '0+4.2x120'           # 0.070 -> below 0.15, must NOT page
+      - series: 'container_pressure_io_stalled_seconds_total{job="kubelet",node="k3s-prod-1",namespace="monitoring",pod="prometheus-kube-prometheus-stack-prometheus-0",container="prometheus"}'
+        values: '0+0.72x120'          # 0.012
+    alert_rule_test:
+      # THE DISCRIMINATOR FIRES, and only for prod-1.
+      - eval_time: 60m
+        alertname: NodeIOPressureNotLocalDisk
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              io_cause: not-local-disk
+              instance: 192.168.2.10:9100
+            exp_annotations:
+              summary: "CAUSE not local disk: 192.168.2.10:9100 is I/O-stalled while every local device is idle and healthy"
+              description: "I/O PSI (full) above 10% for 5m on 192.168.2.10:9100 (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)."
+              diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # AND THE PAGE NAMES THE WORKLOAD — sonarr, not qbittorrent, not prometheus.
+      - eval_time: 60m
+        alertname: ContainerIOStallHigh
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: workload
+              job: kubelet
+              node: k3s-prod-1
+              namespace: sonarr
+              pod: sonarr-6999bc759d-qlzqx
+              container: sonarr
+            exp_annotations:
+              summary: "WHO is stalled: sonarr/sonarr-6999bc759d-qlzqx (sonarr) on k3s-prod-1 is blocked on I/O"
+              description: "Container sonarr in sonarr/sonarr-6999bc759d-qlzqx on k3s-prod-1 spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210."
+              diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # Every local-hardware and NFS-transport and etcd rule stays SILENT. This is the half the
+      # operator had to establish with eight ad-hoc queries.
+      - eval_time: 60m
+        alertname: NodeDiskDeviceSaturated
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeDiskWriteLatencyHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeDiskWriteLatencyCritical
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeDiskReadLatencyHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeNFSRetransmissionRateHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeEtcdRequestsSlow
+        exp_alerts: []
+    promql_expr_test:
+      # The detector that already existed does fire — the problem was never detection.
+      - expr: count(rate(node_pressure_io_stalled_seconds_total[5m]) > 0.10)
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+      # The measured peak, pinned as a band rather than an exact float (promtool compares sample
+      # values bit-for-bit and a counter-increment fixture lands at 2.0979999999999943E-01), so
+      # this fixture cannot quietly drift away from the incident it encodes.
+      - expr: |
+          count(
+            rate(node_pressure_io_stalled_seconds_total[5m]) > 0.2097
+              and rate(node_pressure_io_stalled_seconds_total[5m]) < 0.2099
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+
+  # =============================================================================================
+  # 3. 2026-09-07 — THE FAILING DRIVE, re-queried from prod's own TSDB at offset 164h.
+  #    The opposite verdict from the same alert set: the device rules fire and name the disk, the
+  #    etcd rule fires, and NodeIOPressureNotLocalDisk correctly REFUSES to fire.
+  # =============================================================================================
+  - interval: 1m
+    name: 2026-09-07 failing drive - write-latency and etcd fire, NotLocalDisk correctly silent
+    input_series:
+      - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+44.382x120'        # 0.7397  <- MEASURED at offset 164h
+      - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.11:9100"}'
+        values: '0+6.2x120'           # 0.1033
+      - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.12:9100"}'
+        values: '0+11.322x120'        # 0.1887
+      # Device busy, every value MEASURED at the same window. prod-1's own devices peaked 0.745
+      # (sda) / 0.788 (sdb) — high, but NOT pinned, which is why NodeDiskDeviceSaturated stays
+      # quiet for the dying drive: a failing device services FEW SLOW ops. Nodes 2 and 3 were
+      # rebuilding Longhorn replicas at 0.858 and 0.768, and prod-2's sdb DOES cross 0.85 — so on
+      # 2026-09-07 the correct page set is "saturation on prod-2, dying device on prod-1", which is
+      # exactly the picture the operator lacked. Including nodes 2 and 3 here is also what makes
+      # the NodeIOPressureNotLocalDisk assertion below honest: both of them were stalled above 10%
+      # PSI, and it is their busy devices that rule them out, not the absence of data.
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+44.7x120'          # 0.745
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdb"}'
+        values: '0+47.28x120'         # 0.788
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",device="sdb"}'
+        values: '0+51.48x120'         # 0.858  -> the only device over 0.85 that morning
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",device="nvme0n1"}'
+        values: '0+33.96x120'         # 0.566
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",device="sdd"}'
+        values: '0+46.08x120'         # 0.768
+      # THE SIGNAL THAT WAS MISSING: 1,039 ms per write on prod-1's 2011 C400.
+      - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+900x120'           # 15 writes/s -> above the guard
+      - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+935.01x120'        # 15.5835 s/s / 15 ops/s = 1.0389 s per write
+      # The healthy members, measured the same morning: 4.3 ms and 1.1 ms.
+      - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.11:9100",device="nvme0n1"}'
+        values: '0+1800x120'          # 30 writes/s
+      - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",device="nvme0n1"}'
+        values: '0+7.74x120'          # 4.3 ms per write
+      - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.12:9100",device="sda"}'
+        values: '0+1800x120'          # 30 writes/s
+      - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",device="sda"}'
+        values: '0+1.98x120'          # 1.1 ms per write
+      # The etcd storage path, as the apiserver saw it: 0.54% / 2.05% / 1.65% of requests over 1s.
+      - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="1.0"}'
+        values: '0+596.76x120'        # 9.946/s of 10/s -> fraction 0.0054
+      - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.10:6443",operation="get",le="+Inf"}'
+        values: '0+600x120'
+      - series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.10:6443",operation="get"}'
+        values: '0+600x120'
+      - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.11:6443",operation="get",le="1.0"}'
+        values: '0+587.7x120'         # fraction 0.0205
+      - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.11:6443",operation="get",le="+Inf"}'
+        values: '0+600x120'
+      - series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.11:6443",operation="get"}'
+        values: '0+600x120'
+      - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.12:6443",operation="get",le="1.0"}'
+        values: '0+590.1x120'         # fraction 0.0165
+      - series: 'etcd_request_duration_seconds_bucket{job="apiserver",instance="192.168.2.12:6443",operation="get",le="+Inf"}'
+        values: '0+600x120'
+      - series: 'etcd_request_duration_seconds_count{job="apiserver",instance="192.168.2.12:6443",operation="get"}'
+        values: '0+600x120'
+      # NFS was not the story that morning.
+      - series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+600x120'           # 10 RPC/s
+      - series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+0x120'
+    alert_rule_test:
+      # The device is named, twice, at two urgencies.
+      - eval_time: 60m
+        alertname: NodeDiskWriteLatencyCritical
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              io_cause: device-latency
+              job: node-exporter
+              instance: 192.168.2.10:9100
+              device: sda
+            exp_annotations:
+              summary: "CAUSE device latency CRITICAL: sda on 192.168.2.10:9100 is taking over 50ms per write — etcd is at risk"
+              description: "Average write service time on sda of 192.168.2.10:9100 has exceeded 50ms for 5m under real write load. On these nodes the etcd WAL and every Longhorn replica share one partition, so a device this slow starves etcd's fdatasync and the node leaves the cluster while the host stays up, pingable and SSH-able — exactly the 2026-09-07 failure, where this same figure measured 1,039ms. Treat as HARDWARE until SMART says otherwise. With one etcd member already impaired, losing a second ends quorum: do not reboot or drain another node."
+              diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took'"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      - eval_time: 60m
+        alertname: NodeDiskWriteLatencyHigh
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: device-latency
+              job: node-exporter
+              instance: 192.168.2.10:9100
+              device: sda
+            exp_annotations:
+              summary: "CAUSE device latency: sda on 192.168.2.10:9100 is taking over 20ms per write"
+              description: "Average write service time on sda of 192.168.2.10:9100 has exceeded 20ms for 10m while the device was doing more than 10 writes/s. Healthy on this cluster is 1-3ms (48h maximum 3.1ms). This is the LATENCY class: the device itself is slow, not merely busy. Check SMART for reallocated and pending sectors — prod-1's original 2011 C400 had 61,440 reallocated and measured 1,039ms per write on 2026-09-07 before the node went NotReady. If SMART is clean, suspect a write storm queueing on the shared partition ahead of etcd's fsync."
+              diagnostic_query: "rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]) and rate(node_disk_writes_completed_total[5m]) > 10 — then on the node: sudo smartctl -a /dev/sda | grep -Ei 'Reallocated|Pending|Uncorrectable|Wear'"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # THE ACCEPTANCE CHECK THAT MATTERS MOST: the "not local disk" page does NOT go out for a
+      # failing disk. On prod-1 the write-latency clause of the `unless` chain suppresses it; on
+      # prod-2 and prod-3, which were BOTH stalled above 10% PSI that morning, the busy-device
+      # clause does. Not one of the three raises a misdirecting page.
+      - eval_time: 60m
+        alertname: NodeIOPressureNotLocalDisk
+        exp_alerts: []
+      # Saturation is attributed where it belongs: prod-2's rebuilding sdb at 0.858, and NOT to
+      # prod-1's dying sda at 0.745 — busy is not the same fault as slow.
+      - eval_time: 60m
+        alertname: NodeDiskDeviceSaturated
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: device-saturation
+              job: node-exporter
+              instance: 192.168.2.11:9100
+              device: sdb
+            exp_annotations:
+              summary: "CAUSE device saturation: sdb on 192.168.2.11:9100 is busy >85% of the time"
+              description: "Block device sdb on 192.168.2.11:9100 has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that."
+              diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # THE URGENCY SIGNAL. All three members crossed 0.5% of requests over one second that
+      # morning, so all three page — which is the correct reading: etcd quorum was at risk, not
+      # just one node's disk. This block is pinned as an alert_rule_test and not only as a
+      # promql_expr_test because a promql_expr_test restates the threshold in the TEST file, so it
+      # cannot notice the SHIPPED threshold being changed; mutation M7 (0.005 -> 0.5) survived
+      # until this block existed.
+      - eval_time: 60m
+        alertname: NodeEtcdRequestsSlow
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              io_cause: etcd
+              instance: 192.168.2.10:6443
+            exp_annotations:
+              summary: "CAUSE etcd: over 0.5% of etcd requests from 192.168.2.10:6443 are taking more than a second"
+              description: "The apiserver on 192.168.2.10:6443 is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum."
+              diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+          - exp_labels:
+              severity: critical
+              io_cause: etcd
+              instance: 192.168.2.11:6443
+            exp_annotations:
+              summary: "CAUSE etcd: over 0.5% of etcd requests from 192.168.2.11:6443 are taking more than a second"
+              description: "The apiserver on 192.168.2.11:6443 is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum."
+              diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+          - exp_labels:
+              severity: critical
+              io_cause: etcd
+              instance: 192.168.2.12:6443
+            exp_annotations:
+              summary: "CAUSE etcd: over 0.5% of etcd requests from 192.168.2.12:6443 are taking more than a second"
+              description: "The apiserver on 192.168.2.12:6443 is seeing more than 0.5% of its requests to its LOCAL embedded etcd member take longer than one second, sustained 5m. The healthy measured value on all three members is exactly 0 over 48h; at the 2026-09-07 outage window it was 0.5-2%. This is the precursor to 'Kubelet stopped posting node status' with the host still up. Expect 'slow fdatasync' and 'apply request took too long' in journalctl -u k3s. If a device-latency alert is firing on the same node the cause is the disk; if not, look at etcd leader changes and CPU. Do not reboot or drain a second node — with one member impaired, losing another ends quorum."
+              diagnostic_query: "1 - (sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='1.0'}[5m])) / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job='apiserver',le='+Inf'}[5m]))) — then on the node: journalctl -u k3s --since '30 min ago' | grep -Ei 'fdatasync|apply request took|too many requests|leader'"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      - eval_time: 60m
+        alertname: NodeNFSRetransmissionRateHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeDiskReadLatencyHigh
+        exp_alerts: []
+    promql_expr_test:
+      # All three etcd members raise the urgency signal. Asserted as a count so the arithmetic of
+      # the bucket ratio is pinned without depending on float-exact division.
+      - expr: |
+          count(
+            (
+              1 - (
+                sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="1.0"}[5m]))
+                  / sum by (instance) (rate(etcd_request_duration_seconds_bucket{job="apiserver",le="+Inf"}[5m]))
+              )
+            ) > 0.005
+              and sum by (instance) (rate(etcd_request_duration_seconds_count{job="apiserver"}[5m])) > 0.1
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 3
+      # And the measured per-write service time: 1.0389 s, pinned as a band (see the note in the
+      # tonight fixture on promtool's bit-exact sample comparison).
+      - expr: |
+          count(
+            (
+              rate(node_disk_write_time_seconds_total{device="sda",instance="192.168.2.10:9100"}[5m])
+                / rate(node_disk_writes_completed_total{device="sda",instance="192.168.2.10:9100"}[5m])
+            ) > 1.038
+              and (
+                rate(node_disk_write_time_seconds_total{device="sda",instance="192.168.2.10:9100"}[5m])
+                  / rate(node_disk_writes_completed_total{device="sda",instance="192.168.2.10:9100"}[5m])
+              ) < 1.040
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+
+  # =============================================================================================
+  # 4. LONGHORN REBUILD STORM — a device genuinely pinned, with HEALTHY per-op latency.
+  #    Proves the saturation rule fires on its own cause, that the latency rules do not steal it,
+  #    and that NodeIOPressureNotLocalDisk stays quiet when the local disks ARE the explanation.
+  # =============================================================================================
+  - interval: 1m
+    name: device saturation with healthy latency - saturation fires, NotLocalDisk suppressed
+    input_series:
+      - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+15x120'            # 0.25 — well over the critical PSI threshold
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+58.2x120'          # 0.97 — pinned
+      - series: 'node_disk_writes_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+18000x120'         # 300 writes/s
+      - series: 'node_disk_write_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+90x120'            # 1.5 s/s / 300 ops/s = 5 ms per write — HEALTHY
+      - series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+600x120'
+      - series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+0x120'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NodeDiskDeviceSaturated
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: device-saturation
+              job: node-exporter
+              instance: 192.168.2.10:9100
+              device: sda
+            exp_annotations:
+              summary: "CAUSE device saturation: sda on 192.168.2.10:9100 is busy >85% of the time"
+              description: "Block device sda on 192.168.2.10:9100 has been servicing I/O for more than 85% of wall clock for 10m — it is queue-bound, not slow. This is the SATURATION class of I/O pressure: too much work for the device, while latency per operation may be perfectly normal. Most likely a Longhorn replica rebuild, a Velero backup or a CDI import. The normal cluster maximum is 0.61 (prod-1, measured over 48h). Throttle the writer; do NOT replace a disk on this signal alone — see the per-op latency rules for that."
+              diagnostic_query: "topk(5, rate(node_disk_io_time_seconds_total[5m])) — then compare with the latency rules: saturated-but-fast is a workload problem, slow-but-not-saturated is a hardware problem."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # Saturated but fast: the hardware rules must not claim this event.
+      - eval_time: 60m
+        alertname: NodeDiskWriteLatencyHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NodeDiskWriteLatencyCritical
+        exp_alerts: []
+      # And the discriminator stands down, because the local disk IS the explanation.
+      - eval_time: 60m
+        alertname: NodeIOPressureNotLocalDisk
+        exp_alerts: []
+
+  # =============================================================================================
+  # 5. NFS TRANSPORT — a retransmitting export, plus a node whose op rate is below the guard.
+  # =============================================================================================
+  - interval: 1m
+    name: nfs retransmissions - transport rule fires on prod-1, traffic guard suppresses prod-2
+    input_series:
+      - series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+7200x120'          # 120 RPC/s
+      - series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+300x120'           # 5/s -> 4.17% retransmitted
+      # THE TRAFFIC GUARD'S REASON FOR EXISTING: 10% of six RPCs a minute is not a signal.
+      - series: 'node_nfs_rpcs_total{job="node-exporter",instance="192.168.2.11:9100"}'
+        values: '0+6x120'             # 0.1 RPC/s -> BELOW the >1/s guard
+      - series: 'node_nfs_rpc_retransmissions_total{job="node-exporter",instance="192.168.2.11:9100"}'
+        values: '0+0.6x120'           # 0.01/s -> ratio 0.10, and deliberately ignored
+      - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+0.72x120'          # 0.012 — the node is not even stalled yet
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NodeNFSRetransmissionRateHigh
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-transport
+              instance: 192.168.2.10:9100
+            exp_annotations:
+              summary: "CAUSE NFS transport: 192.168.2.10:9100 is retransmitting over 1% of its NFS RPCs"
+              description: "More than 1% of this node's NFS client RPCs are being retransmitted, sustained for 10m at more than 1 RPC/s. The measured baseline on all three nodes is exactly ZERO over 48h, so any value here is real: the NFS server 192.168.2.102, or the path to it, is dropping or timing out requests. Because I/O PSI counts NFS stalls as I/O stalls, this is the cause class that makes a node look disk-starved while every local device is idle. Investigate the server and the link — not this node's disks."
+              diagnostic_query: "sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m])) / sum by (instance) (rate(node_nfs_rpcs_total[5m])) — and the op rate driving it: sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m]))"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      - eval_time: 60m
+        alertname: NodeIOPressureNotLocalDisk
+        exp_alerts: []
+    promql_expr_test:
+      # The ratio is above threshold on BOTH nodes; only the traffic guard separates them.
+      - expr: |
+          count(
+            sum by (instance) (rate(node_nfs_rpc_retransmissions_total[5m]))
+              / sum by (instance) (rate(node_nfs_rpcs_total[5m])) > 0.01
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 2
+
+  # =============================================================================================
+  # 6. READ LATENCY — a failing device, plus a device below the op-rate guard.
+  # =============================================================================================
+  - interval: 1m
+    name: read latency - fires at 400ms on a busy device, guard suppresses a 1s trickle
+    input_series:
+      - series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
+        values: '0+1200x120'          # 20 reads/s -> above the guard
+      - series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdg"}'
+        values: '0+480x120'           # 8 s/s / 20 ops/s = 400 ms per read
+      # 1 SECOND per read at half a read per second: the artefact the guard exists to drop.
+      - series: 'node_disk_reads_completed_total{job="node-exporter",instance="192.168.2.10:9100",device="sdh"}'
+        values: '0+30x120'            # 0.5 reads/s -> BELOW the guard
+      - series: 'node_disk_read_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sdh"}'
+        values: '0+30x120'            # 1.0 s per read, and deliberately ignored
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NodeDiskReadLatencyHigh
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: device-latency
+              job: node-exporter
+              instance: 192.168.2.10:9100
+              device: sdg
+            exp_annotations:
+              summary: "CAUSE device latency: sdg on 192.168.2.10:9100 is taking over 250ms per read"
+              description: "Average read service time on sdg of 192.168.2.10:9100 has exceeded 250ms for 15m at more than 10 reads/s. The spinning media disks on these nodes peak at 156ms and the SSDs at 6ms in normal operation, so this threshold means a failing or resetting device, not a slow one. Check SMART and dmesg for ATA resets and link errors."
+              diagnostic_query: "rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) and rate(node_disk_reads_completed_total[5m]) > 10"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+    promql_expr_test:
+      # Both devices exceed 250 ms per read; only the guard tells them apart.
+      - expr: |
+          count(
+            rate(node_disk_read_time_seconds_total[5m]) / rate(node_disk_reads_completed_total[5m]) > 0.250
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 2
+
+  # =============================================================================================
+  # 7. THE `unless` SEMANTICS, ASSERTED DIRECTLY.
+  #    NodeIOPressureNotLocalDisk must fire for a stalled node whose devices are SO quiet that the
+  #    op-rate-guarded latency expressions produce NO SERIES AT ALL. An `and <expr> < threshold`
+  #    formulation returns nothing here and the alert vanishes exactly when it is most needed —
+  #    the structurally-inert shape this repo shipped in VeleroNoRecentBackup. `unless` with an
+  #    empty right-hand side removes nothing, so the alert survives. Only PSI and one idle device
+  #    are supplied: no write or read counters exist in this fixture.
+  # =============================================================================================
+  - interval: 1m
+    name: unless semantics - NotLocalDisk still fires when no device is busy enough to measure
+    input_series:
+      - series: 'node_pressure_io_stalled_seconds_total{job="node-exporter",instance="192.168.2.10:9100"}'
+        values: '0+12.588x120'        # 0.2098
+      - series: 'node_disk_io_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",device="sda"}'
+        values: '0+1.2x120'           # 0.02 — practically idle
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NodeIOPressureNotLocalDisk
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              io_cause: not-local-disk
+              instance: 192.168.2.10:9100
+            exp_annotations:
+              summary: "CAUSE not local disk: 192.168.2.10:9100 is I/O-stalled while every local device is idle and healthy"
+              description: "I/O PSI (full) above 10% for 5m on 192.168.2.10:9100 (.10 is k3s-prod-1, .11 is prod-2, .12 is prod-3) while NO local block device is above 50% busy, NO device exceeds 20ms per write and NO device exceeds 250ms per read. The stall is therefore NOT this node's disks. On this cluster that leaves the NFS export at 192.168.2.102: I/O PSI counts NFS stalls, which is why the block-layer metrics look innocent. This is exactly the 2026-09-14 02:40Z profile — PSI 20.98%, busiest device 0.27, 1.6ms writes, 114 NFS reads/s. Do NOT replace a disk and do NOT throttle Longhorn on this alert. The next question is whether OUR OWN NFS op rate is high (our workload is hammering the export) or near zero (something outside this cluster is — the Proxmox media stack still has operational control of the same files)."
+              diagnostic_query: "sum by (instance, method) (rate(node_nfs_requests_total{method=~'Read|Write'}[5m])) — high on this node means our workload, near zero with PSI still high means an external consumer of 192.168.2.102. Then name the pod: topk(5, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m]))"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+    promql_expr_test:
+      # The guarded latency expression really does produce nothing on this fixture — which is what
+      # makes the assertion above a test of `unless` and not an accident.
+      - expr: |
+          count(
+            (rate(node_disk_write_time_seconds_total[5m]) / rate(node_disk_writes_completed_total[5m]))
+              and rate(node_disk_writes_completed_total[5m]) > 10
+          )
+        eval_time: 60m
+        exp_samples: []

platform/monitoring/tests/run.sh

@@ -116,12 +116,24 @@ fi
 # `.git` file points outside the mount, so git cannot read the index there — and with
 # `pipefail` that failure silently emptied the discovery, i.e. a green run over ZERO tests
 # (observed 2026-09-12). `find` needs no repo metadata, so discovery cannot be disabled by
-# how the tree happens to be mounted. .git/ is pruned so a stale object copy is never picked up.
+# how the tree happens to be mounted. .git/ is pruned so a stale object copy is never picked up,
+# and .claude/ with it: .claude/worktrees/agent-*/ holds full repo copies for isolated agent
+# runs, so an unpruned find reported the SAME manifest under test once per worktree (up to 18
+# times), and the L0 check that compares this listing against the tree failed on paths that
+# are not authored surface at all (.claude/ is gitignored, .gitignore:43).
 discover() {
   # Companion convention, anywhere in the tree.
-  find "${ROOT}" -path "${ROOT}/.git" -prune -o -name '*.promtool-test.yaml' -print 2>/dev/null
-  # Suite convention: this directory's own *.test.yaml files.
-  find "${HERE}" -maxdepth 1 -name '*.test.yaml' -print 2>/dev/null
+  find "${ROOT}" \( -path "${ROOT}/.git" -o -path "${ROOT}/.claude" \) -prune -o -name '*.promtool-test.yaml' -print 2>/dev/null
+  # Suite convention, ANYWHERE in the tree — not just this directory.
+  #
+  # This used to be `find "${HERE}" -maxdepth 1`, which made the suite convention a
+  # platform/monitoring-only privilege while the header above documented it as a general one. The
+  # consequence was silent: a suite added under any other component (apps/wave-0/tests/, say) was
+  # never discovered, never run, and still LOOKED like coverage in review — the same shape of hole
+  # as the rules this harness exists to test. Found 2026-09-13 adding apps/wave-*/tests/. Scoped by
+  # NAME only; manifests_for() then refuses (loudly) any *.test.yaml that is not inside a tests/
+  # directory, so a misplaced file fails rather than disappearing.
+  find "${ROOT}" \( -path "${ROOT}/.git" -o -path "${ROOT}/.claude" \) -prune -o -name '*.test.yaml' -print 2>/dev/null
 }
 
 TESTS=()