feat(plex): wire PLEX_CLAIM from a SOPS Secret so the server can be claimed

Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while
the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving
PLEX_CLAIM unwired; the operator has decided to override it. This implements the
wiring in the way that respects the half of that argument which is still true.

WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL,
far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers.
So the committed value is a documented PLACEHOLDER and the live token is injected
by the operator inside that window, with the reconcile FORCED rather than waited
for. runbooks/plex-claim.md is that timed procedure.

WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret
would crashloop the pod". Half of that is wrong, and the difference is the whole
design. Established from plexinc/pms-docker's own first-run script (branch
`master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked
byte-identical against the same file inside the running :latest image:

  * the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the
    PlexOnlineToken pref in /config being empty. Once the server is claimed the
    block is skipped entirely, so the env is a NO-OP on every later start
    whatever it holds ("If server is already logged in, this parameter is
    ignored" -- upstream README).
  * with an invalid/expired token on a still-unclaimed /config, plex.tv answers
    403 with an empty body; curl's status is never checked, the token sed yields
    empty, the success branch at line 120 does not fire, there is no else branch
    and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up
    UNCLAIMED AND SERVING. It does not crashloop.
  * the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even
    if a cont-init script failed), so even a non-zero exit would not halt
    startup.

An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it
is CreateContainerConfigError), which is why the Secret is committed, encrypted
and listed in the wave kustomization rather than hand-created at claim time.

NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had
to tolerate a missing Secret or could not start without a VALID token. Neither
holds: the Secret always exists in the Flux graph, and an invalid token is
non-fatal. schemas/servicedefinition.schema.json is untouched, and so are
catalog/render.py's env emission and the renderer contract.

Changes:
  * apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex,
    key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header
    carries NO dollar-brace expressions on purpose: kustomize preserves body
    comments into the stream envsubst reads (per
    conformance/tests/l0-static/flux-postbuild-substitution), so quoting the
    upstream shell conditions verbatim would have been a live substitution
    hazard on apps-wave-0, not prose.
  * catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the
    "deliberately NOT set" block rewritten around the TTL + one-shot design, and
    the stale crashloop claim in the /config storage comment corrected.
  * apps/wave-0/plex.yaml -- re-rendered with the canonical command
    (`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden
    under conformance/tests/l1-rendering covers plex, so this is the only
    rendered surface that moves.
  * catalog/render.py -- plan_apps_waves places `<svc>-claim-secret.sops.yaml`
    in the app's OWN wave, like the bootstrap secret, plus a selftest assertion.
    Without this, the next `--emit-waves` would have silently dropped the Secret
    from apps/wave-0/kustomization.yaml and put the pod into
    CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned
    subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one
    new line was applied without re-emitting the rest.
  * runbooks/plex-claim.md + its kustomization.yaml generator entry + README
    index -- the operator procedure that fits in 4 minutes, including the single
    forced-reconcile command and the mandatory pod roll (a changed Secret does
    NOT restart anything: PLEX_CLAIM is read only at container start and the pod
    template is byte-identical across this change).

Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02
(ServiceDefinition onboarding contract).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.standup-dns/dns-20260914-pre-externaldns-scope.txt

@@ -0,0 +1,97 @@
+# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev`
+# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied.
+#
+# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers,
+# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not
+# merely left alone. That is a one-way, outward-facing operation against a live authoritative
+# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the
+# reconcile removes can be restored by hand from this file. It is the same discipline as
+# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable
+# change rather than to a migration.
+#
+# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones/<id>/dns_records, all pages) using the
+# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the
+# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the
+# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the
+# devcontainer, which has previously produced false "no records" readings — the API is the only
+# reading that is both complete and authoritative.
+#
+# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the
+# TTL and the proxy flag, not just the value.
+#
+#   <type>|<name>|<content>|<ttl>|<proxied>
+#
+# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here
+# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address.
+#
+# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry
+# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which
+# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy
+# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS
+# cannot touch them regardless of what this branch does:
+#
+#   *.keithjackson.dev            -> 136.51.115.252   legacy stack (the zone wildcard)
+#   brickbreak.keithjackson.dev   -> 20.236.242.14    unrelated, off-prem
+#   dev.brickbreak…               -> 20.236.242.14    unrelated, off-prem
+#
+# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in
+# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of
+# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does
+# NOT cover `<svc>.home.keithjackson.dev`: when a specific `<svc>.home` record is removed, that
+# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended
+# end state for `internal` services and it is also precisely why Pi-hole must already be
+# answering the LAN authoritatively before this branch merges.
+#
+# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog
+# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its
+# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed
+# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an
+# HTTPRoute that already had a published record produced
+# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile
+# interval (~60s) later, and nothing else in the zone moved.
+#
+# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its
+# own): recreate the A record with its value/ttl/proxied from this file, then recreate the
+# matching `a-<name>` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod,
+# external-dns/resource=…` string is what re-establishes ownership. An A record restored without
+# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value
+# points the registry at a source object that does not exist.
+A|*.keithjackson.dev|136.51.115.252|1|false
+A|auth.home.keithjackson.dev|192.168.2.20|1|false
+A|brickbreak.keithjackson.dev|20.236.242.14|1|false
+A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false
+A|crafty.home.keithjackson.dev|192.168.2.20|1|false
+A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false
+A|gitea.home.keithjackson.dev|192.168.2.20|1|false
+A|grafana.home.keithjackson.dev|192.168.2.20|1|false
+A|headlamp.home.keithjackson.dev|192.168.2.20|1|false
+A|home.keithjackson.dev|192.168.2.20|1|false
+A|jackett.home.keithjackson.dev|192.168.2.20|1|false
+A|keycloak.home.keithjackson.dev|192.168.2.20|1|false
+A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false
+A|pihole.home.keithjackson.dev|192.168.2.20|1|false
+A|plex.home.keithjackson.dev|192.168.2.20|1|false
+A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false
+A|radarr.home.keithjackson.dev|192.168.2.20|1|false
+A|seerr.home.keithjackson.dev|192.168.2.20|1|false
+A|semaphore.home.keithjackson.dev|192.168.2.20|1|false
+A|sonarr.home.keithjackson.dev|192.168.2.20|1|false
+A|unifi.home.keithjackson.dev|192.168.2.21|1|false
+TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false
+TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false
+TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false
+TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false
+TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false
+TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false
+TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false
+TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false
+TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false
+TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false
+TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false
+TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false
+TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false
+TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false
+TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false
+TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false
+TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false
+TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false

DECISIONS-LOG.md

@@ -2424,6 +2424,148 @@ container dies with `PermissionError` on every run; my older base had `0400` and
 - The apex flip itself is `migration/dns-apex-inversion`'s job. This change only guarantees the realm
   follows git once the knob moves; it does not move it.
 
+## 2026-09-12 — The silent-gateway outage now has an alert, and the direct signal turned out not to exist (branch `feat/egress-gateway-staleness-alert`)
+
+Hand-off item 4 from the section above: *"A silent gateway outage still has no alert… nothing
+watches the invariant `egress-uplink` label == `cilium-dbg bpf egress list`'s Egress Ifindex."*
+Closed, as far as Prometheus can close it — and the part that cannot be closed is recorded here
+rather than papered over with a plausible-looking metric name.
+
+### The direct signal is not available, verified rather than assumed
+
+The invariant wants the BPF egress map's contents. They are not in Prometheus, for two independent
+reasons, both checked read-only against live prod:
+
+| check | result |
+|---|---|
+| `{__name__=~"cilium.*egress.*"}`, `cilium_drop.*`, `hubble_drop.*` | **empty vector** — no `cilium_*`/`hubble_*` series exist at all |
+| container ports on the cilium DaemonSet | `health=9879`, `peer-service=4244` — **no metrics port**; only the operator has `operator-prometheus-serve-addr :9963` |
+| `kubectl get servicemonitors -A \| grep -iE 'cilium\|hubble'` | none — Hubble/Cilium metrics export is still the open TODO in `platform/cilium/helmrelease.yaml` |
+
+So `HubbleDeniedFlowSpike` cannot fire today either; that is a separate, pre-existing gap, noted
+and left alone. And even with agent metrics turned on, **Cilium OSS publishes no series carrying
+egress-map contents** — the map is readable only through `cilium-dbg bpf egress list`, a CLI over
+the BPF map. Nothing was invented to fill the hole.
+
+What node-exporter *does* export is the live tun0 **ifindex as a metric value**:
+`node_network_iface_id{device="tun0"}` = `129` on `k3s-prod-3`. Read-only on the live cluster, all
+three agree right now:
+
+```
+node label    homelab.keithjackson.dev/egress-uplink = 10.150.192.86-129
+node-exporter node_network_iface_id{device="tun0"}   = 129
+bpf map       Egress IP 10.150.192.86 / Egress Ifindex 129   (all 16 rows, 4 vpn-class pods)
+```
+
+(The map on the two NON-gateway agents reads `Egress IP 0.0.0.0 / Ifindex 0` with
+`Gateway IP 192.168.2.12` — only the gateway node resolves the interface. Worth knowing before
+debugging the wrong agent.)
+
+### PromQL cannot express the equality test — so the invariant is counted instead
+
+The label's value is `<addr>-<ifindex>`; the live ifindex is a metric VALUE. PromQL has no
+label-to-number conversion, so `label == value` is **not expressible**, and there is no
+level-triggered form of this comparison. What is expressible is the same invariant counted over a
+window: **the published label must change at least as often as tun0 is re-created.**
+
+```promql
+(
+    changes(max(node_network_iface_id{device="tun0"})[1h:1m])
+  >
+    (count(count_over_time(max by (label_homelab_keithjackson_dev_egress_uplink) (
+       kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})[1h:1m])) - 1)
+)
+and on()
+(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+```
+
+`count(count_over_time(max by (<label>) (…)[1h:1m]))` is the number of DISTINCT label values in the
+window. The `max by (<label>)` wrapper is load-bearing: it aggregates away `pod`/`instance` so a
+kube-state-metrics restart is not read as a label change. Both sides reduce to a single unlabelled
+series, which is why `and on()` joins — the same reason `VPNTunnelDown` needs `on()`.
+
+### Proof it evaluates — executed against prod's Prometheus, port-forward + `/api/v1/query`
+
+There is no promtool in this repo, so query execution IS the verification. The **rendered** expr was
+extracted from `kubectl kustomize platform-config` output and run verbatim:
+
+| state | query | result |
+|---|---|---|
+| healthy, now | rendered `VPNEgressSteeringStale` | `[]` — not firing |
+| **no gateway node** (`@ 1789245000`, before prod's stand-up: no tun0 series, no labels) | same expr | `[]` — cannot fire |
+| **stale**, constructed from real data | same expr with the already-exported `egress-gateway` label substituted for `egress-uplink`, window 3h | **`1` — FIRES** |
+| same expr, window 1h | | `[]` |
+
+The stale case is honest and cost prod nothing: prod's tun0 really did go **128 → 129** ~2.5h
+earlier when it moved to PIA `.135`, and the `egress-gateway` label really did *not* follow it
+(`changes(...[3h:1m])` = 1, distinct label values = 1 → `1 > 0`). That is bit-for-bit the shape of
+a label that missed a flap. The healthy counterfactual is the one thing left as reasoning rather
+than a query: over that same 3h the REAL `egress-uplink` label held two values
+(`10.72.192.7-128`, then `10.150.192.86-129`), giving `1 > 1` = false. It could not be queried
+because the label was not exported yet — which is the next point.
+
+Supporting queries, also live: the distinct-value idiom returns **1** for a label that held still
+over the window and **2** for one whose value changed inside it (node 3's `egress-gateway` label
+over 12h, which went `""` → `"true"` at stand-up); the gateway guard returns 1; subquery +
+`changes()` returns 0/1/1/1 over 15m/3h/6h/12h as the flap history requires.
+
+### Behaviour at the edges
+
+- **No gateway node:** three independent reasons it cannot fire — no tun0 series, no uplink label,
+  and the `kube_node_labels` guard `VPNTunnelDown` already established. Verified at a past
+  timestamp when prod genuinely had no gateway. This matters: `VPNTunnelDown` fired permanently on
+  prod for exactly this reason and had to be retrofitted.
+- **Convergence:** the up-script rewrites the label within seconds of a connect, worst case ~5m
+  (its 30 × 10s kube-API retry budget on a cold boot, when the API is not up yet). The condition is
+  true during that gap, so `for: 15m` — comfortably past the 2-minute timer and the 5-minute retry
+  budget — is what keeps a normal reconnect quiet.
+- **Bounded, deliberately:** a missed flap alerts from +15m to +1h, then RESOLVES even if egress is
+  still dead. An unbounded critical on `email-critical` is the worse failure for this repo, and 15
+  minutes still beats 80. A level-triggered version needs the map's ifindex as a real metric — the
+  node-exporter textfile collector route, still open.
+
+### The second rule exists because the first one fails OPEN
+
+`VPNEgressSteeringStale` needs `egress-uplink` in kube-state-metrics. Without it the comparison
+degrades to an empty vector and alerts nothing — so `VPNEgressUplinkLabelMissing` (warning,
+`for: 30m`) watches the precondition: a node labelled `egress-gateway=true` with no `egress-uplink`
+label visible. Live right now it returns **1**, correctly: the KSM allowlist extension is in this
+same branch and not yet deployed. It self-clears on the first reconcile, and `for: 30m` covers the
+kube-state-metrics roll. The rule and the allowlist are in the same Flux kustomization on purpose.
+
+`kube-state-metrics.metricLabelsAllowlist` now reads
+`nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]` — a label
+DIMENSION on the same 3 `kube_node_labels` series, not new series, on exactly one node.
+
+### Rejected, with reasons
+
+- **A black-box egress probe** is the most honest signal — it is the only one that answers "does a
+  vpn-class pod's traffic actually leave via the tunnel?" and the only one that catches "the label
+  was written and Cilium ignored it anyway". Rejected **for now**: there is no blackbox-exporter and
+  no probe CronJob in this repo, so it means a new always-on component (itself vpn-class, itself an
+  alerting surface, itself dependent on a third-party endpoint) — and it could not have been
+  verified on prod today. It remains the right next step.
+- **A node-exporter textfile metric from the re-steer script** (hand-off item 4's own suggestion) is
+  the only route to a true level-triggered label-vs-MAP comparison. It needs a textfile directory
+  mounted into the node-exporter DaemonSet plus an Ansible change applied on a prod node — neither
+  of which this branch could execute or verify. Left open, now with the reason recorded.
+- **`absent(node_network_receive_bytes_total{device="tun0"})`-style presence checks** — that IS
+  `VPNTunnelDown`, and it is exactly the blind spot: green while steering points at a dead ifindex.
+
+### Left for the operator / not settled here
+
+1. **The rules are unproven in the firing sense.** The expressions evaluate against live prod data
+   and the stale branch was exercised with a real flap, but no alert has actually reached
+   Alertmanager or the inbox. `conformance/tests/l3-fault-injection/vpn-killswitch` stops and
+   restarts the tunnel and is the natural place to assert `VPNEgressSteeringStale` stays quiet
+   through a healthy reconnect; the alerting L3 test with the SMTP sink is where a fired-and-emailed
+   proof would come from. Neither was fired at prod unasked.
+2. **`HubbleDeniedFlowSpike` cannot fire** — no cilium/hubble metrics are scraped on either cluster.
+   Pre-existing, orthogonal, and the same TODO in `platform/cilium/helmrelease.yaml` blocks it.
+3. **Window vs. stickiness** is a judgement call, not a measurement: 1h chosen so a missed flap
+   alerts for ~45 minutes. If the operator would rather have a sticky alert that needs manual
+   acknowledgement, that is the textfile-collector design, not a tweak to this one.
+
 ## 2026-09-13 — **PROD IS THE APEX.** Dev decommissioned; and a GitOps deadlock found the hard way
 
 Prod serves `home.keithjackson.dev`. Dev's three VMs (Proxmox 101/102/105) are stopped with
@@ -3833,3 +3975,744 @@ Measured baselines for calibration: NFS writes 12.0 / 16.1 / 17.3 MB/s at 1 MB f
 16 MB; NFS RPC retransmissions 0 during the spike. 24h PSI distribution — prod-1 14 min >5%, 11 min
 >10%, 3 min >20%; **prod-2 0 min >5%**; prod-3 5 min. Two distinct prod-1 events in 24h, the earlier
 coinciding with velero's first full volume backup.
+
+**THREE OF MY mountstats FIGURES WERE WRONG — corrected 2026-09-15.** I handed an agent a baseline of
+"server response 4.0–4.8 ms/op, client queue time 0 ms/op, 0 major timeouts over 24h, backlog 0".
+It re-measured and found:
+- **client queue time is 0.013–0.067 ms/op, max 0.333 — not zero.** I read rounded output as zero.
+- **"0 over 24h" was really 0 over about ten minutes.** The node-exporter DaemonSet had just rolled to
+  pick up the collector: `time() - process_start_time_seconds` read 558–561 s, and
+  `count_over_time(...[24h])` returned **18** samples rather than ~2,880. I quoted a 24-hour baseline
+  from ten minutes of data. Same error class as the label-values mistake above — trusting a query
+  whose window was never valid. No shipped rule uses a range longer than 15m for that reason.
+- **the baseline was an IDLE one.** No READ or WRITE traffic was flowing; 4.4 ms is a cached STATFS,
+  not a 1 MB read off a spinning array, and STATFS keepalive is the top operation on every mount. So
+  thresholds could not be derived as a multiple of it and were argued from the link and the array
+  instead.
+
+Two genuine discoveries from that re-measurement:
+- **Mounts sharing an NFS superblock share the kernel's RPC statistics.** prod-1's
+  `/mnt/Storage/plex` and `/mnt/Storage/plex/downloads` report identical op rates to every digit, as
+  do prod-3's two nextcloud mounts. One slow export therefore raises two alerts. Left documented
+  rather than aggregated away, because aggregating hides which path is affected.
+- **`maximum_rpc_slots` high-water: prod-1 586, prod-3 162, prod-2 2.** prod-1's slot table has grown
+  enormously relative to its peers — the client responds to pressure by growing the table rather than
+  queueing, which is also why its queue time stays near zero. This is an independent signal that
+  prod-1 sustains far more concurrent NFS work than the other two nodes, and it is consistent with
+  three of the four NFS-heavy pods being scheduled there.
+
+---
+
+## 2026-09-14 — FINDING (operator): public DNS carries internal RFC1918 addresses
+
+The operator pointed out that public DNS records resolve to internal, non-routable IPs. Verified, and
+it is worse than disclosure alone.
+
+**What is published.** `platform/externaldns/helmrelease.yaml` runs `provider: cloudflare` with
+`sources: [gateway-httproute, service]` and `policy: sync`. external-dns faithfully publishes each
+source's address, and that address is the Cilium Gateway's LoadBalancer VIP `192.168.2.20` (plus the
+L4 service IPs `.21` unifi and `.22` crafty-mc). So ~16 hostnames in a **public authoritative zone**
+answer with RFC1918 space.
+
+**This contradicts the spec.** `docs/09-dns-and-ingress.md` R-DNS-03 states: *"**External**
+(`*.home.keithjackson.dev` from the internet): resolves to the **public IP**"*, *"**PiHole** …
+provides internal LAN DNS and the split-horizon answers"*, and *"`internal` services are unreachable
+from the public internet"*. The deployed arrangement inverts it — the internal VIP is what the world
+is told, and there is no internal authority at all (Pi-hole serves no DNS; see the entry above).
+
+**Provenance, plainly.** The pattern is architectural and predates me: external-dns has always
+published the Gateway's address to Cloudflare, and on dev it did the same with dev's internal VIP
+under `owner=default`. What MY apex migration changed is *which* names carry it — prod moved from
+`prod.home.keithjackson.dev` to the apex, so `home.keithjackson.dev` and its subdomains now publish
+internal addresses where previously they were dev's records or the wildcard. I propagated an existing
+flaw onto the apex; I did not invent it. I also widened prod's `EXTERNALDNS_DOMAIN_FILTER` from its
+own subtree to the zone apex `keithjackson.dev` per finding F-DNS1 (a strict-subdomain filter drops
+the parent zone), which is the documented fallback but leaves `txtOwnerId` as the sole ownership guard.
+
+**Four consequences:**
+1. **Information disclosure.** Anyone can enumerate the internal subnet, the Gateway VIP, both L4
+   service addresses, and the full service inventory — sonarr, radarr, qbittorrent, jackett, gitea,
+   nextcloud, keycloak, semaphore, pihole, plex, seerr, crafty, headlamp, unifi, runbooks, grafana.
+   No credentials leak, but topology and a target list do.
+2. **A DNS-rebind primitive.** A public name answering with a private address is exactly the rebind
+   shape browsers and resolvers defend against — which is *why* the SOHO router at 192.168.2.1 strips
+   these answers, and therefore why LAN clients need per-client dnsmasq overrides. The security
+   control is fighting the misconfiguration.
+3. **The five `public` services are not actually publicly reachable.** plex, gitea, seerr, keycloak and
+   nextcloud declare `exposure: public`, yet their records point at 192.168.2.20, which no internet
+   client can route to. The declaration is unsatisfied.
+4. **Seven `internal` services are advertised publicly** — pihole, radarr, sonarr, jackett, vm-demo,
+   semaphore, qbittorrent — which the spec says must be unreachable from the internet. They are
+   unreachable, but they should not be *listed*.
+
+**Removing the records is NOT a safe first step.** The `*.home.keithjackson.dev` wildcard that used to
+answer for this subtree is **gone** — `nothing-here.home.keithjackson.dev` returns NXDOMAIN — so
+deleting the specific records yields no answer at all rather than falling back. Only
+`*.keithjackson.dev` still resolves, to the legacy public IP `136.51.115.252` (ports 80/443 open,
+serving a placeholder certificate `CN=%D`, so the legacy stack's TLS is not configured for these
+names either).
+
+**Sequenced remediation — NOT started, needs the operator's decisions:**
+1. **Stand up internal DNS first.** Expose Pi-hole's DNS on port 53 via a LoadBalancer IP and make it
+   authoritative for the zone internally. Until this exists, removing public records breaks LAN
+   resolution outright, because the router strips RFC1918 and nothing else answers.
+2. **Then stop publishing internal names publicly.** Either scope external-dns to only the services
+   declared `public`, or point it at an internal provider, or drop the Cloudflare provider for this
+   subtree entirely. This is the step that closes the disclosure.
+3. **Then decide what `public` means for the five.** If they are genuinely meant to be
+   internet-reachable, their records must target the public IP with edge port-forwards and real
+   certificates; if not, they should be re-declared `internal`. Today they are neither.
+
+---
+
+## 2026-09-14 — CORRECTION: unifi/crafty were never up; the ETP defect is real but was not an outage
+
+I told the operator their UniFi controller and Minecraft server had been "unreachable from the LAN for
+up to 8 days" and treated it as a live outage. **That was wrong.** An agent checked the thing I had
+not: it probed the backend pods DIRECTLY, bypassing the VIP, the traffic policy and L2 entirely.
+
+    OPEN      10.0.0.11:8443    crafty's own UI          <- control: the method works
+    REFUSED   10.0.0.11:25565   Minecraft Java
+    REFUSED   10.0.0.74:8443    UniFi
+    REFUSED   10.0.0.74:8080    UniFi inform
+    REFUSED   10.0.0.74:11443   UniFi UOS UI
+    OPEN      10.0.0.114:53     pihole                   <- control
+
+**There is no application behind any of those four ports.** `/crafty/servers` is empty — no Minecraft
+server has ever been created. `apps/vms/unifi/README.md` step 5 still reads *"→ NEXT: install UOS in
+the guest"*, with the unattended install flagged at line 108 as *"STILL OPEN. This is the blocker."*
+The VM is a bare Ubuntu 24.04 with a guest agent. Nothing regressed; the 8-day-old leases are simply
+8-day-old Services. I verified all of this independently before accepting the correction.
+
+**What I got right, and what I got wrong.** The `externalTrafficPolicy: Local` defect is genuine:
+Cilium's L2 announcer elects a lease holder without regard to which nodes hold a backend, so under
+`Local` the frontend is absent from the announcing node's service table and the VIP answers ICMP
+unreachable — demonstrated with `cilium-dbg service list` by flipping the policy and watching the
+entry appear and vanish. What I got wrong was the inference: I observed unreachable ports, found a
+mechanism that would produce unreachable ports, and concluded it was the cause. A control probe of the
+backend would have separated them in one command, and I did not run one before reporting. **A
+plausible mechanism is not a diagnosis.** Same shape as the I/O-pressure spike, where the local disks
+were innocent and I nearly blamed the drive.
+
+The fix merged anyway (`adda6dd`) because it removes a landmine that would have blackholed both VIPs
+the moment those applications exist — and the guard
+`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node` fails on the pre-fix tree naming
+exactly those four Services.
+
+Two further things the agent got right by checking rather than assuming:
+- It nearly added `remote-node` to both CiliumNetworkPolicies, then dumped the endpoint's datapath
+  policy map and found `reserved:remote-node` was ALREADY allowed — Cilium's `cluster` entity subsumes
+  it, and both CNPs already admit `cluster`. Reverted. Pi-hole needs it explicitly only because its
+  policy deliberately omits `cluster`.
+- It did NOT copy Pi-hole's `loadBalancerSourceRanges`. Pi-hole was compensating for a restriction it
+  was giving up; these Services never had one, and adding a LAN-only filter could silently cut off
+  off-LAN players or a remote site informing over WAN.
+
+**The operator's actual next step here is not networking:** install UniFi OS in the guest, and create a
+Minecraft server in Crafty.
+
+---
+
+## 2026-09-14 — Crafty's first Minecraft server exists: Paper 26.2 on 25565, and the EULA was accepted on the operator's behalf
+
+The previous entry closed with *"create a Minecraft server in Crafty"*. Done. `/crafty/servers` is no
+longer empty, and `192.168.2.22:25565` now answers a real Minecraft Server List Ping from the LAN.
+
+**I ACCEPTED MOJANG'S EULA ON THE OPERATOR'S BEHALF.** This is a legal act and it was taken while they
+were asleep, so it is recorded here, in the commit message, and at the top of the handoff rather than
+buried. The agreement is the Minecraft End User Licence Agreement at <https://aka.ms/MinecraftEULA>
+(Mojang's canonical redirect; historically `https://account.mojang.com/documents/minecraft_eula`). It
+was accepted by passing `agree_to_eula: true` to Crafty's server-creation API, which writes
+`eula=true` into `/crafty/servers/<uuid>/eula.txt` (the mechanism is `main_controller.py:457-463`). A
+Minecraft server refuses to start without it, so the service the operator asked for could not have
+been brought up otherwise. **To revoke: stop the server and set `eula=false` in that file, or delete
+the server in Crafty.** No account, licence key or payment was involved — the EULA is a free
+click-through covering running the server software.
+
+**What was created, and why each choice.**
+
+- **Paper, not Vanilla.** Paper is a drop-in Vanilla-compatible server with substantially better tick
+  performance and a plugin API; unmodified Minecraft clients connect to it exactly as they would to
+  Vanilla, so nothing is given up on the player side. Vanilla would only be the right pick if the
+  operator wanted strict parity with Mojang's own jar or a mod loader (Fabric/Forge/NeoForge are also
+  in Crafty's bucket if that changes). Purpur and Folia were available and not chosen: Purpur adds
+  gameplay-changing configuration nobody asked for, and Folia's threaded-regions model targets very
+  large player counts and is the wrong shape for a home server.
+- **Version 26.2.** Mojang's `version_manifest_v2.json` reports `latest.release = 26.2`, released
+  2026-06-16 — about three months mature, so not a fresh drop. The current snapshot is `26.3-rc-2`,
+  and Crafty's bucket also lists `26.2-rc-2`; both were deliberately avoided, as the brief asked for a
+  current stable release and not a snapshot. Crafty verifies the jar against a published SHA256.
+- **A 2 GiB heap (`-Xms2000M -Xmx2000M`), not larger.** This was measured, not guessed. The container's
+  memory limit is 4Gi. With the server idle-but-running, the cgroup reported `memory.current` of
+  2.86Gi against a 4Gi `memory.max` — that is the 2 GiB committed heap, plus roughly half a gigabyte of
+  JVM non-heap (metaspace, code cache, GC structures, thread stacks, memory-mapped region files), plus
+  the Crafty panel itself. A 3 GiB heap would have landed at or above the 4Gi cap and invited an
+  OOMKill. Minimum and maximum heap are set equal, which is the standard Paper recommendation to avoid
+  heap-resize pauses. Node `k3s-prod-3` has about 31 GiB allocatable and was only 38 percent used, so
+  the node was never the constraint — the container limit was.
+
+**The one repo change.** `platform/crafty/deployment.yaml`: the crafty container's memory *request*
+went 1Gi to 3Gi. The limit is untouched at 4Gi. Before a server existed, a 1Gi request was honest;
+now that a 2 GiB heap is permanently committed, a 1Gi request understates real usage roughly threefold,
+which both misleads the scheduler and makes this Burstable pod an early eviction candidate under node
+memory pressure. Nothing else was altered — not the L4 Services, not the NetworkPolicy, not the PVC.
+The existing egress policy already allowed `world`, so the jar download needed no new rule.
+
+**A trap worth recording: the create API silently ignores `autostart`.** The server-creation schema
+accepts `autostart` and `crashdetection` and validates them happily, but the values never reach the
+database — the stored columns are `auto_start` and `crash_detection`, and they came back `false` after
+a creation that had asked for `true`. They must be set with a separate `PATCH /api/v2/servers/<uuid>`.
+Had I trusted the create call, I would have reported autostart working when it was not, and the server
+would have stayed dead after the next pod restart. Both are now `true`, verified by reading them back.
+
+**THIS SERVER IS APP STATE. IT IS NOT IN GIT, AND A REBUILD FROM GIT ALONE WILL NOT RECREATE IT.**
+Same shape as the arr download-client rows. What git holds is only the *capacity* to run Minecraft
+servers: the Deployment, the PVC, the L4 Services and the policies. The server itself lives in two
+places, both on the `crafty-data` Longhorn PVC and neither reconstructible by Flux:
+
+1. **Crafty's SQLite database** (`/crafty/app/config/db/`, the `config` subPath) — the row defining
+   server `13c7b85d-00bb-4a58-8c86-2433ca47891c` ("homelab"): its name, port, jar, launch command,
+   heap sizes, autostart and crash-detection flags.
+2. **The server directory** (`/crafty/servers/13c7b85d-00bb-4a58-8c86-2433ca47891c/`, the `servers`
+   subPath) — `paper.jar`, `eula.txt`, `server.properties`, and **the generated world**, which is the
+   genuinely irreplaceable part. A jar can be re-downloaded; a world cannot be regenerated.
+
+If that PVC is lost and restored from nothing, `flux reconcile` brings back a healthy, *empty* Crafty
+with no servers and a refused connection on 25565 — precisely the state this entry is closing out.
+Recreating it by hand means: create the server in Crafty (Paper, 26.2, port 25565, 2 GiB heap), accept
+the EULA again, PATCH `auto_start` on — and accept that the world starts over. **The world is
+therefore backup-relevant in a way the manifests are not**, which makes the still-open no-backups gap
+(Proxmox migration entry) materially worse than it was yesterday: there is now player-authored state on
+this cluster. Crafty has its own scheduled-backup feature writing to the `backups` subPath, but that is
+same-PVC and so protects against an in-app mistake, not against loss of the volume. Configuring it, and
+getting the volume itself into a real backup, are both still open and both need the operator.
+
+**Proofs, run in the order the brief asked for.** Every one of these is real output, not inference.
+
+    (a) Crafty reports started   running=True  ver='Paper 26.2'  players=0/20
+                                 server log: 'Done (12.737s)! For help, type "help"'
+    (b) in-cluster, cross-node   from a pod on k3s-prod-1 while crafty ran on k3s-prod-3,
+                                 python3 sockets (no curl in these images):
+                                   10.0.1.95:25565     pod IP        SLP OK
+                                   10.43.5.109:25565   ClusterIP     SLP OK
+                                   192.168.2.22:25565  the VIP       SLP OK
+    (c) from the LAN, VIP        workstation host, bash /dev/tcp to 192.168.2.22:25565 -> connect OK
+    (d) Server List Ping         full handshake -> status -> ping/pong, from the workstation:
+                                   MOTD 'A Minecraft Server'   Version 'Paper 26.2'
+                                   Protocol 776                Players 0/20
+                                   round trip 12.3 ms          enforcesSecureChat True
+
+Proof (d) is the one that matters: it is the same exchange a Minecraft client performs to draw the
+server in its list, so it demonstrates the game protocol end to end rather than merely an open port.
+
+**Restart persistence — tested, and it moved nodes.** Deleting the pod rescheduled Crafty from
+`k3s-prod-3` onto `k3s-prod-1`. It came back `2/2 Running`, and with **no manual start issued** the
+server auto-started about 30 seconds later: same UUID, same world directory, `eula=true` intact,
+`Paper 26.2` answering. The LAN Server List Ping was then re-run against the VIP and still succeeded
+with the pod on a different node — which independently re-confirms the `externalTrafficPolicy: Cluster`
+fix from `adda6dd`, since under the old `Local` policy the VIP would have blackholed exactly here.
+
+**Left alone and verified unbroken:** the oauth2-proxy sidecar still returns 302 to
+`keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/auth` for an unauthenticated
+request, so the Keycloak gate on the web UI is intact. Note that `crafty.home.keithjackson.dev` does
+not resolve from the workstation shell — that is the pre-existing split-horizon DNS gap described in
+the entry above, not a regression from this work.
+
+**Still open for the operator:** review the EULA acceptance; decide whether the defaults are wanted
+(MOTD is still "A Minecraft Server", 20 player slots, no whitelist, online-mode on, and **the server is
+open to anyone who can reach the VIP** — a whitelist or `loadBalancerSourceRanges` is a judgement call I
+did not make alone); and get the world into a backup.
+
+---
+
+## 2026-09-14 — Minecraft server live; EULA accepted on the operator's behalf; world backed up
+
+The operator asked for the missing services to be brought up and went to bed. Crafty was healthy but
+had never had a server created — `/crafty/servers` was empty, which is why `192.168.2.22:25565`
+refused. **Paper 26.2 is now running and playable** (merged `9725751`).
+
+**EULA — ACCEPTED ON THE OPERATOR'S BEHALF, recorded here deliberately rather than buried.**
+Mojang's Minecraft EULA (https://aka.ms/MinecraftEULA) was accepted via Crafty's `agree_to_eula`,
+writing `eula=true` into `/crafty/servers/<uuid>/eula.txt`. A server refuses to start without it, so
+the requested service could not come up otherwise. No account, key or payment was involved. **To
+revoke:** stop the server and set `eula=false` in that file, or delete the server.
+
+Proven playable by a real **Server List Ping** handshake from the LAN — MOTD, `Paper 26.2`, protocol
+776, 0/20 players, 12.3 ms RTT — not merely an open port. Also proven across a pod restart: deleting
+the pod moved Crafty prod-3 -> prod-1 and it auto-started with the same world, which independently
+re-confirms the `externalTrafficPolicy: Cluster` fix, since under `Local` the VIP would have
+blackholed exactly there.
+
+**A trap worth remembering:** Crafty's create API ACCEPTS and validates `autostart` but stores
+`auto_start`, so it silently read back `false`. Only reading the record back caught it. Trusting the
+create call would have reported autostart working while the server stayed dead after any restart —
+the same shape as every other "a thing that looks applied and isn't" in this log.
+
+### Correction: the crafty world IS covered by velero — my query was wrong
+I reported "0 PodVolumeBackups for crafty" and treated the world as unprotected. **That was a bad
+query**, not a finding: I grepped PVB *names*, which are `velero-daily-<timestamp>-<random>` and never
+contain the namespace. Crafty had **2** PodVolumeBackups in the last run, among 171 across 24
+namespaces; `velero-daily` includes all namespaces with `defaultVolumesToFsBackup: true`. That is the
+fourth false negative from a sloppy query in two days — see the label-values and 24h-window entries
+above. The pattern is always the same: an empty result from an unvalidated query read as evidence.
+
+What WAS true is narrower: the last scheduled run was 2026-09-13 04:01, **before the world existed**,
+so the new state would have been unprotected until 03:00. I took a one-off backup immediately —
+`crafty-world-202609140709`, **Completed, 0 errors, 54/54 items**, `crafty-data` captured at
+343,898,625 bytes off-cluster in R2.
+
+**Still the operator's call**, left deliberately: server defaults are untouched (MOTD "A Minecraft
+Server", 20 slots, **no whitelist**), so the server is reachable by anyone who can reach the VIP. A
+whitelist or `loadBalancerSourceRanges` would restrict it — but restricting could cut off off-LAN
+players, which is why it is not mine to decide.
+
+---
+
+## 2026-09-14 — UniFi OS was NEVER the blocker: it is installed; the guest lost its IP
+
+Third correction in two days to a claim I made confidently. I told the operator UniFi OS Server had
+never been installed and that installing it was their next step. **Both halves were wrong**, and I had
+taken them from `apps/vms/unifi/README.md`, which was itself stale — I repeated documentation instead
+of checking the machine.
+
+**Verified independently on live prod:**
+
+| claim | evidence |
+|---|---|
+| UOS **is** installed and running | guest console log: `Started uosserver.service — UniFi OS Server Service` and `uosserver-updater.service` |
+| it is a real install, not a stub | guest filesystem `/` holds **6.3 GB of 40 GB**; bare Ubuntu plus our packages is ~2 GB |
+| **the guest has NO address** | `.status.interfaces` → `enp1s0`, `linkState: up`, **no `ipAddress`**; the VMI IP column is empty |
+| confirmed at L2 | `ip neigh` inside virt-launcher: `10.0.2.2 dev k6t-eth0 **FAILED**` — the guest does not answer ARP |
+| VIP / policy exonerated | `/dev/tcp` from virt-launcher to the pod IP *and* to `10.0.2.2`: 11443, 8443, 8080 and **22** all RST |
+
+So the controller runs and nothing can reach it, because its NIC lost its DHCP address. The cause is
+not determinable from outside the guest.
+
+**This nearly caused real damage.** Had the work proceeded on my stated premise it would have
+*reinstalled UniFi OS over a working controller*. The shipped `install-uos` now bails the moment
+systemd knows a `uosserver` unit — a guard that exists only because the agent checked the premise
+instead of executing it.
+
+**Second finding: there is no way into the guest at all.** No SSH key was ever installed, the stock
+`ubuntu` account is locked, and SSH cannot work by construction — KubeVirt masquerade forwards only
+declared ports and 22 is not among them, which the RST above confirms. So the deliverable became
+*console access*: a SOPS secret carrying `installer-url` **and `console-password`**, mounted as a
+read-only NoCloud disk and re-read on every boot.
+
+**The discovery that makes any of this work.** NoCloud's `instance-id` is
+`spec.template.spec.domain.firmware.uuid`, stamped on the VM object — cloud-init compares it against
+the DMI product UUID, so **editing `userData` alone changes nothing on a second boot**: the cached
+datasource is restored and `write_files`/`runcmd` are skipped entirely. That is why a previously
+shipped "unattended install" could never have taken effect, and why an earlier attempt was "tried and
+reverted". The UUID is now pinned in git and bumped once. (Also: the inline `userData` cap is exactly
+**2048 bytes**, established by server-side dry-run — 2048 accepted, 2049 rejected — and the payload
+ships at 1454, so it stayed inline.)
+
+Deliberately NOT done: putting the whole cloud-config behind `secretRef`. That would hide the shell in
+ciphertext, beyond the reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over the
+decrypted resource — exactly the nextcloud-db shape from 2026-09-13. Script plaintext, operator values
+in SOPS.
+
+**Unattended install remains unverified BY US.** Ubiquiti documents no silent flag, answer file or
+environment variables; `yes y | <installer> install` is convergent community practice and is shipped as
+such, labelled third-party. No Ubiquiti credential was touched and no installer was downloaded.
+
+**The operator's real next step is diagnosis, not installation:** set `console-password`, restart the
+VMI, then `virtctl -n unifi console unifi` and find out why `enp1s0` has no address — `ip -4 a`,
+`networkctl status enp1s0`, `journalctl -u systemd-networkd`, `ls /etc/netplan`. Capture state before
+restarting, since a restart may silently restore DHCP and destroy the evidence.
+
+The README's dev-era addressing was corrected throughout (VIP `.1.54`→`192.168.2.21`, pool and Gateway,
+device subnet, node name) along with several stale factual claims, since it doubles as the rebuild
+runbook.
+
+---
+
+## 2026-09-18 — VERIFIED: internal services no longer advertise themselves publicly
+
+The R-DNS-03 disclosure is closed for the `internal` tier. Confirmed at the authority
+(`betty.ns.cloudflare.com`, not a cache):
+
+| name | public DNS | via Pi-hole `.23` |
+|---|---|---|
+| sonarr, radarr, qbittorrent, jackett, semaphore, pihole | **GONE** | 192.168.2.20 |
+| plex, gitea, seerr, keycloak, nextcloud | 192.168.2.20 (untouched, as designed) | — |
+
+So split-horizon now genuinely exists: the names resolve internally and are absent externally, which
+is what `docs/09-dns-and-ingress.md` specified all along. The internal subnet, the Gateway VIP and six
+service names are no longer publicly enumerable, and the DNS-rebind primitive is gone for those six.
+
+**A correction to my own sequencing.** I had insisted this could not land until the operator repointed
+DHCP, on the grounds that removing public records would break LAN resolution. That was wrong, and
+measuring it took one command: the SOHO router at 192.168.2.1 returns **nothing** for all six names, so
+the public records were serving nobody on the LAN. The dependency I asserted did not exist. Landing it
+early closed a security finding a day sooner than my own plan allowed.
+
+**Operator clarifications recorded** (theirs, 2026-09-18):
+1. Pi-hole is **DNS only** — the SOHO router remains the DHCP server. Nothing in the Pi-hole change
+   touches DHCP; it exposes udp/53 and tcp/53 and no DHCP feature is enabled. The remaining action is
+   on the router: hand out `192.168.2.23` as the DNS server, ideally with a public resolver secondary
+   so a Pi-hole outage degrades to "internal names stop resolving" rather than "the LAN loses DNS".
+2. **Reverse proxy by hostname is the accepted design; a separate IP is overkill.** So no second
+   Gateway and no second LoadBalancer address — the existing Gateway already distinguishes by Host/SNI.
+   Per-service public DNS targeting, if wanted, therefore goes through `DNSEndpoint` objects plus the
+   `crd` source rather than a second Gateway carrying a target annotation.
+3. Minecraft stays open with no whitelist — accepted.
+
+**Still open, and it is a fact about the WAN, not a repo decision.** The five `public` services have no
+working public path at all: `136.51.115.252` returns **HTTP 403** for `gitea.home.keithjackson.dev` and
+presents a placeholder certificate (`CN=%D`), so nothing reverse-proxies these names from outside today.
+Either the WAN forwards 80/443 to the Gateway at `192.168.2.20` — in which case their records should
+point at the WAN address — or those five should be re-declared `internal`, because `exposure: public`
+currently asserts something untrue.
+
+**Also fixed:** `apps/vms/unifi/unifi-uos-installer.sops.yaml` had been committed **unencrypted** with
+both fields as literal `REPLACE-ME`, unlike every other `*.sops.yaml` here. `sops <file>` would have
+failed with "sops metadata not found", and Flux would have created the Secret containing the
+placeholder strings. Encrypted in place, round-trip verified. Both fields need setting, not just the
+password — I had omitted `installer-url` from the operator instructions.
+
+---
+
+## 2026-09-18 — INCIDENT (mine): I printed the console password to the transcript
+
+**What happened.** Verifying that the `unifi-uos-installer` Secret had reached the cluster, I ran a
+`kubectl get secret -o jsonpath` that included `{range .data}{@}{end}`. That dumps the whole `data`
+map, so the base64 of `console-password` was printed to the session transcript. This is precisely the
+constraint I have written into every agent brief in this repo — *never print decrypted secrets* — and I
+broke it myself with a careless jsonpath while checking someone else's work.
+
+**Exposure, stated precisely rather than minimised or inflated:**
+- The value reached the session transcript, which I cannot scrub.
+- It did **not** reach git: `git log --all -S<value>` returns **0 commits**.
+- The scratch file that captured it (`tasks/*.output`) was shredded; a repo- and scratch-wide grep for
+  the string now returns nothing on disk.
+
+**Why it is not yet a working credential — the one thing that limits this.** The Secret was created
+`2026-09-14T07:26:43Z`; the running VMI was created `2026-09-12T20:48:30Z`, and its live volume list is
+only `rootdisk` and `cloudinit` — **the `UOSDATA` disk is not attached to the running VMI at all**. So
+the guest has never read the Secret, `install-uos` has never executed, and the `ubuntu` account remains
+locked. The leaked string authenticates nothing today. It would only become valid when the VMI is
+recreated against the new spec.
+
+**Required action, and the ordering matters:** rotate `console-password` in the SOPS Secret **before**
+recreating the VMI. Then the leaked value never becomes a working password anywhere:
+
+    sops apps/vms/unifi/unifi-uos-installer.sops.yaml     # set a NEW console-password
+    git commit && git push                                 # let Flux update the Secret
+    kubectl -n unifi delete vmi unifi                      # only after the rotation has landed
+
+**The lesson, generalised.** Reading a Secret to check *existence* never requires reading its
+*values*. `-o jsonpath='{.metadata.name}'` or listing `.data`'s KEYS answers "did it land?"; dumping
+`.data` answers a question nobody asked. Every previous secret check in this session used a
+keys-only or boolean form — this one regressed because I appended a convenience field to an
+otherwise-safe command without re-reading what it expanded to.
+
+## 2026-09-18 — Two fail-open holes in the gate itself: nested worktrees, and L1 never running
+
+Three pieces of work, one theme: the gate was reporting green over things it had not checked.
+
+### 1. `runbooks/node-memory-swap.md` — the procedure the operator asked for
+
+Keith has a matched 32 GB pair to install and asked which node needs it. **`k3s-prod-1`**: `DIMM1` is a
+Micron `8ATF2G64HZ-3G2E1` (16 GB, DDR4-3200, 1Rx8), `DIMM3` a `16ATF2G64HZ-2G6E1` (16 GB, DDR4-2667,
+2Rx8). Capacity is correct; the pair runs at the slower 2667. Nothing is faulty. `k3s-prod-3` is a
+matched Crucial pair.
+
+The runbook front-loads the go/no-go checks because this is the awkward node to lose: 24 Longhorn
+replicas, one of three etcd members (drain leaves quorum at 2 of 3, no further margin), and its
+rebuilds are the load that starved etcd once. A BLOCKED drain is treated as the safety property
+working, never something to force past.
+
+Three things I had written and then removed, because they were not true:
+
+- **`EtcdIsVoter` does not exist in this cluster.** I had an assertion built on it; it appears nowhere
+  in the repo. The house check is `curl http://127.0.0.1:2381/metrics | grep etcd_server_has_leader`
+  (`runbooks/node-down.md:93`).
+- **`node-drain-policy` is not pinned** in `platform/longhorn/helmrelease.yaml`. The runbook has the
+  operator READ the live Setting and set it before draining rather than assuming the chart default —
+  that Setting is the entire safety story for 24 replicas.
+- I had written "rebuild concurrency is deliberately 1". That is the purge/restore limits, not replica
+  rebuild — the same 5-vs-1 confusion already corrected once in `longhorn-degraded.md`.
+
+Verification hinges on **`Configured Memory Speed`**, not `Speed`: a matched 3200 pair being clocked
+down still reports `Speed: 3200 MT/s`, so matched part numbers alone do not prove the swap worked.
+
+Also recorded: the `node-health` artifacts de-duplicate DIMM part numbers, so they can prove "two
+different modules" but cannot distinguish one module from two identical ones — verification collects
+`dmidecode` live. **Node 2's inventory remains stale and self-contradictory** (documented as the only
+single-DIMM node at 16 GB, yet reporting ~31.1 GiB) and needs re-inventory before anyone claims
+three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was an EFI memattr kernel bug.
+
+### 2. The whole-repo L0 checks were auditing 18 copies of the repo
+
+Six discovery sites walked `ROOT` without pruning dot-directories, so they descended into
+`.claude/worktrees/agent-*/` — full repo copies from isolated agent runs, gitignored at
+`.gitignore:43`.
+
+The visible half was false FAILs: four ServiceMonitors "selecting no Service", nine PrometheusRules
+"under test but not in the repo", every path a worktree path. The whole-repo gate was red for reasons
+with nothing to do with authored surface.
+
+The dangerous half is quieter, and is why all six were fixed rather than only the failing ones: a check
+asking *"does anything in the repo declare X"* can be satisfied by a worktree copy instead of real
+source. The same bug that invents failures can manufacture a pass.
+
+`egress-paths-selected-by-rules` already had the fix and documents hitting it (it once saw 5
+`CiliumEgressGatewayPolicy` objects instead of the 1 authored). Its dot-directory predicate is now
+applied everywhere, plus both `find` calls in `platform/monitoring/tests/run.sh`, whose `--list`
+reported each manifest once per worktree — 21 real pairs where it had been returning them ~19x over.
+
+### 3. L1 rendering never ran on a requirement-scoped gate — closed, with the incident reproduced
+
+`ci/l1.sh` has passed `--include-scope whole-repo` since it was written, with a comment saying no L1
+test declared the scope yet and it was wired ahead of time. Nothing ever declared it, so **the entire
+L1 rendering layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or
+R-NET-05/06/07.**
+
+All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`. The justification is one
+shared code path — `catalog/render.py` against `schemas/servicedefinition.schema.json` — which
+requirement-scoped selection cannot see.
+
+The 2026-09-04 external-dns merge is the proof, reproduced rather than asserted.
+`ci/touched-requirements.sh` attributes `platform/externaldns/helmrelease.yaml` to R-ENV-02,
+R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — **none** of the rendering requirements — yet that
+change added `external-dns.alpha.kubernetes.io/controller: none` to the rendered output of all three
+goldens. Measured with `--print-selection`:
+
+| selection for that change | L1 tests selected |
+|---|---|
+| requirement-scoped only (the 2026-09-04 behaviour) | **0** |
+| with `--include-scope whole-repo` (how `ci/l1.sh` invokes it) | **3** |
+
+### 4. …which exposed two defects in the meta-check that guards all of this
+
+Declaring the scope at L1 made `whole-repo-invariants-wired` (R-TEST-03) fail, and it was right to
+change behaviour — but for the wrong reason, and it could not say so:
+
+- **It was layer-agnostic where the design is layer-bounded.** It compared *every* scope-declaring dir
+  against an `--layer l0` selection, assuming only L0 tests would ever declare the scope — while
+  `ci/l1.sh` was written from the opposite premise, that a whole-repo invariant could appear "at any
+  layer". So it blamed the union for correctly excluding another layer. Now partitioned by layer, with
+  the L1 selection and L1 leakage assertions added — which is what makes the L1 declaration
+  load-bearing instead of decorative.
+- **Every assertion in it failed silently.** The idiom was `<test>` on one line, `chk $?` on the next,
+  under `set -euo pipefail` — so a false test aborted the script *before* `chk` could print. The real
+  failure I hit emitted 26 `ok` lines, rc=1, and no message. All 12 sites now capture the status
+  (`chk $arc`), proven by forcing `MIN_DECLARED=999`: the failure prints its full diagnostic and the
+  run continues to the remaining assertions.
+
+A meta-check that cannot report why it failed is worse than no meta-check, because the silence reads
+as a tooling glitch rather than a finding.
+
+### Gate
+
+Offline `l0,l1,meta --include-scope whole-repo` in the pinned toolbox: **25 tests, 0 failures, 0
+errors, exit 0**. L0 whole-repo went 4 failures → 1 → 0 across the two fixes;
+`whole-repo-invariants-wired` now runs 35 assertions. Host runs report one skip because `promtool` is
+not on `$PATH` there — the toolbox run is the real result, and that difference is worth remembering
+before trusting a host-only gate run.
+
+### PM HANDOFF
+
+Open, in the order I would take them:
+
+1. **The memory swap is a free R-STATE-02 proof.** `node-loss` is Phase 7's last exit item and is still
+   proven only on the decommissioned dev cluster. The swap is a planned, supervised single-node loss on
+   prod — exactly the event R-STATE-02 describes. Capturing evidence during it costs nothing extra;
+   manufacturing the event later costs a second maintenance window.
+2. **Node 2 DIMM re-inventory** — blocked on the vault become password for `node_health`. Ansible only;
+   never `ansible-inventory --list` (it prints decrypted vault values).
+3. **Workstation DNS, one sudo command.** `/etc/dnsmasq.d/10-home.keithjackson.dev.conf` contains
+   `local=/home.keithjackson.dev/` plus `address=/.home.keithjackson.dev/192.168.1.53` — the whole zone
+   hardcoded to the decommissioned dev cluster, with forwarding forbidden so no upstream could correct
+   it. That is one root cause for both the sonarr failure and `gitea` not resolving. Pi-hole answers
+   correctly (`gitea` → `192.168.2.20`); only this file is wrong. Correction to an earlier claim of
+   mine: there is no separate `20-prod.*` file — it is a replace of this one file. Router DHCP option
+   6 → `.23` is the fix that makes it unnecessary per-machine.
+4. **UniFi guest has no IP** — needs console access; `cat /etc/netplan/*.yaml` is the confirming command.
+5. **Five `public` services have no working public path** — waiting on WAN forwarding, which Keith has
+   said is deliberately not ready.
+6. **18 stale agent worktrees** on already-merged branches remain on disk. No longer load-bearing for
+   the gate; cleaning them up is cosmetic and should check each for uncommitted work first.
+7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, 103 unmapped
+   paths, and two rendered manifests nothing applies (the keycloak one would stand up a conflicting
+   instance).
+
+## 2026-09-19 — Overnight: the backup alert was crying wolf, and 49 app manifests were gated by nothing
+
+Worked autonomously while the operator slept. One theme again: things that reported green, or red, over
+something other than what they claimed to measure.
+
+### 1. VeleroNoRecentBackup was firing CRITICAL over backups that protected everything
+
+`velero-daily-20260919030041`: **PartiallyFailed, 3079 of 3079 items backed up, 2 errors** — both
+`PodVolumeBackups` for the emptyDir `plugins` + `scratch` volumes of a
+`homepage-default-kopia-maintain-job-…` pod. Velero runs transient kopia repository-maintenance Jobs in
+its own namespace, one per BackupRepository; under `defaultVolumesToFsBackup: true` it tries to
+fs-backup their volumes, the Job finishes, the pod disappears mid-backup, and the PVB fails. Four such
+failures had accumulated.
+
+Nothing real was ever missed. But `PartiallyFailed` does not advance
+`velero_backup_last_successful_timestamp`, so the alert guarding R-STATE-03 fired nightly over Velero
+cleaning up after itself. **An alert that cries wolf every night is how a real backup failure gets
+ignored** — which is precisely the failure mode that alert exists to prevent.
+
+**First fix was right about the bug and wrong about the layer.** I excluded ns `velero`'s volume data
+(keeping its resources in schedule 2) and explicitly deferred the general fix — skip emptyDir by type —
+because Velero's `volumeTypes` policy condition was unproven here and an unparseable `policies.yaml`
+breaks EVERY backup. I wrote "revisit if the shape shows up elsewhere."
+
+It showed up elsewhere inside the hour. The verification backup came back PartiallyFailed with 4 errors,
+all in ns `runbooks`: the nginx Deployment's emptyDir `tmp` + `cache`, failing with
+`error to expose PVB: error getting client pod …: pods "…" not found`. **Cause worth stating plainly: I
+caused it** — pushing a runbook edit changed the configMap hash, rolled the Deployment, and the backup
+lost the pods mid-snapshot. Self-inflicted, not a standing defect. But the mechanism is not
+self-inflicted: a PVB exposes a volume *through its pod*, so any rollout landing in the backup window
+fails one, and one failed PVB makes the whole Backup PartiallyFailed. On a GitOps cluster that is
+routine, not an edge case.
+
+So the property is what matters, not the namespace: an emptyDir is created empty with its pod and
+destroyed with it, so backing one up cannot protect anything while attempting it can break the backup
+that protects everything else.
+
+**The `volumeTypes` risk was retired by testing, not by hoping.** New policy into a THROWAWAY
+ConfigMap first, live one untouched, proven on prod (velero 1.18.1) with a control:
+
+| ns `runbooks` | PodVolumeBackups |
+|---|---|
+| control, no policy | **4** (2× `tmp`, 2× `cache`), Completed |
+| with the policy | **0**, Backup Completed, 0 errors |
+
+The control is the point — "zero PVBs" alone is equally consistent with nothing having been attempted,
+which is exactly the empty green I shipped earlier in this session with a 24h baseline built from ten
+minutes of data.
+
+Then reverted the namespace exclusion as redundant (`plugins`/`scratch` are emptyDir, verified on a live
+maintain pod) rather than leaving two overlapping fixes.
+
+**Proof, full cluster, policy live:** `pre-memswap-202609190525` — **Completed, 3926/3926 items, 0
+errors, 21/21 PodVolumeBackups Completed, 0 emptyDir volumes backed up.** `VeleroNoRecentBackup`
+cleared. `VeleroBackupPartialFailure` remains at `increase = 1.008` from the pre-fix failure and
+self-clears about an hour after it, by design — the rule is `increase(...[1h]) > 0`, which is correct.
+
+That backup also serves as the pre-maintenance backup the memory-swap runbook requires.
+
+### 2. The runbook claimed the CNPG primaries sit on node 1. They do not.
+
+Verified live: all three CNPG clusters run 3 instances spread one per node by required anti-affinity,
+and **k3s-prod-1 holds a REPLICA of each** (`keycloak-db-2`, `nextcloud-db-2`, `semaphore-db-2`). No
+primary is on node 1, so draining it forces **no failover at all** — the best state this maintenance
+could ask for. The original sentence was stale session state. The runbook now prints the command to
+re-derive it, because primaries move after any restart.
+
+That also settles `CNPGPrimariesCoLocated`, firing for k3s-prod-2 (both `keycloak-db` and
+`semaphore-db` primaries). **Do not tidy it before the swap.** Rebalancing means promoting a replica,
+the only node with spare primary capacity is node 1, so "fixing" the alert would put a primary on the
+node about to be powered off and convert a clean drain into a two-database failover. Rebalance after
+node 1 is back and caught up. Warning is now in the runbook.
+
+### 3. 49 app manifests attributed no requirement, so the merge gate ran no behaviour test for them
+
+`ci/touched-requirements.sh`'s coarse path map had no arm for `apps/*`. A change adding or editing any
+workload selected **zero** requirement-scoped tests — so none of the L2 behaviour tests that enforce
+default-deny, egress class or auth mode ran on exactly the changes that can violate them. Kyverno
+admission and the L0 egress-path check are real backstops, but admission is post-merge and the gate
+exists to catch it before.
+
+Also unmapped: `platform/homepage` and `platform/headlamp` — **the C2 UI**. This was already documented
+as a known gap in `changed_diff()`'s rule-3 comment ("the ONLY route to R-C2-01/02 was rule 3", i.e.
+editing the C2 test itself), and had sat there since.
+
+Twelve arms added: `apps/vms/*` (before `apps/*`, since `case` takes the first match), `apps/*`,
+`platform/homepage|headlamp`, `platform/kubevirt`, `platform/gateway`, `platform/coredns`,
+`platform/crafty`, `platform/external-snapshotter|backup`, `runbooks/*`, `platform-config/*`,
+`platform/kustomization.yaml` and `.sops.yaml`.
+
+`platform/kustomization.yaml` deserves its own note: it is the assembly list, so **one deleted line
+disables an entire platform component** while every file it points at stays valid on disk. That is the
+most consequential single-line edit in the repo and it attributed nothing.
+
+Measured effect for an `apps/` change: **0 → 13 requirement-scoped tests** (3 L1, 8 L2, 2 L3). Coverage
+stays 51/51, so the gate's "attributed IDs must be covered" assertion does not false-block.
+
+Ten new assertions in `ci/tests/touched-requirements.test.sh` (26 total, all passing), including the
+ORDER property — `apps/vms/*` must win over `apps/*` — since that is a real failure mode of `case` and
+not merely a matter of both arms existing.
+
+**A correction on my own numbers.** I first reported "192 unmapped files", then "182". Both were
+inflated by my audit script, not by the repo: the regex used `[^)]*`, which matched across a newline, so
+the first arm parsed as `case "$1" in\n    catalog/*` and never matched anything — and I separately
+mis-parsed alternation arms like `.gitea/*|ci/*`, which made `ci/` look unmapped when it is mapped.
+True remainder after this change is **18 files, all prose or captured artifacts** (`docs/plans/`,
+root READMEs, `.standup-dns/*.txt`).
+
+### 4. Finished `feat/egress-gateway-staleness-alert` — it had been backlogged because it could not pass the gate
+
+Merged the two alerts for the silent egress-gateway failure (`VPNEgressSteeringStale` critical,
+`VPNEgressUplinkLabelMissing` warning) plus the KSM allowlist entry that makes them non-inert.
+
+Confirmed first that they are **live on prod, not decorative**: `k3s-prod-3` carries
+`egress-gateway=true` and `egress-uplink=10.150.192.86-139`, `vpn-egress-steering` has been in place 14
+days, and the label was exported on **0 series** before this change — so the allowlist entry is the
+switch that turns the alerts on.
+
+Three whole-repo checks failed the branch, each a real gap: the runbook the alerts link to did not
+exist (a firing alert would hand the operator a 404); `alerts.yaml` is an already-covered file, so new
+rules in it must be unit-tested; and the scrape-target guard broke. Wrote
+`runbooks/vpn-egress-stale.md` (wired into the generator, four required sections) and
+`platform/monitoring/tests/vpn-egress.test.yaml` (four promtool fixtures — tests 1 and 2 are the SAME
+reconnect with and without the label following it, which isolates the mechanism: a label value change
+retires one series and starts another rather than changing a sample).
+
+**Two of the three failures were defects in the checks, not the branch:**
+
+- `alert-metrics-have-scrape-targets` asserted the allowlist matched `nodes=[...egress-gateway]` with a
+  literal closing bracket — that the list held *exactly* that one label. Correctly adding a second label
+  therefore failed the check while the guarded property stayed satisfied. A guard that breaks when you
+  correctly extend the thing it guards teaches people to edit the guard, which is how it stops guarding.
+- `UNTESTED_ALERTS` flagged `VPNTunnelDown` as stale because "asserted on" is a **bare word match over
+  the test file** — my header *comment* mentioning it counted as coverage. I did not delete the register
+  entry to make that go away; instead the fixtures now assert `VPNTunnelDown` **quiet** in all four
+  cases. That required making them faithful: they had no `node_network_receive_bytes_total{device="tun0"}`
+  at all, so `VPNTunnelDown` would have been firing throughout unnoticed, because promtool only checks
+  the alertnames you name. A healthy tun0 passes traffic — and that silence IS the claim these rules
+  exist for.
+
+**Known, deliberately not fixed:** that word match means any alert named in prose counts as tested.
+Tightening it to `alertname:` positions could expose other prose-only "coverage", and the register is
+shrink-only, so the cascade wants doing deliberately — not at 01:00 unattended.
+
+### 5. Stale worktrees: audited, deliberately not touched
+
+18 agent worktrees remain under `.claude/worktrees/`. **14 clean, 4 dirty** — the dirty ones being
+`fix/gate-does-not-pass-on-skipped-stages`, `fix/postbuild-substitution-catches-dollar-mangling`,
+`fix/scrape-targets-for-inert-alerts` and `fix/qbittorrent-durable-webui-credential`. Every branch is
+already merged into main, so the uncommitted state is leftover agent scratch rather than unmerged work.
+
+Left in place. They no longer affect the gate (the scanner now skips dot-directories), so removal is
+disk hygiene, and deleting someone's uncommitted changes unasked is not a call to make while they are
+asleep. `git worktree remove` preserves branches and their commits, so only that scratch is at risk.
+
+### PM HANDOFF
+
+1. **R-STATE-02 is still the Phase 7 exit item, and it is blocked on permission, not on readiness.**
+   `./conformance/run.sh --layer l3 --requirements R-STATE-02 --record`. The permission classifier
+   refuses it and I did not route around it — hand-rolling the cordon would defeat the guard. I read the
+   test first: it cordons ONE node and evicts only its OWN writer pod, never `kubectl drain`; kubelet
+   and etcd stay up so quorum is untouched; PVC and pods live in the per-test namespace; and `finally`
+   uncordons via a unique label even on failure or panic. Preconditions verified: 3 Ready nodes,
+   `longhorn` default SC, nothing pre-cordoned, 24/24 volumes healthy. Either run it, or add a Bash
+   permission rule.
+2. **The memory swap is still the cheapest R-STATE-02 evidence** — a supervised single-node loss on prod
+   is the event that requirement describes. Node 1 currently holds no DB primary, which is the ideal
+   window.
+3. **Node 2 DIMM re-inventory** — blocked on the vault become password. Never `ansible-inventory --list`.
+4. **Pi-hole needs A records** for `unifi.prod` → `.21` and `crafty-mc.prod` → `.22` before router DHCP
+   option 6 → `.23` is safe. It currently answers `.20` for both, and neither service speaks HTTP
+   through that Gateway VIP, so the LAN-wide change would break UniFi adoption and Minecraft. Until
+   then the per-machine dnsmasq override is load-bearing.
+5. **UniFi guest has no IP** — needs console; `cat /etc/netplan/*.yaml` is the confirming command.
+6. **Five `public` services have no working public path** — waiting on WAN forwarding, deliberately.
+7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, two rendered
+   manifests nothing applies (the keycloak one would stand up a conflicting instance).

README.md

@@ -72,7 +72,7 @@ bug but the honest answer, because proofs are claims about a specific tree on a
 |---|---|
 | 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
 | 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
-| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
+| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
 | 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
 | 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
 | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |

apps/vms/unifi/30-virtualmachine.yaml

@@ -41,10 +41,41 @@ spec:
       # disk KubeVirt cannot migrate and eviction would hard-stop the guest.
       evictionStrategy: LiveMigrate
       domain:
+        # =====================================================================================
+        # FIRMWARE UUID == THE CLOUD-INIT INSTANCE-ID. THIS IS THE RE-RUN LEVER. READ BEFORE EDIT.
+        # =====================================================================================
+        # MEASURED on prod 2026-09-14, not assumed:
+        #   * the NoCloud seed KubeVirt builds for this VM carries
+        #       {"instance-id":"de5c3338-6d32-4956-89d1-12c8272b2c07","local-hostname":"unifi"}
+        #     (read out of noCloud.iso inside virt-launcher-unifi-gp594), and that value was
+        #     EXACTLY `spec.template.spec.domain.firmware.uuid`, which KubeVirt's mutating webhook
+        #     had stamped onto the VM OBJECT — so it is PERSISTENT, not per-VMI (the VMI's own uid
+        #     was b532fd7c-…, a different value). It is not a hash of the VM name either: the four
+        #     standard uuid5 namespaces over "unifi" / "unifi.unifi" all produce something else.
+        #   * cloud-init's NoCloud datasource decides "is this the same instance?" by comparing the
+        #     cached instance-id against the guest's DMI product_uuid — which IS this firmware UUID.
+        #
+        # CONSEQUENCE, and it is the whole reason the previous unattended attempt could never have
+        # worked: EDITING `userData` BELOW AND RESTARTING THE VM CHANGES NOTHING. cloud-init sees an
+        # unchanged instance-id, restores its CACHED datasource, never re-reads the seed, and skips
+        # every per-instance module (write_files, runcmd). The new payload is silently ignored.
+        #
+        # So the UUID is pinned HERE, in git, and BUMPING IT IS THE SUPPORTED WAY to make the guest
+        # process a new cloud-init payload: change the trailing hex, commit, restart the VM. Pinning
+        # it also makes a rebuild reproducible — a fresh apply would otherwise take a random
+        # webhook-generated UUID and give the guest a different identity every time.
+        #
+        # The value below is DELIBERATELY NOT the de5c3338-…c07 the live guest booted with: that
+        # difference is what makes the next boot a NEW cloud-init instance and installs the
+        # `install-uos` helper + its systemd unit. `serial` is pinned to the value the webhook
+        # already stamped, so this change does not also churn the guest's DMI serial.
+        firmware:
+          uuid: "de5c3338-6d32-4956-89d1-12c8272b2c11"
+          serial: "24bb12ac-d311-45f6-a75b-90a9e1b63e18"
         # FLAG (verify-at-bringup): Ubuntu 24.04 amd64 cloud images are hybrid BIOS+UEFI bootable,
         # so KubeVirt's default SeaBIOS SHOULD boot cleanly (unlike the Fedora containerdisk, which
-        # is UEFI-only and needed an explicit efi block). If the guest drops to emergency/no-boot,
-        # add:  firmware: { bootloader: { efi: { secureBoot: false } } }
+        # is UEFI-only and needed an explicit efi block). RESOLVED at bring-up: boots as shipped;
+        # the guest reports Ubuntu 24.04.4 LTS.
         cpu:
           cores: 2
         memory:
@@ -58,9 +89,28 @@ spec:
             - name: cloudinit
               disk:
                 bus: virtio
+            # The operator's installer URL (+ optional SSH key), delivered as a tiny read-only
+            # iso9660 disk — see the `uosdata` volume near the bottom of this file.
+            - name: uosdata
+              disk:
+                bus: virtio
           interfaces:
+            # MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
+            # KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
+            # /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
+            # that file while the NoCloud instance-id is unchanged — and the instance-id here is
+            # domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
+            # handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
+            # administratively DOWN with no address and no DHCP attempt.
+            # That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
+            # device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
+            # (because the per-instance modules were skipped), yet UOS is installed and
+            # qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
+            # Pinning the MAC to the value the current VMI already holds makes the guest's netplan
+            # match permanently, so this cannot recur on the next recreation.
             - name: default
               masquerade: {}
+              macAddress: a6:d7:c0:b3:72:fa
               # Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
               # these and (b) surfaces them as containerPorts on the virt-launcher pod so the
               # LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
@@ -83,41 +133,112 @@ spec:
         - name: rootdisk
           dataVolume:
             name: unifi-rootdisk
+        # =====================================================================================
+        # THE OPERATOR'S ONE FIELD — SOPS Secret `unifi-uos-installer`, delivered as a DISK.
+        # =====================================================================================
+        # KubeVirt exposes a Secret as a small read-only iso9660 disk whose volume label we choose;
+        # each Secret key becomes a file on it. The guest mounts it by label (LABEL=UOSDATA) and
+        # reads `installer-url` plus an optional `console-password`. The Secret ships ENCRYPTED with
+        # REPLACE-ME placeholders, and the guest-side script treats anything that is not an https
+        # URL as "not provided yet" and exits 0 — so the VM boots and stays healthy until the
+        # operator fills it in. See unifi-uos-installer.sops.yaml.example next to this file.
+        #
+        # `console-password` exists because THERE IS CURRENTLY NO WAY INTO THIS GUEST. The stock
+        # Ubuntu cloud image ships a locked `ubuntu` account, this cloud-init never installed a key,
+        # and SSH could not help anyway: the masquerade interface above forwards ONLY the declared
+        # ports, and 22 is not one of them — probed from inside virt-launcher on 2026-09-14, the
+        # guest answers RST on 22 exactly as it does on the UniFi ports. The serial console is the
+        # only door, and a console needs a password. Set it and `virtctl -n unifi console unifi`
+        # works; `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` already gives
+        # read-only console output with no virtctl at all.
+        #
+        # WHY A SEPARATE DISK AND NOT `cloudInitNoCloud.secretRef` (which would also lift the
+        # 2048-byte inline cap): secretRef moves the ENTIRE cloud-config into ciphertext. Two
+        # concrete costs. (a) The shell below stops being reviewable in a diff and falls outside the
+        # L0 dollar-collapse guard, which can only read plaintext. (b) Flux runs its postBuild
+        # envsubst over the DECRYPTED resources, so the escape contract in
+        # clusters/base/apps-vms.yaml would still govern text nobody can see in git — the exact
+        # shape of the 2026-09-13 nextcloud-db incident. Keeping the script in plaintext and putting
+        # ONLY the operator's two values behind SOPS keeps both properties.
+        #
+        # A CHANGED SECRET REACHES THE GUEST ON THE NEXT VMI START with NO firmware-UUID bump: the
+        # disk is rebuilt from the Secret every time the VMI starts, and the systemd unit below
+        # re-reads it on EVERY boot instead of relying on cloud-init (which will not re-run — see
+        # the firmware block above). The UUID bump is a ONE-TIME step, to get that unit installed.
+        - name: uosdata
+          secret:
+            secretName: unifi-uos-installer
+            volumeLabel: UOSDATA
         - name: cloudinit
           cloudInitNoCloud:
-            # INLINE (kept < KubeVirt's 2048-byte userData limit — so NO SSH key here; add one back via
-            # userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
-            # signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
-            # Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
-            # NOTE: apps-vms NOW carries postBuild.substituteFrom cluster-vars (needed for
-            # ${UNIFI_L4_VIP} in 50-service-l4.yaml), so Flux envsubst DOES run over this userData.
-            # Every shell brace-expansion below is therefore escaped as $${...}, which envsubst emits
-            # as a literal $${...} — verified with `flux envsubst`. Bare $VAR and $(...) are left alone
-            # by envsubst and need no escaping. NEVER write an unescaped $${...} here, and never write
-            # an INDEXED expansion unescaped: envsubst hard-fails on `[` ("missing closing brace")
-            # and would break the whole apps-vms apply. CORRECTION (2026-09-01, measured against
-            # flux 2.4.0 `envsubst --strict`): escaping IS total — $${ARR[0]} emits the literal
-            # text safely. Only the UNESCAPED indexed form is fatal. The earlier "escaped or not"
-            # wording in clusters/base/apps-vms.yaml overstated it.
+            # INLINE, and it MUST stay <= 2048 bytes. That ceiling was MEASURED against the
+            # DEPLOYED KubeVirt v1.9.0 rather than quoted: a server-side dry-run apply of an
+            # otherwise-identical VM accepted 2048 bytes of userData and rejected 2049 with
+            # "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for
+            # larger data." The payload below is ~1.5 KB; keep additions small.
+            #
+            # WHAT IT DOES: installs qemu-guest-agent (the KubeVirt AgentConnected signal, which is
+            # also the proof cloud-init ran) and preps Podman for UOS, then writes `install-uos`
+            # plus a systemd unit that runs it on EVERY boot. The unit — not cloud-init — is what
+            # makes a later URL change take effect, because cloud-init will not re-run per-instance
+            # modules while the instance-id is unchanged.
+            #
+            # `install-uos` IS IDEMPOTENT AND WILL NOT REINSTALL OVER A WORKING CONTROLLER. It bails
+            # out the moment systemd knows a `uosserver.service` — which the live guest ALREADY has:
+            # the serial console log for the current boot shows both `uosserver.service` and
+            # `uosserver-updater.service` starting, and the root filesystem is 6.3 GB of 40 GB used.
+            # UOS is installed here; see the README for what is actually wrong (the guest lost its
+            # IP). A stamp file at /var/lib/uos-installed is the second, weaker guard for the case
+            # where the unit name changes in some future UOS release.
+            #
+            # ENVSUBST: apps-vms declares postBuild.substituteFrom (for ${UNIFI_L4_VIP} in
+            # 50-service-l4.yaml), so Flux's envsubst DOES run over this text. ${BASE_DOMAIN} below
+            # is a deliberate substitution. Everything else is bare `$name` or `$(cmd)`, both of
+            # which envsubst leaves untouched — verified against the live VMI, whose rendered
+            # userData still carries its shell variables verbatim. There is deliberately NO
+            # brace-form shell expansion left in this payload, so nothing here needs escaping; if
+            # you add one, escape it per the contract in clusters/base/apps-vms.yaml, and NEVER
+            # write an indexed expansion.
             userData: |-
               #cloud-config
               hostname: unifi
               fqdn: unifi.${BASE_DOMAIN}
               package_update: true
               package_upgrade: false
-              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
+              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl]
               write_files:
                 - path: /usr/local/sbin/install-uos
                   permissions: '0755'
                   content: |
-                    #!/usr/bin/env bash
+                    #!/bin/bash
                     set -euo pipefail
-                    if [ -z "$${UOS_URL:-}" ]; then
-                      UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
-                    fi
-                    [ -n "$${UOS_URL}" ] && [ "$${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
-                    t="$(mktemp)"; echo "downloading $${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
+                    d=/run/uos; mkdir -p $d
+                    mountpoint -q $d || mount -o ro LABEL=UOSDATA $d
+                    p=$(tr -d '\r\n' < $d/console-password || true)
+                    case "$p" in ""|REPLACE-ME) ;; *) echo "ubuntu:$p" | chpasswd ;; esac
+                    systemctl cat uosserver.service >/dev/null 2>&1 && exit 0
+                    s=/var/lib/uos-installed
+                    [ -e $s ] && exit 0
+                    u=$(tr -d '\r\n' < $d/installer-url || true)
+                    case "$u" in https://*) ;; *) echo "UOS installer-url not set yet"; exit 0;; esac
+                    t=$(mktemp)
+                    curl -fL --retry 5 --retry-delay 10 -o $t "$u"
+                    chmod +x $t
+                    yes y | $t install || yes y | $t
+                    rm -f $t; date -Is > $s
+                - path: /etc/systemd/system/uos-install.service
+                  content: |
+                    [Unit]
+                    Description=Install UniFi OS Server
+                    Wants=network-online.target
+                    After=network-online.target
+                    [Service]
+                    Type=oneshot
+                    ExecStart=/usr/local/sbin/install-uos
+                    StandardOutput=journal+console
+                    [Install]
+                    WantedBy=multi-user.target
               runcmd:
-                - [ systemctl, enable, --now, qemu-guest-agent ]
-                - [ systemctl, enable, --now, podman.socket ]
-                - [ bash, -lc, "/usr/local/sbin/install-uos 2>&1 | tee /var/log/uos-install.log > /dev/console" ]
+                - [systemctl, enable, --now, qemu-guest-agent]
+                - [systemctl, enable, --now, podman.socket]
+                - [systemctl, enable, --now, uos-install.service]

apps/vms/unifi/50-service-l4.yaml

@@ -21,8 +21,61 @@
 # k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
 #
 # selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
-# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
-# VIP pinned to the node running the VM (also what L2 announcement does).
+# into the guest.
+#
+# -------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
+# -------------------------------------------------------------------------------------------
+# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
+# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
+# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
+#
+# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
+#    per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
+#    matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
+#    MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
+#    `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
+#    announcing node that holds no backend has nowhere to send the packet and answers ICMP port
+#    unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
+#    (the leases were 8 days old); the operator's controller UI and device inform path were down.
+#    It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
+#    deleting the Lease forces a re-election — and that re-election is random, so it can pick a
+#    backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
+#    reference implementation this file now follows.
+#
+# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
+#    interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
+#    SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
+#    controller has always seen the pod-side masquerade gateway as the source, never the LAN
+#    client. There is a SECOND translation below the Service, and it already destroyed the very
+#    thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
+#    it is giving up an address the guest could not see in the first place.
+#
+#    Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
+#    MAC (and the device reports its own IP inside the inform payload), not the packet source,
+#    and guest-portal authorization runs on the gateway appliance, not on this controller.
+#
+# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
+# is what makes this survive a reschedule and a rebuild.
+#
+# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
+# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
+# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
+# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
+# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
+# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
+# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
+# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
+# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
+#
+# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
+# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
+# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
+# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
+# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
+# fix would be a NEW restriction that could silently break a remote site's device informing over
+# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
+# change, made deliberately and tested against the device fleet.
 #
 # VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
 # default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
@@ -49,7 +102,7 @@ metadata:
     external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:
@@ -72,7 +125,7 @@ metadata:
     lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     kubevirt.io/vm: unifi
   ports:

apps/vms/unifi/README.md

@@ -1,18 +1,48 @@
 # UniFi Network controller (UniFi OS Server) — VM onboarding
 
-> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
-> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
-> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
-> or install cannot stall the app tier.
+> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
+> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
+> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
+> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
+> cannot stall the app tier.
 >
-> **Status (verified 2026-08-26):**
-> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
-> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
-> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
-> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
-> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
->   (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
->   The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
+> **Status (re-verified against live prod 2026-09-14):**
+> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
+> - CDI import — **Succeeded** (100%).
+> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
+>   `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
+> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
+> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
+> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
+>   `:8080` refuse connections.
+
+## What is actually wrong (2026-09-14)
+
+The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
+That was wrong, and acting on it would have meant reinstalling over a working controller.
+
+Evidence, all read-only, from the live cluster:
+
+| Claim | How it was checked | Result |
+|---|---|---|
+| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
+| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
+| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
+| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
+| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
+
+So the controller is installed and its units start; nothing downstream of the guest's own network
+stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
+~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
+UOS taking over host networking, but **that has not been proven and cannot be, from outside the
+guest.** Diagnosing it needs a shell, which is the second problem below.
+
+**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
+no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
+nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
+ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
+guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
+new Secret is for.
 
 ## What this is
 
@@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
 | `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
 | `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
 | `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
-| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
+| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
 | `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
-| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
+| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
+| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
+| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
 | `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
 
-The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
-serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
-this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
+The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
+`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
+version. Browser access is via the L4 VIP. Re-add it if/when that lands.
 
-### Cloud-init delivery — two constraints learned the hard way
+### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)
 
-1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
-   prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
-   shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
-   Secret deliberately.
-2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
-   `$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
-   therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
+1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
+   of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
+   userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
+   payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
+   is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
+   validate as an unknown field.
+2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
+   nothing.** The NoCloud seed's `instance-id` is exactly
+   `spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
+   (so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
+   product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
+   `runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
+   effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
+   to force a re-run. See the long comment in `30-virtualmachine.yaml`.
+3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
+   been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
+   envsubst therefore runs over the cloud-init payload, and the escape contract in
+   `clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
+   against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
+   indexed expansion is a hard error that fails the whole Kustomization.
 
 ### Node prerequisites this VM forced onto the cluster
 
@@ -56,15 +101,31 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 - **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
   3-replica class made the import fail with insufficient storage.
 
-## UOS install method (researched)
+## UOS install method (researched 2026-09-14)
 
-- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
-  2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
-- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
-  `ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
-  **`https://<host>:11443`**.
-- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
-  independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
+- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
+  Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
+  (the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
+  Podman must be at `/usr/bin/podman`.
+- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
+  in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
+- **Is the URL a short-lived signed link?** *Probably not.* The links look like
+  `https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
+  **no signature query string**, and public community installers hard-code them. It is
+  account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
+  account-derived material out of plaintext config — but do **not** build logic around expiry.
+- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
+  Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
+  ecosystem actually does is pipe confirmations in: the widely used community installer
+  (`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
+  either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
+  `yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
+  fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
+  (no URL has been supplied, and UOS is already installed here), so treat it as "established by
+  convergent third-party practice", not as verified-by-us.
+- The previously shipped helper had two defects worth recording: it resolved the URL from
+  `fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
+  `unifi-os-server`, and it invoked the binary with **no `install` subcommand**.
 
 ### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
 
@@ -78,76 +139,143 @@ this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that la
 | 3478 | UDP | **STUN** (device ⇄ controller) |
 | 10001 | UDP | **L2 device discovery** |
 
+**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
+the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.
+
 ## L4 exposure design (the hard part)
 
 The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
 or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
 
 - **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
-  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
-  The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
-  flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
+  Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
+  (`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
+  `default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
+  the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
+  existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
+  / `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
+  decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
 - `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
-  declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
-- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
-  — via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
-  Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
-  reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
-  masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
-  explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
-- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
-  LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
-  `minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
+  declared ports into the guest.
+- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
+  used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
+  Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
+  leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
+  and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
+  it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
+  why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
+- **Device-adoption story:** point devices' **inform host** at the VIP
+  (`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
+  `set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
+  subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
+  broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
+  zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
+  on other VLANs must route to the VIP and be inform-pointed anyway.
+- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
+  per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.
 
-**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
-`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
-(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
+**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
+the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
+`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
+zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
+apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
+flip, so this hostname is current, not a dev leftover.
+
+## The operator's one field — installing UOS without a shell
+
+`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
+the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
+`uos-install.service` reads it **on every boot**:
+
+```bash
+sops apps/vms/unifi/unifi-uos-installer.sops.yaml    # fill installer-url and/or console-password
+git commit && git push                               # merge to the branch prod's Flux tracks
+kubectl -n unifi delete vmi unifi                    # `running: true` recreates it; no virtctl needed
+kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
+  -c guest-console-log                               # watch the boot + the install, read-only
+```
+
+Design notes that matter if you change this:
+
+- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
+  cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
+  L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
+  resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
+  operator values in SOPS.
+- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
+  the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
+  firmware-UUID bump.
+- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
+  a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
+  guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
+  controller on next boot".
+- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
+  `apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
+  `clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
+  guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.
 
 ## Open questions / risks
 
-1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
-   account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
-   **interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
-   the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
-   **Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
-   invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
-2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
+1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
+   requires a shell, which requires `console-password` above plus one restart. First things to run
+   on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
+   `ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
+   own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
+   capture the state before rebooting a second time if you can.
+2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
+   install-method section: no official silent flag exists, `yes y | <installer> install` is the
+   convergent community practice, and it has not been executed on our guest. Treat the first real
+   install as something to watch on the console log, not as a fire-and-forget.
+3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
    import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
    Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
    fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
-3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
-4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
-   `192.168.1.53–.60` pool (`.53` is the shared Gateway).
-5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
+4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
+5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
+   prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
+   (`.23`) or the node addresses (`.10/.11/.12`).
+6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
    migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
-6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
+7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
    installer and for optional remote access.
-7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
+8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
    masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
+9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
+   `kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
+   `virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
+   the cluster's **v1.9.0**.
 
 ## Remaining bring-up
 
-Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
+Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
+is written for a from-scratch cluster; the prod addresses are the live ones.
 
-1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
+1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
 2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
-   Kustomization (not an app wave).
+   Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
 3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
 4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
-   `192.168.1.54`.
-5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
-   and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
-   ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
-6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
+   `192.168.2.21`.
+5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
+   `uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
+   fill `installer-url`, commit, and the guest installs itself on first boot.
+6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
+   console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
+7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
    complete the first-run wizard.
-7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
-8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
+8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
+   split-horizon) and from in-cluster (coredns-custom).
+9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
    connects (green). Test guest portal / throughput if used.
-9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
-10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
+10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
+11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
 
 **Quick health check today:**
 ```bash
 kubectl -n unifi get dv,vmi,svc
 kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
+# the two that would have caught the current failure:
+kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}'   # expect an ipAddress
+kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
+  -c compute -- ip neigh                                                 # expect 10.0.2.2 REACHABLE
 ```

apps/vms/unifi/kustomization.yaml

@@ -11,3 +11,11 @@ resources:
   - 30-virtualmachine.yaml
   - 40-networkpolicy.yaml
   - 50-service-l4.yaml
+  # SOPS-encrypted. Carries the operator's UOS installer URL (+ optional SSH key) and is mounted
+  # into the guest as the UOSDATA disk by 30-virtualmachine.yaml. Decrypted by the `apps-vms` Flux
+  # Kustomization, which clusters/dev and clusters/prod patch with the `decryption` stanza — the
+  # FIRST encrypted content under apps/vms, so that patch was added with this file. The plaintext
+  # template + the operator hand-off steps live in the `.example` sibling of the file below, which
+  # is deliberately NOT named here: the L0 no-plaintext-secrets check fails any kustomization whose
+  # text contains an example template's filename, on the assumption it is being applied.
+  - unifi-uos-installer.sops.yaml

apps/vms/unifi/unifi-uos-installer.sops.yaml

@@ -0,0 +1,29 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: unifi-uos-installer
+    namespace: unifi
+type: Opaque
+stringData:
+    installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
+    console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
+            WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
+            TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
+            cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
+            kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-19T02:47:13Z"
+    mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/vms/unifi/unifi-uos-installer.sops.yaml.example

@@ -0,0 +1,71 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the two values that (a) let the UniFi VM install UniFi OS Server by itself, with
+# nobody typing commands inside the guest, and (b) give the operator a way INTO that guest at all.
+# KubeVirt mounts this Secret into the VM as a tiny read-only iso9660 disk labelled `UOSDATA` (see
+# the `uosdata` volume in 30-virtualmachine.yaml); each key below becomes a file of the same name on
+# that disk, and the in-guest `uos-install.service` reads them on EVERY boot.
+#
+# READ THIS FIRST — UOS IS ALREADY INSTALLED ON THE LIVE PROD GUEST (verified 2026-09-14, see the
+# README "What is actually wrong"). `installer-url` is therefore NOT the fix for today's outage; it
+# is the rebuild path and the "the install never happened" path. `install-uos` refuses to run the
+# installer at all while systemd knows a `uosserver.service`, so filling this in cannot clobber a
+# working controller. `console-password` IS useful today: it is the only way to get a shell.
+#
+# WHY THE URL IS BEHIND SOPS: it is obtained through the operator's own (free) Ubiquiti account
+# session at ui.com/download/software/unifi-os-server, it identifies a specific build, and it is not
+# ours to redistribute. Treat it as account-derived material, not as public configuration.
+# HONEST CAVEAT, checked 2026-09-14: the resulting `https://fw-download.ubnt.com/data/
+# unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` URL carries NO signature query string,
+# and public community install scripts hard-code these URLs — so it does not behave like a
+# short-lived signed link. It does change every release. Encrypting it costs nothing and keeps the
+# repo's "account-derived material is not config" line clean, but do NOT design around "it expires".
+#
+# HAND-OFF STEPS (operator) — this is the whole job:
+#  (a) Sign in at https://ui.com/download/software/unifi-os-server, pick the Linux (x64) build, and
+#      copy the download link (right-click -> Copy link address). Do NOT download it here; the guest
+#      fetches it itself.
+#  (b) Open the encrypted sibling in your editor:
+#         sops apps/vms/unifi/unifi-uos-installer.sops.yaml
+#  (c) Replace the two REPLACE-ME values. Either may be left as the placeholder — the guest skips a
+#      placeholder rather than failing.
+#  (d) Save/quit (sops re-encrypts in place), commit, push, merge to the branch prod tracks.
+#  (e) Restart the VM once so the new cloud-init instance and the new disk are picked up:
+#         kubectl -n unifi delete vmi unifi        # `running: true` recreates it; no virtctl needed
+#      Watch it come up WITHOUT virtctl:
+#         kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher \
+#           -o name | head -1) -c guest-console-log
+#  (f) UI: https://192.168.2.21:11443 (prod) or https://unifi.home.keithjackson.dev:11443.
+#
+# IF YOU LATER NEED A NEW URL (new release, dead link): repeat (b)-(e). No firmware-UUID bump is
+# needed for that — the disk is rebuilt from this Secret on every VMI start and the systemd unit
+# re-reads it every boot. The UUID bump in 30-virtualmachine.yaml is only needed when the cloud-init
+# payload ITSELF changes.
+#
+# The file is already listed under `resources:` in apps/vms/unifi/kustomization.yaml, and `apps-vms`
+# is patched with the SOPS `decryption` stanza in clusters/dev + clusters/prod.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: unifi-uos-installer
+  namespace: unifi
+type: Opaque
+stringData:
+  # KEY NAMES ARE LOAD-BEARING: each key becomes a FILE of that exact name on the UOSDATA disk, and
+  # /usr/local/sbin/install-uos reads them by path. Renaming a key breaks the install silently.
+  #
+  # The signed-in download link for UniFi OS Server, Linux x64. Anything that is not an https URL
+  # (this placeholder included) is treated by the guest as "not provided yet": the unit logs
+  # "UOS installer-url not set yet" and exits 0, so the VM stays healthy and simply does not install.
+  installer-url: REPLACE-ME
+  # Password set on the guest's `ubuntu` account on every boot, for `virtctl -n unifi console unifi`.
+  # THIS IS THE ONLY DOOR INTO THE GUEST. The stock Ubuntu 24.04 cloud image ships `ubuntu` locked,
+  # no key was ever installed, and an SSH key would not help: the VM's masquerade interface forwards
+  # only the ports declared in 30-virtualmachine.yaml and 22 is not among them (probed from inside
+  # virt-launcher, 2026-09-14 — the guest RSTs on 22 just as it does on 11443). Generate one with
+  #     openssl rand -base64 24
+  # Leave it as REPLACE-ME to keep the account locked; the script skips the placeholder by name.
+  console-password: REPLACE-ME

apps/wave-0/jackett.yaml

@@ -268,6 +268,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

apps/wave-0/kustomization.yaml

@@ -14,8 +14,10 @@ resources:
 - nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
+- pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
+- plex-claim-secret.sops.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml

apps/wave-0/pihole-dns.yaml

@@ -0,0 +1,196 @@
+---
+# =================================================================================================
+# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole`
+# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS
+# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition
+# schema genuinely cannot express it: schemas/servicedefinition.schema.json is
+# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned
+# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at
+# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and
+# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this
+# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and
+# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog
+# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the
+# schema has no word for.
+#
+# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is
+# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service
+# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09
+# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on
+# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that
+# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0).
+#
+# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the
+# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config
+# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is
+# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it
+# can never be allowed to move.
+#
+# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53
+# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same
+# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the
+# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a
+# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently
+# and confusingly.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC
+# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself
+# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS,
+# and publishing a resolver's address in a public zone invites exactly the traffic the source-range
+# filter below exists to refuse. Internal name, internal answer, no public record — which is also
+# why this change touches nothing under platform/externaldns/.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to
+# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured,
+# not assumed (Cilium 1.20.1, prod, 2026-09-14):
+#
+#   * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the
+#     CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does
+#     NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services
+#     landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3,
+#     `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1.
+#   * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the
+#     packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three
+#     nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol.
+#   * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod
+#     rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three
+#     minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a
+#     re-election — and that re-election is random, so it may pick a backend-less node again.
+#
+# A resolver that works one time in three and stays broken after any reschedule is not a resolver.
+# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is
+# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement
+# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all
+# answering.
+#
+# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in
+# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the
+# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client
+# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two
+# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole
+# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts
+# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity —
+# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode:
+# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract
+# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
+#
+# -------------------------------------------------------------------------------------------------
+# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
+# -------------------------------------------------------------------------------------------------
+# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
+# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
+# datapath — it never reaches a backend and never reaches a network policy. This is what makes
+# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
+# range to a /24 that excludes the test client, after which the same query that had been answering
+# stopped being answered at all, and restoring it, after which it answered again.
+#
+# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
+# restriction once the source has been rewritten: measured on this cluster, the forwarded query
+# arrives from the ingress node's internal router address and Cilium resolves that source to the
+# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the
+# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction
+# lives here, where it is actually enforceable, and the policy below restricts the PORTS.
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-udp
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+  annotations:
+    lbipam.cilium.io/sharing-key: "pihole-dns"
+    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  loadBalancerSourceRanges:
+    - ${LAN_CIDR}
+  selector:
+    app.kubernetes.io/name: pihole
+  ports:
+    - { name: dns-udp, port: 53, targetPort: 53, protocol: UDP }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-tcp
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+  annotations:
+    lbipam.cilium.io/sharing-key: "pihole-dns"   # same key + same IP => shares the VIP with -udp
+    lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  loadBalancerSourceRanges:
+    - ${LAN_CIDR}
+  selector:
+    app.kubernetes.io/name: pihole
+  ports:
+    - { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP }
+---
+# =================================================================================================
+# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports.
+# =================================================================================================
+# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered
+# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the
+# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything
+# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the
+# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule:
+# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that
+# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for
+# it and no policy for it.
+#
+# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is
+# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's
+# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium
+# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was
+# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced
+# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
+# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
+# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
+# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
+#
+# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
+# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
+# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none
+# needs PiHole on :53) on fewer ports, behind a filter those two do not have.
+#
+# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the
+# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to
+# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same
+# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not
+# an internet-facing open resolver. Say so out loud rather than implying the filter is total.
+#
+# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved
+# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the
+# node-to-node forwarding that IS the Cluster-policy datapath both need them.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-lan-dns-ingress
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+    - fromEntities:
+        - world         # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
+        - host          # this node's kubelet probes
+        - remote-node   # the node-to-node hop that forwards a query to wherever the pod runs
+      toPorts:
+        - ports:
+            - { port: "53", protocol: UDP }
+            - { port: "53", protocol: TCP }

apps/wave-0/pihole.yaml

@@ -81,6 +81,15 @@ spec:
         image: pihole/pihole
         ports:
         - containerPort: 80
+        env:
+        - name: FTLCONF_dns_listeningMode
+          value: ALL
+        - name: FTLCONF_dns_upstreams
+          value: 8.8.8.8;8.8.4.4
+        - name: FTLCONF_dns_rateLimit_count
+          value: '0'
+        - name: FTLCONF_misc_dnsmasq_lines
+          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
         volumeMounts:
         - name: config
           mountPath: /data/config
@@ -223,6 +232,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

apps/wave-0/plex-claim-secret.sops.yaml

@@ -0,0 +1,69 @@
+# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
+# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
+#
+# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
+# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
+# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
+# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
+# not prose, so the upstream conditions below are described in words.
+#
+# THE VALUE SHIPPED WITH THIS FILE IS A PLACEHOLDER, DELIBERATELY — the literal string
+# `replace-me-with-a-fresh-plex-tv-claim-token`, not a credential. A https://plex.tv/claim token has
+# a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll, so no token
+# committed here is ever still valid when the pod starts. This file exists to make the Secret — and
+# therefore the env reference — ALWAYS RESOLVABLE IN THE FLUX GRAPH, not to carry a working token.
+# The real token is injected by the operator inside that 4-minute window: runbooks/plex-claim.md.
+#
+# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
+# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
+# the same file extracted from the running `plexinc/pms-docker:latest` image):
+#   - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
+#     PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
+#     skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
+#     README: "If server is already logged in, this parameter is ignored."
+#   - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
+#     empty body. curl's exit status is never checked, the sed for the authentication-token element
+#     yields the empty string, and the success branch at line 120 simply does not fire. There is no
+#     else branch and no non-zero exit; the script has no `set -e` and ends by touching
+#     /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
+#     It does NOT crashloop.
+#   - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
+#     if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
+# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
+# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
+# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
+# by hand at claim time.
+#
+# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
+#   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+# Never decrypt this file into a shell, a log, or a shared terminal.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: plex-claim
+    namespace: plex
+    labels:
+        app.kubernetes.io/name: plex
+type: Opaque
+stringData:
+    PLEX_CLAIM: ENC[AES256_GCM,data:8O9nvFw+6OR+53mXvHfe321aSmYUQcZuAqLKZpcSlUwNyeU+Nbp/tRPBlg==,iv:EbvvOsXlXjAFOewnWbK28ErmUdWcJJN0Q5Gm53El3ZM=,tag:wXaInGFQ+wqUt765vvjCDA==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
+            cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
+            THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
+            QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
+            RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-09-29T03:25:20Z"
+    mac: ENC[AES256_GCM,data:u5uVmOdXN78UaKUMkPZzqeW84222+tcG57rFKIKcvRKwJkzMTOuSFOeQpKh4eK4KZXVKK7+Q3mFf5G8a+jjuKjYbjSCpR38AF3TJv3xrj8fhnqR+8QHRp8tsgA0jJZBS55N7KKDyOztLFo49sI7mNnlrUwhWTQUkbLgN1LxF5MA=,iv:P4W9G2mI/3VTkvSrDZvnkJ0Qxb1nFmXcF35CrSNhako=,tag:yVsg7xsUALUoCJH0cHfQjw==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/plex.yaml

@@ -132,6 +132,11 @@ spec:
           value: Etc/UTC
         - name: ALLOWED_NETWORKS
           value: 10.0.0.0/8,192.168.0.0/16
+        - name: PLEX_CLAIM
+          valueFrom:
+            secretKeyRef:
+              name: plex-claim
+              key: PLEX_CLAIM
         volumeMounts:
         - name: config
           mountPath: /config

apps/wave-0/qbittorrent.yaml

@@ -344,6 +344,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

apps/wave-1/radarr.yaml

@@ -348,6 +348,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

apps/wave-1/semaphore.yaml

@@ -227,6 +227,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: oidc
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

apps/wave-1/sonarr.yaml

@@ -348,6 +348,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-users,platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

catalog/pihole.yaml

@@ -13,6 +13,93 @@ storage:
     size: 2Gi
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
+# ------------------------------------------------------------------------------------------------
+# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
+# ------------------------------------------------------------------------------------------------
+# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
+# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
+# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
+# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
+# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
+# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
+# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
+# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
+#
+# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
+# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
+# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
+# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
+# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
+# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
+# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
+# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
+# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
+# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
+# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
+# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
+#
+# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
+# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
+# configured through the UI is lost on the next reschedule — which is exactly what happened to the
+# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
+# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
+# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
+# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
+env:
+  # Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
+  # pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
+  # required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
+  # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
+  # enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
+  # nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
+  - name: FTLCONF_dns_listeningMode
+    value: "ALL"
+  # Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
+  # router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
+  # answers, and forwarding to a device that may forward back here invites a loop.
+  - name: FTLCONF_dns_upstreams
+    value: "8.8.8.8;8.8.4.4"
+  # Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
+  # because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
+  # externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
+  # announcement lease and the pihole pod land on different nodes, which is most of the time — and
+  # Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
+  # INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
+  # apparent client's budget, and a normal evening of browsing from a handful of machines would trip
+  # a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
+  # off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
+  # limit becomes meaningful again — and should be restored — if Cilium is ever switched to
+  # `loadBalancer.mode: dsr`, which preserves the client address.
+  - name: FTLCONF_dns_rateLimit_count
+    value: "0"
+  # THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
+  # every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
+  # so the first line is a wildcard for the whole zone and each later line is a specific override.
+  # PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
+  # line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
+  # changes nothing.
+  #
+  # A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
+  # `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
+  # ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
+  # platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
+  # behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
+  # repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
+  # is the annotation that means "this Service publishes its own hostname at its own address":
+  # apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
+  # platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
+  # resolver address, are the complete set of exceptions today.
+  #
+  # AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
+  # (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
+  # NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
+  #
+  # Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
+  # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
+  # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
+  # while resolution for the zone does not work. Confirmed both ways against the live image.
+  - name: FTLCONF_misc_dnsmasq_lines
+    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
 ingressRoutes:
   - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)

catalog/plex.yaml

@@ -32,11 +32,40 @@ env:
   # (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
   - name: ALLOWED_NETWORKS
     value: 10.0.0.0/8,192.168.0.0/16
-  # PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
-  # shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
-  # is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
-  # single API POST) against the running server; the claim then persists on the /config PVC and
-  # survives pod rolls (including the roll this env change causes), so it never belongs in git.
+  # PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
+  # (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
+  # workload). Plex cannot create libraries over the mounted bulk share while the server is
+  # UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
+  # half of that reasoning survived contact with the upstream source.
+  #
+  # THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to
+  # abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push ->
+  # Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a
+  # documented PLACEHOLDER and the live token is injected by the operator inside that window, with an
+  # explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure.
+  #
+  # ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
+  # (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
+  # the running `:latest` image), NOT assumed:
+  #   - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
+  #     PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
+  #     no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
+  #     logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
+  #   - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
+  #     status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
+  #     there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
+  #     SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
+  #     default 0, which continues startup even when a cont-init script fails.
+  # So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
+  # placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
+  # so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
+  # listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
+  # is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
+  - name: PLEX_CLAIM
+    valueFrom:
+      secretKeyRef:
+        name: plex-claim
+        key: PLEX_CLAIM
   # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
   # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
   # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
@@ -56,10 +85,12 @@ storage:
                               # container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
                               # root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
                               # (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
-                              # makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
-                              # UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
-                              # (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
-                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
+                              # makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
+                              # above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
+                              # exchange is then skipped on every later start. (The older note here claimed a
+                              # placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
+                              # otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
+                              # failure, and that is why one is committed.)
   # THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
   # /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
   # downloads/ movies/ photos/ tv/).

catalog/render.py

@@ -80,6 +80,18 @@ BULK_NFS_SERVER = "${BULK_NFS_SERVER}"
 BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retrans=2"]
 GATEWAY_NAME = "shared"
 GATEWAY_NS = "gateway"
+# ExternalDNS per-resource scoping (R-DNS-03; see _externaldns_route_annotations for the full WHY
+# and for the v0.21.0 source citation). The KEY is external-dns's own `ControllerKey`
+# (annotationPrefix + "controller"); external-dns skips any source object whose value is PRESENT
+# and != `ControllerValue` ("dns-controller"), so the VALUE only has to be some other string.
+# `none` is chosen because it reads as intent in `kubectl get httproute -o yaml` rather than as a
+# typo — "no DNS controller owns this route" — and because the one value that must never be
+# emitted here is external-dns's own, which a literal `none` cannot drift into.
+EXTERNALDNS_CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
+EXTERNALDNS_CONTROLLER_DISOWN = "none"
+# The value external-dns claims for itself. Held here ONLY so the renderer and the L0 guard can
+# assert the disown value is not it; never emitted.
+EXTERNALDNS_CONTROLLER_CLAIMED = "dns-controller"
 KP_RELEASE = "kube-prometheus-stack"  # ServiceMonitor/PrometheusRule discovery label
 MONITORING_NS = "monitoring"          # where Prometheus runs; the only admitted scrape source
 
@@ -836,12 +848,15 @@ def r_in_cluster_client_policies(s: dict) -> list:
 
 
 def _route_meta(s: dict, route: dict) -> dict:
+    """HTTPRoute metadata: auth wiring hints PLUS the ExternalDNS scoping derived from
+    `exposure` (_externaldns_route_annotations — R-DNS-03: an `internal` service renders no
+    public DNS record, and now cannot)."""
     meta = {
         "name": f"{s['name']}-{route['port']}",
         "namespace": s["name"],
         "labels": {"app.kubernetes.io/name": s["name"]},
     }
-    ann = _auth_annotations(s)
+    ann = {**_auth_annotations(s), **_externaldns_route_annotations(s)}
     if ann:
         meta["annotations"] = ann
     return meta
@@ -1039,6 +1054,64 @@ def _auth_annotations(s: dict) -> dict:
     return {}
 
 
+def _externaldns_route_annotations(s: dict) -> dict:
+    """ExternalDNS scoping, rendered FROM `exposure` (renderer-contract §exposure; R-DNS-03).
+
+    `internal` renders NO public DNS record. That sentence was in the contract from the start and
+    nothing enforced it: ExternalDNS watched EVERY HTTPRoute the shared Gateway carried, so an
+    `internal` service's route published `<name>.${BASE_DOMAIN}` into the PUBLIC authoritative
+    Cloudflare zone, answering with the Gateway VIP — an RFC1918 address. Six of them did
+    (jackett, pihole, qbittorrent, radarr, semaphore, sonarr), which published the internal
+    subnet, the Gateway VIP and a full service inventory to anyone who asked, and made every one
+    of those names a DNS-rebind primitive that the SOHO router strips on the way back in.
+
+    THE MECHANISM, verified against the DEPLOYED version (chart >=1.21.0 <1.22.0, app v0.21.0 —
+    `registry.k8s.io/external-dns/external-dns:v0.21.0`, read off the live prod Deployment), not
+    from memory. external-dns v0.21.0 `source/annotations/processors.go`:
+
+        func IsControllerMismatch(entity metav1.ObjectMetaAccessor, rType string) bool {
+            value, ok := entity.GetObjectMeta().GetAnnotations()[ControllerKey]
+            if ok && value != ControllerValue { ...; return true }
+            return false
+        }
+
+    with `ControllerKey = "external-dns.alpha.kubernetes.io/controller"` and
+    `ControllerValue = "dns-controller"`. `source/gateway.go` calls it per route inside
+    `Endpoints()`, BEFORE any hostname is resolved:
+
+        if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
+
+    so an HTTPRoute carrying any other value is not merely unpublished — it never becomes an
+    endpoint at all, for any hostname it names. `source/service.go` applies the identical check,
+    so the same annotation scopes the `service` source too if a rendered Service ever grows an
+    external-dns hostname.
+
+    WHY THIS IS STRUCTURAL AND NOT INCIDENTAL. The annotation is derived HERE, from `exposure`,
+    with no per-service list anywhere: every HTTPRoute this renderer emits for an `internal`
+    service carries it, and ci/render-golden.sh proves `apps/` reproduces byte-for-byte from
+    `catalog/`, so an `internal` service cannot EXIST in the applied tree without it. Flipping
+    `exposure: internal` to `public` is the only edit that can publish a name, which is exactly
+    the operator-facing declaration R-DNS-03 is written in terms of. The L0 check
+    conformance/tests/l0-static/internal-services-publish-no-public-dns re-proves all of that on
+    every gate run, including that this function reads `exposure` rather than naming services.
+
+    WHY NOT A CLUSTER-WIDE OPT-IN FILTER (`--label-filter`, chart `labelFilter`). It exists, and
+    v0.21.0's own `--help` names both sources in use as supporting it ("currently supported by
+    source types crd, gateway-httproute, ... service and ambassador-host"), so a "publish only
+    what is labelled" posture is available and is STRONGER by default. It is not taken here
+    because its blast radius is not this defect's: seven published names come from hand-authored
+    platform/apps manifests that carry no `exposure` field and that this renderer never sees
+    (auth/oauth2-proxy, crafty + crafty-mc, grafana, headlamp, the homepage apex, unifi). An
+    opt-in filter DELETES all seven the moment it is set, and whether each should stay published
+    is an operator decision, not a renderer one. Per-resource exclusion has exactly the blast
+    radius of the reported defect. The filter remains the available hardening once those seven
+    are adjudicated — see platform/externaldns/helmrelease.yaml.
+    """
+    if s["exposure"] == "internal":
+        return {EXTERNALDNS_CONTROLLER_KEY: EXTERNALDNS_CONTROLLER_DISOWN}
+    return {}
+
+
 def r_servicemonitor(s: dict):
     """ServiceMonitor — rendered ONLY when the SD says WHERE metrics are served.
 
@@ -1679,6 +1752,25 @@ EXTRA_UNITS: dict = {
             "gitea-actions-runner-secret.sops.yaml",
         ],
     },
+    # PiHole's DNS exposure: the udp/53 + tcp/53 LoadBalancer Services on a pinned VIP, plus the
+    # narrow LAN ingress CiliumNetworkPolicy that admits them (apps/wave-0/pihole-dns.yaml).
+    # `above: False` co-locates it in pihole's OWN wave: these are the pihole pod's own Services and
+    # its own policy, applied in the same wave as the Deployment they select, exactly as the
+    # rendered Service and CNPs in pihole.yaml are. It is NOT a runtime prerequisite of anything, so
+    # there is nothing for a higher wave to gate on.
+    #
+    # It is a non-catalog unit because the ServiceDefinition schema cannot express a LoadBalancer
+    # Service, a pinned lbipam VIP, a sharing key or UDP exposure at all (additionalProperties is
+    # false, and `ingressRoutes` only renders HTTPRoutes on the L7 Gateway) — the same reason
+    # apps/vms/unifi/50-service-l4.yaml and platform/crafty/service-l4.yaml are hand-written. The
+    # split-horizon ANSWERS are in the catalog, as FTLCONF env on catalog/pihole.yaml.
+    "pihole-dns": {
+        "anchor": "pihole",
+        "above": False,
+        "files": [
+            "pihole-dns.yaml",
+        ],
+    },
 }
 
 
@@ -1752,6 +1844,10 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
     convention:
       - `<svc>.yaml`                        -> wave(svc)
       - `<svc>-bootstrap-secret.sops.yaml`  -> wave(svc)   (consumed by svc's env, same apply)
+      - `<svc>-claim-secret.sops.yaml`      -> wave(svc)   (a one-shot upstream-service claim token
+        consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
+        bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
+        in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
       - `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc)  (forward-auth sidecar OIDC creds)
       - `<svc>-db.yaml`                     -> wave(svc-db) (a strictly-lower wave; CNPG DB)
       - `<svc>-db-app.sops.yaml`            -> wave(svc-db) (CNPG initdb owner secret)
@@ -1778,6 +1874,11 @@ def plan_apps_waves(sds: list, present_files: set) -> dict:
         w = waves[name]
         add(w, f"{name}.yaml")
         add(w, f"{name}-bootstrap-secret.sops.yaml")
+        # A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
+        # app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
+        # so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
+        # Placed only when the file exists, so services without one are unaffected.
+        add(w, f"{name}-claim-secret.sops.yaml")
         # forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
         # Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
         # so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
@@ -2078,6 +2179,96 @@ def _selftest() -> int:
         "a non-exposed service must not render a gateway-ingress allow (least privilege)",
     )
 
+    # 4b-dns) exposure -> ExternalDNS SCOPING on the HTTPRoute (R-DNS-03). `internal` renders no
+    #     public DNS record and must be INCAPABLE of rendering one: its route carries
+    #     external-dns's own controller annotation set to something other than `dns-controller`,
+    #     which v0.21.0's gateway-httproute source (source/gateway.go ->
+    #     annotations.IsControllerMismatch) uses to skip the route before any hostname is
+    #     resolved. `public` must NOT carry it, or the five public names would vanish instead.
+    #     Both directions are asserted here because only asserting the internal half would pass
+    #     just as happily if the renderer stamped EVERY route and silently unpublished the zone.
+    dns_int_docs = render(
+        {
+            "name": "di",
+            "source": {"image": "a:1"},
+            "exposure": "internal",
+            "ingressRoutes": [{"host": "di.home.keithjackson.dev", "port": 80}],
+            "auth": {"mode": "none"},
+        }
+    )
+    dns_int_routes = [d[1] for d in dns_int_docs if d[1]["kind"] == "HTTPRoute"]
+    check(
+        len(dns_int_routes) == 1,
+        "an internal exposed service must still render its HTTPRoute (LAN reachability is "
+        "unchanged — only the PUBLIC record is withdrawn)",
+    )
+    if dns_int_routes:
+        iann = dns_int_routes[0]["metadata"].get("annotations", {})
+        check(
+            iann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN,
+            "an internal service's HTTPRoute must carry the ExternalDNS disown annotation",
+        )
+        check(
+            iann.get(EXTERNALDNS_CONTROLLER_KEY) != EXTERNALDNS_CONTROLLER_CLAIMED,
+            "the disown value must never be external-dns's own `dns-controller` (that CLAIMS "
+            "the route rather than releasing it — the one value that inverts the guarantee)",
+        )
+    dns_pub_docs = render(
+        {
+            "name": "dp",
+            "source": {"image": "a:1"},
+            "exposure": "public",
+            "ingressRoutes": [{"host": "dp.home.keithjackson.dev", "port": 80}],
+            "auth": {"mode": "native"},
+        }
+    )
+    dns_pub_routes = [d[1] for d in dns_pub_docs if d[1]["kind"] == "HTTPRoute"]
+    check(len(dns_pub_routes) == 1, "a public service must render its HTTPRoute")
+    if dns_pub_routes:
+        check(
+            EXTERNALDNS_CONTROLLER_KEY
+            not in dns_pub_routes[0]["metadata"].get("annotations", {}),
+            "a public service's HTTPRoute must NOT carry the ExternalDNS disown annotation",
+        )
+    # The DEFAULT posture is `internal` (renderer-contract "Default posture"), so a
+    # ServiceDefinition that simply omits `exposure` must also be unpublishable. Omitting the
+    # field is the likeliest way a new service arrives, and it is the case a per-service
+    # allow-list would miss.
+    dns_dflt_docs = render(
+        {
+            "name": "dd",
+            "source": {"image": "a:1"},
+            "ingressRoutes": [{"host": "dd.home.keithjackson.dev", "port": 80}],
+            "auth": {"mode": "none"},
+        }
+    )
+    dns_dflt_route = next(d[1] for d in dns_dflt_docs if d[1]["kind"] == "HTTPRoute")
+    check(
+        dns_dflt_route["metadata"]["annotations"].get(EXTERNALDNS_CONTROLLER_KEY)
+        == EXTERNALDNS_CONTROLLER_DISOWN,
+        "a ServiceDefinition that OMITS `exposure` defaults to internal and must also be "
+        "unpublishable (default posture, renderer-contract)",
+    )
+    # The scoping annotation must COEXIST with the auth hints rather than replace them: both are
+    # merged into one `annotations` map, and a forward-auth internal app is the overlap case.
+    dns_fa_docs = render(
+        {
+            "name": "df",
+            "source": {"image": "a:1"},
+            "exposure": "internal",
+            "ingressRoutes": [{"host": "df.home.keithjackson.dev", "port": 80}],
+            "auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
+        }
+    )
+    dns_fa_ann = next(
+        d[1] for d in dns_fa_docs if d[1]["kind"] == "HTTPRoute"
+    )["metadata"]["annotations"]
+    check(
+        dns_fa_ann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN
+        and dns_fa_ann.get(f"{LBL}/auth-mode") == "forward-auth",
+        "DNS scoping and auth-mode hints must COEXIST on one internal forward-auth route",
+    )
+
     # 4c) inClusterClients -> one ADDITIVE, least-privilege ingress allow per entry that admits a
     #     PLATFORM controller's NAMESPACE (not the reserved `ingress` identity, not `all`) to ONE
     #     declared port. This is what keeps a cluster whose GitOps source is an in-cluster Service
@@ -2366,7 +2557,8 @@ def _selftest() -> int:
     # 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
     #     EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
     present = {
-        "a.yaml", "svc.yaml", "svc-bootstrap-secret.sops.yaml",
+        "a.yaml", "a-claim-secret.sops.yaml",
+        "svc.yaml", "svc-bootstrap-secret.sops.yaml",
         "svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
         "svc-db-backup-credentials-prod.sops.yaml",
         "gitea.yaml",
@@ -2377,6 +2569,10 @@ def _selftest() -> int:
     plan = plan_apps_waves(plan_sds, present)
     flat = {f: wv for wv, fs in plan.items() for f in fs}
     check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
+    check(flat.get("a-claim-secret.sops.yaml") == 0,
+          "a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
+          "as its app — an env secretKeyRef has no `optional`, so a later wave means "
+          "CreateContainerConfigError")
     check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
           and flat.get("svc-db-backup-credentials.sops.yaml") == 0
           and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,

ci/l1.sh

@@ -30,11 +30,13 @@ req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
 # (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
 # touched subset, then COLLECT its artifacts into the CI artifact dirs.
 # --include-scope: union the touched subset with every L1 test DECLARING
-# `conformance.home.k3s/scope: whole-repo`. No L1 test declares it today, so this is a no-op
-# now — it is wired HERE, once, so the NEXT whole-repo invariant (at any layer) is picked up by
-# declaring the annotation, instead of needing someone to remember to hand-write a stage. That
-# "someone must remember" is the failure mode this whole change removes; see
-# conformance/lib/common.sh. rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
+# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
+# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
+# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
+# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
+# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
+# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
+# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
 : "${WHOLE_REPO_SCOPE:=whole-repo}"
 l1_rc=0
 "$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \

ci/tests/touched-requirements.test.sh

@@ -136,6 +136,65 @@ printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
 assert_present "R-ENV-02"     "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
 assert_present "R-GITOPS-01"  "$OUT4" "clusters/* attributes R-GITOPS-01"
 
+# ---------------------------------------------------------------------------------------
+# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
+#
+# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
+# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
+# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
+# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
+# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
+# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
+#
+# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
+# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
+#
+# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
+# from the PATH and not from the content scan.
+
+# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
+# under test here, not just the presence of both arms.
+git -C "$TMP" checkout -q -b feature-apps-vms main
+mkdir -p "$TMP/apps/vms/demo"
+printf 'kind: VirtualMachine\nmetadata:\n  name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
+OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
+assert_present "R-PLAT-03"  "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
+assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
+assert_absent  "R-PLAT-02"  "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
+
+# A plain ServiceDefinition under apps/ gets the container-workload set.
+git -C "$TMP" checkout -q -b feature-apps main
+mkdir -p "$TMP/apps/wave-1"
+printf 'kind: ServiceDefinition\nmetadata:\n  name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
+OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
+assert_present "R-PLAT-02"  "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
+assert_present "R-NET-01"   "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
+assert_present "R-NET-06"   "$OUT_APPS" "apps/* attributes the egress-class requirements"
+assert_present "R-IDP-02"   "$OUT_APPS" "apps/* attributes the auth-mode requirements"
+
+# homepage / headlamp ARE the command-centre surface.
+git -C "$TMP" checkout -q -b feature-c2 main
+mkdir -p "$TMP/platform/headlamp"
+printf 'kind: Deployment\nmetadata:\n  name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
+OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
+assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
+assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
+
+# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
+git -C "$TMP" checkout -q -b feature-assembly main
+mkdir -p "$TMP/platform"
+printf 'resources:\n  - cilium\n  - longhorn\n' > "$TMP/platform/kustomization.yaml"
+git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
+OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
+printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
+assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
+
 # ---------------------------------------------------------------------------------------
 # THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
 #

ci/touched-requirements.sh

@@ -177,6 +177,23 @@ path_requirements() {
   case "$1" in
     catalog/*)                 echo "R-GITOPS-02" ;;
     schemas/*)                 echo "R-GITOPS-02" ;;
+    # apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
+    # manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
+    # adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
+    # checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
+    # class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
+    # violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
+    # catch it at apply time" is post-merge, and the gate exists to catch it before.
+    #
+    # ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
+    # apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
+    # subject to default-deny (R-NET-01).
+    apps/vms/*)                echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
+    # Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
+    # workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
+    # (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
+    # default-deny (R-NET-01).
+    apps/*)                    echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
     platform/kyverno/*)        echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
     platform/cilium/*)         echo "R-NET-01 R-NET-02 R-NET-05" ;;
     platform/longhorn/*)       echo "R-STATE-01 R-STATE-02" ;;
@@ -208,6 +225,39 @@ path_requirements() {
     platform/keycloak/*)       echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
     platform/oauth2-proxy/*)   echo "R-IDP-02 R-IDP-06" ;;
     platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
+    # The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
+    # rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
+    # platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
+    # itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
+    # docs/12-command-center-ui.md), so editing them must select the C2 tests.
+    platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
+    # KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
+    platform/kubevirt/*)       echo "R-PLAT-03 R-OBS-01" ;;
+    # The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
+    # TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
+    platform/gateway/*)        echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
+    # CoreDNS is cluster DNS resolution (R-DNS-01).
+    platform/coredns/*)        echo "R-DNS-01" ;;
+    # crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
+    # its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
+    platform/crafty/*)         echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
+    # Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
+    platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
+    # runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
+    # resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
+    # is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
+    # runbook change is an observability change.
+    runbooks/*)                echo "R-OBS-04" ;;
+    # platform-config/ is the env-specific configuration Flux substitutes into and applies.
+    platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
+    # platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
+    # Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
+    # file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
+    # change. That is the most consequential single-line edit in the repo and it attributed nothing.
+    platform/kustomization.yaml) echo "R-GITOPS-01" ;;
+    # .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
+    # here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
+    .sops.yaml)                echo "R-SEC-01 R-GITOPS-01" ;;
     conformance/*)             echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
     # infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
     # nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but

clusters/base/cluster-vars.yaml

@@ -99,7 +99,9 @@ data:
   # (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
   # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
   # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
-  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH), velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
+  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
+  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
   # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
   # UNSUBSTITUTED path would ship the literal string into the cluster.
   #
@@ -159,6 +161,30 @@ data:
   # HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
   UNIFI_L4_VIP: "192.168.1.54"               # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
   CRAFTY_L4_VIP: "192.168.1.55"              # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
+  # PiHole's DNS listener: udp/53 + tcp/53 for LAN CLIENTS (apps/wave-0/pihole-dns.yaml). It is a
+  # THIRD pinned L4 VIP for the same reason as the two above — the `shared` Gateway is an L7 HTTPS
+  # terminator and cannot express DNS at all (udp/53 especially). It must NOT be ${GATEWAY_VIP}:
+  # LAN clients are pointed at this address as their resolver, and it has to stay put across
+  # reschedules, so it is pinned by lbipam.cilium.io/ips rather than taken from the pool head.
+  #
+  # WHY A SEPARATE ADDRESS AND NOT THE GATEWAY VIP: every `<svc>.${BASE_DOMAIN}` name resolves TO
+  # the Gateway VIP. Serving DNS from the same address would be legal but conflates "the thing that
+  # answers where services live" with "a service", and it would put LAN resolution behind the
+  # Gateway Service's own lifecycle. Keep them separate.
+  #
+  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
+  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
+  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+
+  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
+  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
+  # from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
+  # CIDR, not to Cilium's `world` entity the way the unifi/crafty game-port allows are. It is the
+  # same /24 the node IPs and ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} come out of.
+  #
+  # dev  -> 192.168.1.0/24  (Proxmox LAN; nodes .50/.51/.52)
+  # prod -> 192.168.2.0/24  (prod LAN; nodes .10/.11/.12)
+  LAN_CIDR: "192.168.1.0/24"                 # ingress scope for the PiHole DNS listener
 
   # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
   # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media

clusters/dev/kustomization.yaml

@@ -122,6 +122,23 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
+  # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
+  # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
+  # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
+  # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
+  # as load-bearing as the file itself. The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-vms
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   - patch: |-
       apiVersion: source.toolkit.fluxcd.io/v1
       kind: GitRepository

clusters/prod/kustomization.yaml

@@ -113,6 +113,23 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
+  # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
+  # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
+  # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
+  # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
+  # as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: apps-vms
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
   #
   # ############################################################################################
@@ -448,6 +465,23 @@ patches:
       - op: replace
         path: /data/CRAFTY_L4_VIP
         value: "192.168.2.22"
+      # PiHole's DNS listener VIP — udp/53 + tcp/53 for LAN clients (apps/wave-0/pihole-dns.yaml).
+      # 192.168.2.23 is the NEXT FREE address in the prod pool 192.168.2.20-.30, verified live
+      # 2026-09-14: the CiliumLoadBalancerIPPool `default-pool` reported IPsTotal 11 / IPsUsed 3 /
+      # IPsAvailable 8, and the only three allocations are .20 (cilium-gateway-shared), .21
+      # (unifi-l4-tcp/-udp) and .22 (crafty-l4-tcp/-udp). ARP-probed from a LAN host in the same
+      # broadcast domain (arping 192.168.2.23, 3 probes, 0 responses), so nothing off-cluster holds
+      # it either, and it is inside the pool block the operator already asserted sits outside the
+      # SOHO router's DHCP scope.
+      - op: replace
+        path: /data/PIHOLE_DNS_VIP
+        value: "192.168.2.23"
+      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
+      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
+      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
+      - op: replace
+        path: /data/LAN_CIDR
+        value: "192.168.2.0/24"
       # ------------------------------------------------------------------------------------------
       # BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
       # ------------------------------------------------------------------------------------------

conformance/runs/20260919T051633Z-6eaaaae/run.log

@@ -0,0 +1,6 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.md and /home/keith/source/home-network-k3s-impl/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+ERROR: cluster reachable but 'chainsaw' (v0.2.12) not on PATH.
+Install the pinned binary or run via the in-cluster Job (incluster/).

conformance/runs/20260919T052225Z-9339d5a/report.junit.xml

@@ -0,0 +1,4 @@
+<testsuites name="chainsaw-report" tests="1" failures="0" errors="0" time="0.000000"><testsuite name="/work/conformance/tests/l3-fault-injection/node-loss" tests="1" failures="0" errors="0" id="0" time="">
+    <testcase name="node-loss" classname="" time="115.855688" />
+  </testsuite>
+</testsuites>

conformance/runs/20260919T052225Z-9339d5a/run.log

@@ -0,0 +1,232 @@
+[run.sh] Coverage report first (always)
+[coverage] 51/51 requirements have a scaffolded test (100.0%); 0 gap(s), 0 safety-critical gap(s).
+[coverage] wrote /work/conformance/_artifacts/coverage/coverage.md and /work/conformance/_artifacts/coverage/coverage.json
+[run.sh] No L0/L1 offline checks match the selection (layer=l3 reqs='R-STATE-02').
+[run.sh] LIGHT pass: no tests selected.
+[run.sh] HEAVY pass: 1 test(s) SERIAL (parallel=1) — node-loss
+Version: 0.2.12
+Loading config (/work/conformance/config/chainsaw-config.yaml)...
+- Using test file: chainsaw-test
+- TestDirs [/work/conformance/tests/l3-fault-injection/node-loss]
+- SkipDelete false
+- FailFast false
+- ReportFormat 'JUNIT-TEST'
+- ReportName 'chainsaw-report-heavy'
+- ReportPath '/work/conformance/_artifacts'
+- Namespace ''
+- FullName false
+- IncludeTestRegex ''
+- ExcludeTestRegex ''
+- ApplyTimeout 1m30s
+- AssertTimeout 3m0s
+- CleanupTimeout 2m0s
+- DeleteTimeout 2m0s
+- ErrorTimeout 1m0s
+- ExecTimeout 3m0s
+- DeletionPropagationPolicy Background
+- Parallel 1
+- Template true
+- NoCluster false
+- PauseOnFailure false
+Loading tests...
+- node-loss (/work/conformance/tests/l3-fault-injection/node-loss)
+Loading values...
+Running tests...
+=== RUN   chainsaw
+=== PAUSE chainsaw
+=== CONT  chainsaw
+=== RUN   chainsaw/node-loss
+=== PAUSE chainsaw/node-loss
+=== CONT  chainsaw/node-loss
+    l.go:52: | 05:22:26 | node-loss | @chainsaw                               | CREATE    | OK    | v1/Namespace @ chainsaw-advanced-boxer
+    l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | TRY       | BEGIN |
+    l.go:52: | 05:22:26 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -euo pipefail
+        NS="$NAMESPACE"
+        
+        # -- Gate 1: skip cleanly (green) if the longhorn tier is not present yet. --
+        if ! kubectl get storageclass longhorn >/dev/null 2>&1; then
+          echo "SKIP: 'longhorn' StorageClass absent — node-loss not applicable on this cluster yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        # -- Gate 2: a 3-replica volume + single-node-loss headroom needs >= 3 Ready nodes. --
+        READY="$(kubectl get nodes \
+          -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
+          | grep -c '^True$' || true)"
+        echo "ready nodes: ${READY}"
+        if [ "${READY:-0}" -lt 3 ]; then
+          echo "SKIP: only ${READY:-0} Ready node(s) (<3) — node-loss resilience not meaningful yet"
+          echo NODE_LOSS_SKIPPED_OK
+          exit 0
+        fi
+        
+        CANARY="node-loss-canary-$(date +%s%N)"
+        
+        # -- GIVEN: a Longhorn PVC (platform default SC = 3 replicas across nodes) + a
+        #    writer pod that writes the canary and flushes it to the replicated volume. --
+        kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: PersistentVolumeClaim
+        metadata:
+          name: survivor
+        spec:
+          accessModes: [ReadWriteOnce]
+          storageClassName: longhorn
+          resources:
+            requests:
+              storage: 1Gi
+        EOF
+        
+        apply_pod() {
+          kubectl -n "$NS" apply -f - <<EOF
+        apiVersion: v1
+        kind: Pod
+        metadata:
+          name: $1
+        spec:
+          restartPolicy: Never
+          containers:
+            - name: app
+              image: busybox:1.36.1
+              command: ["sh", "-c", "sleep 3600"]
+              volumeMounts:
+                - name: data
+                  mountPath: /data
+          volumes:
+            - name: data
+              persistentVolumeClaim:
+                claimName: survivor
+        EOF
+        }
+        
+        apply_pod writer
+        kubectl -n "$NS" wait --for=condition=Ready pod/writer --timeout=240s
+        printf '%s' "$CANARY" | kubectl -n "$NS" exec -i writer -- sh -c 'cat > /data/canary && sync'
+        echo "wrote canary: $(kubectl -n "$NS" exec writer -- cat /data/canary)"
+        
+        # The victim node is the one the attached volume currently lives on.
+        VICTIM="$(kubectl -n "$NS" get pod writer -o jsonpath='{.spec.nodeName}')"
+        [ -n "$VICTIM" ] || { echo "FAIL: could not determine writer's node" >&2; exit 1; }
+        echo "victim node (currently hosts the attached volume): $VICTIM"
+        
+        # Best-effort precondition log: if the Longhorn Volume CR is readable, show the
+        # volume is genuinely 3-replica/healthy before we take a node. Non-fatal (the
+        # runner is least-privilege and may not have longhorn.io read rights) — the
+        # data-availability assertion below is the authoritative proof, not this.
+        PVNAME="$(kubectl -n "$NS" get pvc survivor -o jsonpath='{.spec.volumeName}' 2>/dev/null || true)"
+        if [ -n "$PVNAME" ]; then
+          kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+            -o jsonpath='pre-fault robustness={.status.robustness} replicas={.spec.numberOfReplicas}{"\n"}' \
+            2>/dev/null || echo "(Longhorn Volume CR not readable — continuing; data read is the proof)"
+        fi
+        
+        # -- WHEN: FAULT INJECTION — remove the victim node FROM SERVICE. --
+        #    Label it with THIS test's unique namespace so `finally` can always find and
+        #    uncordon exactly this node; then cordon it; then evict ONLY our own writer
+        #    pod, forcing the volume to detach from the victim. (No `drain` — unrelated
+        #    pods on the node are never touched.)
+        kubectl label node "$VICTIM" "conformance.home.k3s/cordoned-by=$NS" --overwrite
+        kubectl cordon "$VICTIM"
+        echo "cordoned + labelled victim node: $VICTIM"
+        kubectl -n "$NS" delete pod writer --wait=true --timeout=240s
+        echo "evicted writer off the victim node; volume detached from $VICTIM"
+        
+        # -- THEN: a fresh workload re-binds the SAME volume. The victim is unschedulable,
+        #    so the scheduler MUST place the reader on a SURVIVING node; Longhorn re-attaches
+        #    the volume there from its replicas on the other nodes. --
+        apply_pod reader
+        kubectl -n "$NS" wait --for=condition=Ready pod/reader --timeout=360s
+        READER_NODE="$(kubectl -n "$NS" get pod reader -o jsonpath='{.spec.nodeName}')"
+        echo "reader scheduled on surviving node: $READER_NODE (victim was $VICTIM)"
+        if [ "$READER_NODE" = "$VICTIM" ]; then
+          echo "FAIL: reader landed back on the cordoned victim node — node was not removed from service" >&2
+          exit 1
+        fi
+        
+        GOT="$(kubectl -n "$NS" exec reader -- cat /data/canary)"
+        echo "recovered='$GOT' expected='$CANARY'"
+        [ "$GOT" = "$CANARY" ] || { echo "FAIL: data not available after node loss" >&2; exit 1; }
+        echo "volume available + data intact from a surviving node"
+        
+        # Best-effort reconvergence log: if the Longhorn Volume CR is readable, wait
+        # (bounded) for robustness to settle back to available (healthy|degraded — both
+        # serve I/O; a healthy 3-replica volume rebuilds a replica once headroom exists).
+        # Non-fatal for the same least-privilege reason as the pre-fault log.
+        if [ -n "$PVNAME" ]; then
+          for i in $(seq 1 30); do
+            ROB="$(kubectl -n longhorn-system get volumes.longhorn.io "$PVNAME" \
+                  -o jsonpath='{.status.robustness}' 2>/dev/null || true)"
+            [ -z "$ROB" ] && { echo "(Longhorn Volume CR not readable — skipping reconvergence log)"; break; }
+            echo "post-fault robustness[$i]=$ROB"
+            case "$ROB" in healthy|degraded) break ;; esac
+            sleep 10
+          done
+        fi
+        
+        echo NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        ready nodes: 3
+        persistentvolumeclaim/survivor created
+        pod/writer created
+        pod/writer condition met
+        wrote canary: node-loss-canary-1789795346371286043
+        victim node (currently hosts the attached volume): k3s-prod-2
+        pre-fault robustness=healthy replicas=3
+        node/k3s-prod-2 labeled
+        node/k3s-prod-2 cordoned
+        cordoned + labelled victim node: k3s-prod-2
+        pod "writer" deleted from chainsaw-advanced-boxer namespace
+        evicted writer off the victim node; volume detached from k3s-prod-2
+        pod/reader created
+        pod/reader condition met
+        reader scheduled on surviving node: k3s-prod-3 (victim was k3s-prod-2)
+        recovered='node-loss-canary-1789795346371286043' expected='node-loss-canary-1789795346371286043'
+        volume available + data intact from a surviving node
+        post-fault robustness[1]=healthy
+        NODE_LOSS_VOLUME_AVAILABLE_OK
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | TRY       | END   |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | FINALLY   | BEGIN |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c set -uo pipefail
+        for n in $(kubectl get nodes -l "conformance.home.k3s/cordoned-by=$NAMESPACE" -o name 2>/dev/null); do
+          kubectl uncordon "${n#node/}" || true
+          kubectl label "$n" "conformance.home.k3s/cordoned-by-" || true
+          echo "restored node to service: ${n#node/}"
+        done
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        node/k3s-prod-2 uncordoned
+        node/k3s-prod-2 unlabeled
+        restored node to service: k3s-prod-2
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:37 | node-loss | volume-stays-available-across-node-loss | CMD       | RUN   |
+        === COMMAND
+        /usr/bin/sh -c kubectl -n "$NAMESPACE" delete pod writer reader --ignore-not-found --wait=false || true
+        kubectl -n "$NAMESPACE" delete pvc survivor --ignore-not-found --wait=false || true
+        echo "cleanup best-effort; chainsaw namespace teardown finalizes"
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | LOG   |
+        === STDOUT
+        pod "reader" deleted from chainsaw-advanced-boxer namespace
+        persistentvolumeclaim "survivor" deleted from chainsaw-advanced-boxer namespace
+        cleanup best-effort; chainsaw namespace teardown finalizes
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | SCRIPT    | DONE  |
+    l.go:52: | 05:23:38 | node-loss | volume-stays-available-across-node-loss | FINALLY   | END   |
+    l.go:52: | 05:23:38 | node-loss | @chainsaw                               | CLEANUP   | BEGIN |
+    l.go:52: | 05:23:38 | node-loss | @chainsaw                               | DELETE    | OK    | v1/Namespace @ chainsaw-advanced-boxer
+    l.go:52: | 05:24:21 | node-loss | @chainsaw                               | CLEANUP   | END   |
+--- PASS: chainsaw (0.00s)
+    --- PASS: chainsaw/node-loss (115.86s)
+PASS
+Tests Summary...
+- Passed  tests 1
+- Failed  tests 0
+- Skipped tests 0
+Done.
+merge_junit: merged 1 report(s) -> /work/conformance/_artifacts/report.junit.xml (tests=1 failures=0 errors=0 skipped=0)
+[run.sh] Done. JUnit: /work/conformance/_artifacts/report.junit.xml | Coverage: /work/conformance/_artifacts/coverage/coverage.md

conformance/runs/ledger.jsonl

@@ -5,3 +5,4 @@
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9d0814ae61e6a0ba64d3789a71f1cd9c985efbf0", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK, TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-NET-06"], "run_id": "20260913T014923Z-9d0814a", "sentinel": "TUNNEL_RECOVERED_OK", "test": "vpn-killswitch", "ts_utc": "2026-09-13T01:49:23Z"}
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "24b8111d0c39e9417e9f4d782f333f683540cfa7", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK, DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-03"], "run_id": "20260914T021505Z-24b8111", "sentinel": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK", "test": "dr-restore", "ts_utc": "2026-09-14T02:15:05Z"}
 {"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "bd17b65db9fa906f336528e90b429ec1e2377963", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L0", "note": "L0 offline check exited 0; run_l1_offline.py renders its skips as JUnit <skipped/>, so a green JUnit is truthful without a sentinel", "proof": "proven", "requirements": ["R-ENV-02", "R-STATE-03"], "run_id": "20260914T021902Z-bd17b65", "sentinel": null, "test": "placeholder-guard-coupling", "ts_utc": "2026-09-14T02:19:02Z"}
+{"chainsaw_version": "v0.2.12", "env": "prod", "git_sha": "9339d5a79e2301d399e4e20722b164583c53c55d", "junit_result": "passed", "kube_version": "v1.36.4+k3s1", "layer": "L3", "note": "run log OUTPUT carries every declared pass token (NODE_LOSS_VOLUME_AVAILABLE_OK) and no skip token [log scoped to this testcase; tokens read from real-output sections only]", "proof": "proven", "requirements": ["R-STATE-02"], "run_id": "20260919T052225Z-9339d5a", "sentinel": "NODE_LOSS_VOLUME_AVAILABLE_OK", "test": "node-loss", "ts_utc": "2026-09-19T05:22:25Z"}

conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh

@@ -256,7 +256,13 @@ SCRAPE_OWNERS: list[dict] = [
         "file": KPS,
         "require": [
             r"metricLabelsAllowlist:",
-            r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
+              # The label must be IN the allowlist, not BE the whole allowlist. This pattern used to
+              # end in a literal `]`, asserting the list held exactly this one label — so adding a
+              # second one (`egress-uplink`, for the steering-staleness alerts) made this check FAIL
+              # while the property it guards stayed perfectly satisfied. A guard that breaks when you
+              # correctly extend the thing it guards teaches people to edit the guard, which is how it
+              # stops guarding. `[,\\]]` accepts either the end of the list or a further entry.
+              r"nodes=\[homelab\.keithjackson\.dev/egress-gateway[,\]]",
         ],
         "forbid": [],
         "why": "KSM exposes NO custom labels by default. Without the allowlist entry "
@@ -448,9 +454,14 @@ def is_golden_fixture(p: Path) -> bool:
     return "conformance" in parts and "golden" in parts
 
 
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 rule_files = sorted(
     p for p in ROOT.rglob("*.yaml")
-    if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
+    if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
 )
 check(len(rule_files) > 0,
       "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
@@ -619,7 +630,7 @@ def yaml_docs(path: Path) -> list[dict]:
 # job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
 same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
 for p in sorted(ROOT.rglob("*.yaml")):
-    if ".git" in p.parts:
+    if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts):
         continue
     txt = read(p)
     if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:

conformance/tests/l0-static/alert-rules-unit-tested/check.sh

@@ -179,6 +179,10 @@ UNTESTED_FILES = {
 # and NodeIOPressureCritical quiet on the same measured state — the whole point of that fixture is
 # that the two PSI detectors and the eight cause rules must agree about what "healthy" is, so the
 # detectors are no longer unasserted.
+# It SHRANK BY ONE on 2026-09-19: platform/monitoring/tests/vpn-egress.test.yaml pins the two new
+# egress-steering rules, and asserts VPNTunnelDown QUIET in all four fixtures — that silence IS the
+# claim those rules exist for (tun0 up and passing traffic while vpn-class egress is dropped), so it
+# had to become an assertion rather than a sentence in a comment.
 UNTESTED_ALERTS = {
     ("platform/cnpg/alerts.yaml", "CNPGWALArchiveBacklog"),
     ("platform/cnpg/alerts.yaml", "CNPGNoRecoverabilityPoint"),
@@ -195,7 +199,6 @@ UNTESTED_ALERTS = {
     ("platform/monitoring/alerts.yaml", "CertificateExpiringSoon"),
     ("platform/monitoring/alerts.yaml", "CertificateExpiryImminent"),
     ("platform/monitoring/alerts.yaml", "CertificateNotReady"),
-    ("platform/monitoring/alerts.yaml", "VPNTunnelDown"),
     ("platform/monitoring/alerts.yaml", "HubbleDeniedFlowSpike"),
     ("platform/monitoring/alerts.yaml", "KeycloakDown"),
     ("platform/monitoring/alerts.yaml", "KeycloakDatabaseDown"),
@@ -280,11 +283,16 @@ FETCH_RE = re.compile(
 #     versions.env. What is forbidden is a RULE-UNIT RUN acquiring its own engine: then the
 #     version, the network and the machine are decided silently at test time, and a missing
 #     toolchain becomes a green run instead of an exit 3.
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 SELF_DIR = "conformance/tests/l0-static/alert-rules-unit-tested/"
 INSTALLERS = (".gitea/workflows/scripts/install-tools.sh", "infra/devcontainer/")
 for p in sorted(ROOT.rglob("*.sh")):
     rel = str(p.relative_to(ROOT))
-    if rel.startswith((".git/", "conformance/runs/", SELF_DIR)):
+    if rel.startswith((".git/", ".claude/", "conformance/runs/", SELF_DIR)):
         continue
     try:
         text = p.read_text()
@@ -319,7 +327,7 @@ def tracked_yaml() -> list[Path]:
         return [ROOT / p for p in out.stdout.split("\0") if p]
     # No usable index (the gate container mounts a worktree whose .git points outside the
     # mount). Fall back to a filesystem walk rather than auditing nothing.
-    return sorted(p for p in ROOT.rglob("*.y*ml") if ".git/" not in str(p))
+    return sorted(p for p in ROOT.rglob("*.y*ml") if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts))
 
 
 KIND_RE = re.compile(r"^\s*kind:\s*PrometheusRule\s*$", re.M)
@@ -473,7 +481,34 @@ for rel in sorted(alert_files):
           f"{rel} is listed in UNTESTED_FILES but is now covered by {tests} — the register is "
           f"shrink-only: delete the line so the next missing test is caught")
     text = "".join((ROOT / t).read_text() for t in tests if (ROOT / t).is_file())
-    covered = [n for n in names if re.search(rf"\b{re.escape(n)}\b", text)]
+    # "COVERED" MUST MEAN ASSERTED, NOT MENTIONED (tightened 2026-09-19).
+    # This used to be a bare word match over the whole test file, so an alert named anywhere —
+    # including in a PROSE COMMENT — counted as tested. Hit for real while adding
+    # platform/monitoring/tests/vpn-egress.test.yaml: its header comment explains that VPNTunnelDown
+    # stays quiet during an egress-steering failure, and that sentence alone was enough to make this
+    # check declare VPNTunnelDown "asserted on" and demand its removal from UNTESTED_ALERTS. Deleting
+    # the register entry would then have recorded coverage that did not exist — the exact "looks fixed"
+    # outcome this check exists to prevent, produced BY the check.
+    # Now an alert counts only when a test file names it in an `alertname:` position, with comment
+    # lines stripped first so a commented-out assertion does not count either.
+    # MEASURED BLAST RADIUS BEFORE TIGHTENING: of 61 alerts across the PrometheusRule manifests, 32 are
+    # asserted in one of the two idioms below, 29 are not mentioned at all (already declared in the
+    # registers), and ZERO are prose-only — so this change failed nothing on the day it landed. That
+    # mattered because the registers are shrink-only: had any alert been prose-only, tightening would
+    # have needed a register entry it is not allowed to add. (The first measurement said 31/30/0 because
+    # it scanned only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companions.)
+    text_no_comments = "\n".join(
+        l for l in text.split("\n") if not l.lstrip().startswith("#")
+    )
+    # BOTH promtool idioms count, because both are real assertions:
+    #   alert_rule_test:   - alertname: CNPGPrimariesCoLocated
+    #   promql_expr_test:  - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
+    # Matching only the first produced a FALSE NEGATIVE on CNPGPrimariesCoLocated, which
+    # platform/cnpg/alerts.promtool-test.yaml pins across FOUR scenarios entirely through the second
+    # form. Had that been believed, the "fix" would have been a redundant duplicate test for an alert
+    # already better covered than most.
+    covered = [n for n in names
+               if re.search(rf'(?:alertname:\s*|alertname="){re.escape(n)}\b', text_no_comments)]
     covered_total += len(covered)
     for n in names:
         if n in covered:

conformance/tests/l0-static/alert-runbooks-resolvable/check.sh

@@ -126,9 +126,14 @@ def read(p: Path) -> str:
 SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"\']?whole-repo"
 RUNBOOK_URL_RE = re.compile(r"""^\s*runbook_url:\s*["']?([^"'\s#]+)["']?\s*$""")
 
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 rule_files = sorted(
     p for p in ROOT.rglob("*.yaml")
-    if ".git" not in p.parts and "kind: PrometheusRule" in read(p)
+    if not any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) and "kind: PrometheusRule" in read(p)
 )
 check(len(rule_files) > 0,
       "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "

conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml

@@ -0,0 +1,136 @@
+# =============================================================================
+# L0 STATIC — no service declared `internal` can produce a PUBLIC DNS record
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. docs/09 R-DNS-03 says `internal` services are unreachable from the
+# public internet, that external names resolve to the PUBLIC IP, and that Pi-hole provides the
+# internal split-horizon answers. schemas/renderer-contract.md §exposure says an `internal`
+# service renders "no public DNS record". NOTHING ENFORCED EITHER SENTENCE. ExternalDNS
+# (platform/externaldns/helmrelease.yaml, `sources: [gateway-httproute, service]`,
+# `policy: sync`, provider cloudflare) considered EVERY HTTPRoute on the shared Gateway, so six
+# `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — each
+# published `<name>.home.keithjackson.dev` into the PUBLIC authoritative Cloudflare zone,
+# answering with the Gateway VIP 192.168.2.20. Verified at the authority on 2026-09-14; the
+# pre-state is committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt.
+#
+# Three things were wrong at once, and none of them was visible from inside the cluster:
+#   * the internal subnet, the Gateway VIP and a complete service inventory were publicly
+#     enumerable by anyone who could guess or brute-force a label;
+#   * a public name answering with an RFC1918 address IS a DNS-rebind primitive, which is why the
+#     SOHO router at 192.168.2.1 strips those answers and LAN clients need per-client dnsmasq
+#     overrides to reach their own services;
+#   * the guarantee was only ever prose. `exposure` was the operator's one declaration of intent
+#     and it had NO effect on DNS whatsoever, so "internal" meant nothing an attacker had to
+#     respect.
+#
+# WHY THE FIX NEEDS A GUARD AND NOT JUST A COMMIT. The mechanism is an annotation on a rendered
+# HTTPRoute. An annotation is exactly the kind of thing that survives review by looking harmless
+# and then disappears in a refactor of `_route_meta`, a hand-edit of apps/, or a new service added
+# by copying a public one. Nothing about its ABSENCE is loud: the service keeps working on the LAN,
+# the Kustomization stays Ready, and the only symptom is a public record nobody is looking at. The
+# 2026-09-13 ServiceMonitor incident in this same tree is the precedent — twelve monitors pointed
+# at a port that did not exist for weeks, silently, because the only thing asserting the wiring was
+# prose. This test makes the DNS guarantee a gate condition instead.
+#
+# GIVEN every ServiceDefinition in catalog/ and every HTTPRoute and external-dns-annotated
+#       Service this repo authors,
+# WHEN  each catalog service's rendered routes are resolved in the applied tree (apps/),
+# THEN  (1) every route of an `exposure: internal` service carries ExternalDNS's own controller
+#           annotation set to something OTHER than `dns-controller`, which external-dns v0.21.0
+#           uses to skip the source object outright (source/gateway.go ->
+#           annotations.IsControllerMismatch, checked before any hostname is resolved), so the
+#           route cannot become an endpoint for ANY name it declares;
+#       (2) NO route of an `exposure: public` service carries it — asserted explicitly, because a
+#           renderer that stamped every route would satisfy (1) perfectly while silently
+#           withdrawing the five public services' records, and `policy: sync` would delete them;
+#       (3) the same holds for the `service` source: an internal service's rendered Service may
+#           not carry an external-dns hostname annotation without the disown annotation beside it.
+#           Two sources are configured, so closing only the route half is half a guarantee;
+#       (4) the annotation is DERIVED FROM `exposure` in catalog/render.py and from nothing else —
+#           the emitting function must read `exposure`, must not name any catalog service, and
+#           must not be able to emit external-dns's own `dns-controller` value (the one value that
+#           CLAIMS a route rather than releasing it, inverting the guarantee while looking
+#           plausible). A per-service list is how twelve ServiceMonitors acquired one identical
+#           wrong port name in a single edit;
+#       (5) THE RENDERER IS PROVEN END-TO-END on synthetic ServiceDefinitions, by invoking the
+#           SHIPPED catalog/render.py as a subprocess: an `internal` fixture must yield the
+#           annotation, a `public` fixture must not, and a fixture that OMITS `exposure` must also
+#           yield it (the contract's default posture is `internal`, and omitting the field is the
+#           likeliest way a new service arrives). Without this the check would still pass if the
+#           renderer stopped working and someone hand-maintained apps/;
+#       (6) every HTTPRoute or annotated Service in the tree that the catalog does NOT render is
+#           DECLARED in KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why it is out of
+#           scope. Those are hand-authored platform/apps manifests with no `exposure` field for a
+#           renderer to read, and they are the seven names that REMAIN publicly advertised with an
+#           RFC1918 answer after this change. The register is shrink-only: an entry that stops
+#           matching a real object FAILS as stale, so the set of unadjudicated publishers cannot
+#           quietly grow, and adding a new one is a review event;
+#   AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
+#           so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and
+#           ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to
+#           ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
+#
+# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh's path map keys apps/* to NOTHING, and
+# apps/ is exactly where the rendered routes live. A requirement-scoped selection would therefore
+# filter this check out of the change most likely to break it — a hand-edit of a rendered route.
+# Same reasoning as its whole-repo siblings.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and
+# the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it directly,
+# and this Chainsaw Test wraps the SAME script.
+#
+# NOT proven (honest gaps; this is static, and it runs with no cluster and no network):
+#   - That the record is actually GONE from Cloudflare. That is a live read against an external
+#     authority and belongs in L2 beside conformance/tests/l2-behavioral/dns-exposure, which owns
+#     the positive direction (a public name DOES appear). The negative direction was proven
+#     MANUALLY on prod against the deployed v0.21.0 before this landed — a throwaway HTTPRoute
+#     carrying the annotation produced no record at all, and adding the annotation to a route that
+#     already had one produced `action=DELETE` for both its A record and its `a-` TXT registry
+#     entry within one ~60s reconcile — but a manual proof is not a gate.
+#   - That an `internal` name is unreachable from the internet. No DNS record is necessary for
+#     reachability; a port-forward plus a hardcoded IP still reaches the Gateway. This check owns
+#     the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
+#   - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
+#     each publishes so the claim is reviewable, which is all a static check can offer.
+#   - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
+#     rather than a routable address; that is an open operator decision recorded in
+#     platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: internal-services-publish-no-public-dns
+  labels:
+    conformance.home.k3s/R-DNS-02: "true"
+    conformance.home.k3s/R-DNS-03: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — this invariant quantifies over every ServiceDefinition in catalog/, every
+    # HTTPRoute in apps/wave-* and platform/* and runbooks/, and every external-dns-annotated
+    # Service (apps/vms/unifi, platform/crafty). No single requirement path-maps to that set, and
+    # ci/touched-requirements.sh maps apps/* — where the rendered routes live — to no requirement
+    # at all, so a requirement-scoped selection would skip this check on precisely the changes
+    # that can break it.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "true"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      A service the operator declared `internal` cannot produce a public DNS record: the
+      guarantee is derived from `exposure` by catalog/render.py, present on every rendered route
+      in the applied tree, absent from every public one, proven end-to-end on synthetic fixtures,
+      and the hand-authored manifests the catalog does not render are enumerated in a shrink-only
+      register so the unadjudicated set cannot grow silently.
+spec:
+  steps:
+    - name: assert-no-internal-service-can-publish-a-public-dns-record
+      description: >-
+        Resolve every catalog service's rendered HTTPRoutes in apps/, assert the ExternalDNS
+        disown annotation on every internal one and on no public one, extend the same rule to the
+        `service` source, assert catalog/render.py derives the annotation from `exposure` rather
+        than from a service list, prove the shipped renderer on internal/public/default-posture
+        fixtures, and ratchet the register of hand-authored publishers the catalog does not render.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh

@@ -0,0 +1,458 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 STATIC CHECK — no service declared `internal` can produce a PUBLIC DNS record.
+# See chainsaw-test.yaml's header for the full WHY. In one paragraph:
+#
+# docs/09 R-DNS-03 and schemas/renderer-contract.md §exposure both say an `internal` service
+# renders no public DNS record. Nothing enforced it: ExternalDNS watched every HTTPRoute on the
+# shared Gateway, so six internal services published `<name>.home.keithjackson.dev` into the
+# public Cloudflare zone answering with the RFC1918 Gateway VIP (verified at the authority
+# 2026-09-14; pre-state committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt). The fix
+# derives ExternalDNS's controller annotation from `exposure` in catalog/render.py; this check is
+# what keeps it derived, present, and scoped.
+#
+# ASSERTIONS
+#   (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass);
+#   (1) every rendered route of an `internal` service carries the disown annotation;
+#   (2) NO rendered route of a `public` service carries it — a renderer stamping everything would
+#       satisfy (1) and silently delete the five public records under policy: sync;
+#   (3) the `service` source half: an internal service's Service may not carry an external-dns
+#       hostname annotation without the disown annotation beside it;
+#   (4) catalog/render.py DERIVES the annotation from `exposure` — the emitting function reads that
+#       field, names no catalog service, and cannot emit external-dns's own `dns-controller`;
+#   (5) the SHIPPED renderer is proven end-to-end on synthetic ServiceDefinitions (internal,
+#       public, and `exposure` OMITTED — the contract's default posture is internal);
+#   (6) every HTTPRoute / annotated Service the catalog does NOT render is declared in
+#       KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL;
+#   (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by
+#       ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written
+#       ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears).
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
+#   - That the record is gone from Cloudflare. Live read; belongs beside
+#     conformance/tests/l2-behavioral/dns-exposure, which owns the positive direction. The negative
+#     direction was proven manually on prod against the deployed v0.21.0 (a route carrying the
+#     annotation produced no record; adding it to a route that had one produced action=DELETE for
+#     both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
+#   - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
+#     This check owns the NAME axis only.
+#   - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
+#     routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
+# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if
+# catalog/ or apps/ is absent — i.e. the subject tree has vanished, which is not a pass.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if the subject trees are not there at all.
+if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ] || [ ! -f "${ROOT}/catalog/render.py" ]; then
+  echo "check SKIPPED: catalog/, apps/ or catalog/render.py absent — nothing to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: an `internal` ServiceDefinition cannot produce a public DNS record.
+
+PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than
+line-based parsing: an HTTPRoute's annotations and a ServiceDefinition's exposure are nested
+structures, and a line-based reader of them is precisely the approximate parser that silently
+stops matching — the failure class this check exists to prevent.
+"""
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+# external-dns's own constants (v0.21.0 source/annotations/annotations.go):
+#   ControllerKey   = annotationPrefix + "controller"   -> the key below
+#   ControllerValue = "dns-controller"                  -> the value that CLAIMS an object
+# source/annotations/processors.go IsControllerMismatch skips a source object when the key is
+# PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed.
+CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
+CONTROLLER_CLAIMED = "dns-controller"
+HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname"
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+def docs_of(p: Path):
+    """Every YAML document in a file, tolerating the non-manifest files in the tree."""
+    try:
+        return [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
+    except yaml.YAMLError:
+        return []
+
+
+# =============================================================================================
+# REGISTER — KNOWN_NON_CATALOG_PUBLISHERS. Shrink-only.
+#
+# These are HAND-AUTHORED HTTPRoutes and Services that put a name into the same public Cloudflare
+# zone but that catalog/render.py does not render, so there is no `exposure` field for the renderer
+# to derive anything from and this check cannot adjudicate them. That is a real limit, not a pass —
+# so each one is DECLARED, with the name it publishes and why it is out of this check's scope, and
+# the register can only shrink. An entry naming an object that no longer exists FAILS as stale, so
+# a deleted publisher cannot leave a lie behind; a NEW publisher fails until it is declared, so the
+# unadjudicated set cannot grow without a review event. Same ratchet reasoning as
+# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
+#
+# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
+# the honest state after this change, and it is deliberate: converting them is an operator
+# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
+# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
+# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
+#
+# Keyed by "<relpath>:<namespace>/<name>".
+# =============================================================================================
+KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
+    "platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
+        "publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
+        "platform component, not a catalog service; it must be reachable by every browser that "
+        "any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
+        "whether the five public services are — an operator decision, not a renderer one.",
+    "platform/keycloak/httproute.yaml:keycloak/keycloak":
+        "publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
+        "OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
+        "the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
+        "hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
+        "is meant to be public either way; it is listed because the PUBLISHING object is the "
+        "hand-authored platform route, which carries no exposure field.",
+    "platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
+        "publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
+        "certainly should not be publicly resolvable, but it is not a catalog service and "
+        "withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
+        "internal services (Pi-hole must answer first).",
+    "platform/headlamp/httproute.yaml:headlamp/headlamp":
+        "publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
+        "Keycloak OIDC; same adjudication as grafana.",
+    "platform/homepage/httproute.yaml:homepage/homepage":
+        "publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
+        "listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
+        "declaration.",
+    "runbooks/serving/httproute.yaml:runbooks/runbooks":
+        "declares NO hostnames and inherits the https-apex listener's ${BASE_DOMAIN}, so it "
+        "contributes to the apex record rather than a name of its own (the apex TXT registry "
+        "entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
+        "header for why gating runbooks on Keycloak is a loop with no exit.",
+    "platform/crafty/httproute.yaml:crafty/crafty":
+        "publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
+        "oauth2-proxy sidecar). Platform component, no exposure field.",
+    "platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
+        "the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
+        "${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
+        "Gateway cannot express. A game server is the one case where a public name may be "
+        "genuinely wanted; the address it answers with is still RFC1918.",
+    "apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
+        "the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
+        "(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
+        "under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
+        "vm-demo, which declares no ingressRoutes and publishes nothing.",
+}
+
+# =============================================================================================
+# (0) INVENTORY — the catalog, and every HTTPRoute / annotated Service in the tree.
+# =============================================================================================
+CATALOG = ROOT / "catalog"
+sds: dict[str, dict] = {}
+for p in sorted(CATALOG.glob("*.yaml")):
+    for d in docs_of(p):
+        if "name" in d and "source" in d:
+            sds[d["name"]] = d
+
+check(len(sds) > 0, "inventory: catalog/ yielded no ServiceDefinition — parse contract broken")
+
+# The default posture is `internal` (renderer-contract "Default posture"), so an ABSENT field is
+# internal, never "unclassified". Getting this wrong in the check would be the same bug as in the
+# renderer: a new service that omits exposure would go unaudited.
+exposure_of = {n: (sd.get("exposure") or "internal") for n, sd in sds.items()}
+# (namespace, routename) -> service, for the routes the renderer emits: one per ingressRoute,
+# named "<service>-<port>" in namespace "<service>" (catalog/render.py _route_meta).
+expected_routes: dict[tuple[str, str], str] = {}
+for n, sd in sds.items():
+    for r in sd.get("ingressRoutes") or []:
+        expected_routes[(n, f"{n}-{r['port']}")] = n
+
+internal_with_routes = sorted({s for (_, _), s in expected_routes.items()
+                               if exposure_of[s] == "internal"})
+public_with_routes = sorted({s for (_, _), s in expected_routes.items()
+                             if exposure_of[s] == "public"})
+check(len(internal_with_routes) > 0,
+      "inventory: no `internal` catalog service declares an ingressRoute — the subject set of "
+      "this check is empty, which means either the catalog changed shape or the parse is wrong")
+check(len(public_with_routes) > 0,
+      "inventory: no `public` catalog service declares an ingressRoute — assertion (2) would be "
+      "vacuous, so the parse is presumed wrong rather than the repo")
+
+SCAN_DIRS = ["apps", "platform", "runbooks"]
+routes: list[tuple[str, dict]] = []          # (relpath, HTTPRoute doc)
+dns_services: list[tuple[str, dict]] = []    # (relpath, Service doc carrying an external-dns ann)
+for sub in SCAN_DIRS:
+    base = ROOT / sub
+    if not base.is_dir():
+        continue
+    for p in sorted(base.rglob("*.yaml")):
+        rel = p.relative_to(ROOT).as_posix()
+        for d in docs_of(p):
+            kind = d.get("kind")
+            md = d.get("metadata") or {}
+            ann = md.get("annotations") or {}
+            if kind == "HTTPRoute":
+                routes.append((rel, d))
+            elif kind == "Service" and any(
+                a.startswith("external-dns.alpha.kubernetes.io/") for a in ann
+            ):
+                dns_services.append((rel, d))
+
+check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse "
+                       "contract broken (the tree is known to author several)")
+
+
+def ident(d: dict) -> tuple[str, str]:
+    md = d.get("metadata") or {}
+    return (md.get("namespace") or "", md.get("name") or "")
+
+
+def annotations_of(d: dict) -> dict:
+    return (d.get("metadata") or {}).get("annotations") or {}
+
+
+def is_disowned(d: dict) -> bool:
+    """True iff external-dns v0.21.0 would SKIP this object: the controller annotation is present
+    and its value is not external-dns's own. Absent == published (the default), which is the whole
+    point — the annotation is the only thing standing between a route and the public zone."""
+    v = annotations_of(d).get(CONTROLLER_KEY)
+    return v is not None and v != CONTROLLER_CLAIMED
+
+
+# =============================================================================================
+# (1) + (2) every internal service's rendered route is disowned; no public one is.
+# =============================================================================================
+by_ident: dict[tuple[str, str], list[tuple[str, dict]]] = {}
+for rel, d in routes:
+    by_ident.setdefault(ident(d), []).append((rel, d))
+
+for (ns, rname), svc in sorted(expected_routes.items()):
+    found = by_ident.get((ns, rname), [])
+    check(len(found) >= 1,
+          f"({svc}) the catalog declares an ingressRoute rendering HTTPRoute {ns}/{rname}, but no "
+          f"such HTTPRoute exists in the applied tree — apps/ is out of sync with catalog/ "
+          f"(ci/render-golden.sh owns the byte-level half of this)")
+    for rel, d in found:
+        if exposure_of[svc] == "internal":
+            v = annotations_of(d).get(CONTROLLER_KEY)
+            check(
+                v is not None,
+                f"({svc}) R-DNS-03 VIOLATED: {rel} HTTPRoute {ns}/{rname} belongs to an "
+                f"`exposure: internal` service but carries NO `{CONTROLLER_KEY}` annotation, so "
+                f"external-dns publishes {ns}.<zone> into the PUBLIC Cloudflare zone answering "
+                f"with the RFC1918 Gateway VIP. An internal service must render no public record.",
+            )
+            check(
+                v != CONTROLLER_CLAIMED,
+                f"({svc}) {rel} HTTPRoute {ns}/{rname} sets `{CONTROLLER_KEY}: "
+                f"{CONTROLLER_CLAIMED}` — that is external-dns's OWN value, which CLAIMS the route "
+                f"rather than releasing it. It reads like the guard and is its exact inverse.",
+            )
+        else:
+            check(
+                CONTROLLER_KEY not in annotations_of(d),
+                f"({svc}) {rel} HTTPRoute {ns}/{rname} belongs to an `exposure: "
+                f"{exposure_of[svc]}` service yet carries `{CONTROLLER_KEY}`. Under "
+                f"`policy: sync` that DELETES its live public record — the five public services "
+                f"must keep publishing.",
+            )
+
+# =============================================================================================
+# (3) the `service` source half — an internal service's Service may not publish either.
+# =============================================================================================
+for rel, d in dns_services:
+    ns, name = ident(d)
+    svc = ns if ns in sds else None
+    if svc and exposure_of[svc] == "internal":
+        check(
+            is_disowned(d),
+            f"({svc}) {rel} Service {ns}/{name} carries an external-dns annotation "
+            f"({sorted(a for a in annotations_of(d) if a.startswith('external-dns'))}) for an "
+            f"`exposure: internal` service without a disowning `{CONTROLLER_KEY}`. Both "
+            f"`gateway-httproute` AND `service` are configured sources, so closing only the route "
+            f"half is half a guarantee.",
+        )
+
+# =============================================================================================
+# (4) the GENERATOR derives it from `exposure` — not from a list of service names.
+# =============================================================================================
+render_py = ROOT / "catalog" / "render.py"
+src = read(render_py)
+check(bool(src), "catalog/render.py unreadable")
+
+emitter = re.search(
+    r"def _externaldns_route_annotations\(.*?\n(?=\n?def |\n?# =====)", src, re.S
+)
+check(emitter is not None,
+      "catalog/render.py must define `_externaldns_route_annotations` — the single place the "
+      "ExternalDNS scoping is derived. If it was renamed, rename it here too: this check has to "
+      "read the DERIVATION, not just the output, or a hardcoded per-service list would pass.")
+if emitter:
+    body = emitter.group(0)
+    # The derivation must read `exposure`. A body that does not mention it cannot be deriving
+    # anything from the operator's declaration.
+    check('"exposure"' in body or "'exposure'" in body,
+          "`_externaldns_route_annotations` must read the `exposure` field — the guarantee is "
+          "that the OPERATOR'S DECLARATION decides, so `exposure` has to appear in the decision")
+    # and it must not name any catalog service: a per-service list is how twelve ServiceMonitors
+    # acquired one identical wrong port name in a single edit. Only the executable statements are
+    # inspected — the docstring legitimately names the affected services as evidence.
+    code = re.sub(r'"""(?:.|\n)*?"""', "", body, count=1)
+    named = sorted(n for n in sds if re.search(rf"""['"]{re.escape(n)}['"]""", code))
+    check(not named,
+          f"`_externaldns_route_annotations` names catalog service(s) {named} in its CODE. The "
+          f"annotation must be derived from `exposure` alone; a per-service list silently omits "
+          f"the next service anyone adds.")
+    check(CONTROLLER_CLAIMED not in code,
+          f"`_externaldns_route_annotations` must never be able to emit `{CONTROLLER_CLAIMED}` — "
+          f"external-dns's own value CLAIMS the object instead of releasing it")
+
+# =============================================================================================
+# (5) THE SHIPPED RENDERER IS PROVEN on synthetic ServiceDefinitions. Three fixtures, because
+#     "internal is annotated" alone would also pass if the renderer annotated everything, and
+#     because the contract's DEFAULT posture is internal — the case a new service hits by
+#     omitting the field entirely, and the case a per-service list would miss.
+# =============================================================================================
+FIXTURES = [
+    ("fixture-internal", {"exposure": "internal"}, True,
+     "an `internal` fixture must render the disown annotation"),
+    ("fixture-public", {"exposure": "public"}, False,
+     "a `public` fixture must render NO disown annotation"),
+    ("fixture-default", {}, True,
+     "a fixture OMITTING `exposure` defaults to internal (renderer-contract default posture) and "
+     "must also render the disown annotation"),
+]
+with tempfile.TemporaryDirectory() as td:
+    for fname, extra, want, msg in FIXTURES:
+        sd = {
+            "name": fname,
+            "kind": "container",
+            "source": {"image": "example/app:1.0.0"},
+            "egress": "internet",
+            "dependencies": [],
+            "ingressRoutes": [{"host": f"{fname}.home.keithjackson.dev", "port": 8080}],
+            # `native` avoids the oauth2-proxy sidecar path and the schema's groups requirement,
+            # keeping the fixture about DNS scoping and nothing else. A public service may not use
+            # auth.mode none (schema invariant since 1.1.0), so both fixtures use the same mode.
+            "auth": {"mode": "native"},
+            "update": {"channel": "stable", "cadence": "0 4 * * 0",
+                       "autoRollbackOnFailedHealth": True},
+            "observability": {"metrics": True, "slis": ["up"]},
+            "backup": {"policy": "reacquirable"},
+        }
+        sd.update(extra)
+        f = Path(td) / f"{fname}.yaml"
+        f.write_text(yaml.safe_dump(sd, sort_keys=False))
+        proc = subprocess.run(
+            [sys.executable, str(render_py), str(f), "--no-image-automation"],
+            capture_output=True, text=True, cwd=str(ROOT),
+        )
+        if proc.returncode != 0:
+            check(False, f"fixture {fname}: the shipped renderer failed: "
+                         f"{proc.stderr.strip()[:400]}")
+            continue
+        rendered = [d for d in yaml.safe_load_all(proc.stdout)
+                    if isinstance(d, dict) and d.get("kind") == "HTTPRoute"]
+        check(len(rendered) == 1, f"fixture {fname}: expected exactly one rendered HTTPRoute")
+        if rendered:
+            got = is_disowned(rendered[0])
+            check(got is want, f"fixture {fname}: {msg} (rendered annotations: "
+                               f"{annotations_of(rendered[0])})")
+
+# =============================================================================================
+# (6) REGISTER RATCHET — every publisher the catalog does not render is declared, and every
+#     declaration still names a real object.
+# =============================================================================================
+catalog_rendered = set(expected_routes)
+seen_register_keys: set[str] = set()
+for rel, d in routes + dns_services:
+    ns, name = ident(d)
+    if (ns, name) in catalog_rendered:
+        continue
+    key = f"{rel}:{ns}/{name}"
+    seen_register_keys.add(key)
+    check(
+        key in KNOWN_NON_CATALOG_PUBLISHERS,
+        f"UNDECLARED publisher {key}: a {d.get('kind')} the catalog does not render, so it has no "
+        f"`exposure` field this check can adjudicate, and it can put a name into the public "
+        f"Cloudflare zone. Add it to KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why "
+        f"it is out of scope — or render it from a ServiceDefinition instead.",
+    )
+stale = sorted(set(KNOWN_NON_CATALOG_PUBLISHERS) - seen_register_keys)
+check(not stale,
+      f"STALE register entries {stale}: KNOWN_NON_CATALOG_PUBLISHERS is shrink-only, and these "
+      f"name no object in the tree any more. Delete them — a register that keeps entries for "
+      f"deleted objects stops being evidence of anything.")
+
+# =============================================================================================
+# (7) LOAD-BEARING — the scope is DECLARED here (so ci/l0.sh Stage 3b's --include-scope union
+#     always selects this test) and NO stage is hand-written into ci/l0.sh (the
+#     whole-repo-invariants-wired meta-test fails the gate if one is).
+# =============================================================================================
+TEST_DIR = ROOT / "conformance/tests/l0-static/internal-services-publish-no-public-dns"
+test_yaml = read(TEST_DIR / "chainsaw-test.yaml")
+check(bool(test_yaml), "this test's own chainsaw-test.yaml is unreadable")
+check(
+    re.search(r'^\s*conformance\.home\.k3s/scope:\s*"?whole-repo"?\s*$', test_yaml, re.M)
+    is not None,
+    "chainsaw-test.yaml must DECLARE `conformance.home.k3s/scope: \"whole-repo\"` on a real "
+    "mapping line — that declaration is the only thing that unions this check into every gate "
+    "run (ci/l0.sh Stage 3b). A scope named in prose is documentation, not a declaration.",
+)
+l0_sh = read(ROOT / "ci/l0.sh")
+check(
+    "internal-services-publish-no-public-dns" not in l0_sh,
+    "ci/l0.sh must NOT name this test: a whole-repo invariant is selected by its DECLARED scope, "
+    "and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.",
+)
+
+# =============================================================================================
+if FAILURES:
+    for f in FAILURES:
+        print(f"check FAIL: {f}", file=sys.stderr)
+    print(f"check FAILED: {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: {CHECKS} assertions passed "
+      f"({len(sds)} ServiceDefinitions; {len(internal_with_routes)} internal + "
+      f"{len(public_with_routes)} public services with routes; {len(routes)} HTTPRoutes and "
+      f"{len(dns_services)} external-dns-annotated Services inventoried; "
+      f"{len(KNOWN_NON_CATALOG_PUBLISHERS)} declared non-catalog publishers)")
+PY

conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/chainsaw-test.yaml

@@ -0,0 +1,95 @@
+# =============================================================================
+# L0 STATIC — an L2-announced LoadBalancer VIP must be servable from ANY announcing node
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE BUG CLASS THIS CLOSES, AND THE OUTAGE THAT PAID FOR IT.
+# Cilium puts a LoadBalancer VIP on the LAN by electing ONE node per Service to answer ARP for
+# it. That election is a leader lease over the nodes the CiliumL2AnnouncementPolicy's
+# nodeSelector matches, and it is NOT backend-aware — Cilium does not restrict the candidates
+# to nodes running a backend, and has no field that would. Combine that with
+# `externalTrafficPolicy: Local`, which lets a node deliver only to a backend on ITSELF, and
+# the VIP becomes a lottery: an elected announcer holding no backend answers ARP for the
+# address and then has nowhere to send the packet, so the client gets ICMP port unreachable.
+# On three nodes with one replica that is a 1-in-3 chance of working, drawn independently per
+# Service — and it never recovers on its own, because the lease holder keeps renewing after the
+# backend moves away.
+#
+# MEASURED on prod 2026-09-14: unifi-l4-tcp/-udp (VM on k3s-prod-1, both leases on k3s-prod-2)
+# left 192.168.2.21:8443/:8080/:11443 dead from the LAN for up to 8 days, and crafty-l4-tcp
+# (pod on k3s-prod-1, lease on k3s-prod-2) left the Minecraft Java port 192.168.2.22:25565
+# dead — while its twin crafty-l4-udp, same pod, happened to draw k3s-prod-1 and worked. One
+# Service broken and its identical twin working is the tell. The mechanism was then confirmed
+# in both directions on the announcing node with `cilium-dbg service list`: under `Local` the
+# frontend is ABSENT from that node's service table; under `Cluster` it appears with an active
+# backend; flipped back, it vanishes again.
+#
+# WHY A REVIEWER CANNOT CATCH THIS. `Local` reads as the safe, source-IP-preserving choice, and
+# both shipped manifests said so in a comment that also claimed it "pins the VIP to the node
+# running the pod" — the precise thing that is false here. The defect is invisible in review
+# and silent at apply time; it only appears as an outage, days later, after an unrelated
+# reschedule moves a pod. apps/wave-0/pihole-dns.yaml hit it first and is the reference
+# implementation; this check keeps the rest of the tree on that side of the line.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
+#       that would announce its VIP,
+# THEN  (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
+#           node is guaranteed to hold a backend — true only for a DaemonSet-backed Service,
+#           and that exemption is computed from the tree rather than hand-maintained;
+#       (2) the premise is still live (some policy actually announces LoadBalancer VIPs), so
+#           the check cannot pass vacuously if L2 announcement is ever removed;
+#       (3) the announcer is still backend-BLIND — asserted structurally, so that if a future
+#           Cilium gains a backend-aware field the exemption logic is revisited deliberately
+#           instead of going stale in silence;
+#   AND (4) the check is NON-VACUOUS: a real checkout, a non-empty LoadBalancer corpus, and a
+#           floor on the assertion count.
+#
+# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library +
+# PyYAML); conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the
+# SAME script.
+#
+# NOT proven — the runtime half, which needs a cluster: that the VIP is actually reachable,
+# that a backend exists, or that an application is listening behind it. That gap is not
+# academic. On 2026-09-14 all four Services above ALSO had no listening backend at all (UniFi
+# OS had never been installed in the VM; no Minecraft server had ever been created in Crafty),
+# so fixing this configuration defect did NOT by itself make those ports answer. A static
+# check cannot see an absent application; only an L2 behavioural probe can. Nor does this
+# prove `Cluster` is costless — Cilium runs in SNAT mode here, so a cross-node connection
+# reaches the backend bearing the ingress node's address, and whether that is acceptable is a
+# per-service judgement recorded in each manifest.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: l2-announced-vips-serve-from-any-node
+  labels:
+    conformance.home.k3s/R-DNS-01: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-DNS-01"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE, and necessarily so. ci/touched-requirements.sh's path map has NO arm for
+    # `apps/*` and none for `platform/crafty/*`, and nothing in it maps to R-DNS-01 at all. So the
+    # change most likely to break this invariant — authoring or editing an L4 LoadBalancer Service
+    # under apps/ or platform/, exactly where both halves of the 2026-09-14 outage lived — would
+    # attribute NO requirement and be gated by nothing. The invariant also spans three trees at
+    # once (the Services under apps/ and platform/, and the announcement policy under
+    # platform-config/), which no single requirement tag can select.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      No LoadBalancer Service whose VIP is put on the LAN by Cilium L2 announcement may use
+      externalTrafficPolicy Local, because the announcing node is elected without regard to
+      backend placement — so an announcer holding no backend blackholes the VIP and never
+      self-heals. The sole exemption is a DaemonSet-backed Service, where every announcing
+      node is guaranteed a local backend.
+spec:
+  steps:
+    - name: assert-no-l2-announced-vip-is-etp-local
+      description: >-
+        Pair every tracked type=LoadBalancer Service with the CiliumL2AnnouncementPolicy that
+        would announce its VIP, and assert none combines that announcement with
+        externalTrafficPolicy Local unless a DaemonSet guarantees a backend on every node.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/l2-announced-vips-serve-from-any-node/check.sh

@@ -0,0 +1,242 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — l2-announced-vips-serve-from-any-node
+# Serves: R-DNS-01 ("Services MUST be reachable at `<service>.home.keithjackson.dev`."
+#         — spec docs/00-vision-and-scope.md, section "DNS & ingress").
+#
+# THE BUG CLASS THIS CLOSES — it is not hypothetical, it cost an 8-day outage.
+# -----------------------------------------------------------------------------
+# A type=LoadBalancer Service on this cluster gets its VIP from Cilium LB-IPAM and is put on
+# the LAN by a CiliumL2AnnouncementPolicy, which elects ONE node per Service to answer ARP for
+# that VIP. The election is a plain leader lease among the nodes the policy's nodeSelector
+# matches. IT IS NOT BACKEND-AWARE: Cilium does not restrict the candidate set to nodes that
+# actually run a backend, and it has no field that would let it.
+#
+# With `externalTrafficPolicy: Local`, a node may only deliver to a backend on ITSELF. Compose
+# the two and the VIP is a lottery: if the elected announcer holds no backend, it answers ARP
+# for the address and then has nowhere to send the packet, so the client gets ICMP port
+# unreachable. On three nodes with one replica that is a 1-in-3 chance of working, drawn
+# independently PER SERVICE — and it does not self-heal, because the lease holder keeps
+# renewing after the backend moves away. Deleting the Lease only redraws the lottery.
+#
+# MEASURED on prod 2026-09-14, which is why this check exists:
+#   * unifi-l4-tcp / unifi-l4-udp — VM on k3s-prod-1, both leases on k3s-prod-2. 192.168.2.21
+#     :8443, :8080 and :11443 dead from the LAN for up to 8 days (the leases were 8 days old).
+#   * crafty-l4-tcp — pod on k3s-prod-1, lease on k3s-prod-2: 192.168.2.22:25565 (Minecraft
+#     Java) dead. crafty-l4-udp — same pod, lease happened to land on k3s-prod-1, so Bedrock
+#     worked. One Service broken and its twin working, from the same pod, is the whole tell.
+#   * The causal mechanism was confirmed both directions on the announcing node with
+#     `cilium-dbg service list`: under `Local` the frontend 192.168.2.22:25565 is ABSENT from
+#     that node's service table entirely; flipped to `Cluster` it appears as
+#     `1 => 10.0.0.11:25565 (active)`. Flipped back, it vanishes again.
+#
+# `Cluster` is the fix: every node can serve the VIP by forwarding to wherever the backend is,
+# so the announcement and the backend no longer have to coincide. apps/wave-0/pihole-dns.yaml
+# is the reference implementation (and carries the cost analysis); this check keeps the whole
+# tree on that side of the line, because the defect is INVISIBLE in review — `Local` reads as
+# the safe, source-IP-preserving choice, and the manifests that shipped it said so explicitly.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every tracked type=LoadBalancer Service is paired with the CiliumL2AnnouncementPolicy
+#       that would announce its VIP,
+# THEN  (1) no announced Service sets `externalTrafficPolicy: Local`, UNLESS every announcing
+#           node is guaranteed to hold a backend — which is true only for a DaemonSet-backed
+#           Service, and that exemption is computed from the tree, not hand-maintained;
+#       (2) the premise is still live: at least one CiliumL2AnnouncementPolicy actually
+#           announces LoadBalancer VIPs (`loadBalancerIPs: true`). If that ever becomes false
+#           this check is vacuous and must be re-derived rather than silently passing;
+#       (3) the announcer's candidate set really is backend-blind — asserted structurally, by
+#           requiring that the policy carries no field pinning it to backend-holding nodes;
+#   AND (4) the check is NON-VACUOUS: a real checkout is present, the LoadBalancer corpus is
+#           non-empty, and a floor on both the corpus and the assertion count keeps a pass
+#           from degenerating into a no-op when a refactor moves these files.
+#
+# WHY THE EXEMPTION IS NARROW, AND WHY IT IS SAFE. `Local` is correct precisely when the
+# announcing node always has a local backend. Under `default-l2`'s nodeSelector
+# (kubernetes.io/os=linux, i.e. every node) that holds only for a DaemonSet. A Deployment with
+# replicas == node count does NOT qualify: nothing pins one replica per node, a rollout or a
+# cordon breaks it, and the failure mode is this same silent outage. So the exemption keys on
+# DaemonSet backing only.
+#
+# WHAT THIS DOES NOT PROVE (honest gaps):
+#   - That the VIP is actually reachable. This is a static check; it proves the configuration
+#     cannot produce the backend-blind-announcer failure, not that a backend exists, that an
+#     application is listening behind it, or that the LAN can route to it. On 2026-09-14 all
+#     four Services above ALSO had no listening backend (UniFi OS was never installed in the
+#     VM; no Minecraft server had been created in Crafty), and no static check can see that.
+#     The end-to-end claim belongs to an L2 behavioural test against a live cluster.
+#   - That `Cluster` is COSTLESS. It is not: Cilium runs in SNAT mode here, so a cross-node
+#     connection reaches the backend bearing the ingress node's address. Whether that cost is
+#     acceptable is a per-service judgement recorded in each manifest, not something this
+#     check can decide.
+#   - Anything about Services created in-cluster rather than authored here (the Gateway's own
+#     LoadBalancer Service is generated by Cilium from the Gateway resource and is not in the
+#     tree, so it is outside this static scan).
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+import re, subprocess, sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def tracked() -> list[str]:
+    out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
+                         capture_output=True, text=True, check=True).stdout
+    return [p for p in out.split("\0") if p]
+
+
+def docs_of(rel: str):
+    try:
+        return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
+                if isinstance(d, dict)]
+    except Exception:
+        return []
+
+
+FILES = tracked()
+YAML_FILES = [f for f in FILES if re.search(r"\.(ya?ml)(\.example)?$", f)]
+check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
+
+# --------------------------------------------------------------------------- the announcers
+# A CiliumL2AnnouncementPolicy with loadBalancerIPs: true puts LoadBalancer VIPs on the LAN.
+# `serviceSelector` absent means EVERY LoadBalancer Service is announced (Cilium's default).
+announcers: list[dict] = []
+for rel in YAML_FILES:
+    for d in docs_of(rel):
+        if d.get("kind") == "CiliumL2AnnouncementPolicy":
+            spec = d.get("spec") or {}
+            announcers.append({
+                "file": rel,
+                "name": (d.get("metadata") or {}).get("name"),
+                "lb_ips": bool(spec.get("loadBalancerIPs")),
+                "service_selector": spec.get("serviceSelector"),
+                "spec": spec,
+            })
+
+check(len(announcers) > 0,
+      "no CiliumL2AnnouncementPolicy is tracked — this check's premise (VIPs are put on the "
+      "LAN by L2 announcement) no longer holds and the invariant must be re-derived (R-DNS-01)")
+
+lb_announcers = [a for a in announcers if a["lb_ips"]]
+# (2) the premise is live.
+check(len(lb_announcers) > 0,
+      "no CiliumL2AnnouncementPolicy sets loadBalancerIPs: true — nothing announces LoadBalancer "
+      "VIPs, so this check would pass vacuously; re-derive it (R-DNS-01)")
+
+# (3) the candidate set is backend-blind. Cilium has no field that restricts L2 announcement to
+#     nodes holding a backend; assert none has appeared, so that if a future Cilium adds one the
+#     exemption logic here is revisited deliberately instead of silently going stale.
+BACKEND_AWARE_FIELDS = {"backendSelector", "endpointSelector", "localBackendsOnly"}
+for a in lb_announcers:
+    present = BACKEND_AWARE_FIELDS & set(a["spec"].keys())
+    check(not present,
+          f"{a['file']}: CiliumL2AnnouncementPolicy/{a['name']} carries {sorted(present)}, which "
+          "may make announcement backend-aware — the `Local` exemption below must be re-derived "
+          "before this check can keep passing (R-DNS-01)")
+
+# --------------------------------------------------------------------------- DaemonSet backing
+# Service.spec.selector is matched against DaemonSet pod-template labels in the same namespace.
+# A DaemonSet puts a backend on every node, so `Local` cannot strand an announcing node.
+daemonset_labels: list[tuple[str, dict]] = []   # (namespace, template labels)
+for rel in YAML_FILES:
+    for d in docs_of(rel):
+        if d.get("kind") == "DaemonSet":
+            md = d.get("metadata") or {}
+            tmpl = (((d.get("spec") or {}).get("template") or {}).get("metadata") or {})
+            daemonset_labels.append((md.get("namespace"), tmpl.get("labels") or {}))
+
+
+def daemonset_backed(ns, selector) -> bool:
+    if not selector:
+        return False
+    for ds_ns, labels in daemonset_labels:
+        if ds_ns == ns and all(labels.get(k) == v for k, v in selector.items()):
+            return True
+    return False
+
+
+# --------------------------------------------------------------------------- (1) the invariant
+lb_services: list[dict] = []
+for rel in YAML_FILES:
+    for d in docs_of(rel):
+        if d.get("kind") != "Service":
+            continue
+        spec = d.get("spec") or {}
+        if spec.get("type") != "LoadBalancer":
+            continue
+        md = d.get("metadata") or {}
+        lb_services.append({
+            "file": rel,
+            "name": md.get("name"),
+            "ns": md.get("namespace"),
+            "etp": spec.get("externalTrafficPolicy"),
+            "selector": spec.get("selector") or {},
+            "pinned": (md.get("annotations") or {}).get("lbipam.cilium.io/ips"),
+        })
+
+# Non-vacuity floor: these Services are the reason the check exists; if the corpus empties out,
+# something moved and the check must be re-pointed rather than passing on nothing.
+check(len(lb_services) >= 3,
+      f"only {len(lb_services)} tracked type=LoadBalancer Service(s) found — the corpus this "
+      "check quantifies over has moved or emptied; re-point it (R-DNS-01)")
+
+exempt: list[str] = []
+for s in lb_services:
+    where = f"{s['file']}: Service/{s['ns']}/{s['name']}"
+    if s["etp"] != "Local":
+        CHECKS += 1                      # counted: every Service is genuinely inspected
+        continue
+    if daemonset_backed(s["ns"], s["selector"]):
+        exempt.append(f"{s['ns']}/{s['name']}")
+        CHECKS += 1
+        continue
+    pin = f" pinned at {s['pinned']}" if s["pinned"] else ""
+    check(False,
+          f"{where}{pin} combines type=LoadBalancer with externalTrafficPolicy: Local while "
+          f"CiliumL2AnnouncementPolicy/{lb_announcers[0]['name'] if lb_announcers else '?'} "
+          "announces its VIP from nodes chosen WITHOUT regard to backend placement. When the "
+          "elected announcer holds no backend the VIP answers ICMP port unreachable and does "
+          "not self-heal — this is the 8-day unifi/crafty outage of 2026-09-14. Use "
+          "externalTrafficPolicy: Cluster (see apps/wave-0/pihole-dns.yaml for the pattern and "
+          "the client-IP cost it trades away), or back the Service with a DaemonSet so every "
+          "announcing node holds a backend (R-DNS-01)")
+
+# --------------------------------------------------------------------------- (4) non-vacuity
+check(CHECKS >= 8,
+      f"only {CHECKS} assertions ran — too few for this invariant to be meaningful (R-DNS-01)")
+
+# --------------------------------------------------------------------------------- verdict
+if FAILURES:
+    print(f"CHECK FAIL (l2-announced-vips-serve-from-any-node): {len(FAILURES)} of {CHECKS} "
+          "assertions failed", file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+note = f", {len(exempt)} DaemonSet-backed exemption(s): {sorted(exempt)}" if exempt else ""
+print(f"check OK: l2-announced-vips-serve-from-any-node — {CHECKS} assertions over "
+      f"{len(lb_services)} LoadBalancer Service(s) and {len(lb_announcers)} announcing "
+      f"policy(ies){note}")
+PY

conformance/tests/l0-static/placeholder-guard-coupling/check.sh

@@ -238,7 +238,7 @@ def bail(msg):
 
 DEF_RE = re.compile(r"^([a-z0-9_]+_is_placeholder)\(\)\s*\{")
 SELF_DIR = ROOT / "conformance" / "tests" / "l0-static" / "placeholder-guard-coupling"
-SKIP_DIRS = {".git", "_artifacts", "node_modules", ".venv"}
+SKIP_DIRS = {".git", ".claude", "_artifacts", "node_modules", ".venv"}
 
 # ------------------------------------------------------------------ discover the guard FILE
 # Hardcoding a path is what let a file that never landed disable this whole check.

conformance/tests/l0-static/servicemonitor-ports-resolve/check.sh

@@ -154,8 +154,13 @@ services: list[tuple] = []
 monitors: list[tuple] = []
 unparseable: list[str] = []
 
+# NESTED WORKTREES: skip dot-DIRECTORIES (.git, .claude) while still matching dot-FILES such
+# as .sops.yaml. .claude/worktrees/agent-*/ holds full repo copies for isolated agent runs, so
+# a bare rglob from ROOT audited up to 18 duplicates of every file — which both invents FAILs
+# and, worse, can satisfy an "something declares X" check from a copy rather than real source.
+# Same fix as egress-paths-selected-by-rules. .claude/ is gitignored (.gitignore:43).
 for p in sorted(ROOT.rglob("*.yaml")):
-    if ".git" in p.parts or is_test_fixture(p) or ".sops." in p.name:
+    if any(part.startswith(".") for part in p.relative_to(ROOT).parent.parts) or is_test_fixture(p) or ".sops." in p.name:
         continue
     txt = read(p)
     if not txt:

conformance/tests/l0-static/whole-repo-invariants-wired/check.sh

@@ -39,20 +39,20 @@ if [ "$FAILED" -ne 0 ]; then
 fi
 
 grep -q 'CONF_SCOPE_ANNOTATION' "$COMMON_SH" \
-  && grep -q 'conf_dirs_for_scope' "$COMMON_SH"
-chk $? "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
+  && grep -q 'conf_dirs_for_scope' "$COMMON_SH" && arc=0 || arc=$?
+chk $arc "conformance/lib/common.sh resolves the scope annotation (conf_dirs_for_scope)" \
        "conformance/lib/common.sh no longer resolves the scope annotation — no test can be
        selected BY declaring a scope, so every whole-repo invariant is back to being
        selectable away by the requirement filter"
 
-grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH"
-chk $? "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
+grep -q -- '--include-scope' "$RUN_SH" && grep -q 'conf_dirs_for_scopes' "$RUN_SH" && arc=0 || arc=$?
+chk $arc "conformance/run.sh --include-scope unions scope-declared tests into select_dirs" \
        "conformance/run.sh does not union scope-declared tests (--include-scope /
        conf_dirs_for_scopes missing) — the annotation would be inert decoration"
 
 grep -Eq -- '--include-scope[[:space:]]+"?\$\{?WHOLE_REPO_SCOPE' "$L0_SH" \
-  || grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH"
-chk $? "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
+  || grep -Eq -- "--include-scope[[:space:]]+['\"]?${SCOPE}" "$L0_SH" && arc=0 || arc=$?
+chk $arc "ci/l0.sh passes --include-scope ${SCOPE} to the conformance runner" \
        "ci/l0.sh does not pass --include-scope ${SCOPE}: the L0 merge-gate stage is back to a
        purely requirement-scoped run, and every whole-repo invariant is filtered out of
        exactly the change that breaks it"
@@ -63,16 +63,16 @@ while IFS= read -r d; do [ -n "$d" ] && declared_dirs+=("$d"); done < <(
   CONF_ROOT="$CONF" bash -c 'source "${CONF_ROOT}/lib/common.sh"; conf_dirs_for_scope "$1"' _ "$SCOPE"
 )
 MIN_DECLARED="${MIN_DECLARED:-5}"
-[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ]
-chk $? "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
+[ "${#declared_dirs[@]}" -ge "$MIN_DECLARED" ] && arc=0 || arc=$?
+chk $arc "${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED})" \
        "only ${#declared_dirs[@]} test(s) declare scope='${SCOPE}' (floor ${MIN_DECLARED}) — either the
        annotation was stripped from the whole-repo checks, or the resolver stopped matching
        it. Every assertion below would then pass while checking nothing"
 
 for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
   rel="${d#"$ROOT"/}"
-  [ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ]
-  chk $? "${rel}: declares the scope AND carries an executable offline check" \
+  [ -f "$d/check.sh" ] || [ -f "$d/render-check.sh" ] && arc=0 || arc=$?
+  chk $arc "${rel}: declares the scope AND carries an executable offline check" \
          "${rel} declares scope='${SCOPE}' but carries neither check.sh nor render-check.sh — an
          unconditional run of a test with nothing to run is a guaranteed SKIP, i.e. a
          permanent not-green, not a guard"
@@ -91,25 +91,58 @@ BOGUS_REQ="r-nosuchreq-99"
 sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
 sel_with="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" --print-selection 2>/dev/null || true)"
 
-[ -z "$sel_without" ]
-chk $? "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
+[ -z "$sel_without" ] && arc=0 || arc=$?
+chk $arc "a requirement filter matching no test resolves to an EMPTY L0 selection (control)" \
        "the control case is broken: --requirements ${BOGUS_REQ} still selected tests
        ($(printf '%s' "$sel_without" | tr '\n' ' ')), so the assertion below proves nothing
        about --include-scope"
 
+# PARTITIONED BY LAYER, because the union is layer-BOUNDED by design: `--layer l0` must select the
+# scope-declaring L0 tests and NOT the L1 ones (asserted separately below). This loop compared EVERY
+# scope-declaring dir against an `--layer l0` selection, which silently assumed only L0 tests would
+# ever declare the scope — and ci/l1.sh was written from the opposite premise, that a whole-repo
+# invariant could appear "at any layer". The moment the three L1 rendering tests declared it
+# (2026-09-18) this check failed, blaming the union for correctly excluding another layer.
 missing=""
 for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
+  case "$d" in */tests/l0-static/*) ;; *) continue ;; esac
   printf '%s\n' "$sel_with" | grep -qxF "$d" || missing="${missing} ${d#"$ROOT"/}"
 done
-[ -z "$missing" ]
-chk $? "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
+[ -z "$missing" ] && arc=0 || arc=$?
+chk $arc "every scope-declared L0 test is selected DESPITE a requirement filter matching none" \
        "the scope union does not actually select:${missing}. This is the blind spot itself: a
        change that attributes no requirement would not run these checks"
 
+# The SAME property one layer up. This is what makes the L1 declaration load-bearing rather than
+# decorative: ci/l1.sh has always passed --include-scope, but until an L1 test declared the scope
+# nothing proved the union reached L1 at all. The 2026-09-04 external-dns merge is the case in
+# point — it attributed no rendering requirement, selected ZERO L1 tests, gated GREEN, and left all
+# three rendering goldens stale.
+sel_l1="$(bash "$RUN_SH" --layer l1 --requirements "$BOGUS_REQ" --include-scope "$SCOPE" \
+          --print-selection 2>/dev/null || true)"
+missing_l1=""
+declared_l1=0
+for d in ${declared_dirs[@]+"${declared_dirs[@]}"}; do
+  case "$d" in */tests/l1-rendering/*) ;; *) continue ;; esac
+  declared_l1=$((declared_l1+1))
+  printf '%s\n' "$sel_l1" | grep -qxF "$d" || missing_l1="${missing_l1} ${d#"$ROOT"/}"
+done
+[ "$declared_l1" -gt 0 ] && [ -z "$missing_l1" ] && arc=0 || arc=$?
+chk $arc "every scope-declared L1 test is selected under --layer l1 DESPITE a requirement filter matching none" \
+       "scope-declared L1 tests: ${declared_l1}; not selected by --layer l1 --include-scope ${SCOPE}:
+       ${missing_l1:-<none>}. If the count is 0 the rendering goldens are back to being checked only
+       when a touched path happens to attribute R-PLAT-02/R-NET-01/R-NET-05/06/07"
+
+# And the L1 union must not leak the other way either.
+strays_l1="$(printf '%s\n' "$sel_l1" | grep -v '/tests/l1-rendering/' || true)"
+[ -z "$strays_l1" ] && arc=0 || arc=$?
+chk $arc "the scope union stays inside --layer l1 (no cross-layer leakage)" \
+       "--layer l1 --include-scope ${SCOPE} selected non-L1 test(s): $(printf '%s' "$strays_l1" | tr '\n' ' ')"
+
 # The union must not smuggle in another layer's tests when --layer bounds the run.
 strays="$(printf '%s\n' "$sel_with" | grep -v '/tests/l0-static/' || true)"
-[ -z "$strays" ]
-chk $? "the scope union stays inside --layer l0 (no cross-layer leakage)" \
+[ -z "$strays" ] && arc=0 || arc=$?
+chk $arc "the scope union stays inside --layer l0 (no cross-layer leakage)" \
        "--layer l0 --include-scope ${SCOPE} selected non-L0 test(s): $(printf '%s' "$strays" | tr '\n' ' ')"
 
 # ---- (3) a SKIP of a scope-declared check is UNWIRED, NOT GREEN -------------------------
@@ -143,23 +176,23 @@ run_offline() {  # run_offline <dir...> -> echoes rc
 rc_declared="$(run_offline "${WORK}/l0-static/synth-declared")"
 rc_undeclared="$(run_offline "${WORK}/l0-static/synth-undeclared")"
 
-[ "$rc_declared" = 3 ]
-chk $? "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
+[ "$rc_declared" = 3 ] && arc=0 || arc=$?
+chk $arc "a SKIPPED check that DECLARES scope='${SCOPE}' makes the runner exit 3 (UNWIRED, not green)" \
        "a SKIPPED scope-declared check made the offline runner exit ${rc_declared}, not 3. A
        whole-repo invariant reporting 'nothing to audit' means the tree it audits has
        VANISHED; the deleted ci/l0.sh stages called that 'not a pass' and set the gate's
        rc=3. Exit 0 here turns that back into a silent green"
 
-[ "$rc_undeclared" = 0 ]
-chk $? "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
+[ "$rc_undeclared" = 0 ] && arc=0 || arc=$?
+chk $arc "a SKIPPED check with NO scope declaration still exits 0 (unchanged for scaffolds)" \
        "a SKIPPED undeclared check made the runner exit ${rc_undeclared}, not 0 — the strict
        rule leaked onto ordinary scaffolds, which would make every unfinished test block"
 
 # ...and ci/l0.sh must map that rc 3 to the GATE's rc 3, never to a pass.
 l0_static_stage="$(sed -n '/Stage 3b: L0 STATIC/,/^case "\$rc" in/p' "$L0_SH")"
 printf '%s\n' "$l0_static_stage" | grep -Eq '^[[:space:]]*3\)' \
-  && printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3'
-chk $? "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
+  && printf '%s\n' "$l0_static_stage" | grep -Eq 'rc=3' && arc=0 || arc=$?
+chk $arc "ci/l0.sh maps the runner's rc 3 to the gate's own rc 3 (SKIPPED, not a pass)" \
        "ci/l0.sh's L0-static stage has no rc-3 arm setting the gate's rc=3, so a scope-declared
        check that SKIPPED would be reported as a PASS — the exact false green the deleted
        stages were written to prevent"
@@ -193,8 +226,8 @@ done
 
 # ---- (5) the gate still BLOCKS on an l0 failure -----------------------------------------
 # Carried over from the three deleted stages' own checks — asserted once, here.
-grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH"
-chk $? "ci/gate.sh sets the merge block when the l0 stage fails" \
+grep -Eq '^for s in .*\bl0\b.*; do' "$GATE_SH" && grep -Eq 'fail\).*BLOCK=1' "$GATE_SH" && arc=0 || arc=$?
+chk $arc "ci/gate.sh sets the merge block when the l0 stage fails" \
        "ci/gate.sh does not set the merge block on an l0 stage failure — every assertion above
        would hold while a red L0 stage merged anyway"
 

conformance/tests/l1-rendering/container-workload-render/chainsaw-test.yaml

@@ -23,6 +23,17 @@ metadata:
     conformance.home.k3s/requirements: "R-PLAT-02"
     conformance.home.k3s/layer: "L1"
     conformance.home.k3s/safety-critical: "false"
+    # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
+    # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
+    # rendered output breaks these goldens regardless of which requirement the touched path maps to.
+    # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
+    # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
+    # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
+    # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
+    # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
+    # GREEN, and left every golden stale; the staleness surfaced only on a later full run.
+    # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
+    conformance.home.k3s/scope: "whole-repo"
     conformance.home.k3s/intent: >-
       A kind: container ServiceDefinition renders a Deployment + Service (the PaaS path).
 spec:

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -196,6 +196,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: platform-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

conformance/tests/l1-rendering/default-deny-baseline-render/chainsaw-test.yaml

@@ -27,6 +27,17 @@ metadata:
     conformance.home.k3s/requirements: "R-NET-01"
     conformance.home.k3s/layer: "L1"
     conformance.home.k3s/safety-critical: "false"
+    # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
+    # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
+    # rendered output breaks these goldens regardless of which requirement the touched path maps to.
+    # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
+    # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
+    # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
+    # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
+    # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
+    # GREEN, and left every golden stale; the staleness surfaced only on a later full run.
+    # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
+    conformance.home.k3s/scope: "whole-repo"
     conformance.home.k3s/intent: >-
       Rendering any service's namespace yields a default-deny baseline policy.
 spec:

conformance/tests/l1-rendering/egress-class-render/chainsaw-test.yaml

@@ -32,6 +32,17 @@ metadata:
     conformance.home.k3s/requirements: "R-NET-05, R-NET-06, R-NET-07"
     conformance.home.k3s/layer: "L1"
     conformance.home.k3s/safety-critical: "false"
+    # WHY SCOPE, NOT REQUIREMENTS ALONE: every L1 test exercises ONE shared code path —
+    # catalog/render.py against schemas/servicedefinition.schema.json — so anything that changes
+    # rendered output breaks these goldens regardless of which requirement the touched path maps to.
+    # Requirement-scoped selection cannot see that coupling. Concretely (2026-09-04 external-dns
+    # merge): `platform/externaldns/helmrelease.yaml` attributes R-ENV-02, R-OBS-01/02/04,
+    # R-STATE-03 and R-TEST-01/02/03/04 via ci/touched-requirements.sh — NOT R-PLAT-02, R-NET-01 or
+    # R-NET-05/06/07 — yet it added `external-dns.alpha.kubernetes.io/controller: none` to the
+    # rendered output of ALL THREE L1 goldens. A requirement-scoped gate selected ZERO L1 tests, went
+    # GREEN, and left every golden stale; the staleness surfaced only on a later full run.
+    # ci/l1.sh has passed --include-scope since it was written, waiting for this annotation.
+    conformance.home.k3s/scope: "whole-repo"
     conformance.home.k3s/intent: >-
       egress: vpn renders the egress-gateway + kill-switch path; internet does not.
 spec:

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -223,6 +223,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -223,6 +223,7 @@ metadata:
   annotations:
     homelab.keithjackson.dev/auth-mode: forward-auth
     homelab.keithjackson.dev/auth-groups: media-admins
+    external-dns.alpha.kubernetes.io/controller: none
 spec:
   parentRefs:
   - name: shared

platform/coredns/coredns-custom.yaml

@@ -12,6 +12,25 @@
 # works. External/LAN clients keep their own answers (PiHole split-horizon, docs/09) — this ConfigMap
 # only affects CLUSTER DNS (CoreDNS), not LAN DNS.
 #
+# THE LAN HALF IS NOW REAL, AND WAS NOT UNTIL 2026-09-14. This header deferred LAN clients to
+# "PiHole split-horizon" from the day it was written, but PiHole served no DNS at all — its Service
+# was ClusterIP with ports 80 and 4180 only, no port 53 and no LoadBalancer — so the deferral
+# pointed at nothing and LAN clients depended on hand-maintained per-client dnsmasq files (one went
+# stale after the dev decommission and cost the operator sonarr.${BASE_DOMAIN}). PiHole now answers
+# the zone on udp/53 + tcp/53 at its own pinned VIP: exposure in apps/wave-0/pihole-dns.yaml, the
+# answers themselves as FTLCONF env in catalog/pihole.yaml. The two halves are deliberately
+# INDEPENDENT mechanisms over the same intent — this file for pod clients, dnsmasq `address=` lines
+# for LAN clients — and both resolve the whole zone to ${GATEWAY_VIP} by wildcard so a newly
+# onboarded service needs no DNS change on either side.
+#
+# ONE KNOWN DIVERGENCE, recorded rather than papered over: PiHole carries specific overrides for the
+# L4 services that live on their OWN VIPs (unifi -> ${UNIFI_L4_VIP}, crafty-mc -> ${CRAFTY_L4_VIP}),
+# because a LAN client asking for unifi.${BASE_DOMAIN} must reach the UniFi controller's own address.
+# The `template IN A` block below has no such overrides and answers those two names with
+# ${GATEWAY_VIP}, which the Gateway cannot route. No in-cluster workload resolves either name today,
+# so nothing is broken by it; fixing it means per-name `template` blocks or hosts entries here and
+# is a separate change from the LAN work.
+#
 # MECHANISM (k3s-specific, F-DNS / docs/09): k3s CoreDNS ships a Corefile with
 # `import /etc/coredns/custom/*.server`, and mounts the `coredns-custom` ConfigMap (kube-system) at
 # /etc/coredns/custom. Each `*.server` key becomes a top-level server block imported into the

platform/crafty/README.md

@@ -117,3 +117,29 @@ Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subjec
 3. **Server-create + game reach:** in the Crafty UI create a Java server bound to `25565`; start it;
    connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or `192.168.1.55:25565`)
    from the LAN and confirm the world loads. (Bedrock: a server on `19132/udp` reached at the VIP.)
+
+## Runtime state: the servers themselves are NOT in git
+
+As of 2026-09-14 one Java server exists — **"homelab"**, Paper 26.2, TCP 25565, 2 GiB heap, autostart
+on, UUID `13c7b85d-00bb-4a58-8c86-2433ca47891c`. It was created through Crafty's v2 API, and **Mojang's
+EULA (<https://aka.ms/MinecraftEULA>) was accepted on the operator's behalf** to let it start; see the
+DECISIONS-LOG entry of that date.
+
+Everything in this directory describes only the *capacity* to run Minecraft servers. The servers are
+application state on the `crafty-data` Longhorn PVC, in two places, and **Flux cannot recreate either**:
+
+- `/crafty/app/config/db/` (`config` subPath) — Crafty's SQLite row: name, port, jar, launch command,
+  heap sizes, `auto_start` / `crash_detection`.
+- `/crafty/servers/<uuid>/` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and
+  **the generated world**. The jar is re-downloadable; the world is not.
+
+So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565.
+Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH
+`auto_start` to true — see the next note. The world starts over. **Treat this PVC as backup-relevant.**
+
+### Trap: the create API silently ignores `autostart`
+
+`POST /api/v2/servers` accepts and validates `autostart` and `crashdetection`, but never stores them —
+the persisted columns are `auto_start` and `crash_detection`. After a create that asked for `true`,
+both read back `false`. Set them afterwards with `PATCH /api/v2/servers/<uuid>` and **read them back to
+confirm**, or the server will not come up after a pod restart.

platform/crafty/deployment.yaml

@@ -79,8 +79,19 @@ spec:
           resources:
             requests:
               cpu: 250m
-              memory: 1Gi
+              # Raised 1Gi -> 3Gi when the first Minecraft server ("homelab", Paper 26.2) was created:
+              # that JVM runs with -Xms2000M -Xmx2000M, so the container's steady-state floor is now the
+              # panel (~100Mi) PLUS a permanently-committed 2 GiB heap plus JVM overhead. Measured
+              # cgroup memory.current with the server idle-but-running: 2.86Gi. A 1Gi request understated
+              # that by ~3x, which both misleads the scheduler and makes this Burstable pod an early
+              # eviction candidate under node memory pressure (kubelet ranks by usage-over-request).
+              memory: 3Gi
             # Minecraft JVMs are memory-hungry; cap generously for a starter set of small servers.
+            # Headroom check for the 4Gi cap: 2 GiB heap + ~0.5 GiB JVM non-heap (metaspace, code cache,
+            # GC structures, thread stacks, mmap'd region files) + ~0.1 GiB panel = ~2.9Gi observed,
+            # leaving ~1.2Gi. A 3 GiB heap would have landed near 4Gi and risked an OOMKill, which is
+            # why the server was sized at a 2 GiB heap rather than larger. Raise BOTH this limit and the
+            # server's own memory in Crafty together if a bigger heap is ever wanted.
             limits:
               memory: 4Gi
           securityContext:

platform/crafty/service-l4.yaml

@@ -28,9 +28,67 @@
 # documentation) and to the game-port ingress CiliumNetworkPolicy (networkpolicy.yaml). K8s Services
 # enumerate ports individually (no ranges), so add one `{ name, port, targetPort, protocol }` line each.
 #
-# externalTrafficPolicy: Local preserves the LAN client source IP and pins the VIP to the node running
-# the crafty pod (also what L2 announcement does). external-dns (source: service) publishes the A
-# record from the -tcp Service.
+# external-dns (source: service) publishes the A record from the -tcp Service.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
+# -------------------------------------------------------------------------------------------------
+# This file used to say `Local`, justified as "preserves the LAN client source IP and pins the VIP to
+# the node running the crafty pod (also what L2 announcement does)". THE SECOND HALF IS FALSE, and it
+# is what took the Minecraft port off the LAN.
+#
+# L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder per
+# Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which matches ALL
+# linux nodes — and it does NOT filter that set by which nodes hold a backend. MEASURED on prod
+# 2026-09-14: the crafty pod ran on k3s-prod-1, `cilium-l2announce-crafty-crafty-l4-udp` was held by
+# k3s-prod-1 (so Bedrock/UDP happened to work) but `-tcp` was held by k3s-prod-2 — and with `Local` an
+# announcing node holding no backend has nowhere to send the packet and answers ICMP port unreachable.
+# 192.168.2.22:25565, the Java port, was dead from the LAN for up to 8 days. That the two protocols
+# landed on different nodes, one working and one not, is the whole tell: the election is per-Service
+# and random, not backend-aware. It also does NOT self-heal — the holder keeps RENEWING after the pod
+# moves, so only deleting the Lease forces a (still random) re-election. See
+# apps/wave-0/pihole-dns.yaml, which hit this first and is the reference implementation.
+#
+# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the pod is, which is what
+# makes this survive a reschedule and a rebuild.
+#
+# NO CHANGE IS NEEDED IN networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED. Under
+# `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's reserved
+# `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
+# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
+# crafty-allow-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node` (along
+# with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the crafty endpoint's datapath
+# policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
+# `reserved:remote-node` was already an Allow on 25565-25567/TCP and 19132-19134/UDP. Adding
+# `remote-node` to that rule would be a no-op, so it is deliberately NOT added here — do not "fix" it.
+#
+# THE CLIENT-IP COST, STATED PLAINLY AND ACCEPTED. Cilium runs in SNAT mode here (no
+# `loadBalancer.mode: dsr` in platform/cilium/helmrelease.yaml), so unlike UniFi — a KubeVirt VM whose
+# `masquerade` binding already destroyed the client address below the Service — crafty is a plain pod
+# and DOES lose a real client IP it was previously receiving. Every player will appear to the server
+# subprocesses as the ingress node's address. What that actually costs here:
+#   * PLAYER IDENTITY IS UNAFFECTED. Minecraft Java authenticates against Mojang/Microsoft and Bedrock
+#     against Xbox Live; a player is a UUID, not an address. Vanilla `/ban` and the whitelist are
+#     UUID-keyed (banned-players.json, whitelist.json) and keep working exactly as before.
+#   * `/ban-ip` (banned-ips.json) DEGRADES, and would degrade dangerously: with every player behind one
+#     apparent address, an IP ban would hit EVERY player at once. Use `/ban` (by player), not
+#     `/ban-ip`, on this cluster. No IP ban is configured today.
+#   * Per-IP rate limiting / connection throttling degrades the same way — the whole LAN counts as one
+#     client. Nothing in platform/crafty configures any, and Crafty's own web UI is a separate,
+#     Keycloak-gated path that does not ride these Services.
+# This is a real cost and it is worth paying: the alternative on offer is a Java port that is dead
+# outright, and the only fix that keeps BOTH the client IP and a working VIP is `loadBalancer.mode:
+# dsr` on the Cilium HelmRelease — a platform-wide change with an Ansible [parity] contract that rolls
+# the CNI across all three nodes, which belongs in its own change, not in an outage fix.
+#
+# NO loadBalancerSourceRanges HERE, DELIBERATELY — this is where this file departs from
+# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving up a
+# restriction it had and genuinely needed to be LAN-only. These Services never had a source
+# restriction at all (crafty-allow-ingress admits `world` on the game ports, and the block above is
+# explicit that player auth — not network position — is the control). `Cluster` therefore removes no
+# protection and there is nothing to compensate for. Pinning the game ports to ${LAN_CIDR} during an
+# outage fix would be a NEW restriction that would cut off any player reaching these ports from off
+# the LAN, trading this outage for a subtler one.
 apiVersion: v1
 kind: Service
 metadata:
@@ -45,7 +103,7 @@ metadata:
     external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     app.kubernetes.io/name: crafty
   ports:
@@ -65,7 +123,7 @@ metadata:
     lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
 spec:
   type: LoadBalancer
-  externalTrafficPolicy: Local
+  externalTrafficPolicy: Cluster
   selector:
     app.kubernetes.io/name: crafty
   ports:

platform/externaldns/helmrelease.yaml

@@ -26,10 +26,94 @@ spec:
     provider:
       name: cloudflare
     # Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent.
+    #
+    # BOTH SOURCES ARE UNFILTERED HERE, AND THAT IS DELIBERATE — see the WHICH RESOURCES block
+    # below. Scoping is done PER RESOURCE by the renderer, not cluster-wide by this list.
     sources:
       - gateway-httproute
       - service
     # ---------------------------------------------------------------------------------------------
+    # WHICH RESOURCES MAY PUBLISH — `internal` services no longer can (R-DNS-03, 2026-09-14)
+    #
+    # THE DEFECT. `sources` above admits EVERY HTTPRoute on the shared Gateway, and each one's
+    # address is the Gateway VIP. docs/09 R-DNS-03 says `internal` services are unreachable from the
+    # public internet and that Pi-hole serves the internal split-horizon answers; the renderer
+    # contract said an `internal` service renders "no public DNS record". Neither sentence was
+    # enforced by anything, so six `exposure: internal` services (jackett, pihole, qbittorrent,
+    # radarr, semaphore, sonarr) published `<name>.home.keithjackson.dev` into this PUBLIC
+    # authoritative zone answering 192.168.2.20. That exposed the internal subnet, the Gateway VIP
+    # and a complete service inventory to anyone who could guess a label, and made every one of
+    # those names a DNS-rebind primitive — which is exactly why the SOHO router at 192.168.2.1
+    # strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own
+    # services. Pre-state captured at the AUTHORITY and committed:
+    # .standup-dns/dns-20260914-pre-externaldns-scope.txt.
+    #
+    # THE MECHANISM — per-resource exclusion, emitted by catalog/render.py from `exposure`.
+    # An `internal` service's rendered HTTPRoute carries
+    #     external-dns.alpha.kubernetes.io/controller: none
+    # ExternalDNS v0.21.0 (the DEPLOYED app version; read off the live prod Deployment, image
+    # registry.k8s.io/external-dns/external-dns:v0.21.0) skips any source object whose `controller`
+    # annotation is present and != its own `dns-controller`:
+    #     source/annotations/processors.go  IsControllerMismatch()
+    #     source/gateway.go  Endpoints():  if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
+    #     source/service.go  Endpoints():  the identical check
+    # The gateway check runs BEFORE any hostname is resolved, so such a route cannot become an
+    # endpoint for any name it declares. PROVEN LIVE on prod before this landed, with a throwaway
+    # hostname and no existing record at risk: an annotated route published nothing at all, and
+    # ADDING the annotation to a route that already had a record produced
+    #   action=DELETE record=<name> type=A    and   action=DELETE record=a-<name> type=TXT
+    # one ~60s reconcile later, and moved nothing else in the zone.
+    #
+    # WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. The chart's `labelFilter`
+    # (--label-filter) is the stronger default posture — publish only what is labelled — and
+    # v0.21.0's own --help confirms BOTH configured sources support it ("currently supported by
+    # source types crd, gateway-httproute, ... service and ambassador-host"). It is NOT set here
+    # because its blast radius is not this defect's. Nine publishing objects in this repo are
+    # hand-authored platform/apps manifests with no `exposure` field for the renderer to read, and
+    # seven of them hold live records: auth (oauth2-proxy), crafty, crafty-mc, grafana, headlamp,
+    # the apex (homepage), unifi. Setting an opt-in filter DELETES all seven on the next reconcile,
+    # and whether each should be publicly resolvable at all is an OPERATOR decision per name, not a
+    # renderer one. They are enumerated in the shrink-only register in
+    # conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh, so the set
+    # cannot grow silently — adjudicate them there, then this filter becomes the right hardening.
+    #
+    # DELIBERATELY UNTOUCHED: `policy: sync`, `txtOwnerId`, `domainFilters`. Nothing above widens
+    # what this instance can see or own; per-resource exclusion only ever REMOVES an object from
+    # this instance's view, so it cannot make prod capable of deleting a record it does not own.
+    # ---------------------------------------------------------------------------------------------
+    # THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
+    #
+    # The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
+    # that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
+    # actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
+    # this change, for two independent reasons:
+    #
+    #   1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
+    #      (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
+    #      points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
+    #      certificate. Whether prod's public services belong behind it, behind a new forward, or
+    #      are not meant to be public at all is an operator decision. Breaking five services' DNS
+    #      on a guess is worse than leaving a known-wrong-but-stable state.
+    #   2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
+    #      real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
+    #      GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
+    #      gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
+    #      no per-service target override while every public service shares the `shared` Gateway.
+    #      Annotating the Gateway would move EVERY gateway-sourced record, which after this change
+    #      still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
+    #      routable address for admin UIs. That is strictly worse than the status quo.
+    #
+    # SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
+    # is answered, because a knob that cannot take effect invites someone to set it and expect one:
+    #   (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
+    #       port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
+    #   (b) HOW should they be separated from the non-public names that share the Gateway? Options,
+    #       roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
+    #       annotation, with only the public services' routes attached; or publishing the five names
+    #       from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
+    #       which restores per-service targeting without a second Gateway.
+    # Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
+    # ---------------------------------------------------------------------------------------------
     # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
     #
     # On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran

platform/monitoring/alerts.yaml

@@ -394,6 +394,213 @@ spec:
             description: "Container {{ $labels.container }} in {{ $labels.namespace }}/{{ $labels.pod }} on {{ $labels.node }} spent more than 15% of wall clock with every task in its cgroup blocked on I/O, for 5m. This is the WHO, not the why — pair it with the cause rules in this group. If the pod mounts the NFS media export (sonarr, radarr, plex, qbittorrent, nextcloud) and no local-device alert is firing, the stall is on 192.168.2.102. If it is a Longhorn instance-manager, look for a replica rebuild. Measured 2026-09-14 02:40Z: sonarr 0.268 on k3s-prod-1 while the node itself read 0.210."
             diagnostic_query: "topk(10, rate(container_pressure_io_stalled_seconds_total{container!=''}[5m])) — and the write side: topk(10, sum by (namespace, pod) (rate(container_fs_writes_bytes_total[5m])))"
             runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        # ---- CAUSE 6: the NFS export is SLOW (the gap CAUSE 4 left open, closed 2026-09-15) ----
+        # WHAT THIS CLOSES. NodeNFSRetransmissionRateHigh above sees a DROPPED or TIMED-OUT RPC and
+        # nothing else; the comment on it records the honest gap in as many words — "an NFS export
+        # that is merely SLOW cannot be distinguished today" — because `mountstats` was switched on
+        # in the same change and its series did not exist yet. They exist now. VERIFIED on prod
+        # 2026-09-15: all eight families referenced below return series (426 series each for the
+        # five per-operation families, 6 each for the transport families, 1,842 node_mountstats_*
+        # in total, about 0.4% of a 451,650-series head).
+        #
+        # THE DISCRIMINATOR, AND IT IS THE WHOLE POINT OF THESE RULES. mountstats separates the two
+        # halves of a slow NFS operation that every other metric in this repo conflates:
+        #   response_time  the interval the SERVER had the request — 192.168.2.102's problem
+        #   queue_time     the interval the request sat on THIS NODE before it could be sent,
+        #                  waiting for an RPC slot — our problem
+        #   request_time   the sum, i.e. what the application actually waited (not alerted on
+        #                  directly; it cannot tell you which half moved, which is the question)
+        # So: high response with low queue means the SERVER is slow. High queue means WE are
+        # saturating the transport. And high response with low queue AND a LOW op rate of our own
+        # means the server is slow for someone ELSE — which is the operator's standing hypothesis
+        # (the Proxmox-hosted copy of this media stack still has operational control of the same
+        # files on 192.168.2.102) turned into something a rule can assert. That hypothesis was
+        # WEAKENED by the 2026-09-14 event, where prod-1 was itself doing 114 NFS reads/s during
+        # the spike, so the rules below are built to TEST it, not to assume it: NFSExportServerSlow
+        # and NFSExportServerSlowNotOurLoad partition the same slow-server state by whether our own
+        # op rate accounts for it, and exactly one of the two fires.
+        #
+        # MEASURED BASELINE, prod 2026-09-15 (every number below re-measured for this change):
+        #   server response time   4.25 - 4.83 ms/op over 10m, uniform across all six mounts
+        #                          history maximum 4.73 / 5.17 / 5.57 / 6.80 ms/op
+        #   client queue time      0.014 - 0.091 ms/op; history maximum 0.333 ms/op
+        #   our own op rate        0.019 - 0.124 ops/s; history maximum 0.163 ops/s (prod-1 plex)
+        #   major timeouts         0 on every mount
+        #   backlog / sending / pending queue depth   0 on every mount
+        #
+        # THREE HONEST LIMITS ON THAT BASELINE, recorded because they bound how much the thresholds
+        # below are worth:
+        #
+        # (a) THE HISTORY IS MINUTES, NOT DAYS. Every other threshold in this group is cut from 24
+        #     to 48 hours of prod data. These are not. The mountstats series begin when the
+        #     node-exporter DaemonSet rolled, and at measurement time each mount held 18 samples —
+        #     `count_over_time(node_mountstats_nfs_transport_backlog_queue_total[24h])` returned 18,
+        #     not ~2,880, and `time() - process_start_time_seconds{job="node-exporter"}` returned
+        #     558-561 s on all three nodes. So "history maximum" above means a maximum over about
+        #     ten minutes. Nothing below uses a range longer than 15m for that reason: a rule with a
+        #     24h range would also be unreliable for a day after any node-exporter restart.
+        #
+        # (b) IT IS AN IDLE BASELINE, and the loaded one is unknown. The busiest operation on every
+        #     one of the six mounts is STATFS keepalive, at 0.002-0.046 ops/s; there was no READ or
+        #     WRITE traffic at all while these numbers were taken. A 4.3 ms cached STATFS is not the
+        #     cost of a 1 MB READ off a spinning array, so the response-time thresholds below are
+        #     argued from the LINK AND THE ARRAY, with the measured figure used only as a floor.
+        #     They have never been checked against a real media event. runbooks/node-io.md carries
+        #     the query to re-cut them after the first one, and that is follow-up work, not a
+        #     silent assumption.
+        #
+        # (c) SIX MOUNTS, FOUR INDEPENDENT SETS OF COUNTERS. prod-1's /mnt/Storage/plex and
+        #     /mnt/Storage/plex/downloads report byte-identical per-operation counters (op rate
+        #     0.12443893333333335 on both, to every digit), and so do prod-3's two nextcloud
+        #     mounts, because mounts sharing one NFS superblock share the kernel's RPC statistics.
+        #     A single slow export on prod-1 therefore raises TWO alerts differing only in
+        #     `export`. Left alone deliberately: aggregating the label away would hide which mount
+        #     paths are affected, which is the one thing per-export alerting is for, and
+        #     Alertmanager groups them on the page anyway.
+        #
+        # WHY `unless` AND NOT `and ... < threshold` for the "slow but not queueing" clause: the
+        # same reason NodeIOPressureNotLocalDisk gives above. An `unless` whose right-hand side
+        # produces no series removes nothing, so a mount too quiet to have a measurable queue time
+        # still raises the alert; `and queue < 0.005` would instead go silent exactly when the
+        # client side is quietest, which is the state the rule exists to report.
+        # AND THE PRECEDENCE TRAP, again: the guard is its own operand
+        # (`(ratio) > T and rate(ops) > N`), never `(ratio) and rate(ops) > N > T`, which compares
+        # T against the OP RATE. Both forms are asserted in the test file.
+        - alert: NFSExportServerSlow
+          # 50 ms/op is 10x the measured 4.25-4.83 ms mean and 7.4x the 6.80 ms measured maximum,
+          # but the multiple is not the argument (see limit (b)) — the physics is. One outstanding
+          # 1 MB NFS read at 50 ms is 20 MB/s, well under what this gigabit path and this array
+          # deliver, so a mount whose MEAN operation costs 50 ms for 15 minutes is not keeping up.
+          # OP-RATE GUARD >20/s, and it is doing two jobs. It keeps the mean honest (6,000 ops in a
+          # 5m window, so no single unlucky operation can manufacture an average — the artefact the
+          # device rules guard at >10/s for), and it is the load side of the partition with
+          # NFSExportServerSlowNotOurLoad below: above 20 ops/s our own workload plausibly explains
+          # a slow server. 20/s is 123x the busiest measured op rate (0.163/s) and under a fifth of
+          # the 114 NFS reads/s prod-1 itself sustained at the 2026-09-14 02:40Z spike (that figure
+          # is from node_nfs_requests_total, a whole-node family, so the comparison is approximate).
+          expr: |
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.050
+              and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 20
+            unless (
+              (
+                sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
+                  / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+              ) > 0.005
+            )
+          for: 15m
+          labels:
+            severity: warning
+            io_cause: nfs-server-slow
+          annotations:
+            summary: "CAUSE NFS server slow: {{ $labels.export }} is taking over 50ms per operation for {{ $labels.instance }}, under load this node is generating"
+            description: "The NFS server is spending more than 50ms per RPC on {{ $labels.export }} as seen from {{ $labels.instance }}, sustained 15m, while this node is driving more than 20 ops/s at it and its own CLIENT-side queue time is under 5ms per op. Measured healthy is 4.25-4.83ms per op on every mount. The time is on the SERVER, not in our transport — but our own op rate is high enough to explain it, so this reads as self-inflicted: a library scan, an import, a torrent verifying, a Nextcloud reindex. Because I/O PSI counts an NFS wait as an I/O stall, this is a cause class that makes a node look disk-starved while every local device is idle. Do NOT touch local disks. Find the workload with ContainerIOStallHigh and slow it down. If instead our op rate were LOW, NFSExportServerSlowNotOurLoad would be firing and the reading would be the opposite one."
+            diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — split it by operation to see whether metadata or data is slow: sum by (export, operation) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (export, operation) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 0"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        - alert: NFSExportServerSlowNotOurLoad
+          # THE RULE THE OPERATOR ACTUALLY WANTS, and the reason this change exists: the same slow
+          # server, with our own demand too small to account for it. Same 50 ms/op threshold as
+          # NFSExportServerSlow so the two are one statement split by ONE variable — our op rate —
+          # and exactly one of them can fire: this one requires >1 op/s and excludes >20 op/s,
+          # the other requires >20 op/s. No gap between them, no overlap.
+          # >1 op/s is the statistical floor, not a load judgement: 300 operations in a 5m window,
+          # and 6x the busiest op rate ever measured on these mounts (0.163/s, where the top
+          # operation is STATFS keepalive). Below it the mount is idle and cannot stall a node.
+          # `unless` on BOTH exclusions, per the note above: if the op-rate or queue-time series is
+          # missing the alert still fires, which is the safe direction for a rule whose whole
+          # purpose is to report "the load is not ours".
+          expr: |
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.050
+              and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
+            unless (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 20
+            )
+            unless (
+              (
+                sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
+                  / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+              ) > 0.005
+            )
+          for: 15m
+          labels:
+            severity: warning
+            io_cause: nfs-server-slow-external
+          annotations:
+            summary: "CAUSE NFS server slow and the load is NOT OURS: {{ $labels.export }} takes over 50ms per operation while {{ $labels.instance }} asks for under 20 ops/s"
+            description: "{{ $labels.export }} is answering {{ $labels.instance }} in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
+            diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        - alert: NFSClientQueueTimeHigh
+          # THE OTHER HALF OF THE DISCRIMINATOR: the request had not even been sent yet. queue_time
+          # is the interval an RPC waits on this node for a transport slot, so it is OUR saturation,
+          # not the server's.
+          # 5 ms/op is 15x the 0.333 ms history maximum and 55-360x the 0.014-0.091 ms typical, but
+          # the anchor that matters is physical: 5 ms is longer than the server's own measured
+          # answer time (4.25-4.83 ms). A client that waits longer for a slot than the server takes
+          # to answer is unambiguously the bottleneck itself.
+          expr: |
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.005
+              and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
+          for: 10m
+          labels:
+            severity: warning
+            io_cause: nfs-client-backed-up
+          annotations:
+            summary: "CAUSE we are backed up, not the server: {{ $labels.instance }} queues over 5ms per operation before it can send to {{ $labels.export }}"
+            description: "On {{ $labels.instance }} NFS requests to {{ $labels.export }} are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
+            diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        - alert: NFSTransportBacklogDeep
+          # backlog_queue_total is a CUMULATIVE SUM OF QUEUE LENGTH, one sample added per send, so
+          # rate(backlog) / rate(sends) is the mean number of requests sitting in the backlog queue
+          # per send — a depth, not a rate. Measured 0 on all six mounts.
+          # WHY 8 AND NOT 1: this cluster's mounts use a DYNAMIC RPC slot table, and the high-water
+          # marks read 586 on prod-1, 162 on prod-3 and 2 on prod-2 — the client grows the table
+          # rather than queueing, which is why the backlog is 0 and why NFSClientQueueTimeHigh
+          # above, not this rule, is the primary client-saturation signal. A backlog forms only
+          # when even the grown table is full. 8 rather than 1 so a single sampling artefact
+          # cannot page, and it is still 8 deep against a measured floor of zero.
+          expr: |
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m]))
+            ) > 8
+              and sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) > 1
+          for: 10m
+          labels:
+            severity: warning
+            io_cause: nfs-client-backed-up
+          annotations:
+            summary: "CAUSE NFS transport backlog: {{ $labels.instance }} has over 8 requests queued behind a full slot table for {{ $labels.export }}"
+            description: "The RPC backlog queue for {{ $labels.export }} on {{ $labels.instance }} is averaging more than 8 waiting requests per send, for 10m. Measured healthy is exactly 0 on all six mounts. These mounts grow their slot table on demand (high-water 586 on prod-1, 162 on prod-3, 2 on prod-2), so a persistent backlog means even the grown table is full: this node is asking for far more concurrent NFS I/O than the transport can carry. Same remediation as NFSClientQueueTimeHigh — reduce concurrency on this node. This is a client-side ceiling, NOT a statement about the server."
+            diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) — and the two neighbouring depths, sending and pending, derived the same way; pending high with backlog low means the requests are on the wire and the server is holding them."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+        - alert: NFSExportMajorTimeouts
+          # CRITICAL, AND THE ONLY THRESHOLD HERE THAT NEEDS NO MULTIPLE. A major timeout is the
+          # client having exhausted the FULL RPC retransmission backoff on one request without an
+          # answer — tens of seconds of a process blocked in uninterruptible sleep, and the point
+          # at which an NFS mount starts to look hung rather than slow. Measured 0 on every mount,
+          # so any non-zero value is unambiguous and "> 0" is the honest threshold.
+          # 15m range, not 24h, per limit (a) above. `for: 5m` is satisfiable because a single
+          # increment keeps increase() positive for the whole 15m window.
+          expr: sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0
+          for: 5m
+          labels:
+            severity: critical
+            io_cause: nfs-export-failing
+          annotations:
+            summary: "CAUSE NFS export failing: {{ $labels.export }} has timed out RPCs from {{ $labels.instance }} — the mount is hanging, not just slow"
+            description: "{{ $labels.instance }} has recorded NFS major timeouts against {{ $labels.export }} in the last 15m: requests that went unanswered through the entire retransmission backoff. Measured baseline is 0 on all six mounts, so this is never noise. Every process touching that mount is now blocked in uninterruptible sleep, which will show as I/O PSI on the node and as container stalls on the pods — and unlike the latency rules above this one means the export is effectively DOWN for those requests, not slow. Check 192.168.2.102 itself (is it up, is nfsd running, is its array degraded) and the path to it. Pair with NodeNFSRetransmissionRateHigh, which sees the retries that lead here. Do not reboot cluster nodes to clear a hung mount while volumes are not healthy."
+            diagnostic_query: "sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) — which operations: sum by (export, operation) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0 — and the retry that precedes a timeout: sum by (instance, export) (rate(node_mountstats_nfs_operations_transmissions_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
 
 
 # =====================================================================================
@@ -697,6 +904,132 @@ spec:
             summary: "Egress VPN tunnel down"
             description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
+        # -----------------------------------------------------------------------------------------
+        # THE BLIND SPOT `VPNTunnelDown` LEAVES (added 2026-09-12) — R-OBS-04, R-NET-06
+        #
+        # `VPNTunnelDown` goes green the instant a tun0 exists. That is not the same as vpn-class pods
+        # having egress, because Cilium resolves a CiliumEgressGatewayPolicy's `interface: tun0`
+        # exactly ONCE and caches an (egress IP, egress ifindex) pair. Every OpenVPN reconnect
+        # re-creates tun0 with a NEW ifindex, so after the first reconnect the BPF egress map points
+        # at a DESTROYED interface: `bpf_redirect()` fails and the packets are dropped — 0 out tun0,
+        # 0 out the WAN (measured; there is no leak, there is simply no egress). tun0 is present and
+        # carrying bytes the whole time, so `VPNTunnelDown` stays green. The dev gateway sat in
+        # exactly that state for ~80 minutes on 2026-09-12 with nothing alerting.
+        #
+        # The datapath now self-heals: `roles/vpn_egress_gateway` publishes
+        # `homelab.keithjackson.dev/egress-uplink=<tun0 addr>-<ifindex>` on the gateway node from the
+        # OpenVPN up-script on every connect (plus a 2-minute convergence timer for the boot case),
+        # and that node write is what makes Cilium re-resolve. These two rules watch whether it
+        # ACTUALLY converged — if the re-steer mechanism itself fails we are back to a silent outage.
+        #
+        # WHY THIS SIGNAL AND NOT THE MAP ITSELF. The direct invariant is "the gateway node's
+        # `egress-uplink` label == the Egress Ifindex in Cilium's BPF egress map". The map side is NOT
+        # observable in Prometheus, verified on prod 2026-09-12 rather than assumed:
+        #   * the cilium-agent DaemonSet exposes no metrics port at all (`health=9879`,
+        #     `peer-service=4244`; only the OPERATOR has `operator-prometheus-serve-addr :9963`), and
+        #     there is no ServiceMonitor for cilium or hubble on either cluster — Hubble/Cilium
+        #     metrics export is still the open TODO in platform/cilium/helmrelease.yaml. Every
+        #     `cilium_*` and `hubble_*` query returns an empty vector today (which also means
+        #     `HubbleDeniedFlowSpike` above cannot fire yet).
+        #   * and even with agent metrics enabled, Cilium OSS publishes NO series carrying egress-map
+        #     CONTENTS. The map is readable only via `cilium-dbg bpf egress list` — a CLI over the BPF
+        #     map, not an exporter. No metric name is invented here to paper over that.
+        # So the observable is the label: the input to the re-steer, compared against the live tun0
+        # identity that node-exporter does export (`node_network_iface_id{device="tun0"}` — the real
+        # ifindex, 129 on k3s-prod-3 right now, matching both the label `10.150.192.86-129` and
+        # `cilium-dbg bpf egress list`'s `Egress Ifindex 129`, checked by hand on the live cluster).
+        #
+        # WHAT THIS DOES AND DOES NOT CATCH. It catches the whole input half: the up-script not
+        # firing, the label write failing or never being deployed, the convergence timer being dead —
+        # i.e. every way the tun0 identity can move without Cilium being told. It does NOT catch
+        # "the label was written and Cilium still did not re-resolve"; nothing in Prometheus can,
+        # until either the re-steer script exports the map's ifindex through a node-exporter textfile
+        # collector (DECISIONS-LOG 2026-09-12, hand-off item 4) or a vpn-class black-box probe exists.
+        # A probe was considered and rejected for now: there is no blackbox-exporter and no probe
+        # CronJob in this repo, so it means a NEW always-on component whose own failures become a new
+        # alerting surface — worth doing, but not as the thing that closes a hole today.
+        #
+        # WHY A `changes()` COMPARISON RATHER THAN AN EQUALITY TEST. PromQL cannot compare a metric
+        # VALUE (the live ifindex) against a number embedded in a LABEL VALUE (`...-129`): there is no
+        # label-to-number conversion, so the level-triggered equality test is not expressible. What IS
+        # expressible is the same invariant counted over a window: the published label must change at
+        # least as often as tun0 is re-created. `count(count_over_time(max by (<label>) (...)[1h:1m]))`
+        # is the number of DISTINCT label values seen in the last hour (verified live: 1 for a label
+        # that held still, 2 for one that changed value inside the window); the `max by (<label>)`
+        # wrapper is load-bearing — it aggregates away `pod`/`instance`, so a kube-state-metrics
+        # restart is not mistaken for a label change. Both sides reduce to a single unlabelled series,
+        # which is why `and on()` joins cleanly, exactly as `VPNTunnelDown` does.
+        #
+        # Consequence, stated plainly: this is bounded by its window. A missed flap is alerted from
+        # +15m to +1h and then the alert RESOLVES even if egress is still dead. That is a deliberate
+        # trade against the repo's alert-fatigue history (an unbounded critical on the one channel
+        # that must stay trustworthy is worse), and it still turns an 80-minute silent outage into an
+        # email within 15 minutes. The textfile-collector gauge is what would make it level-triggered.
+        - alert: VPNEgressSteeringStale
+          # FIRES when the live tun0 was re-created more times in the last hour than the gateway
+          # node's `egress-uplink` label changed — i.e. Cilium was not told about at least one
+          # tunnel reconnect, so the BPF egress map is pointing at an ifindex that no longer exists
+          # and vpn-class pods have no egress. Convergence is NOT this: the label is rewritten within
+          # seconds of a connect (worst case ~5m, the up-script's 30 x 10s kube-API retry budget on a
+          # cold boot), so `for: 15m` cannot fire on a healthy reconnect.
+          expr: |
+            (
+                changes(max(node_network_iface_id{device="tun0"})[1h:1m])
+              >
+                (
+                  count(
+                    count_over_time(
+                      max by (label_homelab_keithjackson_dev_egress_uplink) (
+                        kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""}
+                      )[1h:1m]
+                    )
+                  ) - 1
+                )
+            )
+            and on()
+            (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "VPN egress steering is stale — vpn-class pods have no egress"
+            description: >-
+              The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
+              Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
+              being dropped, silently, while tun0 still looks healthy. Compare
+              `kubectl get nodes -L homelab.keithjackson.dev/egress-uplink` against
+              `cilium-dbg bpf egress list` on the gateway node; re-applying the CiliumEgressGatewayPolicy
+              (or setting any node label) forces a fresh resolution.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+        - alert: VPNEgressUplinkLabelMissing
+          # The watchdog for the rule above: `VPNEgressSteeringStale` needs the `egress-uplink` label
+          # in kube-state-metrics, and if that label is missing its comparison degrades to an empty
+          # vector — it fails OPEN, alerting nothing. So alert on the precondition itself. A gateway
+          # node with no published uplink means the re-steer mechanism is not in place at all
+          # (roles/vpn_egress_gateway not applied on this node, the up-script never ran, or the
+          # kube-state-metrics label allowlist in kube-prometheus-stack.yaml lost the entry), which is
+          # the configuration that produced the 80-minute dev outage. Warning, not critical: it is a
+          # rollout/config gap to fix in working hours, not a page. `for: 30m` so that the Helm upgrade
+          # that rolls kube-state-metrics — both changes ship in this same kustomization — cannot page
+          # on the way in, and so an Ansible run in progress does not either.
+          expr: |
+            (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+            and on()
+            absent(kube_node_labels{label_homelab_keithjackson_dev_egress_uplink!=""})
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Egress gateway node publishes no egress-uplink label"
+            description: >-
+              A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
+              `homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without it
+              nothing can tell whether Cilium's egress steering followed the last tunnel reconnect, and
+              `VPNEgressSteeringStale` is inert. Check the label on the node, then the
+              `vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
+              metricLabelsAllowlist.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+        # -----------------------------------------------------------------------------------------
         # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
         # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
         # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`

platform/monitoring/kube-prometheus-stack.yaml

@@ -231,7 +231,7 @@ spec:
       # 3 extra series, no cardinality concern. Label name is sanitized by KSM to
       # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
       metricLabelsAllowlist:
-        - "nodes=[homelab.keithjackson.dev/egress-gateway]"
+        - "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
       # --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
       # Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
       # `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that

platform/monitoring/tests/nfs-export-latency.test.yaml

@@ -0,0 +1,672 @@
+# promtool rule-unit tests for the five NFS mountstats rules in the platform-health.node-io-cause
+# group (platform/monitoring/alerts.yaml): NFSExportServerSlow, NFSExportServerSlowNotOurLoad,
+# NFSClientQueueTimeHigh, NFSTransportBacklogDeep, NFSExportMajorTimeouts.
+#
+# Run with: platform/monitoring/tests/run.sh
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHY THIS FILE IS SEPARATE from node-io-cause.test.yaml, which covers the other eight rules of the
+# same group: every fixture in that file was cut from prod's TSDB at a moment when the mountstats
+# collector produced nothing, so not one of its fixtures carries a node_mountstats_* series. These
+# rules need their own, and mixing them in would make it ambiguous which fixture pins which rule.
+#
+# THE QUESTION THESE FIVE RULES ANSWER, and it is one question with four answers:
+#   SERVER SLOW, LOAD IS OURS      response_time high, queue_time low, our op rate high
+#   SERVER SLOW, LOAD IS NOT OURS  response_time high, queue_time low, our op rate LOW
+#   WE ARE BACKED UP               queue_time high (the request had not even been sent)
+#   THE EXPORT IS FAILING          major timeouts, i.e. the request was never answered at all
+# Fixtures 3-7 below are those four states, one apiece, and each asserts the OTHER rules stay quiet.
+# Fixture 4 is the one the operator asked for: it is the in-cluster half of the Proxmox hypothesis.
+#
+#   fixture                       source of the numbers
+#   ----------------------------  --------------------------------------------------------------
+#   1 healthy, measured           prod 2026-09-15, all six mounts, idle (see the two caveats below)
+#   2 healthy under real load     CONSTRUCTED, and it says so — the state that must not page
+#   3 server slow, load ours      CONSTRUCTED from the 2026-09-14 profile (prod-1 at 114 ops/s)
+#   4 server slow, load NOT ours  CONSTRUCTED — the Proxmox-hypothesis profile
+#   5 we are backed up            CONSTRUCTED — client queueing, backlog still empty
+#   6 transport backlog deep      CONSTRUCTED — even the grown slot table is full
+#   7 the export is failing       CONSTRUCTED — a major timeout, measured baseline exactly 0
+#
+# TWO CAVEATS ON FIXTURE 1, both recorded in the rule comments in alerts.yaml and repeated here
+# because they are what bound the value of these tests:
+#   (a) THE MEASURED HISTORY IS MINUTES. The mountstats series start when the node-exporter
+#       DaemonSet rolled; at measurement time each mount held 18 samples, and
+#       time() - process_start_time_seconds{job="node-exporter"} read 558-561 s on all three nodes.
+#       Fixture 1 is therefore a snapshot, not a distribution.
+#   (b) IT IS AN IDLE SNAPSHOT. The busiest operation on all six mounts is STATFS keepalive at
+#       0.002-0.046 ops/s; there was no READ or WRITE traffic while it was taken. A fixture built
+#       only from that would prove the rules are quiet when NOTHING IS HAPPENING, which is worth
+#       very little — an op-rate guard alone would satisfy it. THAT IS WHY FIXTURE 2 EXISTS: it is
+#       explicitly constructed, puts 60 ops/s of healthy-latency traffic on the mount, and is the
+#       fixture that actually pins the response-time and queue-time thresholds in the quiet
+#       direction. It also carries the two non-vacuity hazards below.
+#
+# NON-VACUITY, asserted rather than trusted (the same two hazards node-io-cause.test.yaml records):
+#   1. PROMQL PRECEDENCE. `(ratio) and rate(ops) > 1 > 0.050` parses as
+#      `(ratio) and ((rate(ops) > 1) > 0.050)` — the threshold is compared against the OP RATE and
+#      never touches the ratio, so the rule fires on any mount busier than 1 op/s. Fixture 2
+#      asserts the broken form RETURNS SERIES on a perfectly healthy 60 ops/s mount while the
+#      shipped form returns none. Fixture 1 cannot carry this assertion: at 0.125 ops/s the broken
+#      form is quiet too, for the wrong reason.
+#   2. THE OP-RATE AND SEND-RATE GUARDS ARE LOAD-BEARING. Fixture 2 carries a mount doing HALF an
+#      operation a second whose response time is a full second per op, whose queue time is a full
+#      second per op, and whose backlog queue is 20 deep — 20x, 200x and 2.5x the three thresholds
+#      at once — and asserts that all five rules stay quiet, because thirty operations a minute are
+#      not evidence of anything.
+#   3. THE SHARED-SUPERBLOCK DUPLICATION IS REAL AND IS PINNED. prod-1's /mnt/Storage/plex and
+#      /mnt/Storage/plex/downloads report byte-identical per-operation counters (mounts sharing one
+#      NFS superblock share the kernel's RPC statistics), and so do prod-3's two nextcloud mounts.
+#      Fixture 1 encodes that, and fixture 4 asserts that a single slow export on prod-1 therefore
+#      raises TWO alerts. If a future change aggregates the export label away, fixture 4 fails.
+#   4. THE DISCRIMINATOR ITSELF. Fixture 5 is a client that is queueing 30 ms per op while the
+#      server appears to be taking 60 ms per op — over the server-slow threshold. Both server-slow
+#      rules MUST stay quiet there, and that is asserted. This is the single most important
+#      assertion in the file: without it the `unless` clause could be deleted and nothing would
+#      notice.
+#
+# CLOCK CONVENTION (as in node-io-cause.test.yaml and longhorn.test.yaml): promtool's test clock
+# starts at the unix epoch with a 1m sample interval, so nothing is ever stale inside Prometheus'
+# 5m lookback. Counter fixtures are written as per-MINUTE increments, so `0+7.5x120` is a rate of
+# 7.5/60 = 0.125 per second. Every increment below is annotated with the per-second or per-op
+# figure it encodes.
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. HEALTHY, MEASURED ON PROD 2026-09-15 — all five rules SILENT.
+  #    All six mounts, with the shared-superblock pairs encoded as the identical counters they
+  #    actually are. See caveats (a) and (b) in the header: this fixture is idle, so on its own it
+  #    proves little; fixture 2 is the one that pins the thresholds.
+  # =============================================================================================
+  - interval: 1m
+    name: measured healthy prod state 2026-09-15 - all five NFS mountstats rules quiet
+    input_series:
+      # --- prod-1: /mnt/Storage/plex and /mnt/Storage/plex/downloads. THE SAME NUMBERS TWICE, on
+      #     purpose: measured op rate was 0.12443893333333335 on BOTH, to every digit.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+7.5x120'            # 0.125 ops/s   (measured 0.1244)
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.031875x120'       # / 7.5 = 4.25 ms per op   (MEASURED)
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.000105x120'       # / 7.5 = 0.014 ms per op  (MEASURED)
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0x120'              # zero, on every mount, measured
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+9.48x120'           # 0.158 sends/s (measured 0.1579)
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'              # backlog depth 0, measured on all six mounts
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
+        values: '0+7.5x120'            # identical to the mount above — shared superblock
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
+        values: '0+0.031875x120'       # 4.25 ms per op
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
+        values: '0+0.000105x120'       # 0.014 ms per op
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="STATFS"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
+        values: '0+9.48x120'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
+        values: '0+0x120'
+      # --- prod-2: one mount, /mnt/Storage/plex.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+1.26x120'           # 0.021 ops/s   (measured 0.02086)
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.0064008x120'      # / 1.26 = 5.08 ms per op  (MEASURED)
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.00010458x120'     # / 1.26 = 0.083 ms per op (MEASURED)
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+1.47x120'           # 0.0245 sends/s (measured 0.02456)
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'
+      # --- prod-3: /mnt/Storage/nextcloud and /mnt/Storage/nextcloud/nextcloud — the second shared
+      #     superblock pair, and split across TWO operations so the sum-across-operations in every
+      #     rule is exercised rather than assumed (the real mounts carry 71 operation rows each).
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
+        values: '0+0.63x120'           # half of 1.26/min
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="SEQUENCE"}'
+        values: '0+0.63x120'           # summed: 1.26/min = 0.021 ops/s (measured 0.020944)
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
+        values: '0+0.0030429x120'
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="SEQUENCE"}'
+        values: '0+0.0030429x120'      # summed / 1.26 = 4.83 ms per op (MEASURED)
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
+        values: '0+0.00005229x120'
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="SEQUENCE"}'
+        values: '0+0.00005229x120'     # summed / 1.26 = 0.083 ms per op (MEASURED)
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="STATFS"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
+        values: '0+2.21x120'           # 0.0368 sends/s (measured 0.03684)
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
+        values: '0+0.63x120'
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="SEQUENCE"}'
+        values: '0+0.63x120'
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
+        values: '0+0.0030429x120'
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="SEQUENCE"}'
+        values: '0+0.0030429x120'
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
+        values: '0+0.00005229x120'
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="SEQUENCE"}'
+        values: '0+0.00005229x120'
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud",operation="STATFS"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud"}'
+        values: '0+2.21x120'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud/nextcloud"}'
+        values: '0+0x120'
+      # --- prod-3: /mnt/Storage/plex, its own superblock and its own numbers.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+1.152x120'          # 0.0192 ops/s (measured 0.01920)
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.00544896x120'     # / 1.152 = 4.73 ms per op (MEASURED)
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.000104832x120'    # / 1.152 = 0.091 ms per op (MEASURED, the cluster maximum)
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+2.21x120'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+    promql_expr_test:
+      # THE MEASURED BASELINE IS PINNED, as a band rather than an exact float (promtool compares
+      # sample values bit-for-bit and a counter-increment fixture does not land on a round number).
+      # If someone "tidies" these increments the claim in the rule comments stops being true here.
+      - expr: |
+          count(
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.00424
+              and (
+                sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                  / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+              ) < 0.00426
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 2          # prod-1's two mounts, at the measured 4.25 ms per op
+      # Every one of the six mounts is between 4 and 7 ms per op — the whole measured range.
+      - expr: |
+          count(
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.004
+              and (
+                sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                  / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+              ) < 0.007
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 6
+      # And every one is under 0.1 ms of CLIENT queue time — the half of the discriminator that
+      # says "we are not the bottleneck".
+      - expr: |
+          count(
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) < 0.0001
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 6
+      # THE SHARED-SUPERBLOCK DUPLICATION, pinned: six exports, four distinct sets of counters.
+      # `count(count by (value))` is not expressible, so this asserts the pairs are equal by
+      # subtracting them and counting the zeroes.
+      - expr: |
+          count(
+            (
+              sum by (instance) (rate(node_mountstats_nfs_operations_requests_total{export="192.168.2.102:/mnt/Storage/plex"}[5m]))
+                - sum by (instance) (rate(node_mountstats_nfs_operations_requests_total{export="192.168.2.102:/mnt/Storage/plex/downloads"}[5m]))
+            ) == 0
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 1          # prod-1: the two mounts report the identical op rate
+
+  # =============================================================================================
+  # 2. HEALTHY UNDER REAL LOAD — CONSTRUCTED, and the fixture that actually pins the thresholds
+  #    in the quiet direction. 60 ops/s at healthy per-op latency must NOT page: that is a library
+  #    scan or a stream, and it is the state a badly-chosen threshold would turn into a permanent
+  #    alert. It also carries both non-vacuity hazards.
+  # =============================================================================================
+  - interval: 1m
+    name: constructed healthy under load - 60 ops per second at healthy latency stays quiet
+    input_series:
+      # --- prod-1 plex, genuinely busy and genuinely fine.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+3600x120'           # 60 ops/s — 3x the >20/s split, 60x the >1/s guard
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+16.2x120'           # / 3600 = 4.5 ms per op — healthy, and 11x under threshold
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0.072x120'          # / 3600 = 0.02 ms per op — 250x under threshold
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+3600x120'           # 60 sends/s
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'              # the slot table absorbed it; no backlog
+      # --- THE GUARDS' REASON FOR EXISTING, all three at once: prod-3's plex mount doing HALF an
+      #     operation a second, each taking a full SECOND on the server, a full SECOND in the queue,
+      #     and sitting 20 deep in the backlog. That is 20x, 200x and 2.5x the three thresholds —
+      #     and thirty operations a minute are not evidence of anything, so nothing may fire.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+30x120'             # 0.5 ops/s -> BELOW the >1/s guard
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+30x120'             # 1.0 SECOND per op, and deliberately ignored
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+30x120'             # 1.0 SECOND per op, and deliberately ignored
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+30x120'             # 0.5 sends/s -> BELOW the >1/s send guard
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+600x120'            # / 30 sends = depth 20, and deliberately ignored
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+    promql_expr_test:
+      # HAZARD 1 — PROMQL PRECEDENCE. The BROKEN form returns a series for the perfectly healthy
+      # 60 ops/s mount, because 60 > 0.050 is true and the threshold never reaches the ratio.
+      - expr: |
+          count(
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            )
+              and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1 > 0.050
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+      # The SHIPPED form, same fixture, returns NOTHING. This pair is the regression guard.
+      - expr: |
+          count(
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.050
+              and sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 1
+          )
+        eval_time: 60m
+        exp_samples: []
+      # HAZARD 2 — the trickle mount really is 1 s/op and 20 deep. If a future edit "fixes" this
+      # fixture into something harmless, the guards stop being tested at all.
+      - expr: |
+          sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total{instance="192.168.2.12:9100"}[5m]))
+            / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total{instance="192.168.2.12:9100"}[5m]))
+        eval_time: 60m
+        exp_samples:
+          - labels: '{export="192.168.2.102:/mnt/Storage/plex", instance="192.168.2.12:9100"}'
+            value: 1
+      - expr: |
+          sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total{instance="192.168.2.12:9100"}[5m]))
+            / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total{instance="192.168.2.12:9100"}[5m]))
+        eval_time: 60m
+        exp_samples:
+          - labels: '{export="192.168.2.102:/mnt/Storage/plex", instance="192.168.2.12:9100"}'
+            value: 20
+
+  # =============================================================================================
+  # 3. SERVER SLOW, AND THE LOAD IS OURS — NFSExportServerSlow fires, alone.
+  #    The 2026-09-14 profile: prod-1 hammering the export itself (114 NFS reads/s measured then).
+  # =============================================================================================
+  - interval: 1m
+    name: server slow under our own load - NFSExportServerSlow fires and names the export
+    input_series:
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+7200x120'           # 120 ops/s -> above the >20/s "the load is ours" split
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+576x120'            # / 7200 = 80 ms per op -> 1.6x the 50 ms threshold
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0.36x120'           # / 7200 = 0.05 ms per op -> the client is NOT the problem
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+7200x120'           # 120 sends/s
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'              # no backlog: the transport is keeping up
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-server-slow
+              instance: 192.168.2.10:9100
+              export: 192.168.2.102:/mnt/Storage/plex
+            exp_annotations:
+              summary: "CAUSE NFS server slow: 192.168.2.102:/mnt/Storage/plex is taking over 50ms per operation for 192.168.2.10:9100, under load this node is generating"
+              description: "The NFS server is spending more than 50ms per RPC on 192.168.2.102:/mnt/Storage/plex as seen from 192.168.2.10:9100, sustained 15m, while this node is driving more than 20 ops/s at it and its own CLIENT-side queue time is under 5ms per op. Measured healthy is 4.25-4.83ms per op on every mount. The time is on the SERVER, not in our transport — but our own op rate is high enough to explain it, so this reads as self-inflicted: a library scan, an import, a torrent verifying, a Nextcloud reindex. Because I/O PSI counts an NFS wait as an I/O stall, this is a cause class that makes a node look disk-starved while every local device is idle. Do NOT touch local disks. Find the workload with ContainerIOStallHigh and slow it down. If instead our op rate were LOW, NFSExportServerSlowNotOurLoad would be firing and the reading would be the opposite one."
+              diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — split it by operation to see whether metadata or data is slow: sum by (export, operation) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (export, operation) (rate(node_mountstats_nfs_operations_requests_total[5m])) > 0"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # THE PARTITION HOLDS: the not-our-load variant is excluded by our own 120 ops/s.
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. SERVER SLOW AND THE LOAD IS NOT OURS — THE HEADLINE FIXTURE.
+  #    This is the profile that makes the operator's Proxmox hypothesis testable: the export is
+  #    slow, our queue time is flat, and we are asking it for 5 ops/s. It also asserts the
+  #    shared-superblock duplication — one slow export on prod-1 raises TWO alerts.
+  #    prod-2 sees the SAME slow server on the same export path and does NOT alert, because at
+  #    0.021 ops/s it is below the statistical guard. That is correct and it is asserted.
+  # =============================================================================================
+  - interval: 1m
+    name: server slow while our own demand is small - NFSExportServerSlowNotOurLoad fires on both prod-1 mounts
+    input_series:
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+300x120'            # 5 ops/s -> above the >1/s guard, below the >20/s split
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+36x120'             # / 300 = 120 ms per op -> 25x the measured healthy 4.8 ms
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0.015x120'          # / 300 = 0.05 ms per op -> we are NOT backed up
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+300x120'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'
+      # The shared superblock: identical counters, as measured.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
+        values: '0+300x120'
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
+        values: '0+36x120'
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
+        values: '0+0.015x120'
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
+        values: '0+300x120'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex/downloads"}'
+        values: '0+0x120'
+      # prod-2 mounts the same export path and sees the same slow server — at 0.021 ops/s, which
+      # the guard correctly refuses to average.
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+1.26x120'           # 0.021 ops/s -> below the >1/s guard
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.1512x120'         # / 1.26 = 120 ms per op — just as slow, still not alerted
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0.0000630x120'      # 0.05 ms per op
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex",operation="STATFS"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+1.26x120'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.11:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-server-slow-external
+              instance: 192.168.2.10:9100
+              export: 192.168.2.102:/mnt/Storage/plex
+            exp_annotations:
+              summary: "CAUSE NFS server slow and the load is NOT OURS: 192.168.2.102:/mnt/Storage/plex takes over 50ms per operation while 192.168.2.10:9100 asks for under 20 ops/s"
+              description: "192.168.2.102:/mnt/Storage/plex is answering 192.168.2.10:9100 in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
+              diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-server-slow-external
+              instance: 192.168.2.10:9100
+              export: 192.168.2.102:/mnt/Storage/plex/downloads
+            exp_annotations:
+              summary: "CAUSE NFS server slow and the load is NOT OURS: 192.168.2.102:/mnt/Storage/plex/downloads takes over 50ms per operation while 192.168.2.10:9100 asks for under 20 ops/s"
+              description: "192.168.2.102:/mnt/Storage/plex/downloads is answering 192.168.2.10:9100 in more than 50ms per RPC for 15m, our CLIENT-side queue time is under 5ms per op (so we are not backed up), and THIS NODE is asking for between 1 and 20 ops/s — two orders of magnitude below the 114 NFS reads/s prod-1 sustained on 2026-09-14 when the load WAS ours. Healthy is 4.25-4.83ms per op. Read literally: the server at 192.168.2.102 is slow, and this cluster is not what is making it slow. THE STANDING HYPOTHESIS IS THE PROXMOX-HOSTED COPY OF THIS MEDIA STACK, which still has operational control of the same files — but this alert does not prove that, it only rules US out. What would confirm it is a read on the server itself (nfsstat -s, or per-client counters), which is outside this repo. Do not replace a disk, do not throttle Longhorn, do not move workloads."
+              diagnostic_query: "our own demand, per export: sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) — and the bytes, to be sure a low op rate is not large-block streaming: sum by (instance, export) (rate(node_mountstats_nfs_total_read_bytes_total[5m]))"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # The our-load variant does NOT fire: 5 ops/s is below its >20/s requirement. Exactly one of
+      # the two rules speaks, and it is the one that names the right cause.
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+
+  # =============================================================================================
+  # 5. WE ARE BACKED UP — NFSClientQueueTimeHigh fires AND BOTH SERVER-SLOW RULES STAY QUIET.
+  #    THE MOST IMPORTANT ASSERTION IN THIS FILE. The server here LOOKS slow — 60 ms per op, over
+  #    the 50 ms threshold — because this node is queueing 30 ms per request before it can send.
+  #    If the `unless` clause were dropped from the two server rules, this fixture would page three
+  #    times and blame the wrong machine. The backlog queue is EMPTY here (the dynamic slot table
+  #    grew instead, as prod-1's high-water of 586 slots shows it does), which is also what keeps
+  #    NFSTransportBacklogDeep from being a mere restatement of this rule.
+  # =============================================================================================
+  - interval: 1m
+    name: client transport backed up - queue-time rule fires and both server-slow rules stay quiet
+    input_series:
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+12000x120'          # 200 ops/s
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+720x120'            # / 12000 = 60 ms per op -> OVER the 50 ms server threshold
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+360x120'            # / 12000 = 30 ms per op -> 6x the 5 ms client threshold
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+12000x120'          # 200 sends/s
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+0x120'              # still EMPTY — the slot table absorbed the concurrency
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-client-backed-up
+              instance: 192.168.2.10:9100
+              export: 192.168.2.102:/mnt/Storage/plex
+            exp_annotations:
+              summary: "CAUSE we are backed up, not the server: 192.168.2.10:9100 queues over 5ms per operation before it can send to 192.168.2.102:/mnt/Storage/plex"
+              description: "On 192.168.2.10:9100 NFS requests to 192.168.2.102:/mnt/Storage/plex are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
+              diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # THE `unless` CLAUSE, PINNED. 60 ms per op is over the server threshold and 200 ops/s is over
+      # the our-load split, so NFSExportServerSlow would fire on every clause EXCEPT the queue-time
+      # exclusion. Delete that clause and this assertion fails.
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts: []
+      # And the backlog rule is NOT just this rule in other clothes.
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+    promql_expr_test:
+      # The server half really is over its threshold here — this fixture is only meaningful if it is.
+      - expr: |
+          count(
+            (
+              sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m]))
+                / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))
+            ) > 0.050
+          )
+        eval_time: 60m
+        exp_samples:
+          - labels: '{}'
+            value: 1
+
+  # =============================================================================================
+  # 6. TRANSPORT BACKLOG DEEP — even the grown slot table is full.
+  #    The server is FAST here (8 ms per op), so this is unambiguously a client-side ceiling, and
+  #    both server-slow rules must stay quiet.
+  # =============================================================================================
+  - interval: 1m
+    name: transport backlog deep - backlog rule fires while the server is fast
+    input_series:
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+6000x120'           # 100 ops/s
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+48x120'             # / 6000 = 8 ms per op -> the SERVER is fine
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+240x120'            # / 6000 = 40 ms per op -> all the time is in OUR queue
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex",operation="READ"}'
+        values: '0+0x120'
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+6000x120'           # 100 sends/s -> above the >1/s send guard
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.10:9100",export="192.168.2.102:/mnt/Storage/plex"}'
+        values: '0+72000x120'          # / 6000 sends = depth 12 -> 1.5x the threshold of 8
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-client-backed-up
+              instance: 192.168.2.10:9100
+              export: 192.168.2.102:/mnt/Storage/plex
+            exp_annotations:
+              summary: "CAUSE NFS transport backlog: 192.168.2.10:9100 has over 8 requests queued behind a full slot table for 192.168.2.102:/mnt/Storage/plex"
+              description: "The RPC backlog queue for 192.168.2.102:/mnt/Storage/plex on 192.168.2.10:9100 is averaging more than 8 waiting requests per send, for 10m. Measured healthy is exactly 0 on all six mounts. These mounts grow their slot table on demand (high-water 586 on prod-1, 162 on prod-3, 2 on prod-2), so a persistent backlog means even the grown table is full: this node is asking for far more concurrent NFS I/O than the transport can carry. Same remediation as NFSClientQueueTimeHigh — reduce concurrency on this node. This is a client-side ceiling, NOT a statement about the server."
+              diagnostic_query: "sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m])) — and the two neighbouring depths, sending and pending, derived the same way; pending high with backlog low means the requests are on the wire and the server is holding them."
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # The client is queueing too — the two client-side rules agree, which is expected.
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+              io_cause: nfs-client-backed-up
+              instance: 192.168.2.10:9100
+              export: 192.168.2.102:/mnt/Storage/plex
+            exp_annotations:
+              summary: "CAUSE we are backed up, not the server: 192.168.2.10:9100 queues over 5ms per operation before it can send to 192.168.2.102:/mnt/Storage/plex"
+              description: "On 192.168.2.10:9100 NFS requests to 192.168.2.102:/mnt/Storage/plex are waiting more than 5ms each for a transport slot before they are even sent, sustained 10m at more than 1 op/s. Measured healthy is 0.014-0.091ms per op, maximum 0.333ms. This is CLIENT-side: this node has more NFS work in flight than its RPC transport will carry, so the fix is on this side of the wire — find the workload with ContainerIOStallHigh and reduce its concurrency. It is NOT evidence that 192.168.2.102 is slow; compare the server half with the diagnostic query. When this fires, NFSExportServerSlow and NFSExportServerSlowNotOurLoad deliberately stay quiet, because a queueing client makes the server look slow."
+              diagnostic_query: "client vs server, side by side: sum by (instance, export) (rate(node_mountstats_nfs_operations_queue_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])) and then the same with response_time — plus the slot-table high-water mark, which grows under exactly this pressure: node_mountstats_nfs_transport_maximum_rpc_slots"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # The server is NOT accused. 8 ms per op is a sixth of the threshold.
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+
+  # =============================================================================================
+  # 7. THE EXPORT IS FAILING — a major timeout, on prod-3's nextcloud mount.
+  #    Measured baseline is exactly 0 on all six mounts, so "> 0" is the whole threshold. The
+  #    per-op latency series stay at healthy idle values here on purpose: response_time does not
+  #    accumulate for a request that is never answered, so a hanging mount can look fast right up
+  #    to the timeout. That is precisely why this rule cannot be replaced by a latency rule.
+  # =============================================================================================
+  - interval: 1m
+    name: export failing - major timeouts raise a critical while the latency rules see nothing
+    input_series:
+      - series: 'node_mountstats_nfs_operations_major_timeouts_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
+        values: '0x50 1x90'            # flat zero for 50m, then ONE timeout at t=51m
+      - series: 'node_mountstats_nfs_operations_requests_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
+        values: '0+1.26x140'           # 0.021 ops/s — the measured idle rate
+      - series: 'node_mountstats_nfs_operations_response_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
+        values: '0+0.0060858x140'      # / 1.26 = 4.83 ms per op — the answered requests look fine
+      - series: 'node_mountstats_nfs_operations_queue_time_seconds_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud",operation="WRITE"}'
+        values: '0+0.00010458x140'     # / 1.26 = 0.083 ms per op
+      - series: 'node_mountstats_nfs_transport_sends_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
+        values: '0+2.21x140'
+      - series: 'node_mountstats_nfs_transport_backlog_queue_total{job="node-exporter",instance="192.168.2.12:9100",export="192.168.2.102:/mnt/Storage/nextcloud"}'
+        values: '0+0x140'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              io_cause: nfs-export-failing
+              instance: 192.168.2.12:9100
+              export: 192.168.2.102:/mnt/Storage/nextcloud
+            exp_annotations:
+              summary: "CAUSE NFS export failing: 192.168.2.102:/mnt/Storage/nextcloud has timed out RPCs from 192.168.2.12:9100 — the mount is hanging, not just slow"
+              description: "192.168.2.12:9100 has recorded NFS major timeouts against 192.168.2.102:/mnt/Storage/nextcloud in the last 15m: requests that went unanswered through the entire retransmission backoff. Measured baseline is 0 on all six mounts, so this is never noise. Every process touching that mount is now blocked in uninterruptible sleep, which will show as I/O PSI on the node and as container stalls on the pods — and unlike the latency rules above this one means the export is effectively DOWN for those requests, not slow. Check 192.168.2.102 itself (is it up, is nfsd running, is its array degraded) and the path to it. Pair with NodeNFSRetransmissionRateHigh, which sees the retries that lead here. Do not reboot cluster nodes to clear a hung mount while volumes are not healthy."
+              diagnostic_query: "sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) — which operations: sum by (export, operation) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m])) > 0 — and the retry that precedes a timeout: sum by (instance, export) (rate(node_mountstats_nfs_operations_transmissions_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))"
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/node-io"
+      # BEFORE the timeout the rule is silent — this is the half that proves it is not permanently
+      # firing on a counter that merely EXISTS at zero.
+      - eval_time: 40m
+        alertname: NFSExportMajorTimeouts
+        exp_alerts: []
+      # Nothing else has anything to say: the latency and queue rules see a healthy idle mount.
+      - eval_time: 60m
+        alertname: NFSExportServerSlow
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSExportServerSlowNotOurLoad
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSClientQueueTimeHigh
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: NFSTransportBacklogDeep
+        exp_alerts: []

platform/monitoring/tests/run.sh

@@ -116,10 +116,14 @@ fi
 # `.git` file points outside the mount, so git cannot read the index there — and with
 # `pipefail` that failure silently emptied the discovery, i.e. a green run over ZERO tests
 # (observed 2026-09-12). `find` needs no repo metadata, so discovery cannot be disabled by
-# how the tree happens to be mounted. .git/ is pruned so a stale object copy is never picked up.
+# how the tree happens to be mounted. .git/ is pruned so a stale object copy is never picked up,
+# and .claude/ with it: .claude/worktrees/agent-*/ holds full repo copies for isolated agent
+# runs, so an unpruned find reported the SAME manifest under test once per worktree (up to 18
+# times), and the L0 check that compares this listing against the tree failed on paths that
+# are not authored surface at all (.claude/ is gitignored, .gitignore:43).
 discover() {
   # Companion convention, anywhere in the tree.
-  find "${ROOT}" -path "${ROOT}/.git" -prune -o -name '*.promtool-test.yaml' -print 2>/dev/null
+  find "${ROOT}" \( -path "${ROOT}/.git" -o -path "${ROOT}/.claude" \) -prune -o -name '*.promtool-test.yaml' -print 2>/dev/null
   # Suite convention, ANYWHERE in the tree — not just this directory.
   #
   # This used to be `find "${HERE}" -maxdepth 1`, which made the suite convention a
@@ -129,7 +133,7 @@ discover() {
   # as the rules this harness exists to test. Found 2026-09-13 adding apps/wave-*/tests/. Scoped by
   # NAME only; manifests_for() then refuses (loudly) any *.test.yaml that is not inside a tests/
   # directory, so a misplaced file fails rather than disappearing.
-  find "${ROOT}" -path "${ROOT}/.git" -prune -o -name '*.test.yaml' -print 2>/dev/null
+  find "${ROOT}" \( -path "${ROOT}/.git" -o -path "${ROOT}/.claude" \) -prune -o -name '*.test.yaml' -print 2>/dev/null
 }
 
 TESTS=()

platform/monitoring/tests/vpn-egress.test.yaml

@@ -0,0 +1,188 @@
+# promtool rule-unit tests for VPNEgressSteeringStale + VPNEgressUplinkLabelMissing
+# (platform/monitoring/alerts.yaml)
+#
+# Run with: platform/monitoring/tests/run.sh
+#           (the merge gate runs the same harness via
+#            conformance/tests/l0-static/alert-rules-unit-tested)
+#
+# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
+# `egress-class=vpn` pods leave through one gateway node's tun0. Cilium resolves that interface to an
+# IFINDEX and caches it; an OpenVPN reconnect destroys and re-creates tun0 with a NEW ifindex, and the
+# gateway node's up-script publishes `homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex>` so that
+# node write forces Cilium to re-resolve. If the label does not follow the tunnel, the cache points at a
+# destroyed interface and every vpn-class pod is dropped — while tun0 itself stays up, so VPNTunnelDown
+# says nothing. That shape ran ~80 minutes undetected on the dev gateway.
+#
+# VPNEgressSteeringStale detects it by comparing HOW MANY TIMES tun0's ifindex changed in the last hour
+# against HOW MANY DISTINCT `egress-uplink` label values appeared in the same hour. The subtlety worth
+# testing is that a label VALUE change is not a sample change: it RETIRES one series and STARTS another,
+# so "the label followed the tunnel" shows up as two overlapping series in the window, not as a
+# changing value. Tests 1 and 2 are the same tunnel reconnect with and without that second series, so
+# they isolate exactly that mechanism — anything that broke the series-counting half would make test 2
+# fire, and anything that broke the tun0 half would make test 1 quiet.
+#
+# VPNEgressUplinkLabelMissing is the WATCHDOG for the rule above: with no uplink label anywhere, the
+# staleness comparison has nothing to compare and is structurally INERT — present, evaluating, and
+# incapable of firing. Tests 3 and 4 pin that, because an inert critical alert is the failure mode this
+# whole group exists to prevent.
+#
+# Fixture values are prod's real shape: gateway k3s-prod-3, tun0 ifindex 139 with address
+# 10.150.192.86 (live label value `10.150.192.86-139`), reconnecting to ifindex 140.
+
+rule_files:
+  - generated.rules.yaml
+
+evaluation_interval: 1m
+
+tests:
+  # =============================================================================================
+  # 1. THE BUG — tun0 was re-created, the label did NOT follow. Steering is stale: FIRES.
+  # =============================================================================================
+  - interval: 1m
+    name: tunnel reconnect with no label update - VPNEgressSteeringStale fires
+    input_series:
+      # tun0 re-created at t=41m: ifindex 139 -> 140. One change inside the trailing hour.
+      - series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '139+0x40 140+0x40'
+      # tun0 is UP and passing traffic the whole time — this is what makes the failure deceptive, and
+      # it is asserted below: VPNTunnelDown stays QUIET while egress is actually broken.
+      - series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '0+120000x80'
+      # The node label NEVER changes, so only ONE distinct uplink value exists in the window.
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
+        values: '1+0x80'
+    alert_rule_test:
+      # 1 tunnel change > (1 distinct label value - 1) = 0, held for 15m.
+      - eval_time: 60m
+        alertname: VPNEgressSteeringStale
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+            exp_annotations:
+              summary: "VPN egress steering is stale — vpn-class pods have no egress"
+              description: >-
+                The gateway node's tun0 was re-created without the `egress-uplink` label following it, so
+                Cilium's cached egress ifindex points at a destroyed interface: egress-class=vpn pods are
+                being dropped, silently, while tun0 still looks healthy. Compare `kubectl get nodes -L
+                homelab.keithjackson.dev/egress-uplink` against `cilium-dbg bpf egress list` on the
+                gateway node; re-applying the CiliumEgressGatewayPolicy (or setting any node label)
+                forces a fresh resolution.
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+      # Not yet: at 45m the condition has been true only ~4m, under the 15m `for`.
+      - eval_time: 45m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      # No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
+      # no VPN gateway must not be told its tunnel is down forever.
+      - eval_time: 45m
+        alertname: VPNTunnelDown
+        exp_alerts: []
+      # The watchdog must stay quiet — the label IS present here.
+      - eval_time: 60m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts: []
+      # THE HEADLINE CLAIM, asserted rather than asserted-in-prose: the tunnel alert says NOTHING
+      # while vpn-class egress is completely broken. If VPNTunnelDown covered this failure, the
+      # staleness rule would not need to exist.
+      - eval_time: 60m
+        alertname: VPNTunnelDown
+        exp_alerts: []
+
+  # =============================================================================================
+  # 2. THE HEALTHY RECONNECT — same tunnel change, but the label followed it. STAYS QUIET.
+  #    This is the half that proves the rule is not simply "any reconnect alerts".
+  # =============================================================================================
+  - interval: 1m
+    name: tunnel reconnect WITH the label following - stays quiet
+    input_series:
+      - series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '139+0x40 140+0x40'
+      - series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '0+120000x80'
+      # A label VALUE change retires one series and starts another. Both are inside the trailing
+      # hour, so the window holds TWO distinct uplink values: 2 - 1 = 1, which is NOT < 1 change.
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-139"}'
+        values: '1+0x40 _x40'
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true",label_homelab_keithjackson_dev_egress_uplink="10.150.192.86-140"}'
+        values: '_x41 1+0x39'
+    alert_rule_test:
+      - eval_time: 60m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      - eval_time: 75m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts: []
+      - eval_time: 60m
+        alertname: VPNTunnelDown
+        exp_alerts: []
+
+  # =============================================================================================
+  # 3. THE INERT-ALERT CASE — a gateway node publishing NO uplink label at all.
+  #    VPNEgressSteeringStale cannot fire in this state, which is exactly why this warning exists.
+  # =============================================================================================
+  - interval: 1m
+    name: gateway node with no egress-uplink label - VPNEgressUplinkLabelMissing fires
+    input_series:
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3",label_homelab_keithjackson_dev_egress_gateway="true"}'
+        values: '1+0x60'
+      # tun0 is reconnecting the whole time. The staleness rule STILL cannot fire, because it has no
+      # label series to compare against — that is the inertness being alerted on.
+      - series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '139+0x30 140+0x30'
+      - series: 'node_network_receive_bytes_total{job="node-exporter",instance="192.168.2.12:9100",device="tun0"}'
+        values: '0+120000x60'
+    alert_rule_test:
+      - eval_time: 35m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts:
+          - exp_labels:
+              severity: warning
+            exp_annotations:
+              summary: "Egress gateway node publishes no egress-uplink label"
+              description: >-
+                A node carries `homelab.keithjackson.dev/egress-gateway=true` but no
+                `homelab.keithjackson.dev/egress-uplink` label is visible in kube-state-metrics. Without
+                it nothing can tell whether Cilium's egress steering followed the last tunnel reconnect,
+                and `VPNEgressSteeringStale` is inert. Check the label on the node, then the
+                `vpn-egress-resteer` service/timer on the gateway node, then the kube-state-metrics
+                metricLabelsAllowlist.
+              runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
+      # Under the 30m `for`.
+      - eval_time: 20m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts: []
+      # And the proof of inertness: the critical rule is silent despite a real reconnect.
+      - eval_time: 55m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      - eval_time: 55m
+        alertname: VPNTunnelDown
+        exp_alerts: []
+
+  # =============================================================================================
+  # 4. NO GATEWAY NODE AT ALL — neither rule may fire. Both are guarded by an `and on()` on the
+  #    egress-gateway label so they stay silent on a cluster with no VPN gateway, rather than
+  #    alerting forever about an absent feature.
+  # =============================================================================================
+  - interval: 1m
+    name: no egress-gateway node - both rules silent
+    input_series:
+      - series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
+        values: '1+0x60'
+      - series: 'node_network_iface_id{job="node-exporter",instance="192.168.2.10:9100",device="eth0"}'
+        values: '2+0x60'
+    alert_rule_test:
+      - eval_time: 45m
+        alertname: VPNEgressUplinkLabelMissing
+        exp_alerts: []
+      - eval_time: 45m
+        alertname: VPNEgressSteeringStale
+        exp_alerts: []
+      # No gateway label AND no tun0 history, so VPNTunnelDown's `or` guard is false: a cluster with
+      # no VPN gateway must not be told its tunnel is down forever.
+      - eval_time: 45m
+        alertname: VPNTunnelDown
+        exp_alerts: []

platform/velero/volume-policies.yaml

@@ -24,6 +24,36 @@
 # pod, and nothing tells them. A volume policy tests the PROPERTY that makes a volume bulk-tier —
 # its source is the out-of-cluster NFS export — so it covers volumes that do not exist yet.
 #
+# RULE 2 — emptyDir (added 2026-09-19). Every emptyDir volume is skipped, because an emptyDir holds
+# no durable state BY DEFINITION: created empty with its pod, destroyed with it. Backing one up
+# cannot protect anything, and ATTEMPTING it actively breaks the backup.
+#
+# HOW IT BREAKS THINGS. A PodVolumeBackup exposes the volume through its pod; if that pod is replaced
+# mid-flight the PVB fails with `error to expose PVB: error getting client pod <pod>: pods "<pod>"
+# not found`, and ONE failed PVB makes the whole Backup PartiallyFailed. PartiallyFailed does not
+# advance velero_backup_last_successful_timestamp, so VeleroNoRecentBackup fires CRITICAL over a
+# backup that protected every real volume. Any Deployment rollout landing in the backup window is
+# enough — on a GitOps cluster that is routine, not an edge case. An alert that cries wolf nightly is
+# how a real backup failure gets ignored, and this one guards R-STATE-03.
+#
+# OBSERVED TWICE IN ONE HOUR in unrelated namespaces, which is what justified a general rule over a
+# targeted exclusion:
+#   * ns velero   — Velero's OWN transient kopia repository-maintenance Jobs
+#                   (`<ns>-default-kopia-maintain-job-<ts>`), emptyDir `plugins` + `scratch`. Four
+#                   failed PVBs had accumulated; velero-daily-20260919030041 was PartiallyFailed,
+#                   2 errors, 3079 of 3079 items backed up.
+#   * ns runbooks — the nginx Deployment's emptyDir `tmp` + `cache` (a read-only root filesystem needs
+#                   writable scratch). Four failed PVBs when a runbook edit rolled the Deployment
+#                   while the backup was running.
+#
+# PROVEN ON PROD, WITH A CONTROL (2026-09-19, velero 1.18.1). `volumeTypes` support was the open
+# question, and a policies.yaml Velero cannot parse breaks EVERY backup — so it was tested against a
+# THROWAWAY ConfigMap first, leaving this one untouched until it passed:
+#     control, ns runbooks, NO policy  -> 4 PodVolumeBackups (2x `tmp`, 2x `cache`), Completed
+#     same namespace, WITH this policy -> 0 PodVolumeBackups, Backup Completed, 0 errors
+# The control is the point: "zero PVBs" on its own is equally consistent with nothing being attempted.
+#
+
 # `nfs: {}` is the documented "match any NFS volume source" condition (server/path may be given to
 # narrow it; deliberately not, so a second export is covered too). First matching rule wins.
 #
@@ -44,6 +74,12 @@ data:
   policies.yaml: |
     version: v1
     volumePolicies:
+      # emptyDir first: cheapest condition, and the most common match.
+      - conditions:
+          volumeTypes:
+            - emptyDir
+        action:
+          type: skip
       - conditions:
           nfs: {}
         action:

platform/vpn-egress/README.md

@@ -157,6 +157,31 @@ kubectl -n kube-system exec <cilium-agent-on-the-gateway> -c cilium-agent -- \
 If they disagree, the re-steer path is broken — `journalctl -u vpn-egress-resteer` on the
 gateway node is the next stop.
 
+### And now something watches it (2026-09-12)
+
+The self-heal closed the datapath hole; it did not close the monitoring one, and a mechanism that
+silently stops working is the same outage again. Two rules in `platform/monitoring/alerts.yaml`:
+
+- **`VPNEgressSteeringStale`** (critical, `for: 15m`) — fires when the live tun0 was re-created
+  more times in the last hour than the `egress-uplink` label changed, i.e. Cilium was never told
+  about a reconnect. This is the case `VPNTunnelDown` cannot see: tun0 present, bytes flowing,
+  steering pointed at a destroyed ifindex.
+- **`VPNEgressUplinkLabelMissing`** (warning, `for: 30m`) — fires when a node carries
+  `egress-gateway=true` but no `egress-uplink` label reaches kube-state-metrics, which is the state
+  in which the rule above would be inert. The watchdog for the watchdog.
+
+Both are gated on a gateway node existing (the same `kube_node_labels` guard `VPNTunnelDown` uses),
+so a cluster with no gateway cannot be made to fire them. `egress-uplink` had to be added to
+`kube-state-metrics.metricLabelsAllowlist` in `platform/monitoring/kube-prometheus-stack.yaml` for
+the label to exist as a metric at all — the two changes must ship together.
+
+**What still is not watched:** "the label was written and Cilium still ignored it." Cilium OSS
+exposes no metric carrying the egress map's contents (the map is readable only through
+`cilium-dbg bpf egress list`), and the cilium-agent is not scraped on either cluster in the first
+place — no `cilium_*` or `hubble_*` series exist. Closing that last gap means either exporting the
+map's ifindex from the re-steer script through a node-exporter textfile collector, or a black-box
+probe from a vpn-class pod. Neither exists yet.
+
 ## Resilience — there is NO gateway failover (spec correction, escalated)
 
 **ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)

runbooks/README.md

@@ -6,7 +6,7 @@ Every alert rule in this platform carries an annotation of the form
 
     runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"
 
-46 of them, over the 35 names below. Until this directory existed, that path returned **HTTP 404** on the
+46 of them, over 35 names. Until this directory existed, that path returned **HTTP 404** on the
 live cluster: every alert that reached a human handed them a dead link at the exact moment they needed
 help. These pages are what makes the link resolve.
 
@@ -84,8 +84,10 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`cert-expiry`** — a certificate is expiring. Almost always a late symptom of `cert-issuance`.
 
 ### Networking
-- **`vpn-down`** — the egress VPN tunnel. Includes the silent egress-gateway staleness that this alert
-  cannot see, and what the kill-switch actually does.
+- **`vpn-down`** — the egress VPN tunnel. Explains what the kill-switch actually does, and the silent
+  egress-gateway staleness this alert cannot see — which now has its own alerts and page below.
+- **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached
+  ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
 - **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
 
 ### Backups and DR
@@ -106,6 +108,11 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`keycloak-client-secrets`** — the client/URL reconciler failing or silent. The 2026-09-06 lockout that
   every green signal missed.
 
+### Operator procedures (no alert links here)
+- **`plex-claim`** — claiming the Plex Media Server. The plex.tv claim token lives ~4 minutes, so this
+  is a timed sequence: token → `sops` → push → **forced** `flux reconcile` → pod roll. Includes why a
+  changed Secret does not restart anything, and why an expired token is harmless.
+
 ### Virtualisation
 - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.
 - **`kubevirt-launch-failed`** — a VMI reached the Failed phase. The three k3s-specific launch traps.

runbooks/kustomization.yaml

@@ -80,7 +80,14 @@ configMapGenerator:
       - node-down=node-down.md
       - node-io=node-io.md
       - node-memory=node-memory.md
+      - node-memory-swap=node-memory-swap.md
+      # Operator procedure, not an alert page (no alert links to it — the
+      # alert-runbooks-resolvable check treats that as advisory). It is served
+      # because the 4-minute plex.tv claim window is not a thing to go looking
+      # for in git: /runbooks/plex-claim has to resolve the moment it is needed.
+      - plex-claim=plex-claim.md
       - pvc-capacity=pvc-capacity.md
       - targets-down=targets-down.md
       - update-rollback=update-rollback.md
       - vpn-down=vpn-down.md
+      - vpn-egress-stale=vpn-egress-stale.md

runbooks/node-io.md

@@ -8,7 +8,10 @@
 file — these say WHY, and each names its cause in its own `summary`):
 `NodeDiskDeviceSaturated`, `NodeDiskWriteLatencyHigh`, `NodeDiskWriteLatencyCritical`,
 `NodeDiskReadLatencyHigh`, `NodeNFSRetransmissionRateHigh`, `NodeIOPressureNotLocalDisk`,
-`NodeEtcdRequestsSlow`, `ContainerIOStallHigh`.
+`NodeEtcdRequestsSlow`, `ContainerIOStallHigh`, and — added 2026-09-15, once the `mountstats`
+collector had actually rolled out — the five PER-EXPORT NFS rules that close the "the export is
+merely slow" gap: `NFSExportServerSlow`, `NFSExportServerSlowNotOurLoad`, `NFSClientQueueTimeHigh`,
+`NFSTransportBacklogDeep`, `NFSExportMajorTimeouts`.
 **Dashboard:** `node-pressure`, the *WHY* row — `https://grafana.<BASE_DOMAIN>/d/node-pressure`
 
 ## START HERE — 2-minute triage: which column are you in?
@@ -22,13 +25,16 @@ order, with the numbers they returned on a healthy cluster, during that event, a
 
 Since 2026-09-14 the alerts in `platform-health.node-io-cause` compute this table for you and put
 the verdict in the page's own `summary` (`CAUSE device saturation` / `CAUSE device latency` /
-`CAUSE NFS transport` / `CAUSE not local disk` / `CAUSE etcd` / `WHO is stalled`), every rule
-carries a `diagnostic_query` annotation with the exact PromQL below, and the Grafana dashboard
-`node-pressure` has a **WHY** row that plots all six side by side —
-`https://grafana.<BASE_DOMAIN>/d/node-pressure`. Use this table when you want to check the alert's
-reasoning, or when the answer is "none of the above".
+`CAUSE NFS transport` / `CAUSE not local disk` / `CAUSE etcd` / `WHO is stalled`, and since
+2026-09-15 also `CAUSE NFS server slow` / `CAUSE NFS server slow and the load is NOT OURS` /
+`CAUSE we are backed up, not the server` / `CAUSE NFS transport backlog` / `CAUSE NFS export
+failing`), every rule carries a `diagnostic_query` annotation with the exact PromQL below, and the
+Grafana dashboard `node-pressure` has a **WHY** row that plots the first six side by side —
+`https://grafana.<BASE_DOMAIN>/d/node-pressure`. The per-export NFS numbers (rows 11-15) are NOT on
+that dashboard yet; run the queries. Use this table when you want to check the alert's reasoning, or
+when the answer is "none of the above".
 
-| # | Question / query | HEALTHY (measured 2026-09-14, 24-48h) | 2026-09-14 02:40Z — NFS/unexplained | 2026-09-07 06:05Z — FAILING DRIVE |
+| # | Question / query | HEALTHY (rows 1-10 measured 2026-09-14 over 24-48h; rows 11-15 measured 2026-09-15) | 2026-09-14 02:40Z — NFS/unexplained | 2026-09-07 06:05Z — FAILING DRIVE |
 |---|---|---|---|---|
 | 1 | **Is the node actually stalled?** `max by (instance) (rate(node_pressure_io_stalled_seconds_total[5m]))` | 0.006-0.014; 0 min above 5% on prod-2 | **0.2098** on prod-1; prod-2/3 under 0.015 | **0.7397** on prod-1; prod-2 0.103, prod-3 0.189 |
 | 2 | **Is any device saturated?** `topk(5, rate(node_disk_io_time_seconds_total[5m]))` | 48h max 0.610 / 0.217 / 0.295 | **0.27** (sdi 0.21, sda 0.09) — NOT saturated | 0.745 sda, 0.788 sdb prod-1; 0.858 prod-2 sdb |
@@ -40,6 +46,50 @@ reasoning, or when the answer is "none of the above".
 | 8 | **WHO is stalled?** `topk(10, rate(container_pressure_io_stalled_seconds_total{container!="",namespace!=""}[5m]))` | under 0.02 everywhere | **sonarr 0.268**, qbittorrent 0.070, rest under 0.013 | Longhorn instance-manager and the rebuild set |
 | 9 | **Are volumes degraded?** `kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness \| grep -v healthy` | none | **0 of 24** | all 18 rebuilding |
 | 10 | **Did the node leave?** `min_over_time(kube_node_status_condition{condition="Ready",status="true"}[2h])` | 1 | **1** — never NotReady | **0** — "Kubelet stopped posting node status" |
+| 11 | **Is the NFS SERVER slow, per export?** `sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))` | **4.40-4.81 ms/op** (2026-09-15, 30m window), uniform across all six mounts; snapshot maximum 6.80 | **not collected** — the `mountstats` collector postdates this event | **not collected** |
+| 12 | **Or are WE backed up?** the same expression with `queue_time_seconds_total` in place of `response_time_seconds_total` | **0.013-0.067 ms/op**; snapshot maximum 0.333 ms/op | **not collected** | **not collected** |
+| 13 | **Is the demand ours, per export?** `sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m]))` | 0.019-0.163 ops/s — STATFS keepalive and nothing else | 114 NFS reads/s node-wide (row 6; a different metric family, not per-export) | n/a |
+| 14 | **Is the export failing outright?** `sum by (instance, export) (increase(node_mountstats_nfs_operations_major_timeouts_total[15m]))` | **0** on all six mounts | **not collected** | **not collected** |
+| 15 | **Is the transport queue deep?** `sum by (instance, export) (rate(node_mountstats_nfs_transport_backlog_queue_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_transport_sends_total[5m]))` | **0** on all six mounts (the slot table grows instead: high-water 586 / 162 / 2) | **not collected** | **not collected** |
+
+Rows 11-15 read "not collected" for both incidents and that is not an oversight: `mountstats` was
+switched on on 2026-09-14 and its series begin when the node-exporter DaemonSet rolled, so neither
+historical column can ever be filled in. Their bad-state values are therefore the alert thresholds,
+and rows 11-13 are only meaningful TOGETHER — which is what the next block is for.
+
+### The four-way NFS verdict (rows 11-13 read as one answer)
+
+Rows 11 and 12 split a slow NFS operation into the half the server owns and the half we own; row 13
+then says whether our own demand accounts for it. Four combinations, four different actions, and one
+alert for each:
+
+| row 11 server | row 12 our queue | row 13 our op rate | verdict | alert that says it |
+|---|---|---|---|---|
+| > 50 ms/op | < 5 ms/op | > 20 ops/s | the server is slow **because of us** — a scan, an import, a torrent verifying | `NFSExportServerSlow` |
+| > 50 ms/op | < 5 ms/op | 1-20 ops/s | the server is slow and **we are not the cause** — see the Proxmox note under cause 6 | `NFSExportServerSlowNotOurLoad` |
+| any | > 5 ms/op | > 1 op/s | **we** are the bottleneck: requests are not even being sent yet | `NFSClientQueueTimeHigh` (+ `NFSTransportBacklogDeep` if row 15 is over 8) |
+| any | any | any, with row 14 > 0 | the export is **hanging**, not slow | `NFSExportMajorTimeouts` (critical) |
+
+Below 1 op/s NOTHING here fires, deliberately: the per-op figures are means, and a mount doing a
+handful of operations a minute can produce a one-second "average" from a single unlucky request. An
+idle mount also cannot stall a node. **A flat row 12 is the load-bearing observation** — it is what
+licenses reading row 11 as a statement about 192.168.2.102 rather than about us.
+
+Two caveats before acting on rows 11-15:
+
+* **The healthy numbers are an IDLE baseline.** They were taken with STATFS keepalive as the busiest
+  operation on every mount and no READ or WRITE traffic at all, so 4.4-4.8 ms/op is the cost of a
+  cached metadata call, not of a 1 MB read off a spinning array. The 50 ms threshold is argued from
+  the link and the array (one outstanding 1 MB read at 50 ms is 20 MB/s, well under what this path
+  delivers), not from that measurement. **After the first real media event, re-cut it** —
+  `max_over_time((sum by (instance, export) (rate(node_mountstats_nfs_operations_response_time_seconds_total[5m])) / sum by (instance, export) (rate(node_mountstats_nfs_operations_requests_total[5m])))[24h:1m])`
+  — and if healthy streaming genuinely sits near 50 ms/op, raise it rather than living with a page
+  that means nothing.
+* **Two of the six mounts are duplicates of another two.** prod-1's `/mnt/Storage/plex` and
+  `/mnt/Storage/plex/downloads` report byte-identical per-operation counters, and so do prod-3's two
+  nextcloud mounts, because mounts sharing one NFS superblock share the kernel's RPC statistics. One
+  slow export on prod-1 therefore raises TWO alerts that differ only in `export`. That is expected;
+  it is not two problems.
 
 ### Reading the columns
 
@@ -59,29 +109,38 @@ reasoning, or when the answer is "none of the above".
 
 ### What this table still cannot tell you
 
-Row 5 clears the NFS **transport** — dropped or timed-out RPCs — and nothing more. It cannot
-distinguish *"the export is answering slowly"* from *"we are asking it for a great deal"*, because
-**per-mount NFS round-trip time is not collected on this cluster.** node-exporter's `nfs` collector
-(on by default, host-wide via `/proc/net/rpc/nfs`) gives operation COUNTS and retransmissions only;
-per-mount, per-operation latency needs the **`mountstats`** collector, which is enabled in
-`platform/monitoring/kube-prometheus-stack.yaml` as of 2026-09-14 but **produces nothing until that
-HelmRelease has rolled out**. Until then:
+Row 5 clears the NFS **transport** — dropped or timed-out RPCs — and nothing more. Until 2026-09-15
+that was the whole NFS story here, and this section recorded the gap it left: *"the export is
+answering slowly"* could not be told apart from *"we are asking it for a great deal"*, because
+per-mount NFS round-trip time was not collected. **That gap is now closed** — rows 11-15 and the
+four-way verdict table above are it. The `mountstats` collector, enabled in
+`platform/monitoring/kube-prometheus-stack.yaml` on 2026-09-14, has rolled out and was verified on
+prod 2026-09-15: 1,842 `node_mountstats_*` series (about 0.4% of a 451,650-series head), six mounts,
+426 per-operation series in each of the five operation families and 6 in each transport family.
+`count({__name__=~"node_mountstats_.*"})` and
+`count by (instance, export) (node_mountstats_nfs_operations_requests_total)` are still the two
+queries to run after any node-exporter change; if either comes back empty, **revert the collector**
+rather than leaving it on, because a rule against an unscraped metric is the inert-alert failure this
+repo has already shipped twice (and the L0 gate `alert-metrics-have-scrape-targets` refuses it).
 
-* There is no alert on `node_mountstats_*` — deliberately. A rule against a metric nobody scrapes is
-  the inert-alert failure this repo has already shipped twice, and the L0 gate
-  (`alert-metrics-have-scrape-targets`) refuses it.
-* After the rollout, **verify the series actually arrived** before trusting anything:
-  `count({__name__=~"node_mountstats_.*"})` and
-  `count(node_mountstats_nfs_operations_requests_total) by (export)`. node-exporter reads its own
-  mount namespace, so the kubelet-created NFS mounts are only visible through the `root` hostPath's
-  `mountPropagation: HostToContainer` and will carry a `/host/root/var/lib/kubelet/...` mount point.
-  If the count is zero, **revert the collector** rather than leaving it on.
-* The open question the numbers cannot settle from inside the cluster: the Proxmox-hosted copy of
-  this media stack **still has operational control of the same files on 192.168.2.102**. A second
-  consumer of that export would make the pressure look random from in here and would correlate with
-  nothing we scrape. The in-cluster signal that points at it is row 1 high with rows 2-5 clean; row 6
-  then says whether the traffic is ours. Instrumenting the Proxmox host is out of scope from this
-  repo.
+What is still NOT settled, stated precisely:
+
+* **The Proxmox hypothesis is now testable but still not tested.** The Proxmox-hosted copy of this
+  media stack **still has operational control of the same files on 192.168.2.102**, and a second
+  consumer of that export would make the pressure look random from in here. What the platform can
+  now assert is the in-cluster half: `NFSExportServerSlowNotOurLoad` fires exactly when the export is
+  slow, our transport is not backed up, and our own demand is between 1 and 20 ops/s. That RULES US
+  OUT; it does not rule Proxmox IN. Nothing scraped from inside this cluster can, because the
+  per-client view lives on the server.
+* **What would settle it** is one read on 192.168.2.102 itself while the alert is firing:
+  `nfsstat -s` plus `cat /proc/fs/nfsd/clients/*/info` (or `ss -tn state established '( sport = :2049 )'`)
+  to see which clients are attached and which of them is driving the ops. Two consumers, one
+  export, and only the server can name them. A permanent version of that read is an NFS exporter on
+  the storage host, which is out of scope from this repo — see cause 6.
+* **The thresholds have never seen load.** See the first caveat under the verdict table. This is the
+  one piece of follow-up work these five rules leave behind.
+* The rules are per-export and per-node, so they say WHICH mount and WHICH client node. They do not
+  say which pod — `ContainerIOStallHigh` (row 8) does that.
 
 ### Why the PSI thresholds did not change
 
@@ -290,18 +349,26 @@ reads/s, and `sonarr` at 0.268 container PSI while everything else on the node s
 What to do:
 - **Nothing to the disks.** Do not replace hardware, do not throttle Longhorn, do not raise the PSI
   thresholds. All three would be acting on the wrong subsystem.
-- Ask whether the traffic is ours: row 6 of the triage table. High means our workload is driving the
-  export and the pressure is self-inflicted (an import or library scan). Near zero, with PSI still
-  high, means a consumer **outside this cluster**.
+- Ask whether the traffic is ours: rows 11-13 of the triage table, read together through the
+  four-way verdict table above. That is now an alert rather than a manual query — one of
+  `NFSExportServerSlow` (it is us), `NFSExportServerSlowNotOurLoad` (it is not us),
+  `NFSClientQueueTimeHigh` (we are queueing, the server is fine) or `NFSExportMajorTimeouts` (the
+  export is hanging) will be firing, and each names its own verdict in its `summary`. Row 6 is the
+  older node-wide version of the same question and still works.
 - The standing hypothesis for the outside consumer is that the **Proxmox-hosted copy of this media
-  stack still has operational control of the same files**. It cannot be instrumented from this repo.
-  From in here, the most the platform can say is "the stall is on NFS and our own op rate does not
-  account for it" — which is what the alert's `description` asks you to check.
+  stack still has operational control of the same files**. As of 2026-09-15 the platform can rule
+  ITSELF out — that is what `NFSExportServerSlowNotOurLoad` asserts — but it still cannot point AT
+  Proxmox, because the per-client view only exists on the server. Confirming it needs one read on
+  192.168.2.102: `nfsstat -s` and `cat /proc/fs/nfsd/clients/*/info` while the alert is up. Note that
+  the hypothesis was WEAKENED by the 2026-09-14 event itself, where prod-1 was doing 114 NFS reads/s
+  of its own during the spike — which is exactly the state `NFSExportServerSlow`, not the
+  `NotOurLoad` variant, would have reported.
 - The operator has explicitly **declined** moving the NFS workloads and declined touching the media
   files, so pod anti-affinity and file reorganisation are not on the table. Scope here is
   observability.
-- The measurement gap that would settle it is per-mount RPC round-trip time (`mountstats`), which is
-  switched on but unverified — see "What this table still cannot tell you" above.
+- Per-mount RPC round-trip time (`mountstats`) is no longer the gap: it is collected, verified on
+  prod 2026-09-15, and alerted on by the five `NFS*` rules. The remaining gap is on the SERVER side
+  — see "What this table still cannot tell you" above.
 
 ## When to escalate or stop
 

runbooks/node-memory-swap.md

@@ -0,0 +1,201 @@
+# node-memory-swap — replacing a node's DIMM pair (planned maintenance)
+
+**Not alert-driven.** This is a planned hardware procedure. If a node is *reporting* a memory problem,
+start at `/runbooks/node-memory` instead.
+
+Written for the one known mismatch: **`k3s-prod-1` (192.168.2.10)**.
+
+## Why node 1, and what "mismatched" means here
+
+| slot | part number | size | rated speed | organisation |
+|---|---|---|---|---|
+| `DIMM1` | Micron **8**ATF2G64HZ-**3G2**E1 | 16 GB | DDR4-**3200** | 1Rx8 |
+| `DIMM3` | Micron **16**ATF2G64HZ-**2G6**E1 | 16 GB | DDR4-**2667** | 2Rx8 |
+
+Micron's part numbering carries the proof independently of any inventory doc: the `8ATF` / `16ATF`
+prefix encodes rank organisation, the `-3G2` / `-2G6` suffix encodes 3200 vs 2666 MT/s. Capacity is
+correct — 32 GB, ~31.1 GiB usable — so **nothing is faulty**. The pair simply runs at the slower 2667
+and the ranks differ. `k3s-prod-3` is a matched Crucial pair (`CT16G4SFD8266.C16FE`).
+
+Source: `infra/ansible/_artifacts/node-health/k3s-prod-1.json` records both part numbers; node 3's
+records one. **Caveat on the artifacts:** the extraction de-duplicates part numbers, so it can prove
+"two different modules" but cannot distinguish one module from two identical ones. Only `dmidecode`
+per-slot output settles that, and no raw per-slot block is recorded — which is why verification below
+collects it live.
+
+**Node 2's inventory is stale and contradicts itself** (`infra/PROD-PROVISIONING.md` calls it the only
+single-DIMM node with one 16 GB Samsung module, yet it reports the same ~31.1 GiB as the others). That
+does not change this procedure, but re-inventory node 2 before claiming three-way symmetry.
+
+**DO NOT confuse this with §0b.** The provisioning doc says, correctly, *"do not buy RAM"* — that is
+about node 2's old 2.6-GiB-of-16-GB symptom, which was an **EFI memattr kernel bug, not hardware**.
+Making node 1's pair matched is a different and legitimate change.
+
+## Before you touch anything
+
+*If `kubectl` hangs from the workstation rather than failing, the OIDC context is the reason — prefix
+these with `infra/devcontainer/dev.sh run`.*
+
+**1. Decide what capacity you are installing, because it is a symmetry decision.**
+
+- **2 × 16 GB** keeps all three nodes at 32 GB, which is what `infra/PROD-PROVISIONING.md` §2 assumes
+  (symmetric nodes; replica placement and the scheduler both lean on it).
+- **2 × 32 GB** gives node 1 **64 GB** while nodes 2 and 3 stay at 32 GB — a *new* asymmetry, on the
+  node that is already the asymmetric one on disk. Workloads will schedule there preferentially.
+  Allowed, but make it a decision rather than a side effect.
+
+**2. Confirm the cluster can lose this node.** All three nodes are control-plane **and** etcd members,
+so losing one leaves quorum at 2 of 3 — survivable, but with **zero further margin**. Do not start if
+another node is unhealthy.
+
+    kubectl --context k3s-prod get nodes -o wide
+    # etcd's own view, per node (the house check — embedded etcd metrics on 2381):
+    for n in 1 2 3; do echo "== k3s-prod-$n"; ssh k3s-prod-$n \
+      'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'; done
+
+Every node must report `etcd_server_has_leader 1`. Note which one reports `etcd_server_is_leader 1`:
+if it is node 1, powering it off forces an election, which is normal — you just want to confirm a new
+leader afterwards rather than be surprised by it.
+
+**3. Confirm Longhorn is fully healthy first.** Node 1 held **24 replicas** when last measured. Every
+volume must be healthy *before* you drain, or you are draining into an already-degraded state.
+
+    kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+      -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness,STATE:.status.state
+
+Expect every attached volume `healthy` (a `detached` volume reporting `unknown` is normal for an unused
+PVC). **If anything reads `degraded`, stop and let it rebuild.**
+
+**4. Confirm the drain policy is what protects you**, rather than assuming it — it is not pinned in
+`platform/longhorn/helmrelease.yaml`, so it is whatever the chart defaults to on this version:
+
+    kubectl --context k3s-prod -n longhorn-system get setting node-drain-policy -o jsonpath='{.value}{"\n"}'
+
+You want `block-if-contains-last-replica`. If it reads something more permissive, **set it before
+draining** — this is the setting that stops the drain from stranding a volume whose only healthy
+replica is on node 1.
+
+**5. Take a fresh backup.** There is player-authored state here now (the Minecraft world on
+`crafty-data`). Find the schedule's real resource name rather than guessing it:
+
+    kubectl --context k3s-prod -n velero get schedules
+    kubectl --context k3s-prod -n velero create backup pre-memswap-$(date -u +%Y%m%d%H%M) \
+      --from-schedule <the daily schedule from above>
+
+Wait for `Completed` with `errors: 0` before continuing.
+
+## Drain
+
+    kubectl --context k3s-prod cordon k3s-prod-1
+    kubectl --context k3s-prod drain k3s-prod-1 --ignore-daemonsets --delete-emptydir-data --timeout=30m
+
+**If the drain blocks, that is the policy doing its job.** Do not reach for `--force` or
+`--disable-eviction`. Watch replicas rebuild onto the other two nodes and let the drain proceed:
+
+    kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+      -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -v healthy
+
+Expect real time to pass — this is 24 replicas of actual data movement, and the rebuild path is
+deliberately throttled (see `/runbooks/node-io` for which limits are set and which do not exist on
+this Longhorn version). Watch the pressure it generates while it runs:
+
+    rate(node_pressure_io_stalled_seconds_total{instance="192.168.2.10:9100"}[5m])
+
+The warning threshold is **0.05** and critical is **0.10**; node 1 sat at 0.10 during the incident that
+starved etcd. A rebuild storm pushing this up is expected, sustained time above 0.10 is not.
+
+Note what moves, and one thing that happily does not. Verified live 2026-09-19:
+
+    kubectl --context k3s-prod get pods -A -l cnpg.io/podRole \
+      -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName,ROLE:.metadata.labels.role
+
+**No PostgreSQL primary is on node 1.** All three CNPG clusters run 3 instances spread one per node by
+required anti-affinity, and node 1 holds a *replica* of each (`keycloak-db-2`, `nextcloud-db-2`,
+`semaphore-db-2`). Draining it evicts three replicas and forces **no failover at all** — the best state
+this maintenance could ask for. Re-run the command above before you start rather than trusting this
+paragraph: primaries move on their own after any restart or failover.
+
+`CNPGPrimariesCoLocated` is currently firing for `k3s-prod-2`, which holds both the `keycloak-db` and
+`semaphore-db` primaries. **Do not "fix" that before the swap.** Rebalancing means promoting a replica,
+and the only node with spare primary capacity is node 1 — so tidying the alert would put a primary on
+the very node you are about to power off and convert a no-failover drain into a two-database failover.
+Rebalance *after* node 1 is back and its replicas have caught up.
+
+`sonarr`, `qbittorrent` and `jackett` do sit on node 1 and will reschedule. The UniFi VM
+(`virt-launcher-unifi-*`) is there too — it is not live-migratable in practice today, so expect it to be
+recreated, which also re-runs its cloud-init.
+
+## Power off and swap
+
+    sudo systemctl poweroff   # from the node itself, or via the console
+
+HP EliteDesk 800 G5 Desktop Mini, 2 SODIMM slots. **HP's supported population requires `DIMM1` to be
+populated** — fill `DIMM1` first, then `DIMM3`. Seat both fully; a partially seated SODIMM in these
+machines will POST with one module silently missing.
+
+## Post-swap verification
+
+In order. **Do not uncordon until 1–3 pass.**
+
+**1. The pair is actually matched** — the whole point of the exercise:
+
+    ansible-playbook playbooks/node-health.yml -e node_health_hosts=k3s-prod-1   # read-only
+
+or directly on the node:
+
+    sudo dmidecode -t memory | grep -E 'Locator|Size|Part Number|Speed|Rank'
+
+Both slots must report the **same Part Number**, the **same Rank**, and — the part that proves the
+mismatch is gone — the same `Speed` *and* the same `Configured Memory Speed`. **`Configured Memory
+Speed` is the field that matters:** a pair rated 3200 that is being clocked down still reports
+`Speed: 3200 MT/s` while `Configured Memory Speed` sits lower. Matched part numbers with a
+below-rated configured speed means the job is not done.
+
+**2. Capacity is what you installed.**
+
+    kubectl --context k3s-prod get node k3s-prod-1 -o jsonpath='{.status.capacity.memory}{"\n"}'
+
+Roughly 31.1 GiB reported for 2 × 16 GB, roughly double for 2 × 32 GB. If it reads about half what you
+installed, a module is not seated or not detected — power off and reseat rather than debugging it in
+software, because the scheduler will believe whatever capacity is reported.
+
+**3. No new memory errors.** These machines are non-ECC, so this is a sanity check, not a guarantee:
+
+    sudo journalctl -k -b | grep -iE 'mce|memory error|ECC|Hardware Error'
+
+**4. Rejoin the cluster.**
+
+    kubectl --context k3s-prod uncordon k3s-prod-1
+    kubectl --context k3s-prod get nodes
+    ssh k3s-prod-1 'curl -s http://127.0.0.1:2381/metrics | grep -E "etcd_server_is_leader|etcd_server_has_leader"'
+
+`Ready` on the node and `etcd_server_has_leader 1` from node 1's own etcd before you call this done.
+
+**5. Longhorn returns to fully healthy.** Rebuilds run back onto node 1, and this is the phase that
+historically produced I/O pressure on this node — watch it rather than walking away:
+
+    kubectl --context k3s-prod -n longhorn-system get volumes.longhorn.io \
+      -o custom-columns=NAME:.metadata.name,ROBUST:.status.robustness | grep -vc healthy
+
+with the same PSI query from the drain section. `/runbooks/node-io` records what the post-SSD-swap
+18-replica rebuild measured, for comparison.
+
+**6. Alerts quiet.** The only thing that should be firing is `Watchdog`:
+
+    kubectl --context k3s-prod -n monitoring exec prometheus-kube-prometheus-stack-prometheus-0 \
+      -c prometheus -- /bin/promtool query instant http://localhost:9090 'ALERTS{alertstate="firing"}'
+
+## If it goes wrong
+
+- **Node does not POST**: reseat both modules; failing that, put the original pair back. The old DIMMs
+  are known-good — the mismatch was a consistency/performance issue, not a fault.
+- **Node POSTs with half the memory**: one module not detected. Power off and reseat.
+- **Node rejoins but volumes stay degraded**: `/runbooks/longhorn-degraded`. Do **not** drain a second
+  node while rebuilds are outstanding — that loses quorum and replicas at the same time.
+- **Abort mid-drain**: `kubectl --context k3s-prod uncordon k3s-prod-1` is safe any time before poweroff.
+
+## Update the record when you are done
+
+`infra/PROD-PROVISIONING.md` carries the DIMM inventory table and the "not a matched pair" paragraph.
+Both become wrong the moment this succeeds. Update them in the same change that records the swap —
+stale hardware documentation is exactly what made node 2's inventory untrustworthy.

runbooks/plex-claim.md

@@ -0,0 +1,185 @@
+# plex-claim — claiming the Plex Media Server inside the 4-minute token window
+
+**Alert:** none. This is an **operator procedure**, not an incident page — it is indexed here because
+it is the page you need at the moment you need it, with a stopwatch running.
+**Files:** `catalog/plex.yaml` (env `PLEX_CLAIM`), `apps/wave-0/plex-claim-secret.sops.yaml`
+(the SOPS Secret `plex-claim`, key `PLEX_CLAIM`), `apps/wave-0/plex.yaml` (the rendered golden).
+
+## What this means here
+
+Plex will not let you **create libraries** over the mounted bulk media share while the server is
+**unclaimed** — the 8.33 T NFS tree at `/media` is there, readable, and unusable. Claiming binds the
+server to a plex.tv account. It is a **one-shot** operation: the token is exchanged once for a durable
+`PlexOnlineToken` written into `Preferences.xml` on the `plex-config` Longhorn PVC, and from then on
+the `PLEX_CLAIM` env is **ignored on every start** (upstream gate:
+`[ ! -z "${PLEX_CLAIM}" ] && [ -z "${token}" ]` in `root/etc/cont-init.d/40-plex-first-run`).
+
+**A `https://plex.tv/claim` token lives ~4 minutes.** That is shorter than commit → push → Flux's
+10-minute reconcile interval → pod roll if you let those run on their own timers, which is why this
+page exists: every step below is written to be run back-to-back, with the reconcile **forced** rather
+than waited for. Budget: token in hand at T+0, pod restarted by roughly T+2:00.
+
+**An expired or wrong token does not break anything.** Verified against the image's own first-run
+script (and the copy inside `plexinc/pms-docker:latest`): a failed exchange gets a 403 with an empty
+body, `curl`'s status is never checked, the token `sed` yields empty, the `if` does not fire, there is
+no `else` and no `exit 1`, and the script (no `set -e`) exits 0. The pod comes up **unclaimed and
+serving**. So a burnt token costs you one more attempt, not an outage — and the repo deliberately
+ships a **placeholder** in that Secret for exactly this reason (the env `secretKeyRef` has no
+`optional`, so the Secret must always exist even when its value is useless).
+
+## Running the commands in this runbook
+
+`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
+container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that
+hang is not the cluster being down. Every command below runs as:
+
+    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
+
+Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
+picks a current-context per invocation, and that default has already changed once (k3s-dev until dev
+was decommissioned on 2026-09-13, k3s-prod now). Do not run two `dev.sh` invocations at once.
+
+Mutating `kubectl` and `flux reconcile` are blocked for agents here. **This whole page is operator
+work** — the reconcile and the pod roll below are deliberate operator actions, and the durable change
+is still a git change.
+
+## First commands
+
+**Before spending a token, find out whether the server is already claimed.** This prints only the
+attribute NAME, never its value — do not widen it to print the token:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c \
+     "grep -o PlexOnlineToken= \"/config/Library/Application Support/Plex Media Server/Preferences.xml\" | head -1"'
+```
+
+`PlexOnlineToken=` printed ⇒ **already claimed**; stop, there is nothing to do and no token to burn.
+Empty output ⇒ unclaimed, continue.
+
+Is the workload healthy enough to claim at all, and is the wave reconciling?
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get deploy,pod,pvc'
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0'
+```
+
+## The 4-minute procedure
+
+Have this page open and do not stop between steps.
+
+1. **T+0 — get the token.** Sign in to plex.tv as the account that should OWN this server, open
+   <https://plex.tv/claim>, copy the `claim-…` string. The clock starts now.
+
+2. **T+0:15 — put it in the Secret**, in place, encrypted, without it ever touching a plaintext file:
+
+   ```
+   infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
+   ```
+
+   Replace the `PLEX_CLAIM:` value (the placeholder `replace-me-with-a-fresh-plex-tv-claim-token`, or
+   a previous expired token) and save. `sops` re-encrypts on write to the committed age recipient.
+
+3. **T+0:45 — commit and push.** `git push origin main` **is** the deploy; there is no other apply
+   step.
+
+   ```
+   git add apps/wave-0/plex-claim-secret.sops.yaml
+   git commit -m 'chore(plex): fresh plex.tv claim token'
+   git push origin main
+   ```
+
+4. **T+1:00 — force the reconcile immediately. This is the one command that saves the window** — it
+   pulls the new commit and re-applies wave 0 instead of waiting up to 10 minutes for the interval:
+
+   ```
+   infra/devcontainer/dev.sh run bash -c \
+     'flux --context k3s-prod -n flux-system reconcile kustomization apps-wave-0 --with-source'
+   ```
+
+   `--with-source` is the load-bearing flag: it reconciles the `flux-system` GitRepository first, so
+   the Kustomization applies **your commit** and not the previously-fetched revision.
+
+5. **T+1:30 — roll the pod.** This step is **mandatory and easy to forget**: a changed Secret does
+   **not** restart anything. `PLEX_CLAIM` is injected by `secretKeyRef` and read only at container
+   start, and the Deployment's pod template is byte-identical across this change, so Flux applies the
+   new Secret and the running pod keeps the old value forever.
+
+   ```
+   infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n plex rollout restart deployment/plex'
+   ```
+
+6. **T+2:00 — confirm the exchange happened**, in the new pod's init log:
+
+   ```
+   infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n plex logs deploy/plex --since=5m | grep -i -e claim -e token'
+   ```
+
+   * `Attempting to obtain server token from claim token` **and** `Token obtained successfully`
+     ⇒ **claimed.** Re-run the First-commands `PlexOnlineToken=` check to confirm it persisted, then
+     create the libraries over `/media/movies`, `/media/tv`, `/media/photos`.
+   * `Attempting to obtain server token from claim token` with **no** success line ⇒ the token was
+     expired or invalid. Nothing is broken; go back to step 1 with a new token.
+   * **Neither line** ⇒ the exchange was skipped, which means `/config` already had a
+     `PlexOnlineToken` (already claimed) or the env did not reach the container. Check the pod actually
+     has the variable, by NAME only:
+
+     ```
+     infra/devcontainer/dev.sh run bash -c \
+       'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+          -o jsonpath="{.items[0].spec.containers[0].env[*].name}"'
+     ```
+
+## Likely causes, most likely first
+
+When the claim does not take, in order of how often it is the answer here:
+
+1. **The token expired** — you spent more than ~4 minutes, usually in step 3 or waiting on step 4.
+   Symptom: the "Attempting…" line with no success line. Remedy: a new token, and force the reconcile
+   rather than waiting for it.
+2. **The pod was never restarted** (step 5 skipped). Symptom: no claim log lines at all from a pod
+   whose `AGE` predates your push. The Secret is correct in git and in the cluster and the running
+   container still holds the placeholder.
+3. **Flux had not applied the commit yet** when you rolled the pod — step 5 run before step 4
+   finished, so the new pod read the OLD Secret. Check
+   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0 -o jsonpath='{.status.lastAppliedRevision}'`
+   against your commit, then roll again.
+4. **Already claimed** — the exchange is gated on `PlexOnlineToken` being absent from `/config`, so a
+   server claimed earlier ignores every later token silently. This is the benign case.
+5. **`plex.tv` unreachable.** The upstream `curl` has no `--max-time`/`--connect-timeout`, so a stalled
+   connection can leave the container sitting in cont-init with **nothing listening on 32400** — the
+   one genuine hang risk in this path. Symptom: pod Running but never Ready, log stops after
+   "Attempting…". Check egress (`plex` is `egress: internet`) rather than re-rolling repeatedly.
+6. **Wrong plex.tv account.** The claim succeeds and binds the server to whoever was signed in at
+   step 1. Unclaiming is a plex.tv-side operation on that account — see "escalate or stop".
+
+## When to escalate or stop
+
+- **Stop after two failed token attempts** and fix the timing, not the token. The third rushed attempt
+  is how people start editing live Secrets by hand, which leaves the cluster holding a value that is
+  in no git commit — precisely the drift `qbittorrent-bootstrap` had to be recovered from.
+- **Never `kubectl edit`/`patch` the `plex-claim` Secret in the cluster, and never
+  `kubectl create secret` it.** Everything durable goes through the SOPS file; a live-only Secret is
+  reverted (or adopted with the wrong value) on the next reconcile.
+- **Never `sops -d` this file into a terminal, a log, or a shared screen**, and never widen the
+  `PlexOnlineToken=` check into one that prints the value. The token is a plex.tv account credential.
+- **If the server got claimed to the wrong account, stop and get the operator.** Reversing it means
+  removing the server from that account on plex.tv and clearing `PlexOnlineToken` from
+  `Preferences.xml` on the PVC — a destructive edit of app state on the replicated tier, not a git
+  change.
+- **Do not delete the `plex-config` PVC to "start clean".** It holds the library database, metadata
+  and watch state (`policy: daily-offcluster`), and it is expensive to rebuild. The claim is a
+  two-line pref, not a reason to lose the DB.
+- After a successful claim, **leave the expired token in the SOPS file**. It is inert (the exchange is
+  skipped once claimed) and the Secret must keep existing, because the env `secretKeyRef` has no
+  `optional` — deleting it would put the pod into `CreateContainerConfigError`.
+
+---
+Source of truth: `runbooks/plex-claim.md`. Index: `/runbooks/`
+Upstream behaviour cited here is from `plexinc/pms-docker` branch `master`,
+`root/etc/cont-init.d/40-plex-first-run` (s6-overlay v2), cross-checked against the same file inside
+the running `plexinc/pms-docker:latest` image.

runbooks/vpn-egress-stale.md

@@ -0,0 +1,123 @@
+# vpn-egress-stale — VPN egress steering did not follow the tunnel
+
+**Alerts:** `VPNEgressSteeringStale` (critical, 15m) and `VPNEgressUplinkLabelMissing` (warning, 30m)
+**File:** `platform/monitoring/alerts.yaml`
+**Related:** `/runbooks/vpn-down` is for the tunnel being DOWN. This page is for the tunnel being **up**
+while the pods that must use it have no egress at all.
+
+## What this means here
+
+`egress-class=vpn` pods leave the cluster through one gateway node's `tun0`, selected by the
+`vpn-egress-steering` CiliumEgressGatewayPolicy. Cilium resolves that interface to an **ifindex** and
+caches it. When OpenVPN reconnects it destroys and re-creates `tun0`, which gets a **new ifindex** — and
+Cilium does not re-resolve on its own. The gateway node publishes
+`homelab.keithjackson.dev/egress-uplink=<tun0 address>-<ifindex>` from the OpenVPN up-script on every
+connect, and that node write is what makes Cilium re-resolve.
+
+If the label write does not happen, the cache keeps pointing at a destroyed interface. **`tun0` looks
+perfectly healthy the whole time** — it is up, it has an address, `VPNTunnelDown` stays quiet — while
+every vpn-class pod is silently dropped. That exact shape ran for roughly **80 minutes** undetected on
+the dev gateway, which is why these two alerts exist.
+
+`VPNEgressSteeringStale` compares how many times `tun0`'s ifindex changed in the last hour against how
+many distinct `egress-uplink` label values appeared in the same hour. More tunnel changes than label
+changes means a reconnect happened that the label did not follow.
+
+**`VPNEgressUplinkLabelMissing` is the watchdog for that comparison.** If the label is absent entirely,
+the staleness rule has nothing to compare against and is **inert** — present, evaluating, and incapable
+of firing. So the warning matters more than its severity suggests: it means your critical alert is
+switched off.
+
+Today's gateway is **`k3s-prod-3`** (`homelab.keithjackson.dev/egress-gateway=true`), and the policy has
+been in place since early September. It is a **singleton** — there is no second gateway to fail over to.
+
+## First commands
+
+Run cluster commands through the pinned toolbox, which supplies the kube context:
+
+```
+./infra/devcontainer/dev.sh run kubectl --context k3s-prod get nodes \
+  -L homelab.keithjackson.dev/egress-gateway -L homelab.keithjackson.dev/egress-uplink
+```
+
+You want exactly one node carrying `egress-gateway=true`, and that same node carrying a non-empty
+`egress-uplink` of the form `<address>-<ifindex>`.
+
+```
+# 2. what Cilium currently believes, on the gateway node itself
+ssh k3s-prod-3 'sudo cilium-dbg bpf egress list'
+ssh k3s-prod-3 'ip -o link show tun0'          # the REAL ifindex, first field
+```
+
+Compare the ifindex in the label and in `bpf egress list` against `ip link`. **A mismatch is the whole
+bug.**
+
+```
+# 3. is the re-steer mechanism itself alive?
+ssh k3s-prod-3 'systemctl status vpn-egress-resteer.service vpn-egress-resteer.timer --no-pager'
+ssh k3s-prod-3 'journalctl -u vpn-egress-resteer --since -2h --no-pager | tail -30'
+```
+
+```
+# 4. are vpn-class pods actually broken, or only the metric?
+./infra/devcontainer/dev.sh run kubectl --context k3s-prod get ciliumegressgatewaypolicies.cilium.io \
+  vpn-egress-steering -o yaml | head -40
+```
+
+Prove egress from a pod the policy selects rather than inferring it — the observable symptom is
+"outbound traffic silently disappears", so test outbound traffic:
+
+```
+./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n qbittorrent exec deploy/qbittorrent -- \
+  sh -c 'curl -s --max-time 10 https://ifconfig.me || echo EGRESS-FAILED'
+```
+
+That should return the **VPN provider's** address. The gateway VIP or a timeout both mean egress is
+broken; your own ISP address means the kill-switch is not holding and traffic is leaking, which is worse
+than a drop — see `/runbooks/vpn-down`.
+
+## Likely causes, most likely first
+
+1. **A tunnel reconnect that the label did not follow.** The common case, and what the alert is built to
+   catch. The OpenVPN up-script either did not run or could not write the node label (RBAC, expired
+   kubeconfig on the node, API unreachable at that moment). Check `journalctl -u vpn-egress-resteer` and
+   the up-script's own log around the reconnect time.
+2. **Cilium cached a dead ifindex despite the label changing.** The label write happened but the
+   re-resolution did not take. Forcing any node-label write makes Cilium re-resolve — that is the same
+   lever the up-script pulls, so it is a safe nudge, not a workaround:
+
+       ./infra/devcontainer/dev.sh run kubectl --context k3s-prod annotate node k3s-prod-3 \
+         egress-resteer-nudge="$(date -u +%s)" --overwrite
+
+   Re-applying the CiliumEgressGatewayPolicy has the same effect. Re-check `bpf egress list` after.
+3. **`metricLabelsAllowlist` lost the label** (this fires `VPNEgressUplinkLabelMissing`, not the stale
+   alert). kube-state-metrics exposes **no** custom node labels by default; the allowlist entry in
+   `platform/monitoring/kube-prometheus-stack.yaml` is what publishes them. If that entry is edited or
+   dropped, `kube_node_labels` carries no `label_homelab_keithjackson_dev_egress_uplink` and the critical
+   alert goes quiet without anything looking broken. Confirm what is actually exported:
+
+       ./infra/devcontainer/dev.sh run kubectl --context k3s-prod -n monitoring exec \
+         prometheus-kube-prometheus-stack-prometheus-0 -c prometheus -- \
+         /bin/promtool query instant http://localhost:9090 'kube_node_labels'
+
+   Note that KSM does **not** restart on a ConfigMap-only change; the deployment carries no checksum
+   annotation over it, so a `rollout restart -n monitoring deploy/kube-prometheus-stack-kube-state-metrics`
+   is required after editing the allowlist.
+4. **The gateway node was replaced or relabelled.** If `egress-gateway=true` moved, the up-script must be
+   installed on the new node (`infra/ansible/roles/vpn_egress_gateway`). A gateway label with no
+   up-script behind it produces exactly this alert on the next reconnect.
+
+## When to escalate or stop
+
+- **Stop and do not "fix" it by deleting the policy.** Removing `vpn-egress-steering` does not restore
+  egress; it sends vpn-class traffic out of your **own** ISP connection, which is the leak the class
+  exists to prevent. A broken tunnel that drops traffic is the intended failure mode.
+- **Escalate to a maintenance window** if the ifindex mismatch returns after every reconnect. That is the
+  up-script or its RBAC, not a transient, and chasing it live means repeatedly breaking egress for the
+  vpn-class workloads.
+- **This is a singleton gateway.** There is no failover. If `k3s-prod-3` needs to be drained while this is
+  unresolved, accept that vpn-class egress is down for the duration rather than improvising a second
+  gateway under time pressure.
+- If `VPNEgressUplinkLabelMissing` is firing and you cannot restore the label quickly, say so explicitly
+  in the incident notes: for as long as it is missing, **`VPNEgressSteeringStale` is not watching
+  anything**, so the absence of that critical alert proves nothing.

schemas/renderer-contract.md

@@ -53,6 +53,41 @@ platform must render, and cites the requirement each mapping satisfies. Changing
   ExternalDNS record for `<name>.home.keithjackson.dev`.
 - `internal` renders **no public DNS record** and is never port-forwarded (PiHole resolves it to the
   internal Gateway VIP).
+- **HOW "no public DNS record" IS MADE TRUE, since it was prose only until 2026-09-14.** Every
+  HTTPRoute rendered for an `internal` service carries ExternalDNS's own controller annotation with
+  a value other than the one ExternalDNS claims:
+
+      external-dns.alpha.kubernetes.io/controller: none
+
+  ExternalDNS v0.21.0 (`source/annotations/processors.go`) skips any source object whose
+  `controller` annotation is present and `!= "dns-controller"`, and `source/gateway.go` applies that
+  check **inside `Endpoints()` before any hostname is resolved** — so the route never becomes an
+  endpoint for any name it declares. `source/service.go` applies the identical check, which covers
+  the second configured source. It is derived **from `exposure` and nothing else**
+  (`catalog/render.py` `_externaldns_route_annotations`), so the operator's one declaration decides,
+  a service that OMITS `exposure` inherits the `internal` default and is likewise unpublishable, and
+  there is no per-service list to forget to extend. `ci/render-golden.sh` proves `apps/` reproduces
+  from `catalog/`, so an `internal` service cannot exist in the applied tree without it, and
+  `conformance/tests/l0-static/internal-services-publish-no-public-dns` re-proves both directions
+  (present on internal, **absent** on public) on every gate run.
+  - The route itself, the wildcard TLS and LAN reachability are **unchanged** — only the public
+    record is withdrawn. The internal split-horizon answer is Pi-hole's job (R-DNS-03), which is why
+    withdrawing these records requires Pi-hole to be answering the LAN authoritatively first: the
+    zone has no `*.home.keithjackson.dev` wildcard, and `*.keithjackson.dev` does not match two
+    labels, so a withdrawn name is a public NXDOMAIN with no fallback.
+  - Why not a cluster-wide ExternalDNS opt-in filter (`labelFilter`, which v0.21.0 supports for
+    both configured sources): seven published names come from hand-authored `platform/`/`apps/`
+    manifests that carry no `exposure` field for a renderer to read, and an opt-in filter deletes
+    all seven the moment it is set. Per-resource exclusion has exactly the blast radius of the
+    defect. The filter stays available as hardening once those seven are adjudicated — see
+    `platform/externaldns/helmrelease.yaml` and the register in the L0 check.
+- `public` renders the record but the renderer does **not** yet set a target: the record still
+  resolves to the shared Gateway VIP, i.e. an RFC1918 address, so a `public` service is not in fact
+  publicly reachable. Fixing that is **not** a renderer change on v0.21.0 —
+  `source/gateway.go` reads `external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and
+  a ListenerSet), never from an HTTPRoute, so a per-service target override is not expressible while
+  the public services share the `shared` Gateway. The open decision is recorded in
+  `platform/externaldns/helmrelease.yaml`.
 - Exposure does **not** relax east-west default-deny (docs/09 §"Interaction with default-deny").
 - **Requirements:** R-DNS-02/03 (docs/09), R-NET-04 (docs/03).
 
@@ -131,8 +166,12 @@ platform must render, and cites the requirement each mapping satisfies. Changing
   (Let's Encrypt via **DNS-01**, scoped Cloudflare token), so no per-service cert step — trusted,
   non-self-signed, auto-renewed, including for `internal` services. Per-service `Certificate` only
   where isolation warrants.
-- DNS: **ExternalDNS** watches the HTTPRoute and manages the `<host>` record (public IP for `public`,
-  internal Gateway VIP via PiHole split-horizon for `internal`).
+- DNS: **ExternalDNS** watches the HTTPRoute and manages the `<host>` record — but **only for a
+  `public` service**. An `internal` service's route is rendered with ExternalDNS's controller
+  annotation set to `none`, which removes it from ExternalDNS's view entirely (see §`exposure`), so
+  its `<host>` exists **only** in the internal split-horizon answer PiHole serves, never in the
+  public zone. A `public` record's TARGET is still the Gateway VIP, not a routable address — open
+  operator decision, see §`exposure`.
 - The `host` pattern is enforced to `<label>.home.keithjackson.dev` by the schema.
 - **Requirements:** **R-TLS-01/02/03** (docs/08), **R-DNS-01/02/03** (docs/09).
 
@@ -242,6 +281,11 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 6. **irreplaceable ⇒ off-cluster backup:** DR-critical services must not carry `backup.policy:
    reacquirable` (R-STATE-03). *(Advisory: not enforced by the schema — an L0 policy check should
    assert it; see "Ambiguities / escalations" in schemas/README.md.)*
+7. **`exposure: internal` ⇒ no public DNS record, structurally.** Every HTTPRoute rendered for an
+   `internal` service (including one that omits `exposure` and takes the default) carries
+   `external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
+   Derived from `exposure` alone; enforced on every gate run by
+   `conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
 
 ## Requirement traceability (contract-level)
 Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02

schemas/servicedefinition.schema.json

@@ -2,7 +2,7 @@
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
   "title": "ServiceDefinition",
-  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 / R-IDP-02).",
+  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked prod's GitOps when its GitRepository moved to the in-cluster Gitea Service. Renders one least-privilege ingress CiliumNetworkPolicy per entry — the named namespace and the declared port, nothing wider). 1.4.0 (makes the `bulk` tier REAL: a bulk storage entry now carries `export` (the absolute NFS export path on the ${BULK_NFS_SERVER} host) and renders a STATIC, pre-bound NFS PersistentVolume + PVC over data that already exists outside the cluster, plus optional `subPath` and `readOnly` on any entry. Resolves the OQ-2 bulk-backing stub for STATIC pre-existing datasets; dynamic bulk provisioning remains unimplemented and unneeded). 1.3.0 added optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 / R-IDP-02).",
   "type": "object",
   "additionalProperties": false,
   "required": ["name", "source"],