merge: the substitution check now catches a collapsed doubled dollar

The existing check audited only the inverse — a ${...} undefined or unescaped — so it passed the manifest that broke prod: Postgres dollar-quoting under a postBuild.substituteFrom path, which envsubst halves, delivering 'DO $' and a syntax error on every reconciled run (fixed 16b7fcd). Measured rather than assumed: Flux 2.4.0 embeds fluxcd/pkg/envsubst (a fork of drone/envsubst — the brief credited the wrong module), and --strict output is BYTE-IDENTICAL to non-strict here, so strict mode is blind to this class entirely. The decisive contrast is two commands on the same stream: kubectl kustomize emits valid 'DO $$' — which is what the kubectl-apply verification saw — while the same stream through flux envsubst emits 'DO $'. 357 assertions, 0 unjustified. Legitimate uses genuinely exist (18 ${} escapes plus two reaching shell comments), so the allowlist is a reviewed opt-in rather than a bypass: a stale entry fails, a wildcard is matched verbatim and so fails as stale, a short context or reason is rejected, and a genuine entry covers only the span it matches — a second occurrence in the same file still fails. Strongest proof: restoring the real pre-16b7fcd manifest verbatim makes the guard reject the exact commit that broke production.
merge: Longhorn and cert-manager metrics exist now — 13 alerts were inert
2026-09-13 16:27:51 -05:00 · 2026-09-13 16:26:12 -05:00 · 2026-09-13 16:24:42 -05:00 · 2026-09-13 16:21:44 -05:00 · 2026-09-13 16:09:37 -05:00
13 changed files with 1443 additions and 9 deletions
@@ -0,0 +1,119 @@
+# =============================================================================
+# L0 STATIC — every metric an alert rule queries is CLAIMED by a scrape target this repo configures
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# THE DEFECT THIS CLOSES. An alert rule whose metric is never scraped is not a broken alert — it is
+# an ABSENT alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB with no
+# longhorn_* series returns no vector, so the rule is permanently quiet, permanently "healthy", and
+# indistinguishable from a correct rule on a healthy cluster. It survives review (the PromQL is
+# right), it survives `promtool test rules` (the unit test supplies the series itself), and it
+# survives every dashboard (nobody looks for a panel that was never built).
+#
+# MEASURED ON PROD, 2026-09-13, before this change — 13 rules across three families were inert:
+#   count(longhorn_volume_robustness)                        => (empty)   4 rules
+#   count({__name__=~"certmanager.*"})                       => (empty)   3 rules
+#   count(hubble_drop_total)                                 => (empty)   1 rule
+#   count by (job) (up)                                      => 20 jobs, none Longhorn/cert-manager/Hubble
+# So storage health was unmonitored on a cluster whose prod-1 had just had a failing SSD
+# (DECISIONS-LOG), TLS expiry was unmonitored with the live `wildcard-home` wildcard 89 days from
+# expiry, and the platform had no policy-drop visibility at all. The Flux family (gotk_*, 5 rules)
+# was the same defect and was fixed separately in a79693d.
+#
+# GIVEN every alerting rule in every PrometheusRule in the repo (platform/**, apps/wave-*),
+# WHEN  each rule's PromQL is parsed for the metric names it selects,
+# THEN  (1) every metric name is CLAIMED by exactly one declared scrape owner, and each owner's
+#           claim is PROVEN AGAINST THE REPO — the file that enables that scrape must still contain
+#           the enabling declaration, so deleting `metrics.serviceMonitor.enabled` from
+#           platform/longhorn/helmrelease.yaml fails this check instead of silently re-inerting four
+#           storage alerts;
+#       (2) an owner's NEGATIVE conditions hold too, where the enablement can be defeated without
+#           being deleted — cert-manager's chart calls `fail` if podmonitor and servicemonitor are
+#           both set, Longhorn's own NetworkPolicies drop the scrape unless
+#           networkpolicy-metrics.yaml is wired, and kube_node_labels does not exist at all unless
+#           kube-state-metrics' metricLabelsAllowlist names the label;
+#       (3) a metric with NO owner FAILS, unless it is declared in KNOWN_UNWIRED_METRICS — a
+#           shrink-only register (hubble_drop_total today, with the CNI-rollout reason recorded);
+#       (4) `up{job="X"}` is audited on its own axis, because `up` always exists and so the metric
+#           check above can never see this failure: where the repo authors a Service and a
+#           ServiceMonitor in the SAME file, the monitor's endpoint port must be a port that Service
+#           actually defines. All 12 apps/* `<app>-down` SLI rules fail this today (the monitors name
+#           port `metrics`; no app Service defines one), so they are registered in
+#           KNOWN_UNWIRED_JOBS with the live proof that their `up{job=...}` series do not exist;
+#   AND (5) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`,
+#           so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate
+#           invocation, and ci/gate.sh blocks the merge when the l0 stage fails.
+#
+# WHY BOTH REGISTERS ARE RATCHETS AND NOT BLANKET FAILURES: the same reasoning as
+# alert-rules-unit-tested. Failing outright on the 12 app SLI rules would make the gate red until
+# exporters exist for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr and friends —
+# none of which speak Prometheus natively — so the gate would be disabled, and a disabled gate
+# protects nothing. Merely printing the gap is how these 13 rules stayed inert for three phases.
+# So each gap is DECLARED, counted on every run, and CANNOT GROW: a new alert on an unclaimed
+# metric FAILS, a new Service+ServiceMonitor pair with a mismatched port FAILS, and a STALE
+# register entry (the gap was closed, or the alert deleted) FAILS too, so neither register can rot
+# into fiction.
+#
+# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, the same dependency as
+# catalog/render.py and dev-nodes-provisioned-as-code); conformance/run.sh --layer l0 and ci/l0.sh
+# run it directly, and this Chainsaw Test wraps the SAME script.
+#
+# NOT proven (all named in check.sh's header, which carries the full list). The one worth stating
+# here because it bit this very repo: an owner claims metrics by PREFIX, so this proves the SCRAPE
+# TARGET exists, never that the exporter emits that exact SERIES NAME.
+# `kubevirt_vmi_phase_count{phase="Failed"}` sat inside the scraped kubevirt_ family on an UP target
+# and did not exist at all on the deployed KubeVirt (corrected here to
+# `kubevirt_vmi_info{phase="failed"}`, lowercase — a capital-F matcher is equally inert). Catching
+# that needs the exporter's own metric list, i.e. a live read; it belongs to a future L1/L3 check
+# that diffs alert metrics against /api/v1/label/__name__/values.
+#
+# Also not proven: that a target is UP. This is static — it asserts the
+# enabling declaration is in git, never that the endpoint answers, that the chart still honours the
+# value key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
+# recorded in the commit message (longhorn_volume_robustness 44 series, certmanager_* 104 series,
+# all six new targets up==1). It also cannot check a monitor whose Service comes from a Helm chart
+# rather than from this repo — the port name lives outside the tree — which is why (4) is scoped to
+# same-file Service+ServiceMonitor pairs. And it does not judge thresholds or reachability-to-human
+# (R-OBS-02 / the L3 alerting test).
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: alert-metrics-have-scrape-targets
+  labels:
+    conformance.home.k3s/R-OBS-04: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-OBS-04"
+    conformance.home.k3s/layer: "L0"
+    # WHOLE-REPO SCOPE — this invariant quantifies over every PrometheusRule in the tree AND over
+    # every scrape-enabling declaration that feeds them, which live in eight different components
+    # (platform/monitoring, platform/longhorn, platform/cert-manager, platform/velero,
+    # platform/kubevirt, platform/keycloak, platform/cnpg, apps/wave-*). No single requirement
+    # path-maps to that set.
+    #
+    # WHY IT MUST NOT BE REQUIREMENT-SCOPED: ci/touched-requirements.sh maps apps/* to NOTHING and
+    # platform/monitoring/* to R-OBS-01/R-OBS-02 — not R-OBS-04. A change that adds an alert under
+    # apps/ or platform/kubevirt/, or one that flips a chart's `serviceMonitor.enabled` back to
+    # false under platform/longhorn/, therefore attributes a requirement set that would FILTER THIS
+    # CHECK OUT of exactly the change that re-inerts the alerts. Declaring the scope makes
+    # ci/l0.sh Stage 3b union it into every run instead, and makes an honest SKIP loud (rc 3 =
+    # UNWIRED, never green) rather than silently green.
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every metric name queried by every alert rule is claimed by a scrape target whose enabling
+      declaration is still present in this repo, every `up{job=...}` selector names a job a repo
+      monitor can actually produce, and the two gaps that remain are enumerated in registers that
+      can only shrink — so an alert can never again be authored, reviewed and merged against a
+      metric nothing scrapes.
+spec:
+  steps:
+    - name: audit-every-alert-metric-against-the-configured-scrape-targets
+      description: >-
+        Parse every metric name out of every alerting rule, resolve each against the declared
+        scrape-owner table, prove each owner's enabling declaration (and its negative conditions)
+        still hold in git, audit `up{job=...}` against same-file Service/ServiceMonitor port
+        agreement, and ratchet both registers of declared gaps.
+      try:
+        - script:
+            content: bash ./check.sh
@@ -0,0 +1,702 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — alert-metrics-have-scrape-targets
+# Serves: R-OBS-04 ("The platform alerts on its own health (nodes, storage, GitOps, certs,
+#         VPN)." — spec docs/10-observability-and-alerting.md)
+#
+# WHY THIS EXISTS — AN ALERT ON AN UNSCRAPED METRIC IS AN ABSENT ALERT THAT READS AS A PRESENT ONE
+# -----------------------------------------------------------------------------------------------
+# `longhorn_volume_robustness == 2` evaluated against a TSDB that holds no longhorn_* series
+# returns no vector. The rule is therefore permanently quiet, permanently "healthy", and
+# INDISTINGUISHABLE from a correct rule on a healthy cluster. It passes review (the PromQL is
+# right). It passes `promtool test rules` (the unit test supplies the series itself, so a unit
+# test proves nothing about whether anything scrapes them). It shows up on no dashboard, because
+# nobody looks for a panel that was never built.
+#
+# Measured on prod 2026-09-13, BEFORE the change that added this check — 13 rules were inert:
+#   count(longhorn_volume_robustness)            => (empty)    LonghornVolumeDegraded/Faulted/
+#                                                              CapacityHigh, LonghornNodeStorageHigh
+#   count({__name__=~"certmanager.*"})           => (empty)    CertificateExpiringSoon/
+#                                                              ExpiryImminent/NotReady
+#   count(hubble_drop_total)                     => (empty)    HubbleDeniedFlowSpike
+#   count by (job) (up)                          => 20 jobs, none Longhorn, cert-manager or Hubble
+# Storage health was unmonitored on the cluster whose prod-1 had just had a failing SSD; TLS
+# expiry was unmonitored with the live `wildcard-home` wildcard 89 days out; there was no
+# policy-drop visibility at all. The gotk_* family (5 more rules) was the same defect, fixed
+# separately in a79693d.
+#
+# WHAT MAKES THIS CLASS DIFFERENT from the two failure modes alert-rules-unit-tested catches: a
+# permanently-firing rule is loud, and a rule that behaves wrongly against synthetic series is
+# caught by promtool. This one is caught by NEITHER, because the missing thing is not in the rule
+# file at all — it is a `serviceMonitor.enabled` that was never set in a completely different
+# component's HelmRelease. So the audit has to cross from the alert to the scrape config, which is
+# what the owner table below does.
+#
+# GIVEN every alerting rule in every PrometheusRule in the repo,
+# WHEN  each rule's PromQL is parsed for the metric names it selects,
+# THEN  (1) every metric name is CLAIMED by exactly one declared owner in SCRAPE_OWNERS, and the
+#           owner's `require` regexes still match its file — so deleting the enablement from
+#           platform/longhorn/helmrelease.yaml FAILS here instead of silently re-inerting four
+#           storage alerts;
+#       (2) each owner's `forbid` regexes do NOT match, for the cases where the enablement can be
+#           defeated without being deleted (cert-manager's chart `fail`s if podmonitor and
+#           servicemonitor are both enabled; an `enabled: false` inside the exporter's own block);
+#       (3) a metric with no owner FAILS unless declared in KNOWN_UNWIRED_METRICS (shrink-only);
+#       (4) `up{job="X"}` is audited separately — `up` itself always exists, so (1) is blind to
+#           this — by requiring that where this repo authors a Service and a ServiceMonitor in the
+#           SAME file, the monitor's endpoint port is a port that Service actually defines;
+#           violations must be declared in KNOWN_UNWIRED_JOBS (shrink-only);
+#   AND (5) the check is LOAD-BEARING: this test declares `conformance.home.k3s/scope: whole-repo`
+#           so ci/l0.sh unions it into every gate run, and ci/gate.sh blocks on an l0 failure.
+#
+# WHY THE REGISTERS ARE RATCHETS (the decision and its reasons) — the same three options as
+# alert-rules-unit-tested faced:
+#   * FAIL outright on all 13: correct in principle, but 12 of them are app SLI rules that need
+#     Prometheus exporters for Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr and Seerr —
+#     none of which speak Prometheus natively. The gate would be red for every change until that
+#     work lands, so it would get disabled, and a disabled gate protects nothing.
+#   * merely REPORT: that is precisely how these rules stayed inert across phases 4-6.
+#   * RATCHET (chosen): each gap is declared here, printed with counts on every run, and CANNOT
+#     GROW. A new alert on an unclaimed metric FAILS. A new same-file Service+ServiceMonitor pair
+#     with a mismatched port FAILS. A STALE entry — the gap was closed, or the alert deleted —
+#     also FAILS, so neither register can rot into fiction.
+#
+# WHAT THIS DOES *NOT* PROVE (honest gaps — this is a STATIC check):
+#   - It does not prove a target is UP, that an endpoint answers, that the chart still honours the
+#     values key, or that a NetworkPolicy is sufficient. Those were established by live prod reads
+#     recorded in the commit message: with the enablements applied, longhorn_volume_robustness
+#     returned 44 series, {__name__=~"certmanager.*"} returned 104, and all six new targets read
+#     up==1. A live probe needs a cluster, which L0 by definition does not have.
+#   - It cannot audit a ServiceMonitor whose Service is created by a Helm chart rather than by this
+#     repo: the port name lives outside the tree. That is why (4) is scoped to same-file
+#     Service+ServiceMonitor pairs — the apps/* ServiceDefinition pattern — and why the
+#     platform/* monitors (oauth2-proxy, kubevirt, keycloak, flux) are out of its reach. Their
+#     liveness is covered by the fact that their metrics have owners under (1).
+#   - IT PROVES THE SCRAPE TARGET, NOT THE SERIES NAME. An owner claims a metric by PREFIX, so a
+#     metric in a scraped family whose exact name the exporter does not emit still passes. This is
+#     a real residual hole and it was live in this repo: `kubevirt_vmi_phase_count{phase="Failed"}`
+#     (KubeVirtVMLaunchFailing) sat inside the scraped `kubevirt_` family on an UP target, and the
+#     series simply does not exist on the deployed KubeVirt — verified on prod 2026-09-13, where the
+#     only kubevirt_vmi_* names are `info` and `number_of_outdated`, and the only phase-bearing
+#     names are the phase_transition_time histograms. Corrected in the same change to
+#     `kubevirt_vmi_info{phase="failed"}` (note the LOWERCASE value KubeVirt writes: a capital-F
+#     matcher selects nothing and is just as inert). Catching that class needs the exporter's own
+#     metric list, which is a live read, not a static one — it belongs to a future L1/L3 check that
+#     diffs every alert metric against `/api/v1/label/__name__/values`. The same limit applies to
+#     every LABEL VALUE an alert matches on.
+#   - The metric-name parser is a PromQL approximation, not a parser: it strips string literals,
+#     label-matcher bodies and aggregation label lists, then treats remaining bare identifiers as
+#     metric selectors. A metric referenced ONLY through a `{__name__=~"..."}` regex selector would
+#     be missed. No rule in this repo does that, and assertion (0) fails if the parser stops
+#     finding metrics at all.
+#   - It does not judge thresholds, and does not prove an alert reaches a human (R-OBS-02 and the
+#     L3 alerting test own that).
+#   - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
+#
+# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, and because this test
+# declares the whole-repo scope the runner turns it into rc 3 = UNWIRED, never green) only if the
+# repo contains no PrometheusRule at all — i.e. there is genuinely nothing to audit.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+PYTHON="${PYTHON:-python3}"
+
+# Honest SKIP (never green for a scope-declaring test) if there is nothing to audit at all.
+if ! grep -rl 'kind: PrometheusRule' "${ROOT}/platform" "${ROOT}/apps" >/dev/null 2>&1; then
+  echo "check SKIPPED: no PrometheusRule found under platform/ or apps/ — no alert rules to audit" >&2
+  exit 3
+fi
+
+cd "${ROOT}"
+"${PYTHON}" - "${ROOT}" <<'PY'
+"""Static audit: every alert rule's metrics are claimed by a scrape target this repo configures.
+
+PyYAML is used deliberately rather than the line-based parsing the neighbouring checks favour:
+alert `expr:` values are multi-line block scalars, and a line-based reader of PromQL would be the
+kind of approximate parser that silently stops matching — the exact failure class this check
+exists to prevent. Same dependency as catalog/render.py and dev-nodes-provisioned-as-code.
+"""
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:                                  # pragma: no cover
+    print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
+          file=sys.stderr)
+    raise SystemExit(1)
+
+ROOT = Path(sys.argv[1])
+sys.dont_write_bytecode = True          # keep __pycache__ out of the tracked tree
+
+FAILURES: list[str] = []
+CHECKS = 0
+
+
+def check(cond, msg):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(msg)
+
+
+def read(p: Path) -> str:
+    try:
+        return p.read_text()
+    except (OSError, UnicodeDecodeError):
+        return ""
+
+
+# =============================================================================================
+# THE DECLARED SCRAPE-OWNER TABLE (assertions 1 and 2).
+#
+# Each entry claims a set of metric names (exact names and/or prefixes) and names the file whose
+# declaration is what makes those metrics exist in the TSDB, plus the regexes that prove it. This
+# is the crossing from "an alert queries X" to "something in this repo scrapes X", and it is the
+# part that makes the check a REGRESSION guard rather than a one-time audit: every `require`
+# regex is re-evaluated against the working tree on every gate run.
+#
+# `require`  — ALL must match the file. The enablement itself.
+# `forbid`   — NONE may match. For enablements that can be defeated without being deleted.
+#
+# Longest prefix wins, so kube_velero_schedule_ is resolved before kube_.
+# =============================================================================================
+KPS = "platform/monitoring/kube-prometheus-stack.yaml"
+
+SCRAPE_OWNERS: list[dict] = [
+    {
+        "owner": "kube-prometheus-stack: bundled node-exporter DaemonSet + its ServiceMonitor",
+        "prefixes": ("node_",),
+        "file": KPS,
+        # The subchart block must be present and configured here (it is where the exporter's
+        # resources live). The chart default is enabled:true, so the regression to guard is a
+        # DISABLE, not a missing enable.
+        "require": [r"^\s*prometheus-node-exporter:\s*$"],
+        "forbid": [r"prometheus-node-exporter:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
+        "why": "node_* powers all seven platform-health.nodes rules including the two IO-pressure "
+               "rules that exist because of prod-1's etcd WAL starvation",
+    },
+    {
+        "owner": "kube-prometheus-stack: the kubelet ServiceMonitor (cAdvisor + volume stats)",
+        "prefixes": ("kubelet_",),
+        "file": KPS,
+        # Chart default kubelet.enabled:true. There is no explicit enable to require, so the only
+        # honest assertion is that it has not been turned off the way kubeProxy/kubeEtcd were.
+        "require": [r"kubeProxy:"],   # sentinel: the exporter-toggle block is still authored here
+        "forbid": [r"\n\s*kubelet:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
+        "why": "kubelet_volume_stats_* powers MonitoringVolumeFillingUp and all three "
+               "pvc-capacity-alerts rules",
+    },
+    {
+        "owner": "kube-prometheus-stack: kube-state-metrics custom-resource-state for Velero Schedules",
+        "prefixes": ("kube_velero_schedule_",),
+        "file": KPS,
+        "require": [
+            r"customResourceState:\s*\n\s*enabled:\s*true",
+            r"metricNamePrefix:\s*kube_velero_schedule",
+            # The RBAC is MANDATORY, not cosmetic: without velero.io read access KSM emits the
+            # custom-resource-state metrics as an EMPTY SET and logs a forbidden error — silent,
+            # and the inert-alert failure mode all over again (measured; see the block's header).
+            r'apiGroups:\s*\["velero\.io"\]',
+            r'resources:\s*\["schedules"\]',
+        ],
+        "forbid": [r"customResourceState:\s*\n\s*enabled:\s*false"],
+        "why": "kube_velero_schedule_* is the only thing that can see a Schedule that has NEVER "
+               "backed up (VeleroScheduleNeverBackedUp / SchedulePaused / NoVolumeDataBackup); "
+               "Velero itself publishes nothing until a backup is attempted",
+    },
+    {
+        "owner": "kube-prometheus-stack: bundled kube-state-metrics",
+        "prefixes": ("kube_",),
+        "file": KPS,
+        "require": [r"^\s*kube-state-metrics:\s*$"],
+        "forbid": [r"kube-state-metrics:\s*\n(?:\s*#[^\n]*\n)*\s*enabled:\s*false"],
+        # kube_node_labels is a SPECIAL CASE and is checked on its own below: KSM exposes no
+        # custom node labels by default, so the metric does not exist unless metricLabelsAllowlist
+        # names the label. The prefix claim alone would pass while VPNTunnelDown's join matched
+        # nothing.
+        "why": "kube_* powers the Keycloak, KubeVirt-crashloop, CNPG-colocation and Velero-schedule "
+               "rules",
+    },
+    {
+        "owner": "kube-state-metrics metricLabelsAllowlist (the egress-gateway node label)",
+        "exact": ("kube_node_labels",),
+        "file": KPS,
+        "require": [
+            r"metricLabelsAllowlist:",
+            r"nodes=\[homelab\.keithjackson\.dev/egress-gateway\]",
+        ],
+        "forbid": [],
+        "why": "KSM exposes NO custom labels by default. Without the allowlist entry "
+               "kube_node_labels carries no egress-gateway label and VPNTunnelDown's join silently "
+               "matches nothing — a rule that is present, evaluating, and structurally inert",
+    },
+    {
+        "owner": "CNPG: enablePodMonitor on the Cluster CRs (the operator renders one PodMonitor each)",
+        "prefixes": ("cnpg_",),
+        "file": "platform/keycloak/postgres-cluster.yaml",
+        "require": [r"^\s*enablePodMonitor:\s*true\s*$"],
+        "forbid": [],
+        "why": "cnpg_* powers all six platform-health.cnpg rules, including the WAL-archive backlog "
+               "and no-recoverability-point rules that guard R-STATE-03",
+    },
+    {
+        "owner": "Flux controllers: platform/monitoring/flux-podmonitor.yaml (merged a79693d)",
+        "prefixes": ("gotk_",),
+        "file": "platform/monitoring/flux-podmonitor.yaml",
+        "require": [r"kind:\s*PodMonitor", r"port:\s*http-prom"],
+        "forbid": [],
+        "why": "gotk_* powers FluxReconciliationFailure, FluxResourceSuspended and the three "
+               "automated-update rules — on a platform whose entire delivery model is Flux",
+    },
+    {
+        "owner": "cert-manager: prometheus.servicemonitor.enabled in the HelmRelease",
+        "prefixes": ("certmanager_",),
+        "file": "platform/cert-manager/helmrelease.yaml",
+        "require": [r"prometheus:\s*\n\s*servicemonitor:\s*\n\s*enabled:\s*true"],
+        # The chart calls `fail` when both monitor kinds are enabled, which would break the whole
+        # HelmRelease rather than just the scrape. Measured against v1.21.2.
+        "forbid": [r"podmonitor:\s*\n\s*enabled:\s*true"],
+        "why": "certmanager_* powers CertificateExpiringSoon / ExpiryImminent / NotReady — the only "
+               "warning that an ACME DNS-01 renewal of the live wildcard has quietly failed",
+    },
+    {
+        "owner": "Longhorn: metrics.serviceMonitor.enabled in the HelmRelease",
+        "prefixes": ("longhorn_",),
+        "file": "platform/longhorn/helmrelease.yaml",
+        "require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
+        "forbid": [],
+        "why": "longhorn_* powers the four storage rules on a cluster whose recurring failure mode "
+               "is a disk",
+        # The ServiceMonitor ALONE is not sufficient here — see the companion assertion below.
+        "companion": {
+            "file": "platform/longhorn/kustomization.yaml",
+            "require": [r"^\s*-\s*networkpolicy-metrics\.yaml\s*$"],
+            "why": "Longhorn 1.12.1 ships its own effectively-unconditional NetworkPolicies "
+                   "(guarded on networkPolicies.restrictInternalTraffic, default true — NOT on "
+                   "networkPolicies.enabled) whose longhorn-manager policy admits only other "
+                   "Longhorn components in the same namespace. MEASURED on prod: with the "
+                   "ServiceMonitor applied and no allowance, all three targets were DISCOVERED and "
+                   "every scrape failed with 'context deadline exceeded' after 10.0s — a drop, not "
+                   "a refusal — and the TSDB stayed empty. So without networkpolicy-metrics.yaml "
+                   "the four storage alerts are inert even though the monitor exists",
+        },
+    },
+    {
+        "owner": "KubeVirt/CDI: platform/kubevirt/servicemonitor.yaml",
+        "prefixes": ("kubevirt_",),
+        "file": "platform/kubevirt/servicemonitor.yaml",
+        "require": [r"kind:\s*ServiceMonitor", r"port:\s*metrics"],
+        "forbid": [],
+        "why": "kubevirt_* powers the four platform-health.kubevirt rules (note the CDI rules use "
+               "the kubevirt_cdi_* names, not bare cdi_*)",
+    },
+    {
+        "owner": "Velero: metrics.serviceMonitor.enabled in the HelmRelease",
+        "prefixes": ("velero_",),
+        "file": "platform/velero/helmrelease.yaml",
+        "require": [r"metrics:\s*\n\s*serviceMonitor:\s*\n\s*enabled:\s*true"],
+        "forbid": [],
+        "why": "velero_* powers VeleroBackupFailure / PartialFailure / NoRecentBackup — the rules "
+               "that failed to notice prod running with zero volume backups",
+    },
+]
+
+# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
+# is structurally blind to `up{job="X"}` naming a job nothing produces. That axis is assertion (4).
+UNIVERSAL = {"up"}
+
+# =============================================================================================
+# REGISTER 1 — KNOWN_UNWIRED_METRICS. Shrink-only. A metric here is queried by an alert and
+# claimed by NOTHING, so that alert is inert and the platform is blind to what it describes.
+# Removing an entry means wiring the scrape. An entry that stops being referenced by any alert,
+# or that gains an owner, FAILS as stale.
+# =============================================================================================
+KNOWN_UNWIRED_METRICS: dict[str, str] = {
+    "hubble_drop_total":
+        "HubbleDeniedFlowSpike — the platform has NO policy-drop / NetworkPolicy-regression "
+        "visibility. Unlike Longhorn and cert-manager (whose charts already serve /metrics and "
+        "only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is switched on "
+        "in Cilium's own config: platform/cilium/helmrelease.yaml sets hubble.{enabled,relay,ui} "
+        "but not hubble.metrics.enabled, and not prometheus.enabled for the agent. Turning them on "
+        "OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all three nodes "
+        "— the single most dangerous rollout on this cluster (the 2026-09-01 k8sServiceHost outage; "
+        "cilium was moved to its own Flux Kustomization precisely to isolate that blast radius) — "
+        "and requires a matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same "
+        "commit or a rebuilt node's bootstrap CNI config silently disagrees with the reconciled "
+        "one. Deliberately DEFERRED to its own change with a one-node canary; see the rule's "
+        "header in platform/monitoring/alerts.yaml.",
+}
+
+# =============================================================================================
+# REGISTER 2 — KNOWN_UNWIRED_JOBS. Shrink-only. Each is a `up{job="X"}` selector whose monitor
+# names an endpoint port that the Service it selects does not define, so the monitor discovers
+# ZERO targets, no `up{job="X"}` series exists, and `up{job="X"} == 0` can never fire. This is the
+# nastiest variant of the class: `up` exists in abundance, so nothing about the metric name looks
+# wrong, and `PrometheusTargetsMissing` cannot see it either — a target that was never discovered
+# has no up series to be 0.
+#
+# All twelve are the apps/* ServiceDefinition SLI pattern: the rendered ServiceMonitor names
+# `port: metrics`, and no app Service defines a port of that name (verified: zero occurrences of
+# `name: metrics` across all twelve manifests). VERIFIED on prod 2026-09-13: `count by (job) (up)`
+# listed 20 jobs and none of these twelve appeared; all twelve `<app>-down` rules returned no
+# series. Closing this needs a Prometheus exporter per app — Plex, Pi-hole, Jackett, qBittorrent,
+# Sonarr, Radarr and Seerr do not speak Prometheus natively — so it is its own piece of work, and
+# for keycloak specifically the job labels that DO exist are `keycloak-service`/`keycloak-discovery`
+# (platform/keycloak/servicemonitor.yaml), so that one is a rename away rather than an exporter.
+# =============================================================================================
+KNOWN_UNWIRED_JOBS: dict[str, str] = {
+    "gitea": "apps/wave-0/gitea.yaml — Service defines only port-3000",
+    "jackett": "apps/wave-0/jackett.yaml — no exporter",
+    "pihole": "apps/wave-0/pihole.yaml — needs pihole-exporter",
+    "plex": "apps/wave-0/plex.yaml — no native Prometheus endpoint",
+    "qbittorrent": "apps/wave-0/qbittorrent.yaml — needs qbittorrent-exporter",
+    "vm-demo": "apps/wave-0/vm-demo.yaml — a KubeVirt demo VM, no exporter",
+    "keycloak": "apps/wave-1/keycloak.yaml — the live jobs are keycloak-service / "
+                "keycloak-discovery (platform/keycloak/servicemonitor.yaml); this SLI rule names "
+                "neither, so it is a rename away from being live rather than an exporter away",
+    "nextcloud": "apps/wave-1/nextcloud.yaml — needs nextcloud-exporter (serverinfo token)",
+    "radarr": "apps/wave-1/radarr.yaml — needs exportarr",
+    "semaphore": "apps/wave-1/semaphore.yaml — no exporter",
+    "sonarr": "apps/wave-1/sonarr.yaml — needs exportarr",
+    "seerr": "apps/wave-2/seerr.yaml — no exporter",
+}
+
+# =============================================================================================
+# (0) PARSE EVERY ALERT RULE
+# =============================================================================================
+# PromQL metric-name approximation. Documented as a gap in the header: string literals,
+# label-matcher bodies and aggregation/vector-matching label lists are stripped first, then every
+# remaining bare identifier that is not a PromQL keyword or function is treated as a metric
+# selector. Assertion (0b) fails loudly if this stops finding metrics at all, which is the way an
+# approximate parser silently stops guarding.
+PROMQL_KEYWORDS = set("""
+abs absent absent_over_time ceil changes clamp clamp_max clamp_min day_of_month day_of_week
+day_of_year days_in_month delta deriv exp floor histogram_quantile histogram_count histogram_sum
+histogram_fraction histogram_stddev histogram_stdvar holt_winters hour idelta increase irate
+label_join label_replace ln log2 log10 minute month predict_linear rate resets round scalar sgn
+sort sort_desc sort_by_label sort_by_label_desc sqrt time timestamp vector year avg count
+count_values group max min quantile stddev stdvar sum topk bottomk limitk limit_ratio
+last_over_time avg_over_time min_over_time max_over_time sum_over_time count_over_time
+quantile_over_time stddev_over_time stdvar_over_time present_over_time mad_over_time
+double_exponential_smoothing info ts_of_max_over_time ts_of_min_over_time ts_of_last_over_time
+by without on ignoring group_left group_right offset bool and or unless if atan2 start end
+inf nan NaN Inf
+""".split())
+
+IDENT_RE = re.compile(r"(?<![\w:.])([a-zA-Z_:][a-zA-Z0-9_:]*)")
+UP_JOB_RE = re.compile(r"""\bup\s*\{[^}]*\bjob\s*=\s*["']([^"']+)["']""")
+
+
+def metric_names(expr: str) -> list[str]:
+    e = re.sub(r'"(?:[^"\\]|\\.)*"', '""', expr)
+    e = re.sub(r"'(?:[^'\\]|\\.)*'", "''", e)
+    e = re.sub(r"`(?:[^`\\]|\\.)*`", "``", e)
+    e = re.sub(r"\{[^{}]*\}", " ", e)               # label matchers
+    for _ in range(6):                               # nested aggregations
+        e = re.sub(r"\b(?:by|without|on|ignoring|group_left|group_right)\s*\([^()]*\)", " ", e)
+    out, seen = [], set()
+    for m in IDENT_RE.finditer(e):
+        n = m.group(1)
+        if n in PROMQL_KEYWORDS or re.fullmatch(r"\d+[smhdwy]?", n) or n in seen:
+            continue
+        seen.add(n)
+        out.append(n)
+    return out
+
+
+def is_golden_fixture(p: Path) -> bool:
+    """conformance/tests/**/golden/** holds EXPECTED-OUTPUT fixtures, not deployed manifests.
+
+    conformance/tests/l1-rendering/egress-class-render/golden/*.expected.yaml each contain a
+    rendered `<app>-down` PrometheusRule. They are the assertion data of another test, never
+    reconciled into a cluster, so auditing their scrape targets would be auditing a copy — the
+    same "reference, not declaration" distinction ci/touched-requirements.sh draws for ci/tests/
+    fixture paths. The templates they are rendered FROM are audited through apps/ and catalog/.
+    """
+    parts = p.parts
+    return "conformance" in parts and "golden" in parts
+
+
+rule_files = sorted(
+    p for p in ROOT.rglob("*.yaml")
+    if ".git" not in p.parts and not is_golden_fixture(p) and "kind: PrometheusRule" in read(p)
+)
+check(len(rule_files) > 0,
+      "no file declaring `kind: PrometheusRule` found — the parser or the repo layout changed; "
+      "this check would silently audit nothing")
+
+# metric -> [ "<relpath>:<alert>", ... ] ; and the same shape for up{job="X"} selectors
+refs: dict[str, list[str]] = {}
+job_refs: dict[str, list[str]] = {}
+n_rules = 0
+for f in rule_files:
+    rel = str(f.relative_to(ROOT))
+    try:
+        docs = list(yaml.safe_load_all(read(f)))
+    except yaml.YAMLError as exc:
+        check(False, f"{rel} is not parseable YAML ({exc}); every alert in it is unaudited")
+        continue
+    for d in docs:
+        if not isinstance(d, dict) or d.get("kind") != "PrometheusRule":
+            continue
+        for grp in (d.get("spec") or {}).get("groups") or []:
+            for r in grp.get("rules") or []:
+                if "alert" not in r:
+                    continue                         # recording rules: not this check's subject
+                n_rules += 1
+                site = f"{rel}:{r['alert']}"
+                expr = str(r.get("expr", ""))
+                for n in metric_names(expr):
+                    refs.setdefault(n, []).append(site)
+                for j in UP_JOB_RE.findall(expr):
+                    job_refs.setdefault(j, []).append(site)
+
+check(n_rules > 0,
+      "no alerting rule parsed out of any PrometheusRule — the parse contract broke and this "
+      "check would pass vacuously while every alert's metric went unaudited")
+check(len(refs) >= 20,
+      f"only {len(refs)} distinct metric name(s) parsed out of {n_rules} alerting rule(s). The "
+      f"PromQL approximation has stopped matching — which is exactly how an approximate parser "
+      f"turns into a guard that passes while checking nothing. Fix metric_names() or replace it "
+      f"with a real parser")
+
+# =============================================================================================
+# (1)(2) RESOLVE EVERY METRIC TO AN OWNER, AND PROVE THE OWNER'S CLAIM AGAINST THE REPO
+# =============================================================================================
+def resolve(metric: str) -> dict | None:
+    """Longest-prefix / exact-name match against SCRAPE_OWNERS."""
+    best, best_len = None, -1
+    for o in SCRAPE_OWNERS:
+        if metric in o.get("exact", ()):            # exact names win outright
+            return o
+        for pfx in o.get("prefixes", ()):
+            if metric.startswith(pfx) and len(pfx) > best_len:
+                best, best_len = o, len(pfx)
+    return best
+
+
+# Every owner's FILE must exist, whether or not a metric currently lands on it: an owner pointing
+# at a deleted file is a claim nothing backs.
+for o in SCRAPE_OWNERS:
+    check((ROOT / o["file"]).is_file(),
+          f"SCRAPE_OWNERS names {o['file']} as the scrape owner for "
+          f"{o.get('prefixes') or o.get('exact')}, and that file does not exist in this repo — the "
+          f"claim is unbacked and the alerts that depend on it are inert")
+
+claimed: dict[str, dict] = {}
+unowned: list[str] = []
+for metric in sorted(refs):
+    if metric in UNIVERSAL:
+        continue
+    o = resolve(metric)
+    if o is None:
+        if metric in KNOWN_UNWIRED_METRICS:
+            continue                                 # declared gap; ratcheted below
+        unowned.append(metric)
+        check(False,
+              f"alert metric `{metric}` (queried by {', '.join(sorted(set(refs[metric])))}) is "
+              f"claimed by NO entry in SCRAPE_OWNERS. Nothing in this repo configures a scrape "
+              f"that produces it, so every rule above is STRUCTURALLY INERT: it evaluates to no "
+              f"vector, stays silent forever, and is indistinguishable from a healthy result. "
+              f"Either wire the scrape and add the owner entry, or — if it cannot be wired yet — "
+              f"declare it in KNOWN_UNWIRED_METRICS with the reason, so the gap is reported "
+              f"instead of silent")
+        continue
+    check(metric not in KNOWN_UNWIRED_METRICS,
+          f"`{metric}` is listed in KNOWN_UNWIRED_METRICS but is now claimed by "
+          f"{o['owner']} — the register is shrink-only: delete the entry so the next real gap is "
+          f"the one that shows up")
+    claimed[metric] = o
+
+# The owners that are actually load-bearing right now get their declarations verified.
+for o in SCRAPE_OWNERS:
+    if o not in claimed.values():
+        continue                                     # nothing queries it today; nothing to guard
+    body = read(ROOT / o["file"])
+    for rx in o.get("require", []):
+        check(re.search(rx, body, re.M) is not None,
+              f"{o['file']} no longer matches /{rx}/, which is what makes {o['owner']} scrape. "
+              f"Removing it re-inerts every alert on "
+              f"{o.get('prefixes') or o.get('exact')} — {o['why']}")
+    for rx in o.get("forbid", []):
+        check(re.search(rx, body, re.M) is None,
+              f"{o['file']} matches the FORBIDDEN pattern /{rx}/, which defeats {o['owner']} "
+              f"even though the enablement is still authored. {o['why']}")
+    comp = o.get("companion")
+    if comp:
+        cbody = read(ROOT / comp["file"])
+        check((ROOT / comp["file"]).is_file(),
+              f"{comp['file']} is missing — {comp['why']}")
+        for rx in comp.get("require", []):
+            check(re.search(rx, cbody, re.M) is not None,
+                  f"{comp['file']} no longer matches /{rx}/. The monitor alone is NOT enough here: "
+                  f"{comp['why']}")
+
+# The Longhorn NetworkPolicy must actually grant the path it exists to grant, and no more: the
+# whole point of splitting it out instead of flipping networkPolicies.restrictInternalTraffic off.
+NP = ROOT / "platform/longhorn/networkpolicy-metrics.yaml"
+if any(m.startswith("longhorn_") for m in refs):
+    np = read(NP)
+    check(np != "", f"{NP.relative_to(ROOT)} is missing; the Longhorn scrape is dropped and its "
+                    f"four alerts are inert")
+    check(re.search(r"kind:\s*NetworkPolicy", np) is not None,
+          "platform/longhorn/networkpolicy-metrics.yaml is not a NetworkPolicy")
+    check(re.search(r"app:\s*longhorn-manager", np) is not None,
+          "the Longhorn metrics NetworkPolicy does not select app=longhorn-manager, so it does not "
+          "open the pods that serve /metrics")
+    check(re.search(r"kubernetes\.io/metadata\.name:\s*monitoring", np) is not None,
+          "the Longhorn metrics NetworkPolicy does not admit namespace monitoring, so Prometheus "
+          "still cannot reach longhorn-manager and the storage alerts stay inert")
+    check(re.search(r"port:\s*9500", np) is not None,
+          "the Longhorn metrics NetworkPolicy does not name port 9500 (longhorn-manager's `manager` "
+          "port, which serves /metrics)")
+    # LEAST PRIVILEGE (R-SEC-02): ingress only, one port, and NOT the webhook/recovery ports.
+    check(re.search(r"policyTypes:\s*\n\s*-\s*Ingress\s*$", np, re.M) is not None
+          and "Egress" not in np,
+          "the Longhorn metrics NetworkPolicy is not Ingress-only — it must grant exactly the "
+          "scrape path and nothing else (R-SEC-02)")
+    # Comments are stripped first: the file's header deliberately NAMES 9502/9503 to record that
+    # they stay closed, and a substring search over prose would fail on the documentation of the
+    # very property it is asserting.
+    np_spec = "\n".join(re.sub(r"#.*$", "", ln) for ln in np.splitlines())
+    for closed in ("9502", "9503"):
+        check(closed not in np_spec,
+              f"the Longhorn metrics NetworkPolicy names port {closed} (admission webhook / "
+              f"recovery backend). Only 9500 serves metrics; opening the others to ns monitoring "
+              f"is reach this observability path does not need (R-SEC-02)")
+
+# stale KNOWN_UNWIRED_METRICS entries
+for m in sorted(KNOWN_UNWIRED_METRICS):
+    check(m in refs,
+          f"KNOWN_UNWIRED_METRICS declares `{m}`, which NO alert rule in this repo queries any "
+          f"more — a stale exemption hides the next real gap; delete the entry")
+
+# =============================================================================================
+# (4) up{job="X"} — the axis assertion (1) is structurally blind to.
+#
+# Scoped to files where THIS REPO authors both the Service and the ServiceMonitor (the apps/*
+# ServiceDefinition pattern), because only then is the port name inside the tree. A monitor whose
+# Service comes from a Helm chart cannot be audited statically and is out of scope (header gap).
+# =============================================================================================
+def yaml_docs(path: Path) -> list[dict]:
+    try:
+        return [d for d in yaml.safe_load_all(read(path)) if isinstance(d, dict)]
+    except yaml.YAMLError:
+        return []
+
+
+# job name -> (relpath, monitor endpoint ports, ports the co-authored Services define)
+same_file_monitors: dict[str, tuple[str, set[str], set[str]]] = {}
+for p in sorted(ROOT.rglob("*.yaml")):
+    if ".git" in p.parts:
+        continue
+    txt = read(p)
+    if "kind: ServiceMonitor" not in txt or "kind: Service" not in txt:
+        continue
+    docs = yaml_docs(p)
+    monitors = [d for d in docs if d.get("kind") == "ServiceMonitor"]
+    services = [d for d in docs if d.get("kind") == "Service"]
+    if not monitors or not services:
+        continue
+    svc_ports: set[str] = set()
+    for s in services:
+        for port in (s.get("spec") or {}).get("ports") or []:
+            if isinstance(port, dict) and port.get("name"):
+                svc_ports.add(str(port["name"]))
+    for m in monitors:
+        ep_ports = {str(e["port"]) for e in (m.get("spec") or {}).get("endpoints") or []
+                    if isinstance(e, dict) and e.get("port")}
+        name = (m.get("metadata") or {}).get("name")
+        if name:
+            same_file_monitors[str(name)] = (str(p.relative_to(ROOT)), ep_ports, svc_ports)
+
+for job in sorted(job_refs):
+    sites = ", ".join(sorted(set(job_refs[job])))
+    mon = same_file_monitors.get(job)
+    if mon is None:
+        # No same-file Service+ServiceMonitor pair is named for this job. It may well be produced
+        # by a platform monitor against a chart Service (oauth2-proxy, kubevirt, velero, flux all
+        # are), which this check cannot see — so this is NOT a failure, only unauditable.
+        continue
+    rel, ep_ports, svc_ports = mon
+    missing = sorted(ep_ports - svc_ports)
+    if missing:
+        check(job in KNOWN_UNWIRED_JOBS,
+              f"`up{{job=\"{job}\"}}` ({sites}) can never produce a series: {rel}'s ServiceMonitor "
+              f"scrapes port(s) {missing}, and the Service(s) in that same file define only "
+              f"{sorted(svc_ports) or ['(no named ports)']}. The monitor therefore discovers ZERO "
+              f"targets, there is no `up{{job=\"{job}\"}}` to compare against 0, and the rule is "
+              f"structurally inert — note PrometheusTargetsMissing cannot see this either, because "
+              f"a target that was never discovered has no up series to be 0. Give the Service a "
+              f"named port the workload actually serves metrics on (and an exporter if it has "
+              f"none), or declare it in KNOWN_UNWIRED_JOBS")
+    else:
+        check(job not in KNOWN_UNWIRED_JOBS,
+              f"`{job}` is listed in KNOWN_UNWIRED_JOBS but {rel}'s ServiceMonitor port(s) "
+              f"{sorted(ep_ports)} now all exist on its Service — the register is shrink-only: "
+              f"delete the entry")
+
+# stale KNOWN_UNWIRED_JOBS entries
+for job in sorted(KNOWN_UNWIRED_JOBS):
+    check(job in job_refs,
+          f"KNOWN_UNWIRED_JOBS declares `{job}`, which no alert rule selects via up{{job=...}} any "
+          f"more — a stale exemption hides the next real gap; delete the entry")
+
+# =============================================================================================
+# (5) THE CHECK IS LOAD-BEARING. Asserted against the DECLARATIVE mechanism, in both halves —
+# neither alone is sufficient. The per-test hand-written ci/l0.sh stage is deliberately NOT used
+# (whole-repo-invariants-wired FAILS the gate if one reappears); the property is DECLARED here and
+# HONOURED by ci/l0.sh's --include-scope union.
+# =============================================================================================
+SCOPE_DECL_RE = r"^\s*conformance\.home\.k3s/scope:\s*[\"']?whole-repo"
+SELF = read(ROOT / "conformance" / "tests" / "l0-static" / "alert-metrics-have-scrape-targets"
+            / "chainsaw-test.yaml")
+check(re.search(SCOPE_DECL_RE, SELF, re.M) is not None,
+      "alert-metrics-have-scrape-targets/chainsaw-test.yaml does not declare "
+      "conformance.home.k3s/scope: whole-repo, so the merge gate selects this check only when a "
+      "change happens to touch R-OBS-04. ci/touched-requirements.sh maps apps/* to NOTHING and "
+      "platform/monitoring/* to R-OBS-01/R-OBS-02 — so adding an alert under apps/ or flipping a "
+      "chart's serviceMonitor.enabled back to false would be filtered out of exactly the change "
+      "that re-inerts the alerts")
+L0 = read(ROOT / "ci" / "l0.sh")
+GATE = read(ROOT / "ci" / "gate.sh")
+check("--include-scope" in L0,
+      "ci/l0.sh no longer passes --include-scope to the conformance runner, so the L0 stage is a "
+      "purely requirement-scoped run again and the scope annotation above is inert decoration")
+check(re.search(r"^for s in .*\bl0\b.*; do", GATE, re.M) is not None
+      and re.search(r"fail\).*BLOCK=1", GATE) is not None,
+      "ci/gate.sh does not set the merge block when the l0 stage fails")
+
+# ======================================================================== report
+owners_used = {id(o): o for o in claimed.values()}
+print(f"Alert metric inventory: {n_rules} alerting rule(s) across {len(rule_files)} "
+      f"PrometheusRule file(s) -> {len(refs)} distinct metric name(s)")
+print(f"  claimed by a declared scrape owner: {len(claimed)}/{len(refs) - len(UNIVERSAL & set(refs))}"
+      f"   owners exercised: {len(owners_used)}/{len(SCRAPE_OWNERS)}")
+for o in SCRAPE_OWNERS:
+    ms = sorted(m for m, oo in claimed.items() if oo is o)
+    mark = "  " if ms else "· "
+    print(f"  {mark}{o['file']:<46} {len(ms):>2} metric(s)  {o['owner'][:64]}")
+print(f"\n  DECLARED GAPS (shrink-only registers):")
+print(f"    KNOWN_UNWIRED_METRICS: {len(KNOWN_UNWIRED_METRICS)} — "
+      f"{', '.join(sorted(KNOWN_UNWIRED_METRICS))}")
+print(f"    KNOWN_UNWIRED_JOBS:    {len(KNOWN_UNWIRED_JOBS)} inert up{{job=...}} SLI rule(s) — "
+      f"{', '.join(sorted(KNOWN_UNWIRED_JOBS))}")
+if unowned:
+    print(f"\n  UNCLAIMED (inert, undeclared): {', '.join(unowned)}", file=sys.stderr)
+
+print()
+if not FAILURES:
+    print(f"check OK: alert-metrics-have-scrape-targets — {CHECKS} assertions (R-OBS-04)")
+    raise SystemExit(0)
+print(f"CHECK FAIL (alert-metrics-have-scrape-targets): {len(FAILURES)} failed, "
+      f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
+for f in FAILURES:
+    print(f"  FAIL {f}", file=sys.stderr)
+raise SystemExit(1)
+PY
@@ -8,6 +8,18 @@
 # the empty string while the Kustomization still reports Ready — which emptied Cilium's
 # k8sServiceHost and broke the CNI on 2/3 dev nodes (DECISIONS-LOG 2026-09-01).
 #
+# It is also LOSSY IN THE OTHER DIRECTION, and that is the second incident on this record.
+# envsubst consumes dollars in PAIRS and emits one per pair, unconditionally and everywhere —
+# so no substituted path can carry a literal `$$` at all. On 2026-09-13 a ConfigMap under
+# apps-wave-0 shipped a dollar-quoted PL/pgSQL `DO $$ ... $$;` self-check; Postgres received
+# a half-quoted `DO $` and every Flux-reconciled run of the nextcloud-db grant reconciler died
+# with a syntax error, nine Error pods deep before anyone looked (commit 16b7fcd). It passed
+# review because it was hand-applied with `kubectl apply`, which runs no envsubst: the defect
+# existed ONLY on the rendered path, and it landed on the assertion whose whole job was to
+# make a silent regression visible. Nothing here failed — the manifest rendered
+# "successfully" into something broken, because the check treated any doubled dollar as a
+# deliberate escape and stopped looking.
+#
 # GIVEN every Flux Kustomization under clusters/ that declares postBuild.substituteFrom,
 # WHEN  each `spec.path` is resolved to the exact file set it renders (following
 #       kustomization.yaml `resources:` transitively, not globbing the repo),
@@ -15,7 +27,12 @@
 #       or deliberately ESCAPED `$${...}`; no unescaped expansion carries a default
 #       operator on an undefined name (the silent-empty class strict mode does NOT catch);
 #       no unescaped expansion is unparseable by envsubst (which fails the WHOLE
-#       Kustomization); every path rendering a knob actually substitutes cluster-vars;
+#       Kustomization); NO RUN OF TWO OR MORE DOLLARS IS SILENTLY REWRITTEN — the only shape
+#       that passes unremarked is exactly `$${`, the documented total escape, and every other
+#       run (a bare `$$` that cannot survive verbatim, an ODD `$$${...}` that only LOOKS
+#       escaped and in fact expands, an over-counted `$$$${...}`) is named and justified in
+#       `dollar-collapse.allow`, which is itself held to being exact and non-stale;
+#       every path rendering a knob actually substitutes cluster-vars;
 #       every overlay-patched knob has a base default; and the check is wired into the
 #       merge gate as a blocking, unconditionally-run stage.
 #
@@ -26,6 +43,19 @@
 # only); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
 # Test wraps the SAME script.
 #
+# NOT proven, and deliberately NOT proxied: the wider lesson of 2026-09-13 is that the bug
+# survived review because verification used a path (`kubectl apply`) that runs no
+# substitution. That property is NOT statically decidable from this repo — which command
+# someone ran is a fact about a shell session, and git stores the manifest, never the
+# invocation. Grepping commit prose or runbooks for `kubectl apply` would be trivially
+# satisfiable by rewording and would fire on the break-glass runbooks whose whole content is
+# that command, so no such proxy is attempted. The dollar-run assertion is the honest answer
+# instead: it moves the proof off the verification PATH and onto the SOURCE, where it IS
+# decidable, so the mangling is caught by reading the manifest however a human chose to try
+# it. An end-to-end `kustomize build | flux envsubst --strict` per substituted path would be
+# the real thing, and it is how the arithmetic in check.sh's header was measured — but the L0
+# CI toolchain installs neither tool, so as a gate stage it would SKIP permanently.
+#
 # NOT proven (named in check.sh's header): this is a STATIC scan of the SOURCE manifests
 # a path includes — a deliberate superset of the post-`kustomize build` stream. It does
 # not render, does not prove the feature gate is enabled on the live kustomize-controller,
@@ -63,13 +93,16 @@ metadata:
    conformance.home.k3s/intent: >-
      Every variable a substituted Flux path renders is defined in cluster-vars or
      escaped, so post-build substitution cannot silently empty a value and enabling
-      StrictPostBuildSubstitutions cannot fail a legitimate manifest.
+      StrictPostBuildSubstitutions cannot fail a legitimate manifest; and no dollar run
+      envsubst rewrites reaches the cluster unreviewed, so content that must arrive
+      verbatim cannot be silently mangled on the Flux-rendered path (16b7fcd).
 spec:
  steps:
    - name: audit-every-substituted-flux-path
      description: >-
        Resolve each postBuild.substituteFrom Kustomization to the files it renders and
-        assert every ${...} is defined or escaped, then assert the gate wires it blocking.
+        assert every ${...} is defined or escaped and no dollar run is silently collapsed,
+        then assert the gate wires it blocking.
      try:
        - script:
            content: bash ./check.sh
@@ -18,6 +18,32 @@
 # substituted path renders is either DEFINED or deliberately ESCAPED — and nothing in
 # the repo asserted that. This check does.
 #
+# SECOND INCIDENT, 2026-09-13 — THE INVERSE HAZARD (commit 16b7fcd)
+# ------------------------------------------------------------------
+# The first incident was a variable the author wanted EXPANDED that rendered EMPTY. This one
+# is its mirror: content the author wanted to reach the workload UNTOUCHED that substitution
+# silently REWROTE. A CronJob reconciler in apps/wave-0/nextcloud-db.yaml asserted its own
+# convergence in a dollar-quoted PL/pgSQL block —
+#
+#     DO <dq> BEGIN ... END <dq>;     where <dq> is two literal dollars: standard Postgres
+#
+# — inside a ConfigMap under apps-wave-0, which declares postBuild.substituteFrom. envsubst
+# collapses a doubled dollar into ONE dollar, ALWAYS, so Postgres received a half-quoted
+# `DO <one dollar>` and died with `syntax error at or near "$"`. Three scheduled runs left
+# nine Error pods in nextcloud-db before anyone looked.
+#
+# It PASSED verification because it was hand-applied with `kubectl apply`, which runs NO
+# envsubst. The defect existed ONLY on the Flux-rendered path. And the damage landed
+# precisely on the self-check: the GRANT on line 1 still ran, so the privilege stayed
+# correct and the only thing broken was the assertion meant to make a silent regression
+# visible — the part whose whole job is to stop a blind spot being quiet.
+#
+# THE ASSERTION GAP THIS CLOSES: assertions (1)-(3) below ask whether every `${...}` is
+# defined, escaped, or parseable. They said NOTHING about a doubled dollar that is not an
+# escape, because the old classifier treated a `$` run of length >= 2 as "escaped, therefore
+# fine" and stopped looking. Nothing failed the gate; the manifest rendered "successfully"
+# into something broken. Assertion (7) is that hole, closed.
+#
 # ENVSUBST BEHAVIOUR ASSERTED HERE (empirically verified against the pinned toolchain,
 # `flux envsubst`, flux 2.4.0 — this is measured behaviour, not assumption):
 #   ${DEFINED}        -> substituted with the cluster-vars value.
@@ -33,6 +59,66 @@
 #                        shell/cloud-init consumer. Escaping is total: it neutralises
 #                        even the indexed form.
 #   $VAR, $(cmd)      -> left alone by envsubst; they need no escaping.
+#
+# DOLLAR-RUN ARITHMETIC — the exact rule, MEASURED, not inferred (see assertion (7))
+# ---------------------------------------------------------------------------------
+# envsubst here is `github.com/fluxcd/pkg/envsubst` v1.1.0 — Flux's own maintained FORK of
+# `github.com/drone/envsubst` ("This package is a fork of drone/envsubst", its README) — and it
+# is exposed by the `flux envsubst` CLI ("replicate the behavior of the Flux Kustomization
+# post-build substitutions", its own --help). Named from the binary, not from memory: the
+# pinned flux 2.4.0 in the dev image embeds the Go module path
+# `/home/runner/go/pkg/mod/github.com/fluxcd/pkg/[email protected]/parse`
+# (`grep -ao 'github.com/fluxcd/pkg/envsubst[^ ]*' "$(command -v flux)"`). Its scanner walks
+# the input left to right; on a `$` it looks at the NEXT byte:
+#   * another `$`   -> BOTH are consumed and ONE `$` is emitted. This is UNCONDITIONAL. What
+#                      follows the pair is irrelevant — `{`, a letter, a digit, a semicolon,
+#                      a newline, end of input. `$$` is not "the escape for `${`"; it is the
+#                      escape for `$` itself, everywhere.
+#   * `{`           -> the braced expression is parsed and EXPANDED.
+#   * anything else -> the `$` is emitted literally (hence `$VAR` and `$(cmd)` survive).
+# So a run of N consecutive dollars emits floor(N/2) literal dollars, and when N is ODD the
+# final dollar is still LIVE. Measured with flux 2.4.0 `flux envsubst`, DEFINED=VALUE — the
+# left column is authored bytes, the right is what reaches the cluster:
+#
+#   $                 ->  $                  ${DEFINED}       ->  VALUE
+#   $$                ->  $                  $${DEFINED}      ->  ${DEFINED}   <- the escape
+#   $$$               ->  $$                 $$${DEFINED}     ->  $VALUE   <- ODD: brace LIVE
+#   $$$$              ->  $$                 $$$${DEFINED}    ->  $${DEFINED}
+#   $$$$$             ->  $$$                $$$$${DEFINED}   ->  $$VALUE
+#   DO $$ .. $$;      ->  DO $ .. $;         $$${NOPE:-d}     ->  $d
+#   awk '{print $$2}' ->  awk '{print $2}'   $${ARR[0]}       ->  ${ARR[0]}    (safe)
+#   $$@ $$< $$^       ->  $@ $< $^           $$${ARR[0]}      ->  HARD PARSE ERROR, whole
+#                                                                 Kustomization dies
+#
+# TWO CONSEQUENCES THIS CHECK IS BUILT ON:
+#   (a) THERE IS NO WAY TO SHIP A LITERAL `$$` THROUGH A SUBSTITUTED PATH. To make two
+#       dollars arrive you must author FOUR. So any authored `$$` that is meant to reach the
+#       workload verbatim is a BUG, full stop — that is the 2026-09-13 incident, and Postgres
+#       dollar-quoting is its commonest shape (Makefile `$$@`, awk `$$2` and docker-compose
+#       `$${}` are the same hazard arriving from the other direction: there the collapse is
+#       what the author wanted).
+#   (b) `$$$` + brace is NOT "more escaped" than `$$` + brace — it is NOT escaped at all. An
+#       odd run leaves one live dollar, so the brace expands, or, for an inner envsubst
+#       cannot parse, fails the ENTIRE Kustomization (proven above with $$${ARR[0]}, which
+#       aborted the whole stream while $${ARR[0]} on the line before it was fine).
+# `--strict` reports NONE of this: every row of that table except the parse error was
+# accepted SILENTLY by `flux envsubst --strict`, exit 0, and its output was BYTE-IDENTICAL to
+# non-strict (`cmp`). Strict mode cannot see this class, so a source-level assertion is the
+# only available guard.
+#
+# TO REPRODUCE THE TABLE (30 seconds, no cluster) — it is a measurement, so re-measure it
+# after any FLUX_VERSION bump in infra/devcontainer/versions.env. Open the toolbox with
+# `./infra/devcontainer/dev.sh` and run these INSIDE it (do NOT pipe into `dev.sh run` from
+# the host: it starts the container with `-it`, so a piped stdin hangs):
+#   printf '[$$] [$$$] [$${D}] [$$${D}] [DO $$ x $$;]\n' | D=VALUE flux envsubst
+#   -> [$] [$$] [${D}] [$VALUE] [DO $ x $;]
+# And end-to-end, on the actual delivery path, with the pre-16b7fcd manifest checked out
+# (`git show 16b7fcd^:apps/wave-0/nextcloud-db.yaml > apps/wave-0/nextcloud-db.yaml`):
+#   kubectl kustomize apps/wave-0 | grep 'DO \$'            -> "DO $$"  (the kubectl path)
+#   kubectl kustomize apps/wave-0 | flux envsubst --strict | grep 'DO \$'
+#                                                           -> "DO $"   (the FLUX path), exit 0
+# (`kubectl kustomize` IS the kustomize build here; the image ships no standalone binary.)
+# That two-command contrast IS the 2026-09-13 incident, and is why review missed it.
 #   ${...}, ${a b}, ${ARR[0]}  (UNESCAPED) -> HARD PARSE ERROR ("unable to parse
 #                        variable name" / "missing closing brace") which fails the
 #                        WHOLE Kustomization — every resource in the path, not just
@@ -55,6 +141,12 @@
 #       (5) every knob an overlay patches (`/data/KEY`) exists in the BASE ConfigMap —
 #           the "add the knob with a base default BEFORE referencing it" rule that the
 #           incident's interim discipline depends on;
+#       (7) NO DOLLAR RUN IS SILENTLY REWRITTEN: the only run of length >= 2 that passes
+#           unremarked is exactly `$$` immediately followed by `{` — the documented, total,
+#           already-inventoried escape. EVERY other run of >= 2 dollars (a bare `$$`, an odd
+#           `$$$`, an over-counted `$$$$`) is rewritten by envsubst between git and the
+#           container, so it must be named and justified in `dollar-collapse.allow`, a file
+#           this check also holds to being exact and non-stale;
 #   AND (6) this check is LOAD-BEARING: it DECLARES `conformance.home.k3s/scope: whole-repo`,
 #           so ci/l0.sh's `--include-scope whole-repo` runs it on every gate run, and
 #           ci/gate.sh sets the merge block when the l0 stage fails.
@@ -79,6 +171,21 @@
 #     kustomize-controller. Both are cluster facts (L2), untested here. Tagging
 #     R-GITOPS-01 claims ONLY the static slice: what git says a Kustomization renders
 #     is fully determined, with no variable silently emptied on the way in.
+#   - It CANNOT prove the thing that actually let 16b7fcd's bug reach a cluster: that a
+#     change was VERIFIED through a substituting path. "It was hand-applied with kubectl" is
+#     a fact about a shell session, not about any tracked file — git records the manifest,
+#     never the command someone ran to try it, so the property is not statically decidable
+#     here and no honest formulation of it exists. Nor is a proxy worth having: grepping
+#     commit prose or runbooks for `kubectl apply` would be trivially satisfiable by
+#     rewording, and would fire on the many legitimate break-glass runbooks whose whole
+#     content is that command. So this check does not attempt it, and does not pretend to.
+#     Assertion (7) answers that incident a different way: it moves the proof off the
+#     verification PATH, which is unknowable, and onto the SOURCE, which is not — the
+#     mangling is now caught by reading the manifest, whichever way a human chose to try it.
+#     (The one decidable half-measure, `kustomize build <path> | flux envsubst --strict` per
+#     substituted path in CI, is already ruled out above: the L0 toolchain installs neither
+#     tool, so the stage would SKIP permanently. The dev image has both, and it is how the
+#     arithmetic table above was measured.)
 #   - It does NOT validate cluster-vars VALUES (that GATEWAY_VIP is inside the LB pool,
 #     that an overlay's LAN block is self-consistent). Presence, not correctness.
 #   - A cluster-vars key nothing references is reported as an ADVISORY, never a failure:
@@ -108,6 +215,7 @@ definition of "what a Flux path actually renders" in the harness.
 """
 from __future__ import annotations

+import bisect
 import os
 import re
 import sys
@@ -231,9 +339,18 @@ LEN_RE = re.compile(r"^#([A-Za-z_][A-Za-z0-9_]*)$")


 def classify(dollars: int, inner: str) -> tuple[str, str] | None:
-    """Return (severity-tag, remediation) for a hazardous expansion, else None."""
-    if dollars >= 2:
-        return None                                   # escaped: emitted as a literal
+    """Return (severity-tag, remediation) for a hazardous expansion, else None.
+
+    PARITY, not ">= 2" (the 2026-09-13 correction). envsubst consumes dollars in PAIRS and
+    emits one per pair, so a run of N dollars in front of a brace leaves the brace ESCAPED
+    only when N is EVEN. An ODD run emits floor(N/2) literal dollars and its last dollar is
+    still LIVE, so `$$${X}` expands exactly as `${X}` does — and `$$${ARR[0]}` fails the
+    whole Kustomization where `$${ARR[0]}` is harmless. The old `dollars >= 2 -> return None`
+    called every such run "escaped" and stopped looking. Assertion (7) flags the run itself;
+    this function additionally classifies the still-live expansion an odd run carries.
+    """
+    if dollars % 2 == 0:
+        return None                                   # fully escaped: emitted as a literal
    if NAME_RE.match(inner):
        if inner in KEYS:
            return None                               # defined knob — the happy path
@@ -268,6 +385,9 @@ def rendered_files(path_spec: str) -> tuple[list[str], bool]:
 findings: list[str] = []
 referenced: dict[str, set[str]] = {}
 inventory: list[str] = []
+# rel path -> the Kustomization name(s) that render it, for assertion (7)'s second pass.
+# A file can be rendered by more than one Kustomization; all of them substitute.
+subst_files: dict[str, set[str]] = {}

 for k in kustomizations:
    files, exists = rendered_files(k["path"])
@@ -281,13 +401,14 @@ for k in kustomizations:
            text = f.read_text()
        except (OSError, UnicodeDecodeError):
            continue
+        subst_files.setdefault(rel, set()).add(k["name"])
        for lineno, line in enumerate(text.splitlines(), 1):
            for m in TOKEN_RE.finditer(line):
                dollars, inner = len(m.group(1)), m.group(2)
                verdict = classify(dollars, inner)
                if verdict is None:
-                    if dollars >= 2:
-                        n_esc += 1
+                    if dollars % 2 == 0:              # PARITY, see classify(): an odd run is
+                        n_esc += 1                    # live, not escaped
                    else:
                        n_def += 1
                        referenced.setdefault(re.split(r"[:/#%^,]", inner)[0] or inner,
@@ -313,6 +434,156 @@ for k in kustomizations:
 check(not findings,
      f"{len(findings)} unescaped, undefined or unparseable expansion(s) in substituted paths")

+# ----------------------------------- (7) no dollar run is silently rewritten on the way out
+# THE 2026-09-13 CLASS. envsubst consumes dollars in PAIRS and emits one per pair,
+# unconditionally and everywhere (header: DOLLAR-RUN ARITHMETIC, measured against flux 2.4.0).
+# So every run of >= 2 dollars in a substituted manifest is REWRITTEN between git and the
+# container, and from the text alone the two intents are indistinguishable:
+#
+#   (a) a deliberate escape the author WANTS collapsed  — `$${VAR}` -> literal `${VAR}`;
+#   (b) content that must arrive VERBATIM               — `DO $$ ... $$;` -> `DO $ ... $;`
+#                                                          i.e. the incident. A BUG.
+#
+# THE RULE CHOSEN, AND WHY IT IS NEITHER NOISY NOR VACUOUS.
+# Exactly ONE shape passes unremarked: a run of EXACTLY TWO dollars immediately followed by
+# `{`. That is not a guess about intent, it is the escape contract this repo already
+# documents in clusters/base/apps-vms.yaml, that assertions (1)-(3) already validate the
+# inside of, and that 18 reviewed occurrences already rely on (unifi cloud-init, the gitea
+# and nextcloud OIDC bootstrap scripts, two longhorn comments). Failing those would be a
+# permanently-firing alert on documented, correct intent, and this repo has already been
+# bitten by one of those.
+#
+# EVERY OTHER run of >= 2 dollars FAILS unless it is named in `dollar-collapse.allow`:
+#   * DOLLAR-COLLAPSE   — a run not followed by `{`. Cannot survive verbatim: `$$` -> `$`.
+#                         This is the incident's exact shape and a blanket bug by default.
+#   * DOLLAR-HALF-ESCAPED — an ODD run >= 3 before `{`. Looks escaped, is not: the last
+#                         dollar is live. Occasionally legitimate (`$$${PRICE}` to render a
+#                         literal `$` before a value), so it is allowlistable, not banned.
+#   * DOLLAR-OVER-ESCAPED — an EVEN run >= 4 before `{`. Emits N/2 dollars then a literal
+#                         brace; nearly always a miscount, sometimes deliberate.
+#
+# WHY AN ALLOWLIST AND NOT A BLANKET BAN: a blanket ban was the first choice and was
+# rejected on evidence, not taste — main carries two real non-braced `$$` occurrences today
+# (prose inside embedded shell comments), so a blanket ban would have shipped RED. Nor is
+# the allowlist a hole: an entry is a hand-written edit to a file inside the conformance
+# test, which is loud in review; it must quote a >= 24-character VERBATIM context, so it can
+# only ever cover the occurrence it was written for and never a future one that drifts into
+# the same file; it must carry a >= 40-character reason; and every entry must still MATCH
+# something, so the allowlist cannot rot into a blanket exemption after the code it excused
+# is deleted. A wildcard, a bare `$$` context and a stale entry each FAIL.
+ALLOW_FILE = (ROOT / "conformance" / "tests" / "l0-static"
+              / "flux-postbuild-substitution" / "dollar-collapse.allow")
+MIN_CONTEXT = 24
+MIN_REASON = 40
+DOLLAR_RUN_RE = re.compile(r"\$+")
+
+allow_entries: list[dict] = []
+check(ALLOW_FILE.is_file(),
+      f"{ALLOW_FILE.relative_to(ROOT)} is missing — assertion (7) needs its allowlist to "
+      f"exist even when empty, because an absent file and an empty file are the same green "
+      f"and the next author would not know the mechanism is there")
+if ALLOW_FILE.is_file():
+    for lineno, raw in enumerate(ALLOW_FILE.read_text().splitlines(), 1):
+        if not raw.strip() or raw.lstrip().startswith("#"):
+            continue
+        parts = [x.strip() for x in raw.split("|")]
+        if len(parts) != 3:
+            check(False, f"dollar-collapse.allow:{lineno}: expected exactly 3 '|'-separated "
+                         f"fields (path | verbatim-context | reason), got {len(parts)}")
+            continue
+        allow_entries.append({"line": lineno, "path": parts[0], "ctx": parts[1],
+                              "reason": parts[2], "used": 0})
+
+for e in allow_entries:
+    tag = f"dollar-collapse.allow:{e['line']}"
+    check((ROOT / e["path"]).is_file(),
+          f"{tag}: names '{e['path']}', which is not a file in this repo — a stale entry is "
+          f"a standing exemption for a path someone could recreate")
+    check(len(e["ctx"]) >= MIN_CONTEXT,
+          f"{tag}: context {e['ctx']!r} is {len(e['ctx'])} chars; at least {MIN_CONTEXT} are "
+          f"required so an entry cannot be a near-wildcard that swallows future occurrences")
+    check(DOLLAR_RUN_RE.search(e["ctx"]) is not None
+          and max((len(m.group(0)) for m in DOLLAR_RUN_RE.finditer(e["ctx"])), default=0) >= 2,
+          f"{tag}: context {e['ctx']!r} contains no run of two or more dollars, so it excuses "
+          f"nothing and is either a typo or a decoy")
+    check(len(e["reason"]) >= MIN_REASON,
+          f"{tag}: reason is {len(e['reason'])} chars; at least {MIN_REASON} are required. "
+          f"State what envsubst will actually emit and why that is acceptable — an entry "
+          f"nobody had to explain is an accident wearing an approval")
+
+RUN_TAGS = {
+    "collapse": ("DOLLAR-COLLAPSE",
+                 "envsubst collapses each PAIR of dollars into one, everywhere and "
+                 "unconditionally, so this CANNOT reach the workload verbatim (a literal $$ "
+                 "needs $$$$ in source). This is the 2026-09-13 nextcloud-db shape: a "
+                 "dollar-quoted PL/pgSQL DO block arrived half-quoted and every Flux-"
+                 "reconciled run died. Rewrite without dollars, double them again if the "
+                 "collapse is what you want, or justify it in dollar-collapse.allow"),
+    "half": ("DOLLAR-HALF-ESCAPED",
+             "an ODD run of dollars before a brace is NOT escaped: floor(N/2) literal "
+             "dollars are emitted and the LAST dollar is still live, so this expansion "
+             "EVALUATES at reconcile time (and if its inner is unparseable it fails the "
+             "entire Kustomization). Use an EVEN run to escape, or justify it in "
+             "dollar-collapse.allow if a literal dollar before a substituted value is "
+             "really what you meant"),
+    "over": ("DOLLAR-OVER-ESCAPED",
+             "an EVEN run of four or more dollars emits N/2 literal dollars followed by a "
+             "LITERAL brace — the value is never substituted and extra dollars appear in "
+             "the rendered manifest. Almost always a miscount of the escape; justify it in "
+             "dollar-collapse.allow if it is not"),
+}
+
+run_findings: list[str] = []
+n_runs_ok = 0
+for rel in sorted(subst_files):
+    text = (ROOT / rel).read_text(errors="replace")
+    # Absolute offsets, not per-line: the gitea/nextcloud bootstrap scripts are single
+    # 4000-character folded scalars, so "the line" is not a reviewable unit of context here.
+    nl = [i for i, ch in enumerate(text) if ch == "\n"]
+    spans = []                                       # allowlisted [start, end) regions
+    for e in allow_entries:
+        if e["path"] != rel:
+            continue
+        start = text.find(e["ctx"])
+        while start >= 0:
+            spans.append((start, start + len(e["ctx"]), e))
+            start = text.find(e["ctx"], start + 1)
+    for m in DOLLAR_RUN_RE.finditer(text):
+        n = len(m.group(0))
+        if n < 2:
+            continue
+        braced = text[m.end():m.end() + 1] == "{"
+        if n == 2 and braced:
+            n_runs_ok += 1                           # the documented escape; (1)-(3) own it
+            continue
+        covering = [e for (a, b, e) in spans if a <= m.start() and m.end() <= b]
+        if covering:
+            for e in covering:
+                e["used"] += 1
+            continue
+        kind = "collapse" if not braced else ("half" if n % 2 else "over")
+        tag, fix = RUN_TAGS[kind]
+        lineno = bisect.bisect_right(nl, m.start()) + 1
+        ctx = text[max(0, m.start() - 60):m.end() + 60].replace("\n", "\\n")
+        run_findings.append(
+            f"  [{tag}] {rel}:{lineno}  a run of {n} dollar(s)"
+            f"{' before a brace' if braced else ''}\n"
+            f"        rendered by Flux Kustomization(s) {', '.join(sorted(subst_files[rel]))}"
+            f" — postBuild substitution REWRITES it\n"
+            f"        {fix}\n"
+            f"        | ...{ctx}...")
+
+check(not run_findings,
+      f"{len(run_findings)} dollar run(s) in substituted paths that envsubst rewrites and "
+      f"nothing justifies (the 2026-09-13 nextcloud-db class)")
+
+for e in allow_entries:
+    check(e["used"] > 0,
+          f"dollar-collapse.allow:{e['line']}: STALE — its context is not present in "
+          f"{e['path']} any more (or no longer contains a rewritten dollar run), so the entry "
+          f"now excuses nothing. Delete it. A stale allowlist is how an opt-in decays into a "
+          f"blanket exemption for whatever lands in that file next")
+
 # --------------------------------------------- (5) overlay knobs exist in the base
 # Parse contract: overlays patch cluster-vars with JSON6902 ops (`- op: replace` /
 # `path: /data/KEY`) inside an inline `patch: |-` block. The repo rule (cluster-vars.yaml
@@ -363,6 +634,10 @@ print("Flux postBuild substitution inventory (per Kustomization):")
 for line in inventory:
    print(line)
 print(f"\ncluster-vars knobs defined: {len(KEYS)}   referenced: {len(referenced)}")
+print(f"dollar runs >=2 in substituted files: {n_runs_ok} documented $${{...}} escapes, "
+      f"{sum(e['used'] for e in allow_entries)} justified by dollar-collapse.allow "
+      f"({len(allow_entries)} entr{'y' if len(allow_entries) == 1 else 'ies'}), "
+      f"{len(run_findings)} unjustified")

 if ADVISORIES:
    print("\nADVISORIES (not failures):")
@@ -374,6 +649,12 @@ if findings:
    for f in findings:
        print(f, file=sys.stderr)

+if run_findings:
+    print("\nHAZARDS — dollar runs envsubst rewrites between git and the container:",
+          file=sys.stderr)
+    for f in run_findings:
+        print(f, file=sys.stderr)
+
 print()
 if not FAILURES:
    print(f"check OK: flux-postbuild-substitution — {CHECKS} assertions "
@@ -0,0 +1,52 @@
+# =============================================================================
+# dollar-collapse.allow — the reviewed exceptions to assertion (7) of
+# conformance/tests/l0-static/flux-postbuild-substitution/check.sh.
+#
+# WHY THIS FILE EXISTS
+# --------------------
+# Flux's postBuild substitution runs github.com/fluxcd/pkg/envsubst (Flux's fork of
+# github.com/drone/envsubst; the module path is embedded in the pinned flux 2.4.0
+# binary, and check.sh's header says how to read it back out) over everything a
+# substituted Kustomization renders, and that scanner consumes dollars in PAIRS,
+# emitting ONE dollar per pair — unconditionally, wherever they appear, not only in
+# front of a brace. A run of N dollars becomes floor(N/2) dollars, and an ODD run
+# leaves one live dollar behind (so `$$${X}` is NOT escaped; it expands).
+#
+# Consequence: there is NO way to ship a literal `$$` through a substituted path.
+# On 2026-09-13 that broke the nextcloud-db grant reconciler — a dollar-quoted
+# PL/pgSQL `DO $$ ... $$;` block reached Postgres as `DO $ ... $;` and every
+# Flux-reconciled run died with `syntax error at or near "$"`. It had passed review
+# because it was hand-applied with `kubectl apply`, which runs no envsubst at all.
+# See commit 16b7fcd and check.sh's DOLLAR-RUN ARITHMETIC table (measured against
+# flux 2.4.0, not inferred).
+#
+# From the source text alone, a doubled dollar the author WANTS collapsed and one
+# that must arrive verbatim are indistinguishable. check.sh therefore accepts exactly
+# one shape without comment — a run of EXACTLY TWO dollars immediately followed by
+# `{`, the escape contract clusters/base/apps-vms.yaml already documents and that 18
+# reviewed occurrences already depend on — and FAILS every other run of two or more
+# dollars unless it is named here.
+#
+# ADDING AN ENTRY IS A DECISION, NOT A MUTE BUTTON
+# -----------------------------------------------
+# One entry per line, exactly three `|`-separated fields:
+#
+#     <repo-relative path> | <verbatim context> | <reason>
+#
+#   * the context must be a VERBATIM substring of the file AS STORED IN GIT (for a
+#     folded/JSON-ish scalar that means the escaped bytes, e.g. `\n` and `—`
+#     spelled literally), at least 24 characters long, containing the dollar run it
+#     excuses, and containing no `|` — shift the window if a pipe is in the way;
+#   * the reason must be at least 40 characters and should say what envsubst will
+#     actually EMIT and why that is acceptable;
+#   * an entry covers only the dollar runs inside the region its context matches, so
+#     it can never silently cover a new occurrence that lands elsewhere in the file;
+#   * check.sh FAILS a stale entry — one whose context no longer matches, or which
+#     excuses nothing any more. Delete it rather than leaving it to rot into a
+#     blanket exemption for whatever lands in that file next.
+#
+# If the collapse would change BEHAVIOUR, this file is the wrong answer. Rewrite the
+# content without dollars (16b7fcd) or double them again (`$$$$` -> `$$`).
+# =============================================================================
+apps/wave-0/gitea.yaml | # NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild. | Prose, not code: this is a comment INSIDE the gitea OIDC bootstrap shell script, and the two dollars are the sentence quoting the escape idiom it goes on to explain. envsubst renders it as "the `$` escape below", which is cosmetically wrong and behaviourally inert — a shell comment. Cannot be written correctly at all: `$$$$` would render `$$` but would then be wrong in the source a reader sees in git, and the file has no non-dollar way to name the idiom. Reviewed and accepted as cosmetic.
+apps/wave-1/nextcloud.yaml | # NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild. | Identical case to the gitea entry above, in the nextcloud OIDC bootstrap script: a shell comment quoting the escape idiom, rendered by envsubst as "the `$` escape below". Cosmetic only — the collapse happens inside a `#` comment line that no interpreter reads, and the escaped expansion the comment is describing is on a later line and is unaffected. Reviewed and accepted as cosmetic.
@@ -31,3 +31,53 @@ spec:
    # cluster/split-horizon DNS (docs/09 split-horizon note; realises the TODO that was here).
    dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"
    dns01RecursiveNameserversOnly: true
+    # --- SCRAPE THE TLS ISSUER, or every certificate alert is INERT (added 2026-09-13) ----------
+    # platform/monitoring/alerts.yaml ships CertificateExpiringSoon, CertificateExpiryImminent
+    # (critical) and CertificateNotReady — and NONE of them could ever fire. The chart exposes the
+    # controller's metrics port by default (`prometheus.enabled: true`, confirmed on prod:
+    # Service/cert-manager listens on 9402) but defaults `prometheus.servicemonitor.enabled: false`,
+    # and no ServiceMonitor existed in ns cert-manager. VERIFIED live on prod 2026-09-13:
+    # `count({__name__=~"certmanager.*"})` returned NOTHING and `count by (job) (up)` listed no
+    # cert-manager job — so certmanager_certificate_expiration_timestamp_seconds and
+    # certmanager_certificate_ready_status were never in the TSDB.
+    #
+    # WHY THIS IS PART OF THE TLS PATH: the live wildcard `wildcard-home` (ns gateway) expires
+    # 2026-12-12T02:16:57Z and is renewed by an ACME DNS-01 order that can fail quietly — a Cloudflare
+    # token rotation, a DNS-01 self-check timeout (the very failure the two lines above work around),
+    # a rate limit. With no scrape, the FIRST signal of a failed renewal is a browser TLS error for
+    # every *.${BASE_DOMAIN} service. That is a user-discovered outage on a fully automated path,
+    # which is exactly what R-TLS-01/R-OBS-04 exist to prevent.
+    #
+    # Chart mechanics, verified by rendering cert-manager v1.21.2 (the version this range resolves to):
+    #   * templates/servicemonitor.yaml renders on `prometheus.enabled AND
+    #     prometheus.servicemonitor.enabled`. Measured: `helm template --set
+    #     prometheus.servicemonitor.enabled=true` emits exactly 1 ServiceMonitor, 0 without it.
+    #     `prometheus.enabled` is already true by chart default, so this one line is sufficient.
+    #   * podmonitor and servicemonitor are MUTUALLY EXCLUSIVE — the chart calls `fail` if both are
+    #     set. Do not add `prometheus.podmonitor.enabled` alongside this.
+    #   * the rendered ServiceMonitor selects the controller/webhook/cainjector Services by
+    #     app.kubernetes.io/{name,instance,component} on targetPort `http-metrics`, and sets
+    #     `jobLabel: app.kubernetes.io/name` — so the scrape job lands as `cert-manager`.
+    #   * it carries the label `prometheus: default` (servicemonitor.prometheusInstance). Harmless
+    #     here: platform/monitoring sets serviceMonitorSelectorNilUsesHelmValues false, so Prometheus
+    #     selects every ServiceMonitor regardless of labels. Left at the chart default deliberately —
+    #     overriding it would imply a label-selecting Prometheus this cluster does not have.
+    # Cost: 3 scrape targets, one certmanager_* series set — negligible.
+    #
+    # PROVEN END TO END ON PROD, 2026-09-13 21:0xZ, by applying the rendered ServiceMonitor by hand
+    # (then removing it, so the merge's Helm upgrade owns it) — no NetworkPolicy was needed, as
+    # neither ns cert-manager nor ns monitoring carries one:
+    #   up{namespace="cert-manager"}                  => 3 targets, all 1
+    #                                                    (jobs cert-manager / cainjector / webhook,
+    #                                                     because jobLabel is app.kubernetes.io/name)
+    #   count({__name__=~"certmanager.*"})            => 104    (was EMPTY)
+    #   (certmanager_certificate_expiration_timestamp_seconds - time())/86400
+    #                                                 => 89.2 days for `wildcard-home` (ns gateway,
+    #                                                    ClusterIssuer letsencrypt-prod)
+    #   certmanager_certificate_ready_status{condition="True"} => 1
+    # So the three Certificate* rules now evaluate against a real series and are correctly QUIET:
+    # the wildcard is healthy and 89 days out, which is exactly what CertificateExpiringSoon's 14-day
+    # threshold is meant to catch and could not have.
+    prometheus:
+      servicemonitor:
+        enabled: true
@@ -4,3 +4,8 @@
 # dns01RecursiveNameservers / -Only           # TODO(WS-SEC): may be needed with split-horizon DNS
 # ClusterIssuers (letsencrypt-staging/-prod) live in clusterissuers.yaml, not here.
 # Cloudflare API token: SOPS-encrypted secret `cloudflare-api-token` (WS-SEC, OQ-4) — name only.
+# prometheus.servicemonitor.enabled: true     # scrape the controller's 9402 metrics port; without it
+#                                             # the certmanager_* family is absent from the TSDB and
+#                                             # the three Certificate* alert rules in
+#                                             # platform/monitoring/alerts.yaml are INERT. Mutually
+#                                             # exclusive with prometheus.podmonitor.enabled.
@@ -22,8 +22,22 @@ spec:
      rules:
        # Failed launches: a VMI that reached the Failed phase (scheduling/launch/qemu failure). Any
        # sustained Failed VMI is page-worthy — the IaaS path is broken for that workload.
+        #
+        # METRIC CORRECTED 2026-09-13 (was `kubevirt_vmi_phase_count{phase="Failed"}`). That series
+        # DOES NOT EXIST on the deployed KubeVirt, so this rule was structurally INERT — unlike the
+        # Longhorn/cert-manager/Flux gaps in the same audit, NOT for want of a scrape target: the
+        # `kubevirt-prometheus-metrics` job is up and its siblings below (virt_api_ready_status,
+        # virt_controller_ready_status, cdi_import_pods_high_restart) all resolve. KubeVirt simply
+        # retired the phase-count gauge in favour of an info-style series carrying `phase` as a label.
+        # VERIFIED live on prod 2026-09-13:
+        #   count(kubevirt_vmi_phase_count) => (empty)
+        #   count(kubevirt_vmi_info)        => 1
+        #   kubevirt_vmi_info{...,name="unifi",namespace="unifi",phase="running",...} => 1
+        # Note `phase="running"` — KubeVirt LOWERCASES the phase in this label, so the matcher is
+        # "failed", not "Failed". A capital-F matcher here would silently select nothing and leave the
+        # rule just as inert as the metric name did; that is the whole point of this class of defect.
        - alert: KubeVirtVMLaunchFailing
-          expr: sum by (namespace) (kubevirt_vmi_phase_count{phase="Failed"}) > 0
+          expr: sum by (namespace) (kubevirt_vmi_info{phase="failed"}) > 0
          for: 5m
          labels:
            severity: critical
@@ -102,6 +102,54 @@ spec:
      # Reclaim replica space of deleted/failed volumes promptly in dev.
      orphanAutoDeletion: true

+    # --- SCRAPE THE STORAGE LAYER, or every Longhorn alert is INERT (added 2026-09-13) ----------
+    # platform/monitoring/alerts.yaml ships LonghornVolumeDegraded (critical),
+    # LonghornVolumeFaulted (critical), LonghornVolumeCapacityHigh and LonghornNodeStorageHigh —
+    # and NONE of them could ever fire, because the chart defaults
+    # `metrics.serviceMonitor.enabled: false` and no ServiceMonitor existed in ns longhorn-system.
+    # VERIFIED live on prod 2026-09-13: `count by (job) (up)` listed 19 jobs and none was Longhorn;
+    # `count(longhorn_volume_robustness)` and `count({__name__=~"longhorn.*"})` both returned NOTHING.
+    # So the entire longhorn_* family was absent from the TSDB and four alert rules were decoration.
+    #
+    # WHY THIS IS PART OF THE STORAGE PATH, NOT AN OBSERVABILITY NICE-TO-HAVE: this cluster's
+    # recurring failure mode is a disk. prod-1's failing SSD starved etcd's WAL fsync and wedged the
+    # node (DECISIONS-LOG; the prod-1 drive replacement), and a degraded/faulted replica is precisely
+    # the first symptom Longhorn publishes. R-STATE-02 promises 3-way replication survives a node
+    # loss; with no scrape, a volume can sit at 2 replicas — or faulted — indefinitely, unalerted.
+    # platform/monitoring/pvc-capacity-alerts.yaml's header already flagged this exact gap as "a
+    # separate defect (a missing Longhorn ServiceMonitor/metrics config)". This closes it.
+    #
+    # Chart mechanics, verified by rendering longhorn 1.12.1 (the version this range resolves to):
+    #   * templates/servicemonitor.yaml is gated on `{{- if .Values.metrics.serviceMonitor.enabled }}`
+    #     and NOTHING else — no `.Capabilities` probe. Measured: `helm template --set
+    #     metrics.serviceMonitor.enabled=true` emits exactly 1 ServiceMonitor, and 0 without the flag.
+    #     So this line is both necessary and sufficient; unlike velero's, it needs no API capability.
+    #   * the rendered ServiceMonitor selects `app: longhorn-manager` in the release namespace on the
+    #     port named `manager`. CONFIRMED against prod: Service/longhorn-backend carries
+    #     `app: longhorn-manager` and a port `manager` (9500) — the selector resolves to real
+    #     endpoints, it is not a dangling monitor (which is how the apps/ SLI monitors fail: they name
+    #     a `metrics` port their Service does not have, so they discover zero targets and their
+    #     `up{job=...}` alerts are inert in the very same way).
+    #   * no `additionalLabels` needed: platform/monitoring sets serviceMonitorSelectorNilUsesHelmValues
+    #     false, so Prometheus selects every ServiceMonitor regardless of labels.
+    # Cost: one scrape target per manager pod (3), the longhorn_* families — low hundreds of series
+    # on this cluster's volume count.
+    #
+    # PROVEN END TO END ON PROD, 2026-09-13 21:0xZ. The rendered ServiceMonitor was applied to prod
+    # by hand (then removed again, so the merge's Helm upgrade creates and OWNS it rather than
+    # colliding with an unowned object) with this file's networkpolicy-metrics.yaml already in place:
+    #   count(up{job="longhorn-backend"})              => 3      (all three managers, up==1)
+    #   count({__name__=~"longhorn.*"})                => 8717   (was EMPTY)
+    #   count(longhorn_volume_robustness)              => 44
+    #   count_values("v", longhorn_volume_robustness)  => 24 healthy(1), 72 unknown(0, detached)
+    # and the four alert rules went from "no series, cannot fire" to evaluating for real — one of
+    # them immediately found a true positive (LonghornVolumeCapacityHigh on the Prometheus TSDB
+    # volume at 97.1% ALLOCATED while its filesystem is only 20.4% used: Longhorn is not reclaiming
+    # freed blocks. Reported, NOT silenced by moving the threshold).
+    metrics:
+      serviceMonitor:
+        enabled: true
+
    # Longhorn's own manager/UI stay in-cluster; UI exposure (if any) is gated behind
    # oauth2-proxy forward-auth (ADR-0014) by WS-SEC — not exposed here.

@@ -5,4 +5,10 @@ resources:
  - namespace.yaml
  - helmrepository.yaml
  - helmrelease.yaml
+  # Opens ns monitoring -> longhorn-manager:9500 so Prometheus can actually scrape the metrics the
+  # HelmRelease's `metrics.serviceMonitor.enabled` exposes. Longhorn 1.12.1's own (effectively
+  # unconditional) NetworkPolicies drop that path, so WITHOUT this the ServiceMonitor discovers 3
+  # targets that all sit at up==0 and the Longhorn alerts stay inert. See the file header for the
+  # measured proof. (2026-09-13, from the inert-alert audit.)
+  - networkpolicy-metrics.yaml
  - storageclasses.yaml
@@ -0,0 +1,76 @@
+---
+# Allow Prometheus to reach longhorn-manager's metrics port (R-OBS-04, R-STATE-02).
+#
+# WITHOUT THIS FILE, `metrics.serviceMonitor.enabled: true` IN helmrelease.yaml IS NOT ENOUGH and the
+# four Longhorn alert rules in platform/monitoring/alerts.yaml STAY INERT. This was measured, not
+# predicted — prod, 2026-09-13, with the ServiceMonitor in place:
+#
+#   count by (job) (up)            => {job="longhorn-backend"} => 3      (targets DISCOVERED)
+#   up{job="longhorn-backend"}     => 0, 0, 0                            (every scrape FAILING)
+#   count(longhorn_volume_robustness) => (empty)                         (still nothing in the TSDB)
+#   /api/v1/targets lastError      => 'Get "http://10.0.0.169:9500/metrics": context deadline
+#                                     exceeded'  (10.0007s — a DROP, not a refusal)
+#
+# The endpoint itself is healthy and fast; only the path from ns monitoring is blocked. Proven by
+# curling it from inside the manager pod:
+#   http://<podIP>:9500/metrics -> 200 in 0.031s       (the manager binds the POD IP, not localhost —
+#   http://localhost:9500/metrics -> 000                 which is why a localhost probe misleads here)
+#
+# WHY IT IS BLOCKED: longhorn 1.12.1 ships SIX NetworkPolicies of its own, and the one selecting
+# `app: longhorn-manager` is Ingress-only with a from-list of nothing but other Longhorn components
+# (longhorn-manager, longhorn-ui, longhorn-csi-plugin, the recurring-job/job-task pods,
+# longhorn-driver-deployer) — all bare podSelectors, i.e. same-namespace. Prometheus lives in ns
+# monitoring, so it matches no clause and its packets are dropped.
+#
+# THE CHART OFFERS NO KNOB FOR THIS, and the obvious-looking one is a trap. `networkPolicies.enabled`
+# (default false) does NOT gate these templates — verified by rendering 1.12.1: `helm template` emits
+# the same 6 NetworkPolicies with the flag on AND off, because templates/network-policies/* are
+# guarded on `networkPolicies.restrictInternalTraffic` (default TRUE) instead. So the policies are
+# effectively unconditional, and nothing in the chart adds a Prometheus allowance when
+# metrics.serviceMonitor.enabled is set.
+#
+# WHY NOT `networkPolicies.restrictInternalTraffic: false`: that is the one-line fix and it is the
+# wrong one. It deletes Longhorn's entire internal-traffic restriction across all six components to
+# open one port to one namespace — a straight R-SEC-02 (least network reach) regression, trading a
+# security control for an observability one when the two are not in conflict.
+#
+# THIS IS ADDITIVE AND NARROW. Kubernetes NetworkPolicies are OR'd (Cilium enforces the UNION — see
+# platform/headlamp/networkpolicy.yaml's header), so this grants exactly one new path on top of
+# Longhorn's floor: ns monitoring -> app=longhorn-manager, TCP 9500 only. Longhorn's own policy is
+# left untouched, every other component stays restricted, and nothing else in longhorn-system becomes
+# reachable.
+#
+# TIER: a core networking.k8s.io kind with no CRD dependency, so it belongs to the `platform`
+# Kustomization alongside the HelmRelease (referenced from platform/longhorn/kustomization.yaml) —
+# NOT platform-config, which exists for CRs of platform-installed CRDs.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: longhorn-manager-allow-prometheus-scrape
+  namespace: longhorn-system
+  labels:
+    app.kubernetes.io/name: longhorn
+    # Marks this as an observability path, not a data path — so a future reviewer trimming Longhorn
+    # network reach can see at a glance what breaks (the storage alerts) if they delete it.
+    homelab.keithjackson.dev/purpose: metrics-scrape
+spec:
+  podSelector:
+    matchLabels:
+      app: longhorn-manager
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        # The whole monitoring namespace rather than the Prometheus pod labels: the operator owns
+        # those labels and rewrites them on every kube-prometheus-stack bump (they carry the chart's
+        # instance name and a controller-revision hash). A pod-label selector here would silently
+        # stop matching on an upgrade and put the storage alerts back to inert — the exact regression
+        # class this whole change exists to close. The namespace holds only the monitoring stack.
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: monitoring
+      ports:
+        - protocol: TCP
+          # longhorn-manager's container port `manager` (Service longhorn-backend:9500), which serves
+          # /metrics. NOT 9502/9503 (admission webhook / recovery backend) — those stay closed.
+          port: 9500
@@ -22,3 +22,10 @@
 #   defaultDataLocality: disabled                    # keep replicas spread for durability
 #   orphanAutoDeletion: true                         # reclaim orphaned replica space promptly
 #   backupTarget: <off-cluster>                      # TODO(WS-STORAGE, Phase 4): DR target (ADR-0003)
+#
+# metrics:
+#   serviceMonitor:
+#     enabled: true                                  # scrape longhorn-manager (port `manager`, 9500);
+#                                                    # without it the whole longhorn_* family is absent
+#                                                    # from the TSDB and the four Longhorn alert rules
+#                                                    # in platform/monitoring/alerts.yaml are INERT
@@ -333,6 +333,47 @@ spec:
        # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
        # above (tun0 presence, which also goes absent if the gateway node is down); Cilium datapath
        # health is covered by the Cilium platform alerts.
+        # *** KNOWN INERT — DELIBERATELY NOT FIXED IN THIS CHANGE. DO NOT "fix" it by adding a
+        # *** ServiceMonitor: a monitor alone cannot make this rule fire.
+        # Registered as such in conformance/tests/l0-static/alert-metrics-have-scrape-targets
+        # (KNOWN_UNWIRED_METRICS), which is what keeps this comment from rotting into a lie: a
+        # stale entry there FAILS the gate, so this cannot quietly become fiction.
+        #
+        # VERIFIED live on prod 2026-09-13:
+        #   count(hubble_drop_total)         => (empty)
+        #   count({__name__=~"hubble.*"})    => (empty)
+        #   count({__name__=~"cilium.*"})    => (empty)
+        #   count by (job) (up)              => 20 jobs, none Cilium or Hubble
+        # (re-verified unchanged on prod 2026-09-13 at 21:19Z, ~15h after the first reading)
+        # So the rule evaluates `rate(<absent series>)` and returns no series: it cannot fire, and the
+        # platform has NO policy-drop visibility at all today.
+        #
+        # WHY A MONITOR IS NOT THE FIX. Unlike Longhorn/cert-manager (whose charts already serve
+        # /metrics and only lacked a ServiceMonitor) Hubble publishes NOTHING to scrape until it is
+        # switched on in Cilium's own config. platform/cilium/helmrelease.yaml sets
+        # `hubble.{enabled,relay.enabled,ui.enabled}: true` but NOT `hubble.metrics.enabled`, and not
+        # `prometheus.enabled` for the agent — which is why the cilium_* family is missing too. Turning
+        # them on means:
+        #   * `hubble.metrics.enabled: [drop, ...]` + `hubble.metrics.serviceMonitor.enabled: true`,
+        #     which OPENS PORT 9965 ON EVERY cilium-agent AND ROLLS THE CNI DaemonSet across all
+        #     three nodes. A cilium-agent roll is the single most dangerous rollout on this cluster:
+        #     the 2026-09-01 k8sServiceHost outage is in DECISIONS-LOG, and cilium was moved into its
+        #     OWN Flux Kustomization precisely because a CNI rollout needs an isolated blast radius.
+        #   * a matching change to the ANSIBLE bootstrap values. platform/cilium/helmrelease.yaml
+        #     carries `# [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true`
+        #     against roles/cilium/templates/cilium-values.yaml.j2. Changing only the HelmRelease makes
+        #     a rebuilt node's bootstrap CNI config disagree with the reconciled one — a drift that is
+        #     invisible until the next node rebuild.
+        #   * per-flow label cardinality: `drop` metrics carry source/destination context, and the
+        #     default contexts can multiply series across every pod pair that gets denied — on a
+        #     default-deny cluster, denials are routine, not exceptional.
+        # RECOMMENDATION (for the PM, not to be done silently here): schedule this as its own change
+        # on the cilium Kustomization, with the Ansible parity edit in the same commit, the
+        # labelsContext pinned narrow (e.g. `hubble.metrics.enabled: ["drop:sourceContext=pod;destinationContext=pod"]`),
+        # and a one-node canary before the full DaemonSet roll. Until then this alert is honestly
+        # inert and the gap is: NO policy-drop / NetworkPolicy-regression visibility (docs/07 security
+        # audit), which also means a NetworkPolicy change cannot be observed to have started dropping
+        # legitimate traffic.
        - alert: HubbleDeniedFlowSpike
          # Cilium/Hubble denied (policy-drop) flows. Requires hubble metrics enabled (docs/03).
          expr: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5
Author	SHA1	Message	Date
jakman85	37eea51f24	merge: the substitution check now catches a collapsed doubled dollar gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details The existing check audited only the inverse — a ${...} undefined or unescaped — so it passed the manifest that broke prod: Postgres dollar-quoting under a postBuild.substituteFrom path, which envsubst halves, delivering 'DO $' and a syntax error on every reconciled run (fixed `16b7fcd`). Measured rather than assumed: Flux 2.4.0 embeds fluxcd/pkg/envsubst (a fork of drone/envsubst — the brief credited the wrong module), and --strict output is BYTE-IDENTICAL to non-strict here, so strict mode is blind to this class entirely. The decisive contrast is two commands on the same stream: kubectl kustomize emits valid 'DO $$' — which is what the kubectl-apply verification saw — while the same stream through flux envsubst emits 'DO $'. 357 assertions, 0 unjustified. Legitimate uses genuinely exist (18 ${} escapes plus two reaching shell comments), so the allowlist is a reviewed opt-in rather than a bypass: a stale entry fails, a wildcard is matched verbatim and so fails as stale, a short context or reason is rejected, and a genuine entry covers only the span it matches — a second occurrence in the same file still fails. Strongest proof: restoring the real pre-16b7fcd manifest verbatim makes the guard reject the exact commit that broke production.	2026-09-13 16:27:51 -05:00
jakman85	2f579f103b	merge: Longhorn and cert-manager metrics exist now — 13 alerts were inert Neither family was scraped at all, so every Longhorn alert (degraded, faulted, capacity) and every cert-manager alert (expiry, issuance) was structurally incapable of firing: storage health unmonitored on a cluster whose node just had a failing drive, TLS expiry unmonitored with the live wildcard cert 89 days out. Now proven on prod: longhorn_* 8717 series with longhorn-backend up=3, certmanager_* 104 series across 3 targets. Adds an L0 guard (118 assertions) that crosses from each alert rule to the scrape config and re-proves each enabling declaration against the working tree, including the negatives — cert-manager's monitor mutual-exclusion, the Longhorn metrics NetworkPolicy, and kube_node_labels needing metricLabelsAllowlist. Proven able to fail five ways. Hubble is deliberately NOT enabled: it opens 9965 on every cilium-agent and rolls the CNI DaemonSet across all three nodes, and platform/cilium carries an explicit parity contract with the Ansible template that a HelmRelease-only change would silently break. Registered as a declared gap instead. Two findings this surfaced, neither fixed here and neither absorbed silently: twelve apps/* <app>-down SLI rules discover zero targets because their ServiceMonitors name port 'metrics' and no app Service defines that port; and kube-state-metrics is OOMKilled roughly every ten minutes, making thirteen kube_* rules intermittently inert.	2026-09-13 16:26:12 -05:00
jakman85andClaude Opus 5	830983b791	test(l0): prove the dollar-collapse guard fires — and name the real envsubst The implementation landed in the preceding WIP commit had never been executed. This finishes it: the arithmetic is re-measured, the guard and its allowlist are attacked, and the header's one factual error is corrected. WHAT ENVSUBST ACTUALLY IS. The header credited `github.com/drone/envsubst`. The pinned flux 2.4.0 binary in the dev image embeds `github.com/fluxcd/pkg/[email protected]` — Flux's own fork of drone/envsubst ("This package is a fork of drone/envsubst", its README) — read straight out of the binary with `grep -ao`. Same behaviour, different module; the header now says so and says how to check, because a claim about a dependency that nobody can re-derive is how the last wrong assumption survived. RE-MEASURED, NOT ASSERTED. Every row of the DOLLAR-RUN ARITHMETIC table was re-run through `flux envsubst` (flux 2.4.0, DEFINED=VALUE) and every row holds: `$$`->`$`, `$$$`->`$$`, `$${D}`->`${D}`, `$$${D}`->`$VALUE` (odd run: the brace is LIVE), `$$$${D}`->`$${D}`, `DO $$ x $$;`->`DO $ x $;`, and `$$${ARR[0]}` dies with "missing closing brace" while `$${ARR[0]}` is fine. `--strict` output was byte-identical to non-strict, exit 0 both ways (`cmp`) — strict mode is blind to this whole class, which is why a source assertion is the only guard available. The header now carries the 30-second reproduction, flagged to be re-measured on any FLUX_VERSION bump. END TO END ON THE REAL PATH. With the pre-16b7fcd manifest restored, `kubectl kustomize apps/wave-0` emits `DO $$` — valid SQL, and exactly what the `kubectl apply` verification saw — while the same stream through `flux envsubst --strict` emits `DO $` and `$;` and exits 0. That two-command contrast IS the incident. THE GUARD IS NOT TRIVIALLY SATISFIABLE — 16 adversarial cases, all as intended, run against a throwaway copy of the tree so nothing could be left behind: * an unmarked `$$` under a substituted path FAILS (DOLLAR-COLLAPSE); so do an odd `$$${...}` (DOLLAR-HALF-ESCAPED) and an even `$$$${...}` (OVER-ESCAPED); * a stale entry FAILS three ways — context no longer in the file, context that only covers an already-accepted `$${` escape and so excuses nothing, and a path that is not a file in the repo; * the allowlist cannot blanket-exempt a path: a 2-char context is rejected (<24), a `.` wildcard is matched VERBATIM and therefore rejected as stale, a 24-char context carrying no dollar run is rejected, a 4-char reason is rejected (<40) — and, the central case, a genuine entry covers only the span its context matches, so a SECOND `$$` elsewhere in the same file still fails. Two genuine entries for two real occurrences do pass, so the mechanism is usable rather than merely unsatisfiable; the CLASS, not the instance: the pre-fix `DO $$ ... $$;` reconstructed in a NEW ConfigMap under apps/wave-2 fails at both dollar runs, and restoring the real pre-16b7fcd apps/wave-0/nextcloud-db.yaml verbatim fails the check at lines 384 and 391. The guard rejects the exact commit that broke production. The scratch fixtures lived only in the throwaway copy; `git status` on this branch stayed clean throughout, which is the other half of the last attempt's unfinished business. LEGITIMATE USES EXIST AND STAY GREEN. 18 `$${...}` escapes (unifi cloud-init, the gitea and nextcloud OIDC bootstrap scripts, two longhorn comments) pass unremarked, and the 2 non-braced `$$` occurrences are the two allowlist entries. Those two are shell comments inside block scalars, and `kubectl kustomize` confirms the comment text really does survive into the stream envsubst reads — so they are genuine, reaching content whose collapse is cosmetic, not a hypothetical the allowlist was invented to excuse. STILL NOT PROVEN, AND STILL NOT PROXIED: that a change was verified through a substituting path. Which command a human ran is a fact about a shell session; git stores the manifest, never the invocation, so it is not statically decidable here and no honest formulation exists. Grepping commit prose for `kubectl apply` would be satisfiable by rewording and would fire on the break-glass runbooks whose whole content is that command. The answer is to move the proof off the verification path and onto the source, which is what assertion (7) does. No ci/l0.sh stage: the `conformance.home.k3s/scope: "whole-repo"` annotation is what wires it, and whole-repo-invariants-wired fails the gate on a hand-written stage. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 16:24:42 -05:00
jakman85andClaude Opus 5	a92fa11701	feat(monitoring): scrape Longhorn + cert-manager — 8 alerts were structurally inert An alert rule whose metric is never scraped is not a broken alert, it is an ABSENT alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB with no longhorn_* series returns no vector: permanently quiet, permanently "healthy", indistinguishable from a correct rule. It survives review (the PromQL is right), it survives `promtool test rules` (the unit test supplies the series itself), and it appears on no dashboard. VERIFIED ABSENT ON PROD before this change (re-verified 2026-09-13 21:0xZ, ~15h after the first audit, because a stale reading is not evidence): count(longhorn_volume_robustness) => (empty) 4 rules inert count({__name__=~"certmanager."}) => (empty) 3 rules inert count(hubble_drop_total) / {__name__=~"cilium."} => (empty) 1 rule inert count by (job) (up) => 20 jobs, none of them Longhorn, cert-manager, Cilium or Hubble Consequence: storage health was unmonitored on the cluster whose prod-1 had just had a failing SSD, and TLS expiry was unmonitored with the live `wildcard-home` wildcard 89 days from expiry on an ACME DNS-01 renewal that can fail silently. WHAT THIS CHANGES * platform/longhorn: `metrics.serviceMonitor.enabled: true` (the supported chart way — rendering longhorn 1.12.1 emits exactly 1 ServiceMonitor with the flag and 0 without), plus networkpolicy-metrics.yaml. The NetworkPolicy is LOAD-BEARING, not belt-and-braces: measured with the monitor applied and no allowance, all 3 targets were DISCOVERED and every scrape failed `context deadline exceeded` after 10.0s — a drop, not a refusal — while curling the endpoint from inside the manager pod returned 200 in 0.031s. Longhorn 1.12.1 ships 6 NetworkPolicies of its own, guarded on `networkPolicies.restrictInternalTraffic` (default TRUE) and NOT on `networkPolicies.enabled`, so they are effectively unconditional and the manager policy admits only same-namespace Longhorn components. The one-line alternative (restrictInternalTraffic: false) was rejected: it deletes the internal-traffic restriction across all six components to open one port to one namespace, trading an R-SEC-02 control for an observability one when the two do not conflict. This policy is Ingress-only, ns monitoring -> app=longhorn-manager, TCP 9500 only; 9502/9503 (webhook, recovery backend) stay closed. * platform/cert-manager: `prometheus.servicemonitor.enabled: true`. Chart default already serves 9402; only the monitor was missing. No NetworkPolicy needed — neither ns cert-manager nor ns monitoring carries one. NOT paired with prometheus.podmonitor.enabled: the chart calls `fail` if both are set. * platform/kubevirt: KubeVirtVMLaunchFailing queried `kubevirt_vmi_phase_count{phase="Failed"}`, which DOES NOT EXIST on the deployed KubeVirt — a different flavour of the same defect, with the scrape target up and healthy. Prod has only `kubevirt_vmi_info` and `kubevirt_vmi_number_of_outdated`; the only phase-bearing names are the phase_transition_time histograms. Corrected to `kubevirt_vmi_info{phase="failed"}` — LOWERCASE, because KubeVirt lowercases the value (observed `phase="running"` for the unifi VMI) and a capital-F matcher would select nothing and be exactly as inert. PROVEN, NOT ASSERTED. The rendered ServiceMonitors were applied to prod by hand and the metrics observed to land, then REMOVED again so the merge's Helm upgrade creates and owns them rather than colliding with an unowned object: count(up{job="longhorn-backend"}) => 3 (all three managers, up==1) count({__name__=~"longhorn."}) => 8717 (was empty) count(longhorn_volume_robustness) => 44 up{namespace="cert-manager"} => 3 targets, all 1 count({__name__=~"certmanager."}) => 104 (was empty) wildcard-home days to expiry => 89.2, ready_status{condition="True"}=1 All 58 alert rules in the repo were then evaluated against prod one by one: every one returns a result without error, and exactly ONE fires. TRIAGE OF THE ONE NEWLY-FIRING ALERT — A TRUE POSITIVE, NOT TUNED AWAY. LonghornVolumeCapacityHigh on the Prometheus TSDB volume: `longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes` = 0.971 (48.6 GiB allocated of 50 GiB) while the kubelet's view of the same filesystem is only 20.4% used. Longhorn is not reclaiming freed blocks, so ~48.6 GiB x 3 replicas is committed for ~10 GiB of real data. On a cluster whose failure mode is disk pressure and whose etcd WAL shares a spindle with Longhorn, that is worth knowing and was invisible until now. The threshold is left alone; the follow-up is filesystem trim, not a bigger number. THE GUARD (conformance/tests/l0-static/alert-metrics-have-scrape-targets) Whole-repo L0 invariant, 118 assertions. It crosses from each alert to the scrape config that makes its metric exist: every metric name is claimed by exactly one declared owner, and each owner's enabling declaration is re-proven against the working tree on every gate run, including the negative conditions that defeat an enablement without deleting it. `up{job="X"}` is audited on its own axis, because `up` always exists: where this repo authors a Service and a ServiceMonitor in the same file, the monitor's endpoint port must be a port that Service defines. It DECLARES conformance.home.k3s/scope: "whole-repo" and adds NO ci/l0.sh stage — the whole-repo-invariants-wired meta-test fails the gate if one reappears. PROVEN ABLE TO FAIL, five ways, each reverted: 1. an alert on a fabricated metric -> FAIL "claimed by NO entry" 2. longhorn metrics.serviceMonitor removed -> FAIL "no longer matches" 3. networkpolicy-metrics.yaml unwired -> FAIL "the monitor alone is NOT enough" 4. cert-manager podmonitor also enabled -> FAIL "FORBIDDEN pattern" 5. a register entry made stale / removed -> FAIL in both directions TWO SHRINK-ONLY REGISTERS, because a blanket fail would make the gate red until work that is not this change lands, and a disabled gate protects nothing: * KNOWN_UNWIRED_METRICS (1): hubble_drop_total. See below. * KNOWN_UNWIRED_JOBS (12): every apps/* `<app>-down` SLI rule is ALSO inert, and this was found by the re-audit, not previously known. The rendered ServiceMonitors name `port: metrics` and NOT ONE of the twelve app Services defines a port of that name (zero occurrences of `name: metrics` across all twelve manifests); prod confirms none of the twelve jobs appears in `up`. This is the nastiest variant of the class: `up` exists in abundance so nothing looks wrong, and PrometheusTargetsMissing cannot see it either, because a target that was never discovered has no up series to be 0. Closing it needs an exporter per app (Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr speak no Prometheus) — except keycloak, where the live jobs are keycloak-service / keycloak-discovery and the rule is a rename away from being live. NOT DONE HERE, DELIBERATELY: Hubble. hubble.metrics.enabled RECONFIGURES THE CNI — it opens 9965 on every cilium-agent and rolls the DaemonSet across all three nodes, serialised by this repo's own maxUnavailable: 1 (set after the 2026-09-01 k8sServiceHost outage, where a CNI roll deadlocked 2 of 3 agents). It also needs a matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same commit, because platform/cilium/helmrelease.yaml carries a [parity] contract against it and a rebuilt node would otherwise bootstrap a CNI config that disagrees with the reconciled one. Recommended as its own change on the cilium Kustomization, with the Ansible parity edit, a narrow labelsContext, and a one-node canary — not smuggled in here. ALSO FOUND, NOT FIXED HERE: kube-state-metrics is OOMKilled every ~10 minutes on prod (90 restarts in 16h against a 128Mi limit) while its logs show the custom-resource-state factory reloading every 3 seconds. Thirteen alert rules read kube_* metrics, so those rules are INTERMITTENTLY inert — the same defect class, arriving through a crashloop rather than a missing monitor. Needs its own change. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 16:21:44 -05:00
jakman85andClaude Opus 5	377fa205ea	WIP: catch envsubst collapsing a doubled dollar in substituted paths INCOMPLETE — preserved by the PM after the authoring agent hit the session rate limit mid-task. Do not merge without finishing and gating it. Closes the inverse of what this check already covered. It asserted that every ${...} in a substituted path is defined or escaped; it did NOT catch content containing a literal doubled dollar that the author needs delivered verbatim and that envsubst silently halves. That shipped a broken CronJob (fixed in `16b7fcd`): Postgres dollar-quoting became a single dollar and every Flux-reconciled run died with a syntax error, while passing review because verification used kubectl apply, which runs no envsubst. Introduces a dollar-collapse.allow allowlist so each legitimate use is a reviewed decision rather than an accident. WHAT REMAINS: the agent's own next step was proving it catches the CLASS and not just this instance, via a scratch manifest under a substituted path. Its scratch fixture (apps/wave-2/zz-scratch-class.yaml and the wave-2 kustomization edit) was deliberately temporary and has been reverted here, so the fail-proof is NOT yet demonstrated. Also unanswered: brief point 3, on whether the check can say anything about a manifest verified only through a non-substituting path. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 16:09:37 -05:00