feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>

conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml

@@ -3,8 +3,13 @@
 # Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
 # Spec:    docs/09 (Gateway API + DNS-01 wildcard), ADR-0006 (LE issuers)
 #
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostnames are built from the ${BASE_DOMAIN} of the CLUSTER
+# UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by a
+# test-level chainsaw binding. The same authored test therefore verifies dev's wildcard on dev and
+# prod's on prod, with nothing to pass at run time.
+#
 # GIVEN the shared Gateway terminating TLS with the cert-manager wildcard cert,
-# WHEN  an in-cluster probe dials it by hostname (SNI *.home.keithjackson.dev) and
+# WHEN  an in-cluster probe dials it by hostname (SNI *.${BASE_DOMAIN}) and
 #       verifies the chain against the SYSTEM trust store,
 # THEN  the served cert is non-self-signed, unexpired, covers the wildcard SAN, and
 #       chains to a PUBLICLY-TRUSTED CA root (R-TLS-01). The wildcard SAN also covers
@@ -36,6 +41,11 @@ metadata:
       that chains to a publicly-trusted CA root (LE production); the wildcard SAN
       covers internal-only hostnames too.
 spec:
+  # -- R-ENV-02: the hostname suffix is READ FROM THE CLUSTER UNDER TEST (flux-system/cluster-vars
+  #    — the ConfigMap Flux substitutes ${BASE_DOMAIN} from), never authored per environment.
+  bindings:
+    - name: baseDomain
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
   steps:
     - name: then-gateway-cert-chains-to-trusted-root
       try:
@@ -54,13 +64,16 @@ spec:
                     containers:
                       - name: probe
                         image: curlimages/curl:8.11.1
+                        env:
+                          - name: BASE_DOMAIN
+                            value: ($baseDomain)
                         command:
                           - sh
                           - -c
                           - |
                             set -eu
                             # Probe TWO wildcard hostnames — one "public", one "internal-only". Both
-                            # match *.home.keithjackson.dev, so a clean verify against the SYSTEM CA
+                            # match *.${BASE_DOMAIN}, so a clean verify against the SYSTEM CA
                             # bundle for BOTH proves R-TLS-01 (chains to a publicly-trusted root) AND
                             # R-TLS-03 (the same trusted wildcard covers internal-only names). Names
                             # resolve via in-cluster split-horizon DNS to the Gateway VIP. curl (no
@@ -68,7 +81,7 @@ spec:
                             # the gateway's 404 once TLS+hostname verify; it exits 60 if the chain or
                             # hostname fails. TLS terminates at the gateway before HTTP routing, so no
                             # backend is needed.
-                            for host in conformance.home.keithjackson.dev internal-only.home.keithjackson.dev; do
+                            for host in "conformance.${BASE_DOMAIN}" "internal-only.${BASE_DOMAIN}"; do
                               curl -sS --retry 5 --retry-connrefused --retry-delay 3 --max-time 20 \
                                    -o /dev/null "https://${host}/"
                               echo "TLS_TRUSTED ${host}"
@@ -80,3 +93,20 @@ spec:
               kind: Job
               metadata: { name: tls-probe }
               status: { succeeded: 1 }
+    # -----------------------------------------------------------------------------------
+    # PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
+    # chainsaw already fails the test before reaching this step if any of them fails — but a
+    # green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
+    # GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
+    # `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
+    # forever and NEVER become a recorded proof of R-TLS-01/03 — the TLS half of docs/07's audit proof list
+    # — which is precisely what docs/07 asks for ("These proofs are part of the platform
+    # acceptance suite, not one-time checks"). This step exists only to emit that token, and it
+    # is reached only when everything above has passed. Asserted by
+    # tests/l0-static/auditability-proofs-producible.
+    - name: then-emit-the-proof-of-pass-sentinel
+      try:
+        - script:
+            content: echo TLS_TRUSTED_PROVEN_OK
+            check:
+              (contains($stdout, 'TLS_TRUSTED_PROVEN_OK')): true