feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>

conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml

@@ -2,17 +2,24 @@
 # L2 BEHAVIORAL — split-horizon INTERNAL resolution  (R-DNS-01 / R-DNS-03)   [RUNNABLE]
 # Catalog: docs/14 "DNS/exposure (R-DNS-*)"
 # Spec:    docs/09 "Split-horizon resolution (R-DNS-03)" — internal clients resolve
-#          <svc>.home.keithjackson.dev to the INTERNAL Gateway VIP, not the public hairpin.
+#          <svc>.${BASE_DOMAIN} to the INTERNAL Gateway VIP, not the public hairpin.
 # Impl:    platform/coredns/coredns-custom.yaml (k3s CoreDNS `import *.server`; template plugin
-#          answers the whole home.keithjackson.dev zone with ${GATEWAY_VIP}).
+#          answers the whole ${BASE_DOMAIN} zone with ${GATEWAY_VIP}).
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): this test asks the CLUSTER IT IS POINTED AT for its own
+# ${BASE_DOMAIN} and ${GATEWAY_VIP} — the same flux-system/cluster-vars ConfigMap Flux
+# substitutes from — via test-level chainsaw bindings (see `spec.bindings`). Nothing here is
+# per-environment and no flag is passed at run time, so the suite cannot be run against the
+# wrong expectation: dev asserts dev's zone/VIP, prod asserts prod's.
 #
 # GIVEN the split-horizon coredns-custom ConfigMap is reconciled into k3s CoreDNS,
 # WHEN  an in-cluster pod resolves a platform hostname over its PUBLIC name
-#       (keycloak.home.keithjackson.dev) using the default cluster resolver (CoreDNS 10.43.0.10),
-# THEN  the answer is the in-cluster Gateway VIP (the Cilium L2 `shared` gateway pool 192.168.1.53-.60) — NOT a
-#       public/off-cluster IP. This is what lets pod->public-hostname traffic hairpin through the
-#       Gateway with the correct SNI (*.home.keithjackson.dev) so the wildcard cert validates
-#       (R-DNS-01 "services reachable at <svc>.home..."; R-DNS-03 split-horizon).
+#       (keycloak.${BASE_DOMAIN}) using the default cluster resolver (CoreDNS 10.43.0.10),
+# THEN  the answer is EXACTLY that cluster's ${GATEWAY_VIP} (the address coredns-custom is
+#       templated with, and the `shared` Gateway's own status address) — NOT a public/off-cluster
+#       IP. This is what lets pod->public-hostname traffic hairpin through the
+#       Gateway with the correct SNI (*.${BASE_DOMAIN}) so the wildcard cert validates
+#       (R-DNS-01 "services reachable at <svc>.${BASE_DOMAIN}"; R-DNS-03 split-horizon).
 #
 # Complements dns-exposure (R-DNS-02, the EXTERNAL/public-record side). Together they cover the
 # split-horizon pair: external answer = public record (R-DNS-02); internal answer = Gateway VIP
@@ -34,13 +41,23 @@ metadata:
     conformance.home.k3s/safety-critical: "false"
     conformance.home.k3s/executable: "true"
     conformance.home.k3s/intent: >-
-      An in-cluster pod resolving keycloak.home.keithjackson.dev via cluster DNS gets the internal
-      Gateway VIP (192.168.1.53-.60), not the public hairpin — split-horizon in-cluster resolution.
+      An in-cluster pod resolving keycloak.${BASE_DOMAIN} via cluster DNS gets that cluster's
+      internal Gateway VIP, not the public hairpin — split-horizon in-cluster resolution.
 spec:
   timeouts:
     apply: 60s
     assert: 2m
     exec: 2m
+  # -- R-ENV-02: the environment's identity is READ FROM THE CLUSTER UNDER TEST, never authored
+  #    here. flux-system/cluster-vars is the ConfigMap Flux itself substitutes from
+  #    (clusters/base/cluster-vars.yaml + the dev/prod overlay patch), so these bindings are the
+  #    same source of truth the platform is rendered from. Test-level => in scope for every step
+  #    (unlike step `outputs`, which are step-scoped).
+  bindings:
+    - name: baseDomain
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
+    - name: gatewayVip
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.GATEWAY_VIP)
   steps:
     - name: then-cluster-dns-answers-with-the-internal-gateway-vip
       try:
@@ -59,23 +76,32 @@ spec:
                     containers:
                       - name: probe
                         image: busybox:1.36.1
+                        # The cluster's OWN identity, resolved from flux-system/cluster-vars by the
+                        # test-level bindings — never a literal (R-ENV-02).
+                        env:
+                          - name: BASE_DOMAIN
+                            value: ($baseDomain)
+                          - name: GATEWAY_VIP
+                            value: ($gatewayVip)
                         command:
                           - sh
                           - -c
                           - |
                             set -u
-                            host="keycloak.home.keithjackson.dev"
+                            host="keycloak.${BASE_DOMAIN}"
                             # Resolve via the DEFAULT cluster resolver (CoreDNS, /etc/resolv.conf ->
-                            # 10.43.0.10). Split-horizon must answer with the in-cluster Gateway VIP
-                            # (Cilium L2 `shared` pool 192.168.1.53-.60), proving pods hairpin through the Gateway.
+                            # 10.43.0.10). Split-horizon must answer with THIS cluster's Gateway VIP
+                            # (the address coredns-custom is templated with), proving pods hairpin
+                            # through the Gateway. Exact match on ${GATEWAY_VIP}: stricter than the
+                            # old hardcoded 192.168.1.53-.60 range AND correct in both environments.
                             out="$(nslookup "$host" 2>/dev/null)"
                             echo "$out"
-                            ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -E '^192\.168\.1\.(5[3-9]|60)$' | head -1)"
+                            ip="$(echo "$out" | awk '/^Address: /{print $2}' | grep -Fx "${GATEWAY_VIP}" | head -1)"
                             if [ -n "$ip" ]; then
                               echo "INTERNAL_RESOLUTION_OK $host -> $ip (in-cluster Gateway VIP)"
                               exit 0
                             fi
-                            echo "FAIL: $host did not resolve to the in-cluster Gateway VIP (192.168.1.53-.60);"
+                            echo "FAIL: $host did not resolve to this cluster's Gateway VIP (${GATEWAY_VIP});"
                             echo "      split-horizon coredns-custom not in effect (got a public/off-cluster answer)."
                             exit 1
         - assert: