feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>

conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml

@@ -3,11 +3,18 @@
 # Catalog: docs/14 "DNS/exposure (R-DNS-*)"
 # Spec:    docs/09 (ExternalDNS auto-manages records; split-horizon foundation)
 #
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostname is built from the ${BASE_DOMAIN} of the CLUSTER
+# UNDER TEST (read from flux-system/cluster-vars via a test-level chainsaw binding), and the
+# expected answer is the LB IP that THIS Service was actually assigned (read back from its own
+# status) — so neither the name nor the address is authored per environment, and no run-time flag
+# can be forgotten. Requesting a record under the cluster's own suffix is also what keeps the two
+# environments from ever describing the same public name (the 2026-09-04 incident precondition).
+#
 # GIVEN a LoadBalancer Service annotated with an external-dns hostname,
 # WHEN  ExternalDNS reconciles (source=service, provider=cloudflare, policy=sync),
 # THEN  a PUBLIC DNS record for that hostname is auto-created, pointing at the
-#       Service's Cilium-assigned LB IP (R-DNS-02). The record resolves to the
-#       cluster LB IP (Cilium L2 `shared` pool 192.168.1.53-.60), i.e. clients reach the cluster endpoint.
+#       Service's Cilium-assigned LB IP (R-DNS-02) — asserted as an EXACT match against that
+#       Service's own `status.loadBalancer.ingress[0].ip`, i.e. clients reach the cluster endpoint.
 #
 # CLEANUP: chainsaw deletes the per-test namespace (Service included) at teardown;
 # ExternalDNS policy=sync then REMOVES the record from Cloudflare on its next pass.
@@ -38,6 +45,13 @@ spec:
     apply: 90s
     assert: 5m
     exec: 6m
+  # -- R-ENV-02: the environment's hostname suffix is READ FROM THE CLUSTER UNDER TEST.
+  #    flux-system/cluster-vars is the ConfigMap Flux substitutes ${BASE_DOMAIN} from, so this
+  #    binding is the same source of truth the platform is rendered from — dev asks dev, prod
+  #    asks prod, and there is no per-environment fork and no flag to forget.
+  bindings:
+    - name: baseDomain
+      value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
   steps:
     - name: given-a-service-requesting-a-dns-name
       try:
@@ -48,7 +62,8 @@ spec:
               metadata:
                 name: dns-probe-svc
                 annotations:
-                  external-dns.alpha.kubernetes.io/hostname: conformance-dns.home.keithjackson.dev
+                  # conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
+                  external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
               spec:
                 type: LoadBalancer
                 # No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and
@@ -59,7 +74,27 @@ spec:
                   - name: http
                     port: 80
                     targetPort: 80
+        # Wait for Cilium LB-IPAM to actually assign the VIP: it is the value ExternalDNS will
+        # publish and the value the probe below asserts EXACTLY. `|| ` + empty-array literal
+        # keeps the expression nil-safe while the address is still absent, so the assert POLLS
+        # (fails and retries) instead of erroring out on the first evaluation.
+        - assert:
+            resource:
+              apiVersion: v1
+              kind: Service
+              metadata:
+                name: dns-probe-svc
+              status:
+                (length(loadBalancer.ingress || `[]`)): 1
     - name: then-externaldns-creates-the-public-record
+      # Step-scoped binding: read back the LB IP THIS Service was assigned (the previous step's
+      # assert guarantees it exists by now — step bindings are evaluated when the step runs).
+      # Expecting that exact address is both stricter than the old hardcoded 192.168.1.53-.60
+      # range and correct in every environment: dev's pool is 192.168.1.53-.60, prod's is
+      # 192.168.2.20-.30, and neither is written down here.
+      bindings:
+        - name: lbIp
+          value: (x_k8s_get($client, 'v1', 'Service', $namespace, 'dns-probe-svc').status.loadBalancer.ingress[0].ip)
       try:
         - apply:
             resource:
@@ -76,20 +111,27 @@ spec:
                     containers:
                       - name: probe
                         image: busybox:1.36.1
+                        # Both values come from the cluster under test: the suffix from
+                        # flux-system/cluster-vars, the address from this Service's own status.
+                        env:
+                          - name: BASE_DOMAIN
+                            value: ($baseDomain)
+                          - name: LB_IP
+                            value: ($lbIp)
                         command:
                           - sh
                           - -c
                           - |
                             set -u
-                            host="conformance-dns.home.keithjackson.dev"
+                            host="conformance-dns.${BASE_DOMAIN}"
                             # Poll PUBLIC DNS (1.1.1.1) until the auto-created A record appears and
-                            # points into the cluster LB range (Cilium L2 `shared` pool 192.168.1.53-.60). Proves ExternalDNS created it.
+                            # points at THIS Service's assigned LB IP. Proves ExternalDNS created it.
                             for i in $(seq 1 28); do
-                              ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | grep -oE '192\.168\.1\.(5[3-9]|60)' | head -1)"
+                              ip="$(nslookup "$host" 1.1.1.1 2>/dev/null | awk '/^Address: /{print $2}' | grep -Fx "${LB_IP}" | head -1)"
                               if [ -n "$ip" ]; then echo "RECORD_FOUND $host -> $ip"; exit 0; fi
-                              echo "poll $i: record not yet published"; sleep 10
+                              echo "poll $i: record not yet published as ${LB_IP}"; sleep 10
                             done
-                            echo "FAIL: ExternalDNS did not publish the record within timeout"; exit 1
+                            echo "FAIL: ExternalDNS did not publish ${host} -> ${LB_IP} within timeout"; exit 1
         - assert:
             resource:
               apiVersion: batch/v1