feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>

ci/l0.sh

@@ -137,6 +137,44 @@ else
   esac
 fi
 
+# ---- Stage 3d: Keycloak client-secret declaration + reconciler (UNCONDITIONAL) ----
+# The Keycloak realm import resolves each OIDC client's secret from an `(env:NAME)`
+# placeholder. When the operator cannot resolve one it stores the LITERAL placeholder as
+# the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS, then rests at Done=True and
+# never re-runs. On 2026-09-06 that had all 12 prod clients unusable ("Invalid client or
+# Invalid client credentials") behind an entirely green cluster; dev had one. The check
+# asserts the declaration is self-consistent (every placeholder defined, every Secret key
+# present, names matching the reconciler's contract), that the Flux ordering is a REAL
+# dependsOn edge rather than a comment, and — by extracting the shipped script from its
+# ConfigMap and running its --selftest — that the reconciler is compare-then-write,
+# idempotent, non-logging and blast-radius-bounded.
+#
+# WHY ITS OWN UNCONDITIONAL STAGE, same reasoning as 3c: Stage 3b scopes the conformance
+# run to the TOUCHED requirements (R-TEST-03). This check's invariant spans platform/,
+# platform-config/ and clusters/, so a change that breaks it need not attribute R-IDP-03/04
+# and the requirement-scoped selection would filter it out of exactly the change that
+# introduces the bug. It is a sub-second offline scan plus a loopback selftest, with no
+# side effects, so running it twice when 3b also selects it is not worth avoiding.
+KEYCLOAK_SECRET_CHECK="${KEYCLOAK_SECRET_CHECK:-${REPO_ROOT}/conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh}"
+log "Stage: Keycloak client-secret reconciler safety (${KEYCLOAK_SECRET_CHECK})"
+if [ ! -f "$KEYCLOAK_SECRET_CHECK" ]; then
+  skip "keycloak client-secret check not present at '${KEYCLOAK_SECRET_CHECK}' — UNWIRED, not green"
+  summary "- keycloak-client-secrets: **SKIPPED** (check missing)"
+  if [ "$rc" -eq 0 ]; then rc=3; fi
+else
+  kc_rc=0
+  bash "$KEYCLOAK_SECRET_CHECK" || kc_rc=$?
+  case "$kc_rc" in
+    0) ok "Keycloak client-secret declaration + reconciler passed"
+       summary "- keycloak-client-secrets: **passed**" ;;
+    3) skip "keycloak client-secret check SKIPPED (platform/keycloak absent) — not a pass"
+       summary "- keycloak-client-secrets: **SKIPPED** (nothing to audit)"
+       if [ "$rc" -eq 0 ]; then rc=3; fi ;;
+    *) err "Keycloak client-secret declaration + reconciler FAILED"
+       summary "- keycloak-client-secrets: **FAILED**"; rc=1 ;;
+  esac
+fi
+
 case "$rc" in
   0) ok "L0 stage complete" ;;
   3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;