feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform-config/kustomization.yaml

@@ -43,6 +43,11 @@ resources:
   - ../platform/longhorn/volumesnapshotclass.yaml
   # monitoring: PrometheusRule alert rules (need monitoring.coreos.com CRDs from kube-prometheus-stack)
   - ../platform/monitoring/alerts.yaml
+  # Cluster-wide PVC capacity rules. A SEPARATE FILE from alerts.yaml, whose only PVC-fill rule
+  # (MonitoringVolumeFillingUp) is scoped namespace="monitoring" — so before this, no database PVC
+  # was covered by any fill alert. Partitioned by namespace so the two never double-fire; see the
+  # file header. (2026-09-06, from the CNPG WAL/archiving incident.)
+  - ../platform/monitoring/pvc-capacity-alerts.yaml
   # kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the
   # operators register; the ServiceMonitors/alerts consume monitoring.coreos.com. All CRs of platform-
   # installed CRDs -> applied here, gated dependsOn platform (F-MON1). The dashboard is a plain
@@ -73,9 +78,21 @@ resources:
   # keycloak: the CNPG DB Cluster + Keycloak instance + realm import (CRs of the CNPG / Keycloak
   # operator CRDs installed by platform/cnpg + platform/keycloak) + its ingress + ServiceMonitor.
   # ${KEYCLOAK_REPLICAS} in keycloak-instance.yaml is resolved by this Kustomization's substituteFrom.
+  # cnpg: CloudNativePG platform-health alerts (WAL archiving / recoverability). PrometheusRule is a
+  # CR of monitoring.coreos.com, so it belongs to this pass, not the platform controllers pass —
+  # identical wiring to platform/kubevirt/alerts.yaml. Added 2026-09-06 after all three prod CNPG
+  # clusters ran 38h+ with ContinuousArchiving=False and firstRecoverabilityPoint=null, unalerted.
+  - ../platform/cnpg/alerts.yaml
   - ../platform/keycloak/postgres-cluster.yaml
   - ../platform/keycloak/keycloak-instance.yaml
   - ../platform/keycloak/keycloak-realm-import.yaml
+  # The realm import is a ONE-SHOT that fails OPEN: an unresolved secret placeholder is
+  # stored LITERALLY and the import still reports success, then rests at Done=True and
+  # never re-runs. This CronJob is the level-triggered loop that corrects that drift
+  # (2026-09-06 incident; see the file header). It belongs in THIS tier, not ./platform,
+  # so it applies after the keycloak-realm-clients Secret (rendered by ./platform) and
+  # alongside the import whose failure mode it covers.
+  - ../platform/keycloak/client-secret-reconciler.yaml
   - ../platform/keycloak/k8s-oidc-rbac.yaml   # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
   - ../platform/keycloak/httproute.yaml
   - ../platform/keycloak/servicemonitor.yaml