feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/monitoring/alerts.yaml

@@ -247,8 +247,37 @@ spec:
         # is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent()
         # also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering
         # gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24.
+        #
+        # FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO
+        # egress-gateway node it could never clear. Prod has no gateway node, and this alert fired
+        # continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel
+        # that must stay trustworthy. The expression is now GATED on the cluster actually being supposed
+        # to have a tunnel, via two independent guards OR'd together:
+        #
+        #   1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the
+        #      CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the
+        #      steering datapath has a target. Requires the kube-state-metrics node-label allowlist
+        #      enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is
+        #      what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no
+        #      manual re-enable to forget. The Node object survives a NotReady node, so this stays true
+        #      while the gateway node is DOWN: gateway-node-down is still alerted.
+        #   2. `tun0` was observed at all in the last hour — holds the alert armed through a
+        #      kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently
+        #      of the KSM rollout.
+        #
+        # `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the
+        # guards carry no labels, so an unqualified `and` would never match. Both clauses verified live
+        # against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on
+        # dev (tunnel up), and clause 2 already true on dev.
         - alert: VPNTunnelDown
-          expr: absent(node_network_receive_bytes_total{device="tun0"})
+          expr: |
+            absent(node_network_receive_bytes_total{device="tun0"})
+            and on()
+            (
+                (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+              or
+                (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
+            )
           for: 5m
           labels:
             severity: critical
@@ -359,6 +388,56 @@ spec:
             summary: "Keycloak database unavailable"
             description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
+        # --- The client-secret reconciler (platform/keycloak/client-secret-reconciler.yaml) ---
+        # WHY THESE EXIST: on 2026-09-06 every OIDC login on prod failed because the realm import
+        # had stored the LITERAL `(env:...)` placeholder as all 12 client secrets — and every
+        # signal the cluster offered was GREEN (the import Job succeeded, the CR read Done=True,
+        # KeycloakDown/KeycloakDatabaseDown were quiet, the conformance test that asserts
+        # Done=True passed). The only observable was a human failing to log in. The CronJob is
+        # the healer; these two rules make its SILENCE observable, so the class cannot sit
+        # undetected again. They are the runtime counterpart of the L2 audit test.
+        #
+        # UNVERIFIED-ON-CLUSTER at authoring time: no CronJob existed on either cluster yet, so
+        # these series had no samples to check against. `kube_job_status_failed` /
+        # `kube_job_status_succeeded` and the `job_name` label WERE confirmed present (the realm
+        # import Job exports them); the reconciler's own series appear with its first run.
+        # Confirm with: kube_job_status_succeeded{namespace="keycloak",
+        # job_name=~"keycloak-client-secret-reconciler-.*"}
+        - alert: KeycloakClientSecretReconcilerFailing
+          # A reconciler run failed. Either it found drift it could not repair, or it could not
+          # reach the admin API. Fires alongside KeycloakDown during an IdP outage — that is
+          # deliberate (the reconciler genuinely is not running), so treat KeycloakDown as the
+          # cause when both are lit. Self-clears when the failed Jobs age out (TTL 1h).
+          expr: |
+            max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Keycloak client-secret reconciler failing"
+            description: >-
+              A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client
+              secret may differ from git (unresolved import placeholder, or an unrepairable
+              drift) and SSO logins for that client will fail. Read the Job log: it names the
+              client and never prints a secret.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
+        - alert: KeycloakClientSecretReconcilerStale
+          # No SUCCESSFUL run in scope. Catches the failure modes the rule above cannot see: the
+          # CronJob suspended, never applied by Flux, or unable to schedule a pod at all — i.e.
+          # the healer being quietly absent, which is exactly the "looks applied and isn't"
+          # failure this whole change exists to end.
+          expr: |
+            absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
+          for: 2h
+          labels:
+            severity: warning
+          annotations:
+            summary: "Keycloak client-secret reconciler has not succeeded"
+            description: >-
+              No successful keycloak-client-secret-reconciler run in 2h (schedule is every 15m).
+              The CronJob may be suspended, unapplied, or unable to schedule — client-secret
+              drift would go uncorrected and unnoticed (docs/15).
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
 
 # =====================================================================================
 # The monitor's own capacity — don't let the monitor blind itself (docs/10)