feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>

platform/monitoring/alerts.yaml

@@ -247,8 +247,37 @@ spec:
         # is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent()
         # also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering
         # gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24.
+        #
+        # FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO
+        # egress-gateway node it could never clear. Prod has no gateway node, and this alert fired
+        # continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel
+        # that must stay trustworthy. The expression is now GATED on the cluster actually being supposed
+        # to have a tunnel, via two independent guards OR'd together:
+        #
+        #   1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the
+        #      CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the
+        #      steering datapath has a target. Requires the kube-state-metrics node-label allowlist
+        #      enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is
+        #      what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no
+        #      manual re-enable to forget. The Node object survives a NotReady node, so this stays true
+        #      while the gateway node is DOWN: gateway-node-down is still alerted.
+        #   2. `tun0` was observed at all in the last hour — holds the alert armed through a
+        #      kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently
+        #      of the KSM rollout.
+        #
+        # `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the
+        # guards carry no labels, so an unqualified `and` would never match. Both clauses verified live
+        # against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on
+        # dev (tunnel up), and clause 2 already true on dev.
         - alert: VPNTunnelDown
-          expr: absent(node_network_receive_bytes_total{device="tun0"})
+          expr: |
+            absent(node_network_receive_bytes_total{device="tun0"})
+            and on()
+            (
+                (count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
+              or
+                (count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
+            )
           for: 5m
           labels:
             severity: critical
@@ -359,6 +388,56 @@ spec:
             summary: "Keycloak database unavailable"
             description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
             runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
+        # --- The client-secret reconciler (platform/keycloak/client-secret-reconciler.yaml) ---
+        # WHY THESE EXIST: on 2026-09-06 every OIDC login on prod failed because the realm import
+        # had stored the LITERAL `(env:...)` placeholder as all 12 client secrets — and every
+        # signal the cluster offered was GREEN (the import Job succeeded, the CR read Done=True,
+        # KeycloakDown/KeycloakDatabaseDown were quiet, the conformance test that asserts
+        # Done=True passed). The only observable was a human failing to log in. The CronJob is
+        # the healer; these two rules make its SILENCE observable, so the class cannot sit
+        # undetected again. They are the runtime counterpart of the L2 audit test.
+        #
+        # UNVERIFIED-ON-CLUSTER at authoring time: no CronJob existed on either cluster yet, so
+        # these series had no samples to check against. `kube_job_status_failed` /
+        # `kube_job_status_succeeded` and the `job_name` label WERE confirmed present (the realm
+        # import Job exports them); the reconciler's own series appear with its first run.
+        # Confirm with: kube_job_status_succeeded{namespace="keycloak",
+        # job_name=~"keycloak-client-secret-reconciler-.*"}
+        - alert: KeycloakClientSecretReconcilerFailing
+          # A reconciler run failed. Either it found drift it could not repair, or it could not
+          # reach the admin API. Fires alongside KeycloakDown during an IdP outage — that is
+          # deliberate (the reconciler genuinely is not running), so treat KeycloakDown as the
+          # cause when both are lit. Self-clears when the failed Jobs age out (TTL 1h).
+          expr: |
+            max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Keycloak client-secret reconciler failing"
+            description: >-
+              A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client
+              secret may differ from git (unresolved import placeholder, or an unrepairable
+              drift) and SSO logins for that client will fail. Read the Job log: it names the
+              client and never prints a secret.
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
+        - alert: KeycloakClientSecretReconcilerStale
+          # No SUCCESSFUL run in scope. Catches the failure modes the rule above cannot see: the
+          # CronJob suspended, never applied by Flux, or unable to schedule a pod at all — i.e.
+          # the healer being quietly absent, which is exactly the "looks applied and isn't"
+          # failure this whole change exists to end.
+          expr: |
+            absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
+          for: 2h
+          labels:
+            severity: warning
+          annotations:
+            summary: "Keycloak client-secret reconciler has not succeeded"
+            description: >-
+              No successful keycloak-client-secret-reconciler run in 2h (schedule is every 15m).
+              The CronJob may be suspended, unapplied, or unable to schedule — client-secret
+              drift would go uncorrected and unnoticed (docs/15).
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
 
 # =====================================================================================
 # The monitor's own capacity — don't let the monitor blind itself (docs/10)

platform/monitoring/kube-prometheus-stack.yaml

@@ -221,6 +221,17 @@ spec:
         limits:
           memory: 256Mi
     kube-state-metrics:
+      # Expose the egress-gateway NODE LABEL as a metric label so alerts can ask "does this cluster
+      # have a VPN egress gateway at all?". KSM exposes NO custom labels by default, so without this
+      # `kube_node_labels` does not exist and any join against it silently matches nothing.
+      # Consumed by `VPNTunnelDown` (platform/monitoring/alerts.yaml): the alert is armed only on a
+      # cluster that actually has a labelled gateway node, and ARMS ITSELF the moment one is labelled
+      # — which is how prod (no gateway today) stops emailing a permanently-firing critical alert
+      # without anyone having to remember to re-enable it later. Scoped to one label on one resource:
+      # 3 extra series, no cardinality concern. Label name is sanitized by KSM to
+      # `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
+      metricLabelsAllowlist:
+        - "nodes=[homelab.keithjackson.dev/egress-gateway]"
       resources:
         requests:
           cpu: 20m

platform/monitoring/pvc-capacity-alerts.yaml

@@ -0,0 +1,148 @@
+---
+# Cluster-wide PersistentVolume capacity alerts (docs/10, R-OBS-04).
+#
+# WHY THIS FILE EXISTS, AND WHY IT IS NOT IN alerts.yaml. platform/monitoring/alerts.yaml has exactly
+# ONE PVC-fill rule, `MonitoringVolumeFillingUp`, and it is scoped `namespace="monitoring"`. Verified
+# by grep 2026-09-06: it is the repo's ONLY use of `kubelet_volume_stats_*`, so before this file NO
+# database PVC — and in fact no PVC outside the monitoring namespace — was covered by any volume-fill
+# alert at all. That gap is half of the 2026-09-06 CNPG archiving incident (see platform/cnpg/
+# alerts.yaml): keycloak-db's 5 GiB PVC was accumulating unrecycled WAL toward a disk-full PANIC on
+# the SSO database with nothing watching it.
+#
+# NOTE, because it changes what "covered" means: `platform-health.longhorn`'s
+# `LonghornVolumeCapacityHigh` looks like it might have covered these PVCs. It does not — the
+# longhorn_* series are ABSENT from Prometheus on BOTH clusters (verified read-only 2026-09-06:
+# `count(longhorn_volume_capacity_bytes)` is empty on dev and on prod), so that whole rule group is
+# currently inert. That is a separate defect (a missing Longhorn ServiceMonitor/metrics config) and
+# is NOT fixed here; these rules deliberately key on `kubelet_volume_stats_*`, which is present and
+# working on both clusters.
+#
+# The rules live in a NEW FILE rather than in alerts.yaml on purpose: alerts.yaml is under concurrent
+# edit, and a second file needs no coordination. Wiring is identical either way — PrometheusRule is a
+# CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack, so it is applied by the
+# `platform-config` Flux Kustomization (dependsOn: platform), NOT the controllers pass (F-MON1), and
+# `release: kube-prometheus-stack` is what makes the operator's Prometheus select it. It IS referenced
+# from platform-config/kustomization.yaml.
+#
+# ------------------------------------------------------------------------------------------------
+# HOW DOUBLE-FIRING WITH MonitoringVolumeFillingUp IS AVOIDED — by exact partition, not by luck.
+#
+# Two facts, both established live rather than assumed:
+#
+#  1. `kubelet_volume_stats_*` is published by TWO scrape jobs on these clusters, and they carry
+#     DIFFERENT labels for the same PVC:
+#        job="kubelet"    -> namespace=<the PVC's real namespace>, node=<node>
+#        job="apiserver"  -> namespace="default", exported_namespace=<the PVC's real namespace>
+#     Both jobs report the same 18 PVCs. `MonitoringVolumeFillingUp{namespace="monitoring"}` therefore
+#     only ever matches the job="kubelet" copy (the apiserver copy says namespace="default"). Every
+#     rule below pins `job="kubelet"` too, so both rule sets range over the SAME series universe and
+#     `namespace!="monitoring"` is an EXACT complement of `namespace="monitoring"`. Without the job
+#     pin, `namespace!="monitoring"` would also match the apiserver copy of the monitoring PVCs (they
+#     say namespace="default") and this file WOULD double-fire on them — and would report the wrong
+#     namespace while doing it.
+#
+#  2. Every rule below is scoped `namespace!="monitoring"`, including the critical one. So no PVC is
+#     covered twice and no PVC is uncovered. The deliberate consequence: monitoring's own PVCs keep
+#     only the existing 15% warning and get no critical tier. Raising one belongs in alerts.yaml,
+#     which this file does not touch — recorded here as an explicit hand-off, not a silent gap.
+#
+# ------------------------------------------------------------------------------------------------
+# WHY `and kubelet_volume_stats_inodes > 0` IS ON EVERY RULE — this guard is load-bearing.
+#
+# Block-mode PVCs (volumeMode: Block — the KubeVirt VM root disks) have no filesystem for the kubelet
+# to stat, so it reports available=0, used=0, inodes=0 against a real capacity. `available/capacity`
+# is therefore exactly 0 FOREVER. Verified live 2026-09-06: `unifi/unifi-rootdisk` (40 GiB, Block, on
+# both clusters) reads 0.0. Without this guard, the warning AND the critical rule below would have
+# fired permanently on it from the moment they were applied — precisely the permanently-firing-critical
+# alert fatigue this repo has already been burned by (see the VPNTunnelDown post-mortem in
+# alerts.yaml). `inodes > 0` is the "is there a real filesystem here?" test: any ext4/xfs volume has
+# a non-zero inode total; a raw block device reports 0. Proven both ways on both clusters — with the
+# guard the rules return the 15 filesystem PVCs outside monitoring and exclude unifi-rootdisk; without
+# it, unifi-rootdisk is the only match at every threshold.
+#
+# LIVE STATE AT AUTHORING (read-only, both clusters, 2026-09-06): every rule below returns EMPTY on
+# dev and on prod. Nothing here fires on day one; these are lead-time alarms, not a description of
+# today. Fullest filesystem PVC outside monitoring: prod keycloak-db-1 at 93% free, dev keycloak-db-1
+# at 88% free.
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: platform-health-pvc-capacity
+  namespace: monitoring
+  labels:
+    release: kube-prometheus-stack
+    platform-health: "true"
+spec:
+  groups:
+    - name: platform-health.pvc-capacity
+      rules:
+        # Level alarm, warning tier. Thresholds deliberately IDENTICAL to MonitoringVolumeFillingUp
+        # (<15% free, 15m) so the two halves of the partition behave the same way; only the namespace
+        # scope differs.
+        - alert: PersistentVolumeFillingUp
+          expr: |
+            (
+              kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
+                / kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
+            ) < 0.15
+            and kubelet_volume_stats_inodes{job="kubelet"} > 0
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "PVC filling up: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
+            description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
+        # Level alarm, critical tier. This is the ONLY critical in this change: it fires only when a
+        # volume is genuinely about to run out, which for a Postgres data volume means an imminent
+        # disk-full PANIC. Nothing on either cluster is anywhere near 5% today, so it does not add a
+        # permanently-firing critical. There is no counterpart for this tier anywhere else in the
+        # repo, so it cannot duplicate one.
+        - alert: PersistentVolumeAlmostFull
+          expr: |
+            (
+              kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
+                / kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
+            ) < 0.05
+            and kubelet_volume_stats_inodes{job="kubelet"} > 0
+          for: 15m
+          labels:
+            severity: critical
+          annotations:
+            summary: "PVC almost full: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
+            description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
+        # TREND alarm — the lead-time rule, and the point of the exercise. A level threshold is useless
+        # for a slow leak on a mostly-empty volume: prod keycloak-db sat at 93% FREE while leaking
+        # ~6.8 MiB/h of unrecyclable WAL, so <15% was ~29 days away and would have arrived as a
+        # surprise. This rule instead asks "is this volume on course to hit zero?".
+        #
+        # 24h fit / 7d horizon: the 24h window is long enough to average out a daily backup or log
+        # rotation (a 6h window, as the kubernetes-mixin uses, chases those and flaps), and a 7d
+        # horizon gives a full week to react. `for: 1h` requires the projection to stay negative for an
+        # hour, so a single burst of writes cannot notify on its own.
+        #
+        # The `< 0.50` gate is what keeps this quiet: a volume with more than half its space free is
+        # not "about to fill" in any actionable sense, and predict_linear on a nearly-empty volume is
+        # dominated by noise. It costs some lead time on the WAL case — the CNPG rules in
+        # platform/cnpg/alerts.yaml are the day-one signal there, and this is the generic backstop for
+        # every OTHER PVC in the cluster, which is what was missing entirely.
+        #
+        # NOTE the monitoring namespace is excluded here too (partition rule above). That also
+        # side-steps predict_linear's classic false positive: a Prometheus TSDB filling toward its
+        # retention high-water mark looks exactly like an unbounded leak until it plateaus.
+        - alert: PersistentVolumeFillingUpPredicted
+          expr: |
+            (
+              kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
+                / kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
+            ) < 0.50
+            and predict_linear(kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}[24h], 7 * 24 * 3600) < 0
+            and kubelet_volume_stats_inodes{job="kubelet"} > 0
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "PVC predicted to fill within 7 days: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
+            description: "Based on the last 24h of growth, PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is on course to run out of space within 7 days. Act now rather than at the 15%/5% thresholds: resize the PVC, prune data, or fix whatever is growing without bound (for a *-db-N volume that is usually failing WAL archiving — see platform/cnpg/alerts.yaml)."
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"