feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
1 parent
d1a6628a97
commit
6a8953e288
46 files changed
+7898
-71
No files matched your search
@@ -180,9 +180,41 @@ path_requirements() {
|
||||
platform/externaldns/*) echo "R-DNS-02" ;;
|
||||
platform/velero/*) echo "R-STATE-03" ;;
|
||||
platform/monitoring/*) echo "R-OBS-01 R-OBS-02" ;;
|
||||
# platform/cnpg was UNMAPPED until 2026-09-06. It is the PostgreSQL DR path for keycloak-db /
|
||||
# nextcloud-db / semaphore-db (barmanObjectStore WAL archiving + ScheduledBackup), so a change
|
||||
# there is an R-IDP-05 / R-STATE-03 change; it now also carries the platform-health alert rules
|
||||
# for that path (R-OBS-04). Without this line an edit under platform/cnpg/ contributed ZERO
|
||||
# requirements unless it happened to retype an R-ID on a non-comment line, and escaped the gate.
|
||||
platform/cnpg/*) echo "R-IDP-05 R-STATE-03 R-OBS-04" ;;
|
||||
# vpn-egress was UNMAPPED until 2026-09-06. platform/vpn-egress/egress-steering.yaml IS the
|
||||
# R-NET-04 datapath (the rule that selects between the two egress paths) and the R-NET-05/06/07
|
||||
# steering policy — it even self-declares those IDs in its `homelab.keithjackson.dev/requirements`
|
||||
# annotation. But the content scan only reads ADDED/REMOVED lines, so any edit that does not
|
||||
# happen to retype an R-ID (a selector change, an excludedCIDRs change, a comment) contributed
|
||||
# ZERO requirements and escaped the gate entirely. Mapped explicitly; the annotation still wins
|
||||
# when a diff does carry the tags.
|
||||
platform/vpn-egress/*) echo "R-NET-04 R-NET-05 R-NET-06 R-NET-07" ;;
|
||||
# Keycloak is the whole SSO surface: the realm import (declarative identity config),
|
||||
# the SOPS client secrets, and the reconciler that heals them. Added 2026-09-06 after
|
||||
# the client-secret incident, in which editing platform/keycloak/ attributed NO
|
||||
# requirement, so the L2 keycloak tests were filtered out of the merge gate's touched
|
||||
# subset — the change that could break every OIDC login was gated by nothing.
|
||||
# R-IDP-05 is in scope because platform/keycloak/ also carries the CNPG DB Cluster and
|
||||
# its off-cluster backup credential, not only the realm.
|
||||
platform/keycloak/*) echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
|
||||
platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;;
|
||||
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
|
||||
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
|
||||
clusters/*) echo "R-GITOPS-01 R-ENV-02" ;;
|
||||
# bootstrap/ was UNMAPPED until 2026-09-06, and the omission was load-bearing:
|
||||
# bootstrap/standup.sh carries the placeholder guards that decide whether `velero` may be
|
||||
# enabled against a real backup endpoint. A change touching standup.sh ALONGSIDE any
|
||||
# already-mapped path scoped the L0 run to that other path's requirements and silently
|
||||
# excluded conformance/tests/l0-static/placeholder-guard-coupling — the check that exists
|
||||
# precisely to catch a guard/value mismatch. The 2026-09-05 guard bug (a prefix glob that
|
||||
# stopped matching when the value it guards changed, in a different file) is exactly the
|
||||
# class that omission would hide.
|
||||
bootstrap/*) echo "R-STATE-03 R-GITOPS-01" ;;
|
||||
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
|
||||
*) : ;;
|
||||
esac
|
||||
|
||||
Reference in new issue
Block a user