2 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	4687521d65	fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity ... LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume (actual_size 52243681280 / capacity 53687091200) while the kubelet reported the SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one user_created=false system snapshot from prod-1's replica rebuild holds 40510484480 bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same shape as CNPGNoRecoverabilityPoint. DELETED, not re-thresholded. Raising the threshold would tune away a true statement about allocation. Redundancy proven, not asserted: of 24 Longhorn volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp / PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block — no filesystem for anyone to measure, and the deleted rule was not covering it either. AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum the rules assumed: count(longhorn_volume_robustness) => 96 (24 volumes x 4 states) count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24 count(longhorn_volume_robustness == 2) => (empty) count(longhorn_volume_robustness == 3) => (empty) So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the "44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml was the same misreading and is corrected in place. The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a disk leaves the scheduling pool at 75% used — the old threshold could only fire after scheduling had already been refused. Replaced by: LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so one full disk could hide behind a half-empty sibling. Measured prod usage 0.2254 / 0.2306 / 0.2369 on prod-1/2/3. PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml adds 6 cases pinning every rule quiet on today's measured prod state and firing on a constructed bad state, with every `for:` asserted, plus regression assertions that the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on prod: all four new expressions return empty, while the sum of their selector cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58 alert rules unit-tested), which also brings MonitoringVolumeFillingUp, PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the redundancy claim rests on them. REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a 50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the longhorn StorageClass is feasible but not advisable now — remove-snapshots-during- filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB anyway, and continuous discard on nodes where the etcd WAL shares the spindle points at this cluster's known failure mode. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 17:19:03 -05:00
jakman85andClaude Opus 5	6a8953e288	feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing ... Minimal prod delivery, authored directly on main (= prod's live revision d1a6628) rather than by merging work/prod-standup-campaign. That branch carries the DNS apex inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun rewriting apex records dev owns while the non-k3s prod still serves 443 there. Operator sequencing is "dev migrates first". Nothing under clusters/ is touched here at all, which is what makes this safe to land now. Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of platform/velero/ and the CNPG destinationPath changes. Those land as feat/r2-backup-prod together with the endpoint and bucket knobs. WHAT THIS FIXES 1. Nothing was watching CNPG. All three prod databases have run since bootstrap with ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed anywhere. Because archive_mode=on and the archive command fails, Postgres refuses to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within the hour, and evaluates to 3 firing series on prod today. Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because nextcloud-db's exporter cannot log in to its own database (permission denied for database "nextcloud", both clusters) so the pg_stat_archiver family does not exist for it — the obvious metric would have been blind to one database in three. 2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to job="kubelet" (the apiserver copy of the same series reports namespace=default), and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report available=0 forever and would otherwise have fired critical permanently. 3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment a node carries the egress-gateway label, with dev's behaviour unchanged today. 4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored LITERALLY and the import still reports success, then sits at Done=True and never re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was green. The CronJob is the level-triggered correction; two alerts make its silence observable. Repaired live already — this stops it recurring. Also: the R-NET-04 conformance scan descended into nested agent worktrees under .claude/, counting 4 egress policies where 1 is authored. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 00:59:22 -05:00