jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
149 lines
10 KiB
YAML
149 lines
10 KiB
YAML
---
|
|
# Cluster-wide PersistentVolume capacity alerts (docs/10, R-OBS-04).
|
|
#
|
|
# WHY THIS FILE EXISTS, AND WHY IT IS NOT IN alerts.yaml. platform/monitoring/alerts.yaml has exactly
|
|
# ONE PVC-fill rule, `MonitoringVolumeFillingUp`, and it is scoped `namespace="monitoring"`. Verified
|
|
# by grep 2026-09-06: it is the repo's ONLY use of `kubelet_volume_stats_*`, so before this file NO
|
|
# database PVC — and in fact no PVC outside the monitoring namespace — was covered by any volume-fill
|
|
# alert at all. That gap is half of the 2026-09-06 CNPG archiving incident (see platform/cnpg/
|
|
# alerts.yaml): keycloak-db's 5 GiB PVC was accumulating unrecycled WAL toward a disk-full PANIC on
|
|
# the SSO database with nothing watching it.
|
|
#
|
|
# NOTE, because it changes what "covered" means: `platform-health.longhorn`'s
|
|
# `LonghornVolumeCapacityHigh` looks like it might have covered these PVCs. It does not — the
|
|
# longhorn_* series are ABSENT from Prometheus on BOTH clusters (verified read-only 2026-09-06:
|
|
# `count(longhorn_volume_capacity_bytes)` is empty on dev and on prod), so that whole rule group is
|
|
# currently inert. That is a separate defect (a missing Longhorn ServiceMonitor/metrics config) and
|
|
# is NOT fixed here; these rules deliberately key on `kubelet_volume_stats_*`, which is present and
|
|
# working on both clusters.
|
|
#
|
|
# The rules live in a NEW FILE rather than in alerts.yaml on purpose: alerts.yaml is under concurrent
|
|
# edit, and a second file needs no coordination. Wiring is identical either way — PrometheusRule is a
|
|
# CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack, so it is applied by the
|
|
# `platform-config` Flux Kustomization (dependsOn: platform), NOT the controllers pass (F-MON1), and
|
|
# `release: kube-prometheus-stack` is what makes the operator's Prometheus select it. It IS referenced
|
|
# from platform-config/kustomization.yaml.
|
|
#
|
|
# ------------------------------------------------------------------------------------------------
|
|
# HOW DOUBLE-FIRING WITH MonitoringVolumeFillingUp IS AVOIDED — by exact partition, not by luck.
|
|
#
|
|
# Two facts, both established live rather than assumed:
|
|
#
|
|
# 1. `kubelet_volume_stats_*` is published by TWO scrape jobs on these clusters, and they carry
|
|
# DIFFERENT labels for the same PVC:
|
|
# job="kubelet" -> namespace=<the PVC's real namespace>, node=<node>
|
|
# job="apiserver" -> namespace="default", exported_namespace=<the PVC's real namespace>
|
|
# Both jobs report the same 18 PVCs. `MonitoringVolumeFillingUp{namespace="monitoring"}` therefore
|
|
# only ever matches the job="kubelet" copy (the apiserver copy says namespace="default"). Every
|
|
# rule below pins `job="kubelet"` too, so both rule sets range over the SAME series universe and
|
|
# `namespace!="monitoring"` is an EXACT complement of `namespace="monitoring"`. Without the job
|
|
# pin, `namespace!="monitoring"` would also match the apiserver copy of the monitoring PVCs (they
|
|
# say namespace="default") and this file WOULD double-fire on them — and would report the wrong
|
|
# namespace while doing it.
|
|
#
|
|
# 2. Every rule below is scoped `namespace!="monitoring"`, including the critical one. So no PVC is
|
|
# covered twice and no PVC is uncovered. The deliberate consequence: monitoring's own PVCs keep
|
|
# only the existing 15% warning and get no critical tier. Raising one belongs in alerts.yaml,
|
|
# which this file does not touch — recorded here as an explicit hand-off, not a silent gap.
|
|
#
|
|
# ------------------------------------------------------------------------------------------------
|
|
# WHY `and kubelet_volume_stats_inodes > 0` IS ON EVERY RULE — this guard is load-bearing.
|
|
#
|
|
# Block-mode PVCs (volumeMode: Block — the KubeVirt VM root disks) have no filesystem for the kubelet
|
|
# to stat, so it reports available=0, used=0, inodes=0 against a real capacity. `available/capacity`
|
|
# is therefore exactly 0 FOREVER. Verified live 2026-09-06: `unifi/unifi-rootdisk` (40 GiB, Block, on
|
|
# both clusters) reads 0.0. Without this guard, the warning AND the critical rule below would have
|
|
# fired permanently on it from the moment they were applied — precisely the permanently-firing-critical
|
|
# alert fatigue this repo has already been burned by (see the VPNTunnelDown post-mortem in
|
|
# alerts.yaml). `inodes > 0` is the "is there a real filesystem here?" test: any ext4/xfs volume has
|
|
# a non-zero inode total; a raw block device reports 0. Proven both ways on both clusters — with the
|
|
# guard the rules return the 15 filesystem PVCs outside monitoring and exclude unifi-rootdisk; without
|
|
# it, unifi-rootdisk is the only match at every threshold.
|
|
#
|
|
# LIVE STATE AT AUTHORING (read-only, both clusters, 2026-09-06): every rule below returns EMPTY on
|
|
# dev and on prod. Nothing here fires on day one; these are lead-time alarms, not a description of
|
|
# today. Fullest filesystem PVC outside monitoring: prod keycloak-db-1 at 93% free, dev keycloak-db-1
|
|
# at 88% free.
|
|
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: platform-health-pvc-capacity
|
|
namespace: monitoring
|
|
labels:
|
|
release: kube-prometheus-stack
|
|
platform-health: "true"
|
|
spec:
|
|
groups:
|
|
- name: platform-health.pvc-capacity
|
|
rules:
|
|
# Level alarm, warning tier. Thresholds deliberately IDENTICAL to MonitoringVolumeFillingUp
|
|
# (<15% free, 15m) so the two halves of the partition behave the same way; only the namespace
|
|
# scope differs.
|
|
- alert: PersistentVolumeFillingUp
|
|
expr: |
|
|
(
|
|
kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
|
|
/ kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
|
|
) < 0.15
|
|
and kubelet_volume_stats_inodes{job="kubelet"} > 0
|
|
for: 15m
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "PVC filling up: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
|
|
description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
|
# Level alarm, critical tier. This is the ONLY critical in this change: it fires only when a
|
|
# volume is genuinely about to run out, which for a Postgres data volume means an imminent
|
|
# disk-full PANIC. Nothing on either cluster is anywhere near 5% today, so it does not add a
|
|
# permanently-firing critical. There is no counterpart for this tier anywhere else in the
|
|
# repo, so it cannot duplicate one.
|
|
- alert: PersistentVolumeAlmostFull
|
|
expr: |
|
|
(
|
|
kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
|
|
/ kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
|
|
) < 0.05
|
|
and kubelet_volume_stats_inodes{job="kubelet"} > 0
|
|
for: 15m
|
|
labels:
|
|
severity: critical
|
|
annotations:
|
|
summary: "PVC almost full: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
|
|
description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
|
# TREND alarm — the lead-time rule, and the point of the exercise. A level threshold is useless
|
|
# for a slow leak on a mostly-empty volume: prod keycloak-db sat at 93% FREE while leaking
|
|
# ~6.8 MiB/h of unrecyclable WAL, so <15% was ~29 days away and would have arrived as a
|
|
# surprise. This rule instead asks "is this volume on course to hit zero?".
|
|
#
|
|
# 24h fit / 7d horizon: the 24h window is long enough to average out a daily backup or log
|
|
# rotation (a 6h window, as the kubernetes-mixin uses, chases those and flaps), and a 7d
|
|
# horizon gives a full week to react. `for: 1h` requires the projection to stay negative for an
|
|
# hour, so a single burst of writes cannot notify on its own.
|
|
#
|
|
# The `< 0.50` gate is what keeps this quiet: a volume with more than half its space free is
|
|
# not "about to fill" in any actionable sense, and predict_linear on a nearly-empty volume is
|
|
# dominated by noise. It costs some lead time on the WAL case — the CNPG rules in
|
|
# platform/cnpg/alerts.yaml are the day-one signal there, and this is the generic backstop for
|
|
# every OTHER PVC in the cluster, which is what was missing entirely.
|
|
#
|
|
# NOTE the monitoring namespace is excluded here too (partition rule above). That also
|
|
# side-steps predict_linear's classic false positive: a Prometheus TSDB filling toward its
|
|
# retention high-water mark looks exactly like an unbounded leak until it plateaus.
|
|
- alert: PersistentVolumeFillingUpPredicted
|
|
expr: |
|
|
(
|
|
kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
|
|
/ kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
|
|
) < 0.50
|
|
and predict_linear(kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}[24h], 7 * 24 * 3600) < 0
|
|
and kubelet_volume_stats_inodes{job="kubelet"} > 0
|
|
for: 1h
|
|
labels:
|
|
severity: warning
|
|
annotations:
|
|
summary: "PVC predicted to fill within 7 days: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
|
|
description: "Based on the last 24h of growth, PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is on course to run out of space within 7 days. Act now rather than at the 15%/5% thresholds: resize the PVC, prune data, or fix whatever is growing without bound (for a *-db-N volume that is usually failing WAL archiving — see platform/cnpg/alerts.yaml)."
|
|
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|