feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
d1a6628a97
commit
6a8953e288
46 files changed
+7898
-71
No files matched your search
@@ -0,0 +1,182 @@
|
||||
---
|
||||
# CloudNativePG platform-health alerts (docs/10, R-OBS-04) — the DATABASE DURABILITY signals.
|
||||
#
|
||||
# WHY THIS FILE EXISTS (incident, 2026-09-06). All three prod CNPG clusters (keycloak/keycloak-db,
|
||||
# nextcloud-db/nextcloud-db, semaphore-db/semaphore-db) had `ContinuousArchiving=False` with
|
||||
# `barman-cloud-wal-archive: exit status 4` and `firstRecoverabilityPoint: null` from BOOTSTRAP
|
||||
# (38h+) and NOTHING TOLD ANYONE — it was found by hand. Before this file there was no CNPG alert
|
||||
# of any kind in platform/ or clusters/ (verified by grep, 2026-09-06).
|
||||
#
|
||||
# WHY IT IS NOT MERELY "NO BACKUPS": `archive_mode=on` + a FAILING archive_command means Postgres
|
||||
# REFUSES TO RECYCLE WAL. pg_wal grows monotonically until the data PVC is full, at which point the
|
||||
# postmaster PANICs. Measured live on prod keycloak-db: 17 unarchived segments / ~268 MiB on a 5 GiB
|
||||
# PVC (93% free), growing ~6.8 MiB/h — roughly 29 days to a disk-full PANIC on the cluster's SSO
|
||||
# database, which takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth down with it.
|
||||
# So the archiving rules below are BOTH a DR-coverage alarm (R-IDP-05/R-STATE-03) AND the early
|
||||
# warning for a storage outage. The generic PVC-fill backstop is the separate
|
||||
# platform/monitoring/pvc-capacity-alerts.yaml.
|
||||
#
|
||||
# WIRING: PrometheusRule is a CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack,
|
||||
# so this is applied by the `platform-config` Flux Kustomization (dependsOn: platform), NOT by the
|
||||
# controllers pass (F-MON1) — identical wiring to platform/monitoring/alerts.yaml and
|
||||
# platform/kubevirt/alerts.yaml. The `release: kube-prometheus-stack` label is what makes the
|
||||
# operator's Prometheus select these rules (ruleSelector). It IS referenced from
|
||||
# platform-config/kustomization.yaml — an alert file nobody includes is the "looks applied and isn't"
|
||||
# failure this repo keeps hitting.
|
||||
#
|
||||
# WHICH METRIC FAMILY, AND WHY IT MATTERS (empirically settled against BOTH live clusters, read-only,
|
||||
# 2026-09-06). CNPG's exporter publishes TWO independent families:
|
||||
#
|
||||
# * `cnpg_collector_*` — emitted by the instance manager itself (reads pg_wal/, the Cluster status
|
||||
# and the backup catalogue). Present for EVERY instance.
|
||||
# * `cnpg_pg_stat_archiver_*` — emitted only if the exporter's SQL user queries SUCCEED.
|
||||
#
|
||||
# Every rule below keys on `cnpg_collector_*`, deliberately. On BOTH clusters the nextcloud-db
|
||||
# exporter cannot log in to its own database:
|
||||
# FATAL: permission denied for database "nextcloud" (SQLSTATE 42501) [user cnpg_metrics_exporter]
|
||||
# so `cnpg_pg_stat_archiver_failed_count` / `_archived_count` DO NOT EXIST for nextcloud-db at all.
|
||||
# An archiving alert built on pg_stat_archiver — the obvious choice — would have been SILENTLY BLIND
|
||||
# to one of the three prod databases. `CNPGMetricsCollectorErroring` below alerts on that condition
|
||||
# itself so the blind spot cannot recur unnoticed.
|
||||
#
|
||||
# LIVE PROOF (read-only PromQL, both clusters, 2026-09-06 — prod fires, dev does not):
|
||||
# cnpg_collector_pg_wal_archive_status{value="ready"} prod: keycloak 17, nextcloud 7, semaphore 3
|
||||
# dev: 0, 0, 0 (min over 24h also 0)
|
||||
# cnpg_collector_first_recoverability_point prod: 0, 0, 0
|
||||
# dev: 1788018752 / 1788018762 / 1788018770
|
||||
# cnpg_collector_last_available_backup_timestamp prod: 0, 0, 0 dev: 0.1-3.1h old
|
||||
# cnpg_collector_pg_wal{value="count"} vs {value="max"} prod: 17 vs 64 dev: 36 vs 64 (neither fires yet)
|
||||
# cnpg_last_error prod+dev: nextcloud-db = 1
|
||||
#
|
||||
# SEVERITIES: every rule here is `warning` (-> the email-warning receiver). This is deliberate.
|
||||
# A database heading for disk-full deserves a warning with weeks of lead time, and this repo already
|
||||
# has alert fatigue from a permanently-firing `critical` (see the VPNTunnelDown post-mortem in
|
||||
# platform/monitoring/alerts.yaml). The `critical` tier for this failure mode is the PVC actually
|
||||
# running out of space — PersistentVolumeAlmostFull in platform/monitoring/pvc-capacity-alerts.yaml —
|
||||
# which fires only when an outage is genuinely imminent.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: platform-health-cnpg
|
||||
namespace: monitoring
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
platform-health: "true"
|
||||
spec:
|
||||
groups:
|
||||
- name: platform-health.cnpg
|
||||
rules:
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# THE DAY-ONE RULE. This is the one that would have caught the 2026-09-06 incident within
|
||||
# the hour instead of 38 hours later.
|
||||
#
|
||||
# pg_wal/archive_status holds one marker file per WAL segment: `.ready` = written, NOT yet
|
||||
# archived; `.done` = archived, safe to recycle. A HEALTHY archiver drains .ready within
|
||||
# seconds, so a persistent non-zero ready count means archiving is stuck — and every stuck
|
||||
# segment is 16 MiB that Postgres may not recycle.
|
||||
#
|
||||
# THRESHOLD/DURATION: `> 0` with `for: 1h`, not a count threshold. `for` requires the
|
||||
# condition to hold at EVERY evaluation for the full hour, so a write burst that transiently
|
||||
# queues a segment resets the timer and never notifies; but a genuinely stuck archiver never
|
||||
# returns to zero. Calibrated live: dev (working archiving) has ready=0 at every sample over
|
||||
# 24h; prod has never been below 3 (min_over_time 24h: keycloak 9, nextcloud 5, semaphore 3).
|
||||
# A count threshold like `> 5` would have delayed semaphore-db's alert indefinitely — its
|
||||
# backlog sits at 3 because it writes almost nothing.
|
||||
- alert: CNPGWALArchiveBacklog
|
||||
expr: |
|
||||
max by (namespace, pod) (cnpg_collector_pg_wal_archive_status{value="ready"}) > 0
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "CNPG WAL archiving stuck: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments unarchived)"
|
||||
description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} has had {{ $value }} WAL segment(s) waiting in pg_wal/archive_status for over an hour — barman-cloud-wal-archive is failing. TWO consequences: (1) there is no continuous archive, so PITR is broken (R-IDP-05/R-STATE-03); (2) archive_mode=on means Postgres will NOT recycle WAL, so pg_wal grows without bound until the data PVC fills and the postmaster PANICs. Check `kubectl logs -n {{ $labels.namespace }} {{ $labels.pod }} -c postgres | grep archive` and the Cluster's ContinuousArchiving condition."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# The DR-COVERAGE statement, independent of the mechanism above: this cluster has NO point in
|
||||
# time it can be recovered to. CNPG reports 0 when the Cluster status' firstRecoverabilityPoint
|
||||
# is null — exactly what prod showed for 38h. Distinct from CNPGWALArchiveBacklog: archiving can
|
||||
# be healthy while no base backup has ever completed (ScheduledBackup broken -> only this fires),
|
||||
# and an old base backup can exist while archiving has just broken (only the other fires).
|
||||
#
|
||||
# `for: 6h` because a legitimately fresh Cluster reads 0 until its first base backup lands
|
||||
# (ScheduledBackup immediate:true, but MinIO/bucket setup can lag) — 6h keeps a normal cold
|
||||
# start quiet while still catching a bootstrap that never produced a recoverability point.
|
||||
- alert: CNPGNoRecoverabilityPoint
|
||||
expr: |
|
||||
max by (namespace, pod) (cnpg_collector_first_recoverability_point) == 0
|
||||
for: 6h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "CNPG cluster has NO recoverability point: {{ $labels.namespace }}/{{ $labels.pod }}"
|
||||
description: "PostgreSQL instance {{ $labels.namespace }}/{{ $labels.pod }} reports firstRecoverabilityPoint = null for 6h — there is NO point-in-time this database can be restored to. Off-cluster DR for this database does not exist (R-IDP-05 / R-STATE-03). Usually the barmanObjectStore credentials/endpoint/path are wrong; check `kubectl get cluster -n {{ $labels.namespace }} -o yaml` for the ContinuousArchiving condition."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-archiving"
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# The daily ScheduledBackup silently stopped. Nothing else covers this: the Velero rules in
|
||||
# platform/monitoring/alerts.yaml watch Velero's own schedules, which do NOT back up these
|
||||
# databases (CNPG owns their DR path — a Velero PVC snapshot of a live Postgres is not a
|
||||
# backup). Guarded with `> 0` so a cluster that has NEVER had a backup is reported by
|
||||
# CNPGNoRecoverabilityPoint above and does not double-notify here (time()-0 would otherwise be
|
||||
# ~57 years and always true). 36h = one missed daily run plus slack.
|
||||
- alert: CNPGNoRecentBaseBackup
|
||||
expr: |
|
||||
(time() - max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp)) > (36 * 3600)
|
||||
and max by (namespace, pod) (cnpg_collector_last_available_backup_timestamp) > 0
|
||||
for: 2h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "No recent CNPG base backup: {{ $labels.namespace }}/{{ $labels.pod }}"
|
||||
description: "The most recent completed base backup for {{ $labels.namespace }}/{{ $labels.pod }} is over 36h old — the daily ScheduledBackup has stopped producing backups, so the PITR window is ageing out against the 30d retentionPolicy. Check `kubectl get backup -n {{ $labels.namespace }}`."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-backup-missed"
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# WAL-VOLUME-SPECIFIC ESCALATION. pg_wal holds more segments than max_wal_size allows for,
|
||||
# which means checkpoints are no longer reclaiming space — the mechanism that actually fills
|
||||
# the PVC. Both series come from the same metric under different `value` labels, so each side
|
||||
# is aggregated `by (namespace, pod)` to drop `value` and give the comparison a clean matching
|
||||
# label set. On CNPG the units are SEGMENTS (16 MiB each): `max` = max_wal_size (64 = 1 GiB on
|
||||
# both clusters), `count` = segments currently in pg_wal.
|
||||
#
|
||||
# This does NOT fire today (prod keycloak-db 17 vs 64, dev 36 vs 64) and it is not meant to:
|
||||
# CNPGWALArchiveBacklog is the day-one signal. This one is the "the backlog is now large enough
|
||||
# to be a storage problem" escalation, and it still leads the PVC-fill alerts by weeks —
|
||||
# at prod's observed ~6.8 MiB/h keycloak-db reaches 64 segments ~40 days before the 5 GiB PVC
|
||||
# runs out. Note pg_wal shares the DATA PVC here (no separate `walStorage:` is configured on
|
||||
# any Cluster), which is why the PVC-fill rules cover it at all; the CNPG-native
|
||||
# `cnpg_collector_pg_wal{value="volume_size"}` is NaN on both clusters for that reason and is
|
||||
# therefore NOT usable as a signal.
|
||||
- alert: CNPGWALVolumeNotRecycling
|
||||
expr: |
|
||||
max by (namespace, pod) (cnpg_collector_pg_wal{value="count"})
|
||||
> max by (namespace, pod) (cnpg_collector_pg_wal{value="max"})
|
||||
for: 2h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "CNPG WAL not being recycled: {{ $labels.namespace }}/{{ $labels.pod }} ({{ $value }} segments)"
|
||||
description: "pg_wal on {{ $labels.namespace }}/{{ $labels.pod }} holds {{ $value }} segments, more than max_wal_size, for 2h — checkpoints are not reclaiming WAL. With archive_mode=on this is what a failing archive_command looks like once it has been failing for a while; pg_wal shares the data PVC, so this ends in a disk-full PANIC. Fix archiving (see CNPGWALArchiveBacklog) — do NOT delete WAL by hand."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-wal-growth"
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# THE MONITOR'S OWN BLIND SPOT, for databases. cnpg_last_error=1 means the exporter's SQL user
|
||||
# queries are failing, which BLANKS the whole `cnpg_pg_stat_*` family for that instance while
|
||||
# `up` stays 1 and `cnpg_collector_up` stays 1 — a failure that looks exactly like health.
|
||||
#
|
||||
# THIS FIRES TODAY, ON BOTH CLUSTERS, FOR nextcloud-db. It is a TRUE POSITIVE, verified live
|
||||
# 2026-09-06: `cnpg_errors_total` carries
|
||||
# pg_stat_archiver on db nextcloud: failed to connect to `user=cnpg_metrics_exporter
|
||||
# database=nextcloud`: FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
|
||||
# i.e. nextcloud-db has been publishing NO pg_stat_archiver / pg_stat_database metrics at all.
|
||||
# Remedy: grant the exporter role CONNECT on the application database. `for: 6h` keeps a pod
|
||||
# restart or a brief connection blip quiet; severity stays `warning` so it never becomes the
|
||||
# permanently-firing `critical` this repo has been burned by.
|
||||
- alert: CNPGMetricsCollectorErroring
|
||||
expr: |
|
||||
max by (namespace, pod) (cnpg_last_error) == 1
|
||||
for: 6h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "CNPG metrics collector erroring: {{ $labels.namespace }}/{{ $labels.pod }}"
|
||||
description: "The CloudNativePG exporter on {{ $labels.namespace }}/{{ $labels.pod }} has been failing its SQL collection queries for 6h, so the entire cnpg_pg_stat_* family (archiver, database, replication) is MISSING for this instance while the scrape target still reports up=1. Inspect the reason with `cnpg_errors_total` (its `errorUserQueries` label carries the message) — commonly the cnpg_metrics_exporter role lacks CONNECT on the application database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cnpg-exporter"
|
||||
@@ -29,6 +29,7 @@ files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate mai
|
||||
| `keycloak-instance.yaml` (`Keycloak` CR, `${KEYCLOAK_REPLICAS}`) | **platform-config** | config |
|
||||
| `postgres-cluster.yaml` (CNPG `Cluster` on longhorn) | **platform-config** | config |
|
||||
| `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
|
||||
| `client-secret-reconciler.yaml` (CronJob: client secrets ← git, every 15m) | **platform-config** | config |
|
||||
| `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
|
||||
| `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
|
||||
|
||||
@@ -47,10 +48,14 @@ operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO
|
||||
|
||||
## Operator hand-offs (fill before / during deploy)
|
||||
|
||||
1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. Two client
|
||||
secrets (`grafana-client-secret`, `oauth2-proxy-client-secret`), `openssl rand -hex 32` each. The
|
||||
realm import injects them into the Keycloak clients via `spec.placeholders`. Add to
|
||||
`kustomization.yaml` once filled. The realm import stays PENDING until this exists.
|
||||
1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. One
|
||||
`<clientId>-client-secret` key per confidential client, `openssl rand -hex 32` each. The realm
|
||||
import injects them into the Keycloak clients via `spec.placeholders`. Add to
|
||||
`kustomization.yaml` once filled.
|
||||
**The key name MUST be `<clientId>-client-secret`** — the reconciler below derives the clientId
|
||||
from it, and CI rejects any other name.
|
||||
*(This step used to read "the realm import stays PENDING until this exists". It does not — see
|
||||
the next section. That false belief is what allowed the 2026-09-06 outage.)*
|
||||
2. **`../oauth2-proxy/oauth2-proxy-credentials.sops.yaml`** (oauth2-proxy ns) — `client-id: oauth2-proxy`,
|
||||
`client-secret` = the SAME `oauth2-proxy-client-secret` above, `cookie-secret` = `openssl rand -base64 32`.
|
||||
3. **`../monitoring/grafana-oidc-credentials.sops.yaml`** (monitoring ns) — `client-secret` = the SAME
|
||||
@@ -60,6 +65,45 @@ operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO
|
||||
import in `keycloak`, once for the consuming app.) See the two `.sops.yaml.example` templates for the
|
||||
exact keys + steps. No plaintext secret is ever committed.
|
||||
|
||||
## Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)
|
||||
|
||||
**The realm import is a one-shot that fails open.** Measured on both clusters:
|
||||
|
||||
* When the operator **cannot resolve** a `$(env:...)` placeholder it stores **the literal placeholder
|
||||
text** as the client secret — and the import Job still exits 0. There is no PENDING state, no
|
||||
error condition, and `HasErrors` stays `False`.
|
||||
* Once complete the CR rests at **`Done=True` / `Started=False` and never re-runs**, so a Secret key
|
||||
added later is never picked up.
|
||||
|
||||
Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder;
|
||||
every login rejected with *"Invalid client or Invalid client credentials"*) and a single broken client
|
||||
on dev (`oauth2-proxy`, whose key was added after dev's import had already completed). Every cluster
|
||||
signal was green throughout, including the conformance test that asserts `Done=True`.
|
||||
|
||||
Three layers now cover it:
|
||||
|
||||
| Layer | Where | What it does |
|
||||
|---|---|---|
|
||||
| **Prevent** | `conformance/tests/l0-static/keycloak-client-secret-reconciler` (runs on every merge, unconditionally, from `ci/l0.sh`) | Every placeholder is defined, reads `keycloak-realm-clients`, and names an **existing** key called `<clientId>-client-secret`; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive `dependsOn` ancestor (`wait: true`) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable. |
|
||||
| **Heal** | `client-secret-reconciler.yaml` — CronJob, every 15m | Compares each client's **stored** secret with the mounted `keycloak-realm-clients` value and PUTs the **minimal** `{clientId, secret}` representation only where they differ. Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (`automountServiceAccountToken: false`). |
|
||||
| **Detect** | `conformance/tests/l2-behavioral/keycloak-client-secrets-match-git` + alerts `KeycloakClientSecretReconcilerFailing` / `…Stale` | Runs the same script in read-only `--audit` mode against the live realm and fails on any mismatch or literal placeholder; the alerts make the reconciler's silence observable in prod. |
|
||||
|
||||
The reconciler's own `--selftest` (run by the L0 gate, which extracts the script from the shipped
|
||||
ConfigMap) proves compare-then-write, **idempotence** (zero writes on a converged realm),
|
||||
no-secret-logging, the endpoint allow-list, and fail-closed behaviour.
|
||||
|
||||
**Manual repair / one-off run** (the CronJob normally does this):
|
||||
|
||||
```sh
|
||||
kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
|
||||
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1
|
||||
```
|
||||
|
||||
**What the reconciler does NOT fix:** anything other than client secrets. A NEW client, a changed
|
||||
redirect URI or a new protocol mapper still needs the realm import to actually re-run (it will not on
|
||||
its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that a realm edit
|
||||
landed.
|
||||
|
||||
## Exceptions that keep native auth (do not force SSO) — ADR-0014
|
||||
|
||||
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
|
||||
@@ -0,0 +1,720 @@
|
||||
---
|
||||
# =============================================================================
|
||||
# Keycloak CLIENT-SECRET RECONCILER (R-IDP-03, R-IDP-04)
|
||||
#
|
||||
# CARRIED BY platform-config (dependsOn platform, wait:true), so it applies only
|
||||
# after the `platform` tier has applied the SOPS-encrypted keycloak-realm-clients
|
||||
# Secret and the Keycloak Operator, and alongside the realm import itself.
|
||||
#
|
||||
# THE FAILURE THIS EXISTS FOR (measured on both clusters, 2026-09-06)
|
||||
# -------------------------------------------------------------------
|
||||
# keycloak-realm-import.yaml declares 12 confidential clients whose secrets are
|
||||
# import placeholders resolved from the keycloak-realm-clients Secret via
|
||||
# spec.placeholders. When the operator cannot resolve a placeholder it stores the
|
||||
# LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS
|
||||
# SUCCESS. The KeycloakRealmImport then rests at Done=True / Started=False and
|
||||
# NEVER RE-RUNS, so nothing ever corrects it:
|
||||
#
|
||||
# * prod: all 12 clients held the literal placeholder. Every OIDC login on the
|
||||
# cluster failed with "Invalid client or Invalid client credentials".
|
||||
# * dev: exactly one (oauth2-proxy) was broken -- that Secret key was added
|
||||
# AFTER dev's realm import had already completed, and a completed import is
|
||||
# never re-run, so the new key was never picked up.
|
||||
#
|
||||
# Both were repaired live via the admin API on 2026-09-06 and verified. This
|
||||
# CronJob is what makes the class non-recurring: a ONE-SHOT that fails open is
|
||||
# replaced by a LEVEL-TRIGGERED loop that converges every 15 minutes, the same
|
||||
# posture Flux gives every other resource in this repo (R-GITOPS-03 drift
|
||||
# correction, applied to the one piece of state Flux itself cannot reach --
|
||||
# Keycloak's own database).
|
||||
#
|
||||
# ORDERING IS NOT ENOUGH, WHICH IS WHY THIS IS THE PRIMARY FIX. The tier graph
|
||||
# already guarantees the Secret is applied before the realm import (the Secret is
|
||||
# rendered by ./platform; the import by ./platform-config, which dependsOn
|
||||
# platform with wait:true, and Flux's dependsOn is REVISION-gated so revision N of
|
||||
# the Secret lands before revision N of the import). Ordering still cannot help,
|
||||
# because the operator will not re-run a completed import when a LATER revision
|
||||
# adds a key -- which is precisely how dev broke. Only a reconciler closes that.
|
||||
#
|
||||
# WHY A CronJob AND NOT A KYVERNO POLICY OR AN OPERATOR PATCH: the drifted state
|
||||
# lives in Keycloak's Postgres database, not in a Kubernetes object, so no
|
||||
# admission or mutation policy can see it. The admin REST API is the only
|
||||
# interface to it.
|
||||
#
|
||||
# WHAT IT MAY TOUCH: the reconciler holds a hard endpoint allow-list (token,
|
||||
# client lookup, client-secret read, minimal client PUT). Users, groups, roles,
|
||||
# authentication flows and realm settings are structurally unreachable from it --
|
||||
# prod's `homelab` realm carries a real user and the groups media-users /
|
||||
# platform-admins / viewers, and they must survive every run. It never logs a
|
||||
# secret, and it writes only where the stored value differs from git.
|
||||
#
|
||||
# PROOF: the script's own `--selftest` drives these real code paths against an
|
||||
# in-process stub Keycloak and asserts compare-then-write, idempotence (a second
|
||||
# run over a converged realm issues ZERO writes), no-secret-logging, the endpoint
|
||||
# allow-list and fail-closed behaviour. The L0 static gate EXTRACTS the script
|
||||
# from this ConfigMap and runs it, so what CI proves is byte-for-byte what ships:
|
||||
# conformance/tests/l0-static/keycloak-client-secret-reconciler/.
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: keycloak-client-secret-reconciler
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak-client-secret-reconciler
|
||||
app.kubernetes.io/part-of: keycloak
|
||||
data:
|
||||
# SINGLE SOURCE OF TRUTH for the reconciler. Do not fork it into a sidecar copy:
|
||||
# the L0 gate extracts THIS block and executes its --selftest, so a second copy
|
||||
# would be untested. Contains no dollar sign at all, on purpose -- this file is
|
||||
# rendered by a Flux Kustomization with postBuild.substituteFrom, so an
|
||||
# unescaped dollar-brace expansion here would be rewritten before any container
|
||||
# ran (the trap documented at length in apps/wave-1/nextcloud.yaml).
|
||||
reconcile-client-secrets.py: |
|
||||
#!/usr/bin/env python3
|
||||
# =============================================================================
|
||||
# reconcile-client-secrets.py -- make every Keycloak OIDC client's STORED secret
|
||||
# equal the value git declares for it, idempotently, without touching anything
|
||||
# else in the realm.
|
||||
#
|
||||
# WHY THIS EXISTS (the incident it prevents)
|
||||
# ------------------------------------------
|
||||
# keycloak-realm-import.yaml declares each confidential client's secret as an
|
||||
# `(env:<NAME>_CLIENT_SECRET)` import placeholder and maps <NAME> to a key of the
|
||||
# SOPS-encrypted `keycloak-realm-clients` Secret via spec.placeholders. When the
|
||||
# Keycloak Operator cannot resolve a placeholder at import time it stores the
|
||||
# LITERAL placeholder text as the client secret AND THE IMPORT JOB STILL REPORTS
|
||||
# SUCCESS. The KeycloakRealmImport then sits at Done=True / Started=False and
|
||||
# never re-runs, so it never self-heals. Measured 2026-09-06: on prod all 12
|
||||
# clients with secrets held the literal placeholder and every OIDC login failed
|
||||
# with "Invalid client or Invalid client credentials"; on dev exactly one
|
||||
# (oauth2-proxy) was broken, because that Secret key was added after dev's import
|
||||
# had already completed.
|
||||
#
|
||||
# So the import is a ONE-SHOT that fails OPEN. This is the level-triggered
|
||||
# reconciler that closes it.
|
||||
#
|
||||
# CONTRACT
|
||||
# --------
|
||||
# Desired state files in SECRETS_DIR named `<clientId>-client-secret`, each
|
||||
# holding exactly the secret for that clientId (this is the
|
||||
# `keycloak-realm-clients` Secret, mounted).
|
||||
# Actual state GET /admin/realms/<realm>/clients/<uuid>/client-secret
|
||||
# Action only where they differ: PUT /admin/realms/<realm>/clients/<uuid>
|
||||
# with the MINIMAL representation {"clientId":..,"secret":..}.
|
||||
#
|
||||
# WHY MINIMAL: Keycloak merges non-null fields of a client representation, so a
|
||||
# two-field PUT rewrites the secret and nothing else. PUTting the full
|
||||
# round-tripped representation back has a far larger blast radius and is in fact
|
||||
# rejected -- 400 {"error":"unknown_error","error_description":"Cannot parse the
|
||||
# JSON"} (measured during the live repair). The stub in --selftest models that
|
||||
# rejection, so a regression to the fat PUT fails in CI, not on a live realm.
|
||||
#
|
||||
# SAFETY PROPERTIES, each enforced in code and proved by --selftest:
|
||||
# 1. compare-then-write: a client already correct is SKIPPED, no PUT issued.
|
||||
# 2. idempotent: a second run over a converged realm issues zero writes.
|
||||
# 3. no secret is ever printed, in any mode, on any path.
|
||||
# 4. the ONLY endpoints this program may touch are the master-realm token
|
||||
# endpoint, the client LOOKUP, the client-secret READ and the minimal client
|
||||
# PUT. Every other URL is refused by _http() before a socket is opened, so
|
||||
# users, groups, roles, authentication flows and realm settings are
|
||||
# structurally unreachable from here -- prod's `homelab` realm carries a
|
||||
# real user and the groups media-users / platform-admins / viewers, and they
|
||||
# MUST survive.
|
||||
# 5. public clients (e.g. `kubernetes`: PKCE, no secret) are skipped.
|
||||
#
|
||||
# MODES
|
||||
# (default) reconcile: compare, and write only what differs.
|
||||
# --audit READ-ONLY. Report drift and exit 1 if any client's stored secret
|
||||
# differs from the declared one. Never writes. This is what the L2
|
||||
# conformance test runs.
|
||||
# --selftest the offline behavioural proof, against an in-process stub
|
||||
# Keycloak. No cluster; loopback only. This is what the L0 static
|
||||
# gate runs, so the code the gate proves is the code that ships.
|
||||
#
|
||||
# NOT ONE DOLLAR SIGN IN THIS FILE, DELIBERATELY -- not even a regex end-anchor
|
||||
# (hence re.fullmatch throughout), and not even inside these comments. It ships
|
||||
# inside a ConfigMap that the `platform-config` Flux Kustomization renders with
|
||||
# postBuild.substituteFrom, so Flux envsubst rewrites this text before any
|
||||
# container runs: an unescaped dollar-brace expansion would be silently emptied at
|
||||
# reconcile time. That is the exact trap that made apps/wave-1/nextcloud.yaml's
|
||||
# OIDC guard dead code for months -- read the comment block there. Python needs no
|
||||
# dollar signs at all, so the trap is avoided STRUCTURALLY rather than escaped, and
|
||||
# the character is banned outright so no future edit has to reason about escaping.
|
||||
# Enforced by conformance/tests/l0-static/keycloak-client-secret-reconciler/check.sh.
|
||||
# =============================================================================
|
||||
from __future__ import annotations
|
||||
|
||||
import hmac
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
SECRETS_DIR = os.environ.get("CLIENT_SECRETS_DIR", "/etc/keycloak-realm-clients")
|
||||
KEY_SUFFIX = "-client-secret"
|
||||
TIMEOUT = 20
|
||||
READY_ATTEMPTS = int(os.environ.get("READY_ATTEMPTS", "24"))
|
||||
READY_SLEEP = float(os.environ.get("READY_SLEEP", "5"))
|
||||
|
||||
# Endpoint allow-list (safety property 4). fullmatch-ed against the path of every
|
||||
# URL _http() is asked to open; anything else raises before a socket exists.
|
||||
_ALLOWED = (
|
||||
r"/realms/master/protocol/openid-connect/token",
|
||||
r"/admin/realms/[^/]+/clients",
|
||||
r"/admin/realms/[^/]+/clients/[^/]+",
|
||||
r"/admin/realms/[^/]+/clients/[^/]+/client-secret",
|
||||
)
|
||||
|
||||
|
||||
class Refused(RuntimeError):
|
||||
"""A URL outside the allow-list was requested. Never expected in a normal run."""
|
||||
|
||||
|
||||
def _http(method, base, path, query=None, headers=None, form=None, body=None):
|
||||
"""Return (status, text). Refuses any path outside the allow-list."""
|
||||
if not any(re.fullmatch(p, path) for p in _ALLOWED):
|
||||
raise Refused("endpoint not on the allow-list: %s %s" % (method, path))
|
||||
url = base.rstrip("/") + path
|
||||
if query:
|
||||
url = url + "?" + urllib.parse.urlencode(query)
|
||||
hdrs = dict(headers or {})
|
||||
data = None
|
||||
if form is not None:
|
||||
data = urllib.parse.urlencode(form).encode()
|
||||
hdrs["Content-Type"] = "application/x-www-form-urlencoded"
|
||||
elif body is not None:
|
||||
data = json.dumps(body).encode()
|
||||
hdrs["Content-Type"] = "application/json"
|
||||
req = urllib.request.Request(url, data=data, headers=hdrs, method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
|
||||
return resp.status, resp.read().decode("utf-8", "replace")
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code, e.read().decode("utf-8", "replace")
|
||||
|
||||
|
||||
def declared(secrets_dir):
|
||||
"""{clientId: secret} from the mounted Secret.
|
||||
|
||||
Kubernetes projects a Secret volume as symlinks beside dot-prefixed internal
|
||||
entries (`..data`, `..2026_09_06_...`); the leading-dot filter skips those. A
|
||||
key must be named `<clientId>-client-secret`: that naming contract is what
|
||||
lets this reconciler derive the client set from git without a second
|
||||
declaration, and the L0 static check enforces it against
|
||||
keycloak-realm-import.yaml so the two can never drift apart.
|
||||
"""
|
||||
out = {}
|
||||
for name in sorted(os.listdir(secrets_dir)):
|
||||
if name.startswith(".") or not name.endswith(KEY_SUFFIX):
|
||||
continue
|
||||
path = os.path.join(secrets_dir, name)
|
||||
if not os.path.isfile(path):
|
||||
continue
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
value = fh.read()
|
||||
# A trailing newline in the SOPS source would otherwise become part of the
|
||||
# secret and mismatch forever. Keycloak secrets never contain whitespace.
|
||||
out[name[: -len(KEY_SUFFIX)]] = value.strip()
|
||||
return out
|
||||
|
||||
|
||||
def admin_token(base, user, password, log):
|
||||
"""Bearer token for admin-cli in the master realm, with a bounded readiness wait."""
|
||||
last = ""
|
||||
for attempt in range(1, READY_ATTEMPTS + 1):
|
||||
try:
|
||||
st, text = _http(
|
||||
"POST", base, "/realms/master/protocol/openid-connect/token",
|
||||
form={"client_id": "admin-cli", "grant_type": "password",
|
||||
"username": user, "password": password})
|
||||
except OSError as e:
|
||||
st, text = 0, "connection error (%s)" % e.__class__.__name__
|
||||
if st == 200:
|
||||
token = json.loads(text).get("access_token")
|
||||
if token:
|
||||
return token
|
||||
last = "http 200 but no access_token in the response"
|
||||
else:
|
||||
# Deliberately do NOT echo the response body: it is harmless today, but
|
||||
# the request that produced it carried the admin password, and keeping
|
||||
# that class of mistake impossible by habit is cheaper than reviewing it.
|
||||
last = "http %s" % st
|
||||
log("waiting for the Keycloak admin API (attempt %d/%d): %s"
|
||||
% (attempt, READY_ATTEMPTS, last))
|
||||
if attempt < READY_ATTEMPTS:
|
||||
time.sleep(READY_SLEEP)
|
||||
raise RuntimeError("Keycloak admin API not usable after %d attempt(s): %s"
|
||||
% (READY_ATTEMPTS, last))
|
||||
|
||||
|
||||
def reconcile(base, realm, token, want, audit, log):
|
||||
"""Return (changed, problems). Writes only when audit is False."""
|
||||
hdrs = {"Authorization": "Bearer " + token}
|
||||
changed = 0
|
||||
problems = []
|
||||
for cid in sorted(want):
|
||||
st, text = _http("GET", base, "/admin/realms/%s/clients" % realm,
|
||||
query={"clientId": cid}, headers=hdrs)
|
||||
if st != 200:
|
||||
problems.append("%s: client lookup failed (http %s)" % (cid, st))
|
||||
continue
|
||||
found = json.loads(text)
|
||||
if not isinstance(found, list) or not found:
|
||||
problems.append("%s: NOT FOUND in realm %s -- git declares a secret for it "
|
||||
"but the realm has no such client (the operator runs the "
|
||||
"realm import exactly once)" % (cid, realm))
|
||||
continue
|
||||
client = found[0]
|
||||
uuid = client.get("id")
|
||||
if client.get("publicClient"):
|
||||
log("%-14s skipped (public client, holds no secret)" % cid)
|
||||
continue
|
||||
st, text = _http("GET", base,
|
||||
"/admin/realms/%s/clients/%s/client-secret" % (realm, uuid),
|
||||
headers=hdrs)
|
||||
if st != 200:
|
||||
problems.append("%s: could not read the stored secret (http %s)" % (cid, st))
|
||||
continue
|
||||
stored = json.loads(text).get("value") or ""
|
||||
if hmac.compare_digest(stored, want[cid]):
|
||||
log("%-14s ok (already correct, no write)" % cid)
|
||||
continue
|
||||
# Describe the drift WITHOUT quoting either value. The literal-placeholder
|
||||
# case is named because it is the failure this program exists for, and
|
||||
# saying so reveals nothing secret.
|
||||
kind = ("stored value is the UNRESOLVED IMPORT PLACEHOLDER"
|
||||
if "env:" in stored else "stored value differs from the declared one")
|
||||
if audit:
|
||||
problems.append("%s: DRIFT -- %s" % (cid, kind))
|
||||
continue
|
||||
log("%-14s DRIFT (%s) -- repairing" % (cid, kind))
|
||||
# MINIMAL representation only -- see the header. Keycloak merges non-null
|
||||
# fields, so protocol mappers, redirect URIs, flows, groups, roles and users
|
||||
# are all untouched by this write.
|
||||
st, text = _http("PUT", base, "/admin/realms/%s/clients/%s" % (realm, uuid),
|
||||
headers=hdrs, body={"clientId": client.get("clientId"),
|
||||
"secret": want[cid]})
|
||||
if not 200 <= st < 300:
|
||||
problems.append("%s: PUT failed (http %s) %s" % (cid, st, text[:160]))
|
||||
continue
|
||||
st, text = _http("GET", base,
|
||||
"/admin/realms/%s/clients/%s/client-secret" % (realm, uuid),
|
||||
headers=hdrs)
|
||||
readback = (json.loads(text).get("value") or "") if st == 200 else ""
|
||||
if not hmac.compare_digest(readback, want[cid]):
|
||||
problems.append("%s: PUT was accepted but the readback still does not match"
|
||||
% cid)
|
||||
continue
|
||||
changed += 1
|
||||
log("%-14s REPAIRED and verified by readback" % cid)
|
||||
return changed, problems
|
||||
|
||||
|
||||
def run(argv, log):
|
||||
audit = "--audit" in argv
|
||||
base = os.environ.get("KEYCLOAK_BASE_URL", "").strip()
|
||||
realm = os.environ.get("KEYCLOAK_REALM", "").strip()
|
||||
user = os.environ.get("KEYCLOAK_ADMIN_USER", "")
|
||||
password = os.environ.get("KEYCLOAK_ADMIN_PASSWORD", "")
|
||||
for name, value in (("KEYCLOAK_BASE_URL", base), ("KEYCLOAK_REALM", realm),
|
||||
("KEYCLOAK_ADMIN_USER", user),
|
||||
("KEYCLOAK_ADMIN_PASSWORD", password)):
|
||||
if not value:
|
||||
log("FATAL: %s is empty" % name)
|
||||
return 1
|
||||
log("mode=%s realm=%s base=%s"
|
||||
% ("audit (read-only)" if audit else "reconcile", realm, base))
|
||||
try:
|
||||
want = declared(SECRETS_DIR)
|
||||
except OSError as e:
|
||||
log("FATAL: cannot read %s (%s)" % (SECRETS_DIR, e.__class__.__name__))
|
||||
return 1
|
||||
if not want:
|
||||
log("FATAL: no `<clientId>%s` keys found in %s -- the keycloak-realm-clients "
|
||||
"Secret is missing or empty, and reporting success here would repeat the "
|
||||
"very fail-open the realm import already has"
|
||||
% (KEY_SUFFIX, SECRETS_DIR))
|
||||
return 1
|
||||
log("declared clients (%d): %s" % (len(want), ", ".join(sorted(want))))
|
||||
try:
|
||||
token = admin_token(base, user, password, log)
|
||||
changed, problems = reconcile(base, realm, token, want, audit, log)
|
||||
except Refused as e:
|
||||
log("FATAL: %s" % e)
|
||||
return 1
|
||||
except (RuntimeError, OSError, ValueError) as e:
|
||||
log("FATAL: %s: %s" % (e.__class__.__name__, e))
|
||||
return 1
|
||||
if problems:
|
||||
log("")
|
||||
for p in problems:
|
||||
log("PROBLEM %s" % p)
|
||||
log("")
|
||||
if audit:
|
||||
log("AUDIT FAIL: %d client secret(s) do not match git" % len(problems))
|
||||
else:
|
||||
log("RECONCILE FAIL: %d client secret(s) could not be reconciled"
|
||||
% len(problems))
|
||||
return 1
|
||||
if audit:
|
||||
log("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK: all %d declared client secret(s) match "
|
||||
"git" % len(want))
|
||||
else:
|
||||
log("KEYCLOAK_CLIENT_SECRETS_RECONCILED_OK: %d client(s) checked, %d repaired"
|
||||
% (len(want), changed))
|
||||
return 0
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# --selftest -- the offline behavioural proof. It drives the REAL code paths above
|
||||
# against an in-process stub Keycloak, so what the gate proves is what ships.
|
||||
# =============================================================================
|
||||
def _refused(path):
|
||||
"""True iff _http() refuses `path` outright (allow-list, safety property 4)."""
|
||||
try:
|
||||
_http("PUT", "http://127.0.0.1:1", path, body={})
|
||||
except Refused:
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
def _selftest():
|
||||
import contextlib
|
||||
import http.server
|
||||
import io
|
||||
import tempfile
|
||||
import threading
|
||||
|
||||
global SECRETS_DIR, READY_ATTEMPTS, READY_SLEEP
|
||||
|
||||
SECRET_A = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
|
||||
SECRET_B = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
|
||||
SECRET_C = "cccccccccccccccccccccccccccccccc"
|
||||
STALE = "0ld-r0tated-value"
|
||||
ADMIN_PW = "stub-admin-password"
|
||||
# The exact value the broken clusters stored. chr(36) is the dollar sign,
|
||||
# spelled this way so the file stays free of the character envsubst reacts to.
|
||||
LITERAL = chr(36) + "(env:PIHOLE_CLIENT_SECRET)"
|
||||
|
||||
state = {"clients": {}, "writes": [], "touched": []}
|
||||
|
||||
class Stub(http.server.BaseHTTPRequestHandler):
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
def _send(self, code, obj):
|
||||
payload = json.dumps(obj).encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(payload)))
|
||||
self.end_headers()
|
||||
self.wfile.write(payload)
|
||||
|
||||
def _route(self, method):
|
||||
parsed = urllib.parse.urlparse(self.path)
|
||||
path, qs = parsed.path, urllib.parse.parse_qs(parsed.query)
|
||||
state["touched"].append("%s %s" % (method, path))
|
||||
if path == "/realms/master/protocol/openid-connect/token":
|
||||
return self._send(200, {"access_token": "stub-token"})
|
||||
if method == "GET" and re.fullmatch(r"/admin/realms/homelab/clients", path):
|
||||
cid = (qs.get("clientId") or [""])[0]
|
||||
c = state["clients"].get(cid)
|
||||
return self._send(200, [] if c is None else [dict(c, id="uuid-" + cid)])
|
||||
m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)/client-secret",
|
||||
path)
|
||||
if m and method == "GET":
|
||||
c = state["clients"].get(m.group(1))
|
||||
if c is None:
|
||||
return self._send(404, {"error": "not found"})
|
||||
return self._send(200, {"type": "secret", "value": c.get("secret", "")})
|
||||
m = re.fullmatch(r"/admin/realms/homelab/clients/uuid-([^/]+)", path)
|
||||
if m and method == "PUT":
|
||||
raw = self.rfile.read(int(self.headers.get("Content-Length") or 0))
|
||||
rep = json.loads(raw or b"{}")
|
||||
# The live server rejects a full round-tripped representation with
|
||||
# 400 "Cannot parse the JSON"; modelled so a regression to the fat
|
||||
# PUT fails here rather than against a real realm.
|
||||
if set(rep) != {"clientId", "secret"}:
|
||||
return self._send(400, {
|
||||
"error": "unknown_error",
|
||||
"error_description": "Cannot parse the JSON"})
|
||||
state["writes"].append(m.group(1))
|
||||
state["clients"][m.group(1)]["secret"] = rep["secret"]
|
||||
return self._send(204, {})
|
||||
return self._send(500, {"error": "an endpoint the reconciler must not touch"})
|
||||
|
||||
def do_GET(self):
|
||||
self._route("GET")
|
||||
|
||||
def do_PUT(self):
|
||||
self._route("PUT")
|
||||
|
||||
def do_POST(self):
|
||||
self._route("POST")
|
||||
|
||||
srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Stub)
|
||||
threading.Thread(target=srv.serve_forever, daemon=True).start()
|
||||
base = "http://127.0.0.1:%d" % srv.server_address[1]
|
||||
|
||||
tmp = tempfile.mkdtemp()
|
||||
for cid, secret in (("grafana", SECRET_A), ("pihole", SECRET_B),
|
||||
("kubernetes", SECRET_C), ("gitea", SECRET_A)):
|
||||
with open(os.path.join(tmp, cid + KEY_SUFFIX), "w", encoding="utf-8") as fh:
|
||||
fh.write(secret + "\n") # trailing newline: must be tolerated
|
||||
# Kubernetes' own projection entries must be ignored, not read as clients.
|
||||
os.mkdir(os.path.join(tmp, "..data"))
|
||||
|
||||
SECRETS_DIR = tmp
|
||||
READY_ATTEMPTS, READY_SLEEP = 2, 0.01
|
||||
os.environ.update({"KEYCLOAK_BASE_URL": base, "KEYCLOAK_REALM": "homelab",
|
||||
"KEYCLOAK_ADMIN_USER": "admin",
|
||||
"KEYCLOAK_ADMIN_PASSWORD": ADMIN_PW})
|
||||
|
||||
failures = []
|
||||
checks = [0]
|
||||
|
||||
def expect(cond, what):
|
||||
checks[0] += 1
|
||||
if not cond:
|
||||
failures.append(what)
|
||||
|
||||
def invoke(*argv):
|
||||
state["writes"] = []
|
||||
state["touched"] = []
|
||||
buf = io.StringIO()
|
||||
with contextlib.redirect_stdout(buf):
|
||||
rc = run(list(argv), print)
|
||||
return rc, buf.getvalue()
|
||||
|
||||
def reset():
|
||||
state["clients"] = {
|
||||
# already correct
|
||||
"grafana": {"clientId": "grafana", "publicClient": False,
|
||||
"secret": SECRET_A},
|
||||
# THE INCIDENT: the literal, unresolved import placeholder
|
||||
"pihole": {"clientId": "pihole", "publicClient": False, "secret": LITERAL},
|
||||
# public client: no secret, must be skipped
|
||||
"kubernetes": {"clientId": "kubernetes", "publicClient": True},
|
||||
# ordinary drift (rotated in git, stale in the realm)
|
||||
"gitea": {"clientId": "gitea", "publicClient": False, "secret": STALE},
|
||||
}
|
||||
|
||||
def leaked(text):
|
||||
return [s for s in (SECRET_A, SECRET_B, SECRET_C, STALE, ADMIN_PW)
|
||||
if s in text]
|
||||
|
||||
# -- 1. audit of a drifted realm: fails, names the placeholder, writes NOTHING
|
||||
reset()
|
||||
rc, out = invoke("--audit")
|
||||
expect(rc == 1, "audit must exit 1 when a stored secret differs from git")
|
||||
expect(state["writes"] == [],
|
||||
"audit mode must never write (writes=%r)" % state["writes"])
|
||||
expect("UNRESOLVED IMPORT PLACEHOLDER" in out,
|
||||
"audit must name the unresolved-placeholder case explicitly")
|
||||
expect("pihole: DRIFT" in out and "gitea: DRIFT" in out,
|
||||
"audit must report BOTH drifted clients, not stop at the first")
|
||||
expect(re.search(r"grafana\s+ok \(already correct", out),
|
||||
"an already-correct client must be reported as such")
|
||||
expect(re.search(r"kubernetes\s+skipped \(public client", out),
|
||||
"a public client must be skipped, not treated as drift")
|
||||
expect(not leaked(out), "audit output leaked a secret: %r" % leaked(out))
|
||||
|
||||
# -- 2. reconcile: repairs exactly the drifted clients, verified by readback
|
||||
reset()
|
||||
rc, out = invoke()
|
||||
expect(rc == 0, "reconcile must exit 0 after repairing (tail=%r)" % out[-300:])
|
||||
expect(sorted(state["writes"]) == ["gitea", "pihole"],
|
||||
"reconcile must write exactly the drifted clients, got %r" % state["writes"])
|
||||
expect("grafana" not in state["writes"],
|
||||
"compare-then-write: an already-correct client must not be written")
|
||||
expect("kubernetes" not in state["writes"], "a public client must not be written")
|
||||
expect("REPAIRED and verified by readback" in out,
|
||||
"every repair must be readback-verified")
|
||||
expect("2 repaired" in out, "the summary must count the repairs")
|
||||
expect(not leaked(out), "reconcile output leaked a secret: %r" % leaked(out))
|
||||
|
||||
# -- 3. IDEMPOTENCE: re-running against the converged realm writes nothing
|
||||
rc, out = invoke()
|
||||
expect(rc == 0, "the second reconcile must exit 0")
|
||||
expect(state["writes"] == [],
|
||||
"IDEMPOTENCE: a converged realm must take ZERO writes, got %r"
|
||||
% state["writes"])
|
||||
expect("0 repaired" in out, "the second run must report 0 repaired")
|
||||
rc, out = invoke("--audit")
|
||||
expect(rc == 0, "audit must pass once the realm has converged")
|
||||
expect("KEYCLOAK_CLIENT_SECRETS_AUDIT_OK" in out,
|
||||
"a clean audit must emit the sentinel the L2 conformance test greps for")
|
||||
|
||||
# -- 4. BLAST RADIUS: only the allow-listed endpoint shapes are ever touched
|
||||
reset()
|
||||
invoke()
|
||||
for touched in state["touched"]:
|
||||
expect(re.fullmatch(
|
||||
r"POST /realms/master/protocol/openid-connect/token"
|
||||
r"|GET /admin/realms/homelab/clients"
|
||||
r"|(GET|PUT) /admin/realms/homelab/clients/uuid-[^/]+(/client-secret)?",
|
||||
touched), "the reconciler touched an off-list endpoint: %s" % touched)
|
||||
expect(not [t for t in state["touched"]
|
||||
if "/users" in t or "/groups" in t or "/roles" in t
|
||||
or "/authentication" in t],
|
||||
"the reconciler must never touch users/groups/roles/authentication")
|
||||
expect(_refused("/admin/realms/homelab/users"),
|
||||
"the allow-list must refuse a users endpoint before opening a socket")
|
||||
expect(_refused("/admin/realms/homelab/groups"),
|
||||
"the allow-list must refuse a groups endpoint before opening a socket")
|
||||
expect(_refused("/admin/realms/homelab"),
|
||||
"the allow-list must refuse a whole-realm update endpoint")
|
||||
|
||||
# -- 5. a declared client the realm never got is LOUD, not a silent pass
|
||||
reset()
|
||||
del state["clients"]["gitea"]
|
||||
rc, out = invoke("--audit")
|
||||
expect(rc == 1, "a declared client missing from the realm must fail the audit")
|
||||
expect("NOT FOUND" in out, "the missing client must be named in the output")
|
||||
|
||||
# -- 6. an empty / unmounted Secret must fail closed, never report success
|
||||
SECRETS_DIR = tempfile.mkdtemp()
|
||||
rc, out = invoke()
|
||||
expect(rc == 1, "an empty secrets dir must fail closed")
|
||||
expect("keys found in" in out, "the empty-secret failure must say what is missing")
|
||||
SECRETS_DIR = tmp
|
||||
|
||||
srv.shutdown()
|
||||
print("")
|
||||
if failures:
|
||||
print("SELFTEST FAIL: %d of %d assertions failed" % (len(failures), checks[0]))
|
||||
for f in failures:
|
||||
print(" FAIL %s" % f)
|
||||
return 1
|
||||
print("SELFTEST OK: %d assertions (compare-then-write, idempotence, "
|
||||
"no-secret-logging, endpoint allow-list, fail-closed)" % checks[0])
|
||||
return 0
|
||||
|
||||
|
||||
def main():
|
||||
argv = sys.argv[1:]
|
||||
if "--selftest" in argv:
|
||||
return _selftest()
|
||||
return run(argv, lambda m: print(m, flush=True))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
---
|
||||
# CronJob — the reconcile loop. Every 15 minutes, so a cold start (or a Secret key
|
||||
# added after the realm import already completed) converges within one interval
|
||||
# instead of waiting for a human to notice a failed login. Runs are cheap: a
|
||||
# converged realm performs 12 GETs and ZERO writes.
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: keycloak-client-secret-reconciler
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak-client-secret-reconciler
|
||||
app.kubernetes.io/part-of: keycloak
|
||||
spec:
|
||||
schedule: "*/15 * * * *"
|
||||
# Forbid: a run that is still waiting for Keycloak must not be lapped by the next
|
||||
# tick. Two concurrent reconcilers would be harmless (both converge to the same
|
||||
# value) but would double the write attempts during an outage recovery.
|
||||
concurrencyPolicy: Forbid
|
||||
startingDeadlineSeconds: 300
|
||||
successfulJobsHistoryLimit: 1
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
activeDeadlineSeconds: 600
|
||||
ttlSecondsAfterFinished: 3600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak-client-secret-reconciler
|
||||
app.kubernetes.io/part-of: keycloak
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
# The reconciler talks ONLY to the Keycloak admin API. It needs no
|
||||
# Kubernetes API access at all — its two inputs are mounted Secrets — so
|
||||
# it gets no ServiceAccount token (R-SEC-02, least privilege).
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: reconciler
|
||||
# Stock upstream python; the reconciler is pure standard library, so
|
||||
# nothing is installed at runtime. Pinned exactly, like busybox /
|
||||
# curlimages elsewhere in the repo. Renovate does not manage plain
|
||||
# `image:` tags (renovate.json disables image-tag updates so it never
|
||||
# fights Flux image-automation), so this is bumped by hand.
|
||||
image: python:3.13.15-alpine3.23
|
||||
imagePullPolicy: IfNotPresent
|
||||
command:
|
||||
- python3
|
||||
- /opt/reconciler/reconcile-client-secrets.py
|
||||
env:
|
||||
- name: KEYCLOAK_BASE_URL
|
||||
value: http://keycloak-service.keycloak.svc.cluster.local:8080
|
||||
- name: KEYCLOAK_REALM
|
||||
value: homelab
|
||||
# readOnlyRootFilesystem is on; keep CPython from trying to write
|
||||
# bytecode caches beside the read-only ConfigMap mount.
|
||||
- name: PYTHONDONTWRITEBYTECODE
|
||||
value: "1"
|
||||
- name: PYTHONUNBUFFERED
|
||||
value: "1"
|
||||
# Break-glass admin (R-IDP-04). Same Secret the Keycloak CR uses to
|
||||
# bootstrap the admin; keys admin-user / admin-password.
|
||||
- name: KEYCLOAK_ADMIN_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-admin-credentials
|
||||
key: admin-user
|
||||
- name: KEYCLOAK_ADMIN_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-admin-credentials
|
||||
key: admin-password
|
||||
volumeMounts:
|
||||
- name: script
|
||||
mountPath: /opt/reconciler
|
||||
readOnly: true
|
||||
# The DESIRED state: one file per declared client, named
|
||||
# `<clientId>-client-secret`. Mounting the Secret (rather than
|
||||
# reading it through the API) is what lets this pod run with no
|
||||
# ServiceAccount token at all.
|
||||
- name: client-secrets
|
||||
mountPath: /etc/keycloak-realm-clients
|
||||
readOnly: true
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 48Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
volumes:
|
||||
- name: script
|
||||
configMap:
|
||||
name: keycloak-client-secret-reconciler
|
||||
defaultMode: 0444
|
||||
- name: client-secrets
|
||||
secret:
|
||||
secretName: keycloak-realm-clients
|
||||
defaultMode: 0400
|
||||
@@ -247,8 +247,37 @@ spec:
|
||||
# is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent()
|
||||
# also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering
|
||||
# gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24.
|
||||
#
|
||||
# FIXED 2026-09-06 — the bare `absent(...)` was evaluated CLUSTER-WIDE, so on a cluster with NO
|
||||
# egress-gateway node it could never clear. Prod has no gateway node, and this alert fired
|
||||
# continuously from 2026-09-05T21:23Z into `email-critical` — unbounded noise on the one channel
|
||||
# that must stay trustworthy. The expression is now GATED on the cluster actually being supposed
|
||||
# to have a tunnel, via two independent guards OR'd together:
|
||||
#
|
||||
# 1. A node carries `homelab.keithjackson.dev/egress-gateway=true` — the same label the
|
||||
# CiliumEgressGatewayPolicy nodeSelector uses, so the alert is armed exactly when the
|
||||
# steering datapath has a target. Requires the kube-state-metrics node-label allowlist
|
||||
# enabled in kube-prometheus-stack.yaml (KSM exposes no custom labels by default). This is
|
||||
# what makes a gateway-less cluster SELF-ARM the moment a gateway node is labelled — no
|
||||
# manual re-enable to forget. The Node object survives a NotReady node, so this stays true
|
||||
# while the gateway node is DOWN: gateway-node-down is still alerted.
|
||||
# 2. `tun0` was observed at all in the last hour — holds the alert armed through a
|
||||
# kube-state-metrics outage, and keeps dev's behaviour unchanged even before/independently
|
||||
# of the KSM rollout.
|
||||
#
|
||||
# `and on()` joins on the empty label set deliberately: absent() carries `device="tun0"` while the
|
||||
# guards carry no labels, so an unqualified `and` would never match. Both clauses verified live
|
||||
# against dev and prod Prometheus (read-only) on 2026-09-06: empty on prod (cannot fire), empty on
|
||||
# dev (tunnel up), and clause 2 already true on dev.
|
||||
- alert: VPNTunnelDown
|
||||
expr: absent(node_network_receive_bytes_total{device="tun0"})
|
||||
expr: |
|
||||
absent(node_network_receive_bytes_total{device="tun0"})
|
||||
and on()
|
||||
(
|
||||
(count(kube_node_labels{label_homelab_keithjackson_dev_egress_gateway="true"}) > 0)
|
||||
or
|
||||
(count(count_over_time(node_network_receive_bytes_total{device="tun0"}[1h])) > 0)
|
||||
)
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
@@ -359,6 +388,56 @@ spec:
|
||||
summary: "Keycloak database unavailable"
|
||||
description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
|
||||
# --- The client-secret reconciler (platform/keycloak/client-secret-reconciler.yaml) ---
|
||||
# WHY THESE EXIST: on 2026-09-06 every OIDC login on prod failed because the realm import
|
||||
# had stored the LITERAL `(env:...)` placeholder as all 12 client secrets — and every
|
||||
# signal the cluster offered was GREEN (the import Job succeeded, the CR read Done=True,
|
||||
# KeycloakDown/KeycloakDatabaseDown were quiet, the conformance test that asserts
|
||||
# Done=True passed). The only observable was a human failing to log in. The CronJob is
|
||||
# the healer; these two rules make its SILENCE observable, so the class cannot sit
|
||||
# undetected again. They are the runtime counterpart of the L2 audit test.
|
||||
#
|
||||
# UNVERIFIED-ON-CLUSTER at authoring time: no CronJob existed on either cluster yet, so
|
||||
# these series had no samples to check against. `kube_job_status_failed` /
|
||||
# `kube_job_status_succeeded` and the `job_name` label WERE confirmed present (the realm
|
||||
# import Job exports them); the reconciler's own series appear with its first run.
|
||||
# Confirm with: kube_job_status_succeeded{namespace="keycloak",
|
||||
# job_name=~"keycloak-client-secret-reconciler-.*"}
|
||||
- alert: KeycloakClientSecretReconcilerFailing
|
||||
# A reconciler run failed. Either it found drift it could not repair, or it could not
|
||||
# reach the admin API. Fires alongside KeycloakDown during an IdP outage — that is
|
||||
# deliberate (the reconciler genuinely is not running), so treat KeycloakDown as the
|
||||
# cause when both are lit. Self-clears when the failed Jobs age out (TTL 1h).
|
||||
expr: |
|
||||
max(kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"}) > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Keycloak client-secret reconciler failing"
|
||||
description: >-
|
||||
A keycloak-client-secret-reconciler Job has been failing for 15m — an OIDC client
|
||||
secret may differ from git (unresolved import placeholder, or an unrepairable
|
||||
drift) and SSO logins for that client will fail. Read the Job log: it names the
|
||||
client and never prints a secret.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
|
||||
- alert: KeycloakClientSecretReconcilerStale
|
||||
# No SUCCESSFUL run in scope. Catches the failure modes the rule above cannot see: the
|
||||
# CronJob suspended, never applied by Flux, or unable to schedule a pod at all — i.e.
|
||||
# the healer being quietly absent, which is exactly the "looks applied and isn't"
|
||||
# failure this whole change exists to end.
|
||||
expr: |
|
||||
absent(kube_job_status_succeeded{namespace="keycloak", job_name=~"keycloak-client-secret-reconciler-.*"} == 1)
|
||||
for: 2h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Keycloak client-secret reconciler has not succeeded"
|
||||
description: >-
|
||||
No successful keycloak-client-secret-reconciler run in 2h (schedule is every 15m).
|
||||
The CronJob may be suspended, unapplied, or unable to schedule — client-secret
|
||||
drift would go uncorrected and unnoticed (docs/15).
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-client-secrets"
|
||||
|
||||
# =====================================================================================
|
||||
# The monitor's own capacity — don't let the monitor blind itself (docs/10)
|
||||
|
||||
@@ -221,6 +221,17 @@ spec:
|
||||
limits:
|
||||
memory: 256Mi
|
||||
kube-state-metrics:
|
||||
# Expose the egress-gateway NODE LABEL as a metric label so alerts can ask "does this cluster
|
||||
# have a VPN egress gateway at all?". KSM exposes NO custom labels by default, so without this
|
||||
# `kube_node_labels` does not exist and any join against it silently matches nothing.
|
||||
# Consumed by `VPNTunnelDown` (platform/monitoring/alerts.yaml): the alert is armed only on a
|
||||
# cluster that actually has a labelled gateway node, and ARMS ITSELF the moment one is labelled
|
||||
# — which is how prod (no gateway today) stops emailing a permanently-firing critical alert
|
||||
# without anyone having to remember to re-enable it later. Scoped to one label on one resource:
|
||||
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
|
||||
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
|
||||
metricLabelsAllowlist:
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway]"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 20m
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
---
|
||||
# Cluster-wide PersistentVolume capacity alerts (docs/10, R-OBS-04).
|
||||
#
|
||||
# WHY THIS FILE EXISTS, AND WHY IT IS NOT IN alerts.yaml. platform/monitoring/alerts.yaml has exactly
|
||||
# ONE PVC-fill rule, `MonitoringVolumeFillingUp`, and it is scoped `namespace="monitoring"`. Verified
|
||||
# by grep 2026-09-06: it is the repo's ONLY use of `kubelet_volume_stats_*`, so before this file NO
|
||||
# database PVC — and in fact no PVC outside the monitoring namespace — was covered by any volume-fill
|
||||
# alert at all. That gap is half of the 2026-09-06 CNPG archiving incident (see platform/cnpg/
|
||||
# alerts.yaml): keycloak-db's 5 GiB PVC was accumulating unrecycled WAL toward a disk-full PANIC on
|
||||
# the SSO database with nothing watching it.
|
||||
#
|
||||
# NOTE, because it changes what "covered" means: `platform-health.longhorn`'s
|
||||
# `LonghornVolumeCapacityHigh` looks like it might have covered these PVCs. It does not — the
|
||||
# longhorn_* series are ABSENT from Prometheus on BOTH clusters (verified read-only 2026-09-06:
|
||||
# `count(longhorn_volume_capacity_bytes)` is empty on dev and on prod), so that whole rule group is
|
||||
# currently inert. That is a separate defect (a missing Longhorn ServiceMonitor/metrics config) and
|
||||
# is NOT fixed here; these rules deliberately key on `kubelet_volume_stats_*`, which is present and
|
||||
# working on both clusters.
|
||||
#
|
||||
# The rules live in a NEW FILE rather than in alerts.yaml on purpose: alerts.yaml is under concurrent
|
||||
# edit, and a second file needs no coordination. Wiring is identical either way — PrometheusRule is a
|
||||
# CR of the monitoring.coreos.com CRD installed by kube-prometheus-stack, so it is applied by the
|
||||
# `platform-config` Flux Kustomization (dependsOn: platform), NOT the controllers pass (F-MON1), and
|
||||
# `release: kube-prometheus-stack` is what makes the operator's Prometheus select it. It IS referenced
|
||||
# from platform-config/kustomization.yaml.
|
||||
#
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# HOW DOUBLE-FIRING WITH MonitoringVolumeFillingUp IS AVOIDED — by exact partition, not by luck.
|
||||
#
|
||||
# Two facts, both established live rather than assumed:
|
||||
#
|
||||
# 1. `kubelet_volume_stats_*` is published by TWO scrape jobs on these clusters, and they carry
|
||||
# DIFFERENT labels for the same PVC:
|
||||
# job="kubelet" -> namespace=<the PVC's real namespace>, node=<node>
|
||||
# job="apiserver" -> namespace="default", exported_namespace=<the PVC's real namespace>
|
||||
# Both jobs report the same 18 PVCs. `MonitoringVolumeFillingUp{namespace="monitoring"}` therefore
|
||||
# only ever matches the job="kubelet" copy (the apiserver copy says namespace="default"). Every
|
||||
# rule below pins `job="kubelet"` too, so both rule sets range over the SAME series universe and
|
||||
# `namespace!="monitoring"` is an EXACT complement of `namespace="monitoring"`. Without the job
|
||||
# pin, `namespace!="monitoring"` would also match the apiserver copy of the monitoring PVCs (they
|
||||
# say namespace="default") and this file WOULD double-fire on them — and would report the wrong
|
||||
# namespace while doing it.
|
||||
#
|
||||
# 2. Every rule below is scoped `namespace!="monitoring"`, including the critical one. So no PVC is
|
||||
# covered twice and no PVC is uncovered. The deliberate consequence: monitoring's own PVCs keep
|
||||
# only the existing 15% warning and get no critical tier. Raising one belongs in alerts.yaml,
|
||||
# which this file does not touch — recorded here as an explicit hand-off, not a silent gap.
|
||||
#
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# WHY `and kubelet_volume_stats_inodes > 0` IS ON EVERY RULE — this guard is load-bearing.
|
||||
#
|
||||
# Block-mode PVCs (volumeMode: Block — the KubeVirt VM root disks) have no filesystem for the kubelet
|
||||
# to stat, so it reports available=0, used=0, inodes=0 against a real capacity. `available/capacity`
|
||||
# is therefore exactly 0 FOREVER. Verified live 2026-09-06: `unifi/unifi-rootdisk` (40 GiB, Block, on
|
||||
# both clusters) reads 0.0. Without this guard, the warning AND the critical rule below would have
|
||||
# fired permanently on it from the moment they were applied — precisely the permanently-firing-critical
|
||||
# alert fatigue this repo has already been burned by (see the VPNTunnelDown post-mortem in
|
||||
# alerts.yaml). `inodes > 0` is the "is there a real filesystem here?" test: any ext4/xfs volume has
|
||||
# a non-zero inode total; a raw block device reports 0. Proven both ways on both clusters — with the
|
||||
# guard the rules return the 15 filesystem PVCs outside monitoring and exclude unifi-rootdisk; without
|
||||
# it, unifi-rootdisk is the only match at every threshold.
|
||||
#
|
||||
# LIVE STATE AT AUTHORING (read-only, both clusters, 2026-09-06): every rule below returns EMPTY on
|
||||
# dev and on prod. Nothing here fires on day one; these are lead-time alarms, not a description of
|
||||
# today. Fullest filesystem PVC outside monitoring: prod keycloak-db-1 at 93% free, dev keycloak-db-1
|
||||
# at 88% free.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: platform-health-pvc-capacity
|
||||
namespace: monitoring
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
platform-health: "true"
|
||||
spec:
|
||||
groups:
|
||||
- name: platform-health.pvc-capacity
|
||||
rules:
|
||||
# Level alarm, warning tier. Thresholds deliberately IDENTICAL to MonitoringVolumeFillingUp
|
||||
# (<15% free, 15m) so the two halves of the partition behave the same way; only the namespace
|
||||
# scope differs.
|
||||
- alert: PersistentVolumeFillingUp
|
||||
expr: |
|
||||
(
|
||||
kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
|
||||
/ kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
|
||||
) < 0.15
|
||||
and kubelet_volume_stats_inodes{job="kubelet"} > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "PVC filling up: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
|
||||
description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is under 15% free for 15m. If this is a CloudNativePG data volume (*-db-N), check WAL archiving FIRST — a failing archive_command stops Postgres recycling WAL and fills the volume (see platform/cnpg/alerts.yaml); deleting WAL by hand destroys the database."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
# Level alarm, critical tier. This is the ONLY critical in this change: it fires only when a
|
||||
# volume is genuinely about to run out, which for a Postgres data volume means an imminent
|
||||
# disk-full PANIC. Nothing on either cluster is anywhere near 5% today, so it does not add a
|
||||
# permanently-firing critical. There is no counterpart for this tier anywhere else in the
|
||||
# repo, so it cannot duplicate one.
|
||||
- alert: PersistentVolumeAlmostFull
|
||||
expr: |
|
||||
(
|
||||
kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
|
||||
/ kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
|
||||
) < 0.05
|
||||
and kubelet_volume_stats_inodes{job="kubelet"} > 0
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "PVC almost full: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
|
||||
description: "PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} has under 5% free for 15m — the workload is about to fail writes. A CloudNativePG data volume at this point will PANIC the postmaster and take the database down."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
# TREND alarm — the lead-time rule, and the point of the exercise. A level threshold is useless
|
||||
# for a slow leak on a mostly-empty volume: prod keycloak-db sat at 93% FREE while leaking
|
||||
# ~6.8 MiB/h of unrecyclable WAL, so <15% was ~29 days away and would have arrived as a
|
||||
# surprise. This rule instead asks "is this volume on course to hit zero?".
|
||||
#
|
||||
# 24h fit / 7d horizon: the 24h window is long enough to average out a daily backup or log
|
||||
# rotation (a 6h window, as the kubernetes-mixin uses, chases those and flaps), and a 7d
|
||||
# horizon gives a full week to react. `for: 1h` requires the projection to stay negative for an
|
||||
# hour, so a single burst of writes cannot notify on its own.
|
||||
#
|
||||
# The `< 0.50` gate is what keeps this quiet: a volume with more than half its space free is
|
||||
# not "about to fill" in any actionable sense, and predict_linear on a nearly-empty volume is
|
||||
# dominated by noise. It costs some lead time on the WAL case — the CNPG rules in
|
||||
# platform/cnpg/alerts.yaml are the day-one signal there, and this is the generic backstop for
|
||||
# every OTHER PVC in the cluster, which is what was missing entirely.
|
||||
#
|
||||
# NOTE the monitoring namespace is excluded here too (partition rule above). That also
|
||||
# side-steps predict_linear's classic false positive: a Prometheus TSDB filling toward its
|
||||
# retention high-water mark looks exactly like an unbounded leak until it plateaus.
|
||||
- alert: PersistentVolumeFillingUpPredicted
|
||||
expr: |
|
||||
(
|
||||
kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}
|
||||
/ kubelet_volume_stats_capacity_bytes{job="kubelet", namespace!="monitoring"}
|
||||
) < 0.50
|
||||
and predict_linear(kubelet_volume_stats_available_bytes{job="kubelet", namespace!="monitoring"}[24h], 7 * 24 * 3600) < 0
|
||||
and kubelet_volume_stats_inodes{job="kubelet"} > 0
|
||||
for: 1h
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "PVC predicted to fill within 7 days: {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }}"
|
||||
description: "Based on the last 24h of growth, PVC {{ $labels.namespace }}/{{ $labels.persistentvolumeclaim }} on node {{ $labels.node }} is on course to run out of space within 7 days. Act now rather than at the 15%/5% thresholds: resize the PVC, prune data, or fix whatever is growing without bound (for a *-db-N volume that is usually failing WAL archiving — see platform/cnpg/alerts.yaml)."
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/pvc-capacity"
|
||||
@@ -28,6 +28,45 @@ Three layers, two owners:
|
||||
`vpn`-class traffic whenever `tun0` is down (fail-closed, R-NET-06), scoped to vpn-class only so
|
||||
`internet`-class and node traffic are unaffected.
|
||||
|
||||
## Resilience — there is NO gateway failover (spec correction, escalated)
|
||||
|
||||
**ADR-0015 §Decision says "Two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)
|
||||
provide the active/standby the earlier ADRs asked for", and spec `docs/03` repeats it. That feature
|
||||
does not exist in Cilium OSS.** The supported topology is **exactly one gateway node**.
|
||||
|
||||
Verified against the CRD actually deployed on this cluster
|
||||
(`kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`, apiVersion `cilium.io/v2`,
|
||||
Cilium **v1.20.1**) — the field's own description:
|
||||
|
||||
> **`egressGateway`** — "EgressGateway is the gateway node responsible for SNATing traffic.
|
||||
> **In case multiple nodes are a match for the given set of labels, the first node in lexical
|
||||
> ordering based on their name will be selected.**"
|
||||
|
||||
Consequences, stated plainly:
|
||||
|
||||
- **One gateway node is the supported topology.** `infra/ansible/inventory/hosts.ini`'s
|
||||
`[vpn_egress_gateway]` group is singleton by design.
|
||||
- **A second labelled node does NOT provide failover.** It is not a standby; it carries no traffic
|
||||
and takes over nothing when the first node dies. Gateway-node loss = `vpn`-class egress down
|
||||
(fail-closed, per R-NET-06 — no leak, but no service).
|
||||
- **Lexical ordering decides who wins if two are ever labelled.** Labelling `k3s-prod-1` while
|
||||
`k3s-prod-3` is the live gateway would **silently** move the role to `k3s-prod-1` — because `1`
|
||||
sorts before `3` — taking the tunnel and the public egress IP with it, with no event and no alert.
|
||||
- **The plural `egressGateways` field (Cilium 1.20) is not a workaround.** It *shards* endpoints
|
||||
across entries by CiliumEndpoint UID rather than running active/standby: losing one entry's node
|
||||
blackholes that entry's share and re-shards the remainder (breaking live connections), and each
|
||||
shard leaves from a *different* public IP. Upstream confirmation:
|
||||
[cilium/cilium#30157](https://github.com/cilium/cilium/issues/30157) ("egress gateway doesn't
|
||||
support failover between nodes") and
|
||||
[cilium/cilium#18230](https://github.com/cilium/cilium/issues/18230) ("egress gateway does not
|
||||
tolerate single node failure").
|
||||
|
||||
**Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents
|
||||
(spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited
|
||||
here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in
|
||||
`DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file
|
||||
and `egress-steering.yaml` are the accurate statement of what the implementation can actually do.
|
||||
|
||||
## Secrets — host-side now
|
||||
PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not**
|
||||
a cluster SOPS Secret. SOPS still governs in-cluster secrets; VPN egress simply no longer needs one.
|
||||
|
||||
@@ -69,6 +69,26 @@ spec:
|
||||
# ADR-0015) is the authoritative fail-closed backstop that DROPS this vpn-class traffic — scoped to
|
||||
# vpn-class only, so default/internet-class is unaffected (invariants 3 & 4: vpn fails CLOSED,
|
||||
# internet fails OPEN).
|
||||
#
|
||||
# SINGLE GATEWAY ONLY — there is NO failover, and a second labelled node is NOT a standby.
|
||||
# `egressGateway` (singular) resolves the nodeSelector to ONE node. Verbatim from the CRD
|
||||
# deployed on this cluster (`ciliumegressgatewaypolicies.cilium.io`, v2, Cilium 1.20.1 —
|
||||
# `kubectl get crd ciliumegressgatewaypolicies.cilium.io -o json`):
|
||||
#
|
||||
# "EgressGateway is the gateway node responsible for SNATing traffic.
|
||||
# In case multiple nodes are a match for the given set of labels, the first node
|
||||
# in lexical ordering based on their name will be selected."
|
||||
#
|
||||
# So labelling a second node does not create an active/standby pair — it just hands the role to
|
||||
# whichever node sorts FIRST BY NAME. A labelled `k3s-prod-1` would silently take the role from a
|
||||
# labelled `k3s-prod-3`, moving the tunnel (and the public egress IP) with no event and no alert.
|
||||
# LABEL EXACTLY ONE NODE. The plural `egressGateways` field (also present in 1.20) is NOT a fix:
|
||||
# it SHARDS endpoints across entries by CiliumEndpoint UID, so losing one entry's node blackholes
|
||||
# that shard and re-shards the rest (breaking live connections), and each shard exits from a
|
||||
# DIFFERENT public IP. Upstream: cilium/cilium#30157, cilium/cilium#18230.
|
||||
# ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)"
|
||||
# is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see
|
||||
# platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06.
|
||||
egressGateway:
|
||||
nodeSelector:
|
||||
matchLabels:
|
||||
|
||||
Reference in new issue
Block a user