0 Fork 0

Files

T

jakman85andClaude Opus 5 6a8953e288 feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-07 00:59:22 -05:00

7.5 KiB

Raw Blame History

platform/keycloak

Keycloak — the platform Identity Provider (SSO via OIDC + SAML), onboarded early (Phase 6 IdP).

Governing ADR: ADR-0014 — IdP: Keycloak
Spec: docs/15 — Identity & Access
Serves: R-IDP-01..06
Owner: WS-SEC

Image / tooling strategy (F-KY1 landmine avoided)

The original stub used the Bitnami Keycloak chart + bundled Bitnami PostgreSQL. Bitnami's free Docker Hub images were purged (DECISIONS-LOG F-KY1) → that path ImagePullBackOffs. Replaced with:

Keycloak: the official Keycloak Operator (quay.io/keycloak/keycloak-operator + quay.io/keycloak/keycloak, v26.0.6) driving a Keycloak CR + a KeycloakRealmImport CR for config-as-code (ADR-0014 permits Operator OR Helm; Operator gives clean CR-driven realm reconcile).
Database: CloudNativePG (ghcr.io/cloudnative-pg, see ../cnpg) — a Cluster on the 3-replica longhorn StorageClass (DR-critical, R-IDP-05), with declarative backup hooks.

quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.

Layout (controllers / configs split — F-MON1)

File	Applied by	What
`kustomization.yaml` (namespace, operator CRDs+Deployment, admin secret, dashboard)	platform	controllers
`keycloak-instance.yaml` (`Keycloak` CR, `${KEYCLOAK_REPLICAS}`)	platform-config	config
`postgres-cluster.yaml` (CNPG `Cluster` on longhorn)	platform-config	config
`keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA)	platform-config	config
`client-secret-reconciler.yaml` (CronJob: client secrets ← git, every 15m)	platform-config	config
`httproute.yaml` (keycloak.home.keithjackson.dev)	platform-config	config
`servicemonitor.yaml` (Prometheus scrape)	platform-config	config

Realm config (groups platform-admins/media-users/viewers, OIDC clients grafana+oauth2-proxy, brute-force ON, and a conditional MFA browser flow that REQUIRES OTP for platform-admins, R-IDP-04) reconciles purely from git — no console clicking (R-IDP-03).

Admin credential + break-glass (R-IDP-04 / R-IDP-05)

The provisioned keycloak-admin-credentials.sops.yaml (keys admin-user/admin-password) is injected as the temporary bootstrap/break-glass admin via KC_BOOTSTRAP_ADMIN_* env (podTemplate escape hatch) — this consumes the secret as provisioned, without re-keying it (the operator's native bootstrapAdmin wants username/password keys, which would require editing the encrypted file). The Keycloak bootstrap admin is the documented break-glass path so an IdP/SSO outage does not lock the operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO is proven.

Operator hand-offs (fill before / during deploy)

keycloak-realm-clients.sops.yaml (keycloak ns) — from .sops.yaml.example. One <clientId>-client-secret key per confidential client, openssl rand -hex 32 each. The realm import injects them into the Keycloak clients via spec.placeholders. Add to kustomization.yaml once filled. The key name MUST be <clientId>-client-secret — the reconciler below derives the clientId from it, and CI rejects any other name. (This step used to read "the realm import stays PENDING until this exists". It does not — see the next section. That false belief is what allowed the 2026-09-06 outage.)
../oauth2-proxy/oauth2-proxy-credentials.sops.yaml (oauth2-proxy ns) — client-id: oauth2-proxy, client-secret = the SAME oauth2-proxy-client-secret above, cookie-secret = openssl rand -base64 32.
../monitoring/grafana-oidc-credentials.sops.yaml (monitoring ns) — client-secret = the SAME grafana-client-secret above.

(Cross-namespace Secrets can't be shared, so each client secret is stored twice — once for the realm import in keycloak, once for the consuming app.) See the two .sops.yaml.example templates for the exact keys + steps. No plaintext secret is ever committed.

Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)

The realm import is a one-shot that fails open. Measured on both clusters:

When the operator cannot resolve a $(env:...) placeholder it stores the literal placeholder text as the client secret — and the import Job still exits 0. There is no PENDING state, no error condition, and HasErrors stays False.
Once complete the CR rests at Done=True / Started=False and never re-runs, so a Secret key added later is never picked up.

Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder; every login rejected with "Invalid client or Invalid client credentials") and a single broken client on dev (oauth2-proxy, whose key was added after dev's import had already completed). Every cluster signal was green throughout, including the conformance test that asserts Done=True.

Three layers now cover it:

Layer	Where	What it does
Prevent	`conformance/tests/l0-static/keycloak-client-secret-reconciler` (runs on every merge, unconditionally, from `ci/l0.sh`)	Every placeholder is defined, reads `keycloak-realm-clients`, and names an existing key called `<clientId>-client-secret`; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive `dependsOn` ancestor (`wait: true`) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable.
Heal	`client-secret-reconciler.yaml` — CronJob, every 15m	Compares each client's stored secret with the mounted `keycloak-realm-clients` value and PUTs the minimal `{clientId, secret}` representation only where they differ. Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (`automountServiceAccountToken: false`).
Detect	`conformance/tests/l2-behavioral/keycloak-client-secrets-match-git` + alerts `KeycloakClientSecretReconcilerFailing` / `…Stale`	Runs the same script in read-only `--audit` mode against the live realm and fails on any mismatch or literal placeholder; the alerts make the reconciler's silence observable in prod.

The reconciler's own --selftest (run by the L0 gate, which extracts the script from the shipped ConfigMap) proves compare-then-write, idempotence (zero writes on a converged realm), no-secret-logging, the endpoint allow-list, and fail-closed behaviour.

Manual repair / one-off run (the CronJob normally does this):

kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1

What the reconciler does NOT fix: anything other than client secrets. A NEW client, a changed redirect URI or a new protocol mapper still needs the realm import to actually re-run (it will not on its own) or a deliberate admin-API/kcadm change. Done=True is never evidence that a realm edit landed.

Exceptions that keep native auth (do not force SSO) — ADR-0014

Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).

7.5 KiB Raw Blame History