jakman85
f338e91770
All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication saves the DATA when a node dies; it does not save the SERVICE — a single instance has to be rescheduled, re-attach its volume and replay WAL before Postgres accepts a connection again. With nothing in the spec keeping them apart, all three primaries had drifted onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give each Cluster a streaming standby CNPG can promote in seconds, repointing the -rw Service with no volume re-attach and no restore. UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate corruption and deletions faithfully; only the Barman archive is a backup, and adding standbys while archive_mode=on with a failing archive_command would have multiplied the disk-full clock, not hedged it. All three Clusters on both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real firstRecoverabilityPoint (verified read-only, 2026-09-07). Scaling out is non-disruptive on its own: CNPG creates -2 then -3 sequentially, each seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are not disconnected. The anti-affinity change that DOES cause a switchover is deliberately the next commit. DEV IMPACT: identical — dev is on the same shape (three single-instance DBs, all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes and Longhorn headroom for it (see the next commit's message for the numbers). Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning. Verified: yamllint clean; `kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0` both render `instances: 3`. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02)
platform/keycloak
Keycloak — the platform Identity Provider (SSO via OIDC + SAML), onboarded early (Phase 6 IdP).
- Governing ADR: ADR-0014 — IdP: Keycloak
- Spec: docs/15 — Identity & Access
- Serves: R-IDP-01..06
- Owner: WS-SEC
Image / tooling strategy (F-KY1 landmine avoided)
The original stub used the Bitnami Keycloak chart + bundled Bitnami PostgreSQL. Bitnami's free Docker Hub images were purged (DECISIONS-LOG F-KY1) → that path ImagePullBackOffs. Replaced with:
- Keycloak: the official Keycloak Operator (
quay.io/keycloak/keycloak-operator+quay.io/keycloak/keycloak, v26.0.6) driving aKeycloakCR + aKeycloakRealmImportCR for config-as-code (ADR-0014 permits Operator OR Helm; Operator gives clean CR-driven realm reconcile). - Database: CloudNativePG (
ghcr.io/cloudnative-pg, see../cnpg) — aClusteron the 3-replicalonghornStorageClass (DR-critical, R-IDP-05), with declarative backup hooks.
quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.
Layout (controllers / configs split — F-MON1)
| File | Applied by | What |
|---|---|---|
kustomization.yaml (namespace, operator CRDs+Deployment, admin secret, dashboard) |
platform | controllers |
keycloak-instance.yaml (Keycloak CR, ${KEYCLOAK_REPLICAS}) |
platform-config | config |
postgres-cluster.yaml (CNPG Cluster on longhorn) |
platform-config | config |
keycloak-realm-import.yaml (KeycloakRealmImport: realm/groups/clients/MFA) |
platform-config | config |
client-secret-reconciler.yaml (CronJob: client secrets ← git, every 15m) |
platform-config | config |
httproute.yaml (keycloak.home.keithjackson.dev) |
platform-config | config |
servicemonitor.yaml (Prometheus scrape) |
platform-config | config |
Realm config (groups platform-admins/media-users/viewers, OIDC clients grafana+oauth2-proxy,
brute-force ON, and a conditional MFA browser flow that REQUIRES OTP for platform-admins, R-IDP-04)
reconciles purely from git — no console clicking (R-IDP-03).
Admin credential + break-glass (R-IDP-04 / R-IDP-05)
The provisioned keycloak-admin-credentials.sops.yaml (keys admin-user/admin-password) is injected
as the temporary bootstrap/break-glass admin via KC_BOOTSTRAP_ADMIN_* env (podTemplate escape
hatch) — this consumes the secret as provisioned, without re-keying it (the operator's native
bootstrapAdmin wants username/password keys, which would require editing the encrypted file).
The Keycloak bootstrap admin is the documented break-glass path so an IdP/SSO outage does not lock the
operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO is proven.
Operator hand-offs (fill before / during deploy)
keycloak-realm-clients.sops.yaml(keycloak ns) — from.sops.yaml.example. One<clientId>-client-secretkey per confidential client,openssl rand -hex 32each. The realm import injects them into the Keycloak clients viaspec.placeholders. Add tokustomization.yamlonce filled. The key name MUST be<clientId>-client-secret— the reconciler below derives the clientId from it, and CI rejects any other name. (This step used to read "the realm import stays PENDING until this exists". It does not — see the next section. That false belief is what allowed the 2026-09-06 outage.)../oauth2-proxy/oauth2-proxy-credentials.sops.yaml(oauth2-proxy ns) —client-id: oauth2-proxy,client-secret= the SAMEoauth2-proxy-client-secretabove,cookie-secret=openssl rand -base64 32.../monitoring/grafana-oidc-credentials.sops.yaml(monitoring ns) —client-secret= the SAMEgrafana-client-secretabove.
(Cross-namespace Secrets can't be shared, so each client secret is stored twice — once for the realm
import in keycloak, once for the consuming app.) See the two .sops.yaml.example templates for the
exact keys + steps. No plaintext secret is ever committed.
Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)
The realm import is a one-shot that fails open. Measured on both clusters:
- When the operator cannot resolve a
$(env:...)placeholder it stores the literal placeholder text as the client secret — and the import Job still exits 0. There is no PENDING state, no error condition, andHasErrorsstaysFalse. - Once complete the CR rests at
Done=True/Started=Falseand never re-runs, so a Secret key added later is never picked up.
Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder;
every login rejected with "Invalid client or Invalid client credentials") and a single broken client
on dev (oauth2-proxy, whose key was added after dev's import had already completed). Every cluster
signal was green throughout, including the conformance test that asserts Done=True.
Three layers now cover it:
| Layer | Where | What it does |
|---|---|---|
| Prevent | conformance/tests/l0-static/keycloak-client-secret-reconciler (runs on every merge, unconditionally, from ci/l0.sh) |
Every placeholder is defined, reads keycloak-realm-clients, and names an existing key called <clientId>-client-secret; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive dependsOn ancestor (wait: true) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable. |
| Heal | client-secret-reconciler.yaml — CronJob, every 15m |
Compares each client's stored secret with the mounted keycloak-realm-clients value and PUTs the minimal {clientId, secret} representation only where they differ. Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (automountServiceAccountToken: false). |
| Detect | conformance/tests/l2-behavioral/keycloak-client-secrets-match-git + alerts KeycloakClientSecretReconcilerFailing / …Stale |
Runs the same script in read-only --audit mode against the live realm and fails on any mismatch or literal placeholder; the alerts make the reconciler's silence observable in prod. |
The reconciler's own --selftest (run by the L0 gate, which extracts the script from the shipped
ConfigMap) proves compare-then-write, idempotence (zero writes on a converged realm),
no-secret-logging, the endpoint allow-list, and fail-closed behaviour.
Manual repair / one-off run (the CronJob normally does this):
kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1
What the reconciler does NOT fix: anything other than client secrets. A NEW client, a changed
redirect URI or a new protocol mapper still needs the realm import to actually re-run (it will not on
its own) or a deliberate admin-API/kcadm change. Done=True is never evidence that a realm edit
landed.
Exceptions that keep native auth (do not force SSO) — ADR-0014
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).