jakman85
cac413e87a
Three instances replicating ASYNCHRONOUSLY still lose data on failover: COMMIT returns as soon as the primary has flushed locally, so a primary that dies before its standbys catch up takes the last few transactions with it and the promoted standby comes up quietly missing them. On keycloak-db that is a just-created user, a rotated credential or a consent vanishing at exactly the moment of a failover. `method: any` + `number: 1` makes COMMIT wait for ONE of the two standbys to confirm — the point of having them. dataDurability: preferred, NEVER required. `required` is a strict quorum: fewer than `number` healthy standbys and writes BLOCK. Postgres here shares three nodes with etcd and the CNI, so one degraded standby — a node reboot, a Longhorn replica rebuild, a Cilium restart — would turn a partial degradation into a total write outage, and on the SSO database that takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth with it. `preferred` keeps synchronous commit while a healthy standby is there and relaxes the required count dynamically when one is not. dataDurability is only honoured while standbyNamesPre/standbyNamesPost are unset — both are. NON-DISRUPTIVE: this compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter. No pod restart, no switchover, no client disconnect. It is the last of the three commits because it wants standbys that already exist and are already on their final nodes. Field shape verified against the live clusters.postgresql.cnpg.io CRD before relying on it (CNPG 1.30.0, byte-identical schema on k3s-prod and k3s-dev): spec.postgresql.synchronous requires `method` (enum any|first) and `number` (x-kubernetes-validation self > 0); `dataDurability` is enum required| preferred. None of the three Clusters previously had a `postgresql:` block, so this adds one rather than merging into existing parameters. DEV IMPACT: identical and non-disruptive, same reload-only path. Verified: yamllint clean; both tiers render the block. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit f1ce19461415e77eb2f1ca5109ebab49edf0860b)
platform/keycloak
Keycloak — the platform Identity Provider (SSO via OIDC + SAML), onboarded early (Phase 6 IdP).
- Governing ADR: ADR-0014 — IdP: Keycloak
- Spec: docs/15 — Identity & Access
- Serves: R-IDP-01..06
- Owner: WS-SEC
Image / tooling strategy (F-KY1 landmine avoided)
The original stub used the Bitnami Keycloak chart + bundled Bitnami PostgreSQL. Bitnami's free Docker Hub images were purged (DECISIONS-LOG F-KY1) → that path ImagePullBackOffs. Replaced with:
- Keycloak: the official Keycloak Operator (
quay.io/keycloak/keycloak-operator+quay.io/keycloak/keycloak, v26.0.6) driving aKeycloakCR + aKeycloakRealmImportCR for config-as-code (ADR-0014 permits Operator OR Helm; Operator gives clean CR-driven realm reconcile). - Database: CloudNativePG (
ghcr.io/cloudnative-pg, see../cnpg) — aClusteron the 3-replicalonghornStorageClass (DR-critical, R-IDP-05), with declarative backup hooks.
quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.
Layout (controllers / configs split — F-MON1)
| File | Applied by | What |
|---|---|---|
kustomization.yaml (namespace, operator CRDs+Deployment, admin secret, dashboard) |
platform | controllers |
keycloak-instance.yaml (Keycloak CR, ${KEYCLOAK_REPLICAS}) |
platform-config | config |
postgres-cluster.yaml (CNPG Cluster on longhorn) |
platform-config | config |
keycloak-realm-import.yaml (KeycloakRealmImport: realm/groups/clients/MFA) |
platform-config | config |
client-secret-reconciler.yaml (CronJob: client secrets ← git, every 15m) |
platform-config | config |
httproute.yaml (keycloak.home.keithjackson.dev) |
platform-config | config |
servicemonitor.yaml (Prometheus scrape) |
platform-config | config |
Realm config (groups platform-admins/media-users/viewers, OIDC clients grafana+oauth2-proxy,
brute-force ON, and a conditional MFA browser flow that REQUIRES OTP for platform-admins, R-IDP-04)
reconciles purely from git — no console clicking (R-IDP-03).
Admin credential + break-glass (R-IDP-04 / R-IDP-05)
The provisioned keycloak-admin-credentials.sops.yaml (keys admin-user/admin-password) is injected
as the temporary bootstrap/break-glass admin via KC_BOOTSTRAP_ADMIN_* env (podTemplate escape
hatch) — this consumes the secret as provisioned, without re-keying it (the operator's native
bootstrapAdmin wants username/password keys, which would require editing the encrypted file).
The Keycloak bootstrap admin is the documented break-glass path so an IdP/SSO outage does not lock the
operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO is proven.
Operator hand-offs (fill before / during deploy)
keycloak-realm-clients.sops.yaml(keycloak ns) — from.sops.yaml.example. One<clientId>-client-secretkey per confidential client,openssl rand -hex 32each. The realm import injects them into the Keycloak clients viaspec.placeholders. Add tokustomization.yamlonce filled. The key name MUST be<clientId>-client-secret— the reconciler below derives the clientId from it, and CI rejects any other name. (This step used to read "the realm import stays PENDING until this exists". It does not — see the next section. That false belief is what allowed the 2026-09-06 outage.)../oauth2-proxy/oauth2-proxy-credentials.sops.yaml(oauth2-proxy ns) —client-id: oauth2-proxy,client-secret= the SAMEoauth2-proxy-client-secretabove,cookie-secret=openssl rand -base64 32.../monitoring/grafana-oidc-credentials.sops.yaml(monitoring ns) —client-secret= the SAMEgrafana-client-secretabove.
(Cross-namespace Secrets can't be shared, so each client secret is stored twice — once for the realm
import in keycloak, once for the consuming app.) See the two .sops.yaml.example templates for the
exact keys + steps. No plaintext secret is ever committed.
Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)
The realm import is a one-shot that fails open. Measured on both clusters:
- When the operator cannot resolve a
$(env:...)placeholder it stores the literal placeholder text as the client secret — and the import Job still exits 0. There is no PENDING state, no error condition, andHasErrorsstaysFalse. - Once complete the CR rests at
Done=True/Started=Falseand never re-runs, so a Secret key added later is never picked up.
Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder;
every login rejected with "Invalid client or Invalid client credentials") and a single broken client
on dev (oauth2-proxy, whose key was added after dev's import had already completed). Every cluster
signal was green throughout, including the conformance test that asserts Done=True.
Three layers now cover it:
| Layer | Where | What it does |
|---|---|---|
| Prevent | conformance/tests/l0-static/keycloak-client-secret-reconciler (runs on every merge, unconditionally, from ci/l0.sh) |
Every placeholder is defined, reads keycloak-realm-clients, and names an existing key called <clientId>-client-secret; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive dependsOn ancestor (wait: true) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable. |
| Heal | client-secret-reconciler.yaml — CronJob, every 15m |
Compares each client's stored secret with the mounted keycloak-realm-clients value and PUTs the minimal {clientId, secret} representation only where they differ. Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (automountServiceAccountToken: false). |
| Detect | conformance/tests/l2-behavioral/keycloak-client-secrets-match-git + alerts KeycloakClientSecretReconcilerFailing / …Stale |
Runs the same script in read-only --audit mode against the live realm and fails on any mismatch or literal placeholder; the alerts make the reconciler's silence observable in prod. |
The reconciler's own --selftest (run by the L0 gate, which extracts the script from the shipped
ConfigMap) proves compare-then-write, idempotence (zero writes on a converged realm),
no-secret-logging, the endpoint allow-list, and fail-closed behaviour.
Manual repair / one-off run (the CronJob normally does this):
kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1
What the reconciler does NOT fix: anything other than client secrets. A NEW client, a changed
redirect URI or a new protocol mapper still needs the realm import to actually re-run (it will not on
its own) or a deliberate admin-API/kcadm change. Done=True is never evidence that a realm edit
landed.
Exceptions that keep native auth (do not force SSO) — ADR-0014
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).