jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
110 lines
7.5 KiB
Markdown
110 lines
7.5 KiB
Markdown
# platform/keycloak
|
|
|
|
**Keycloak** — the platform Identity Provider (SSO via OIDC + SAML), onboarded early (Phase 6 IdP).
|
|
|
|
- **Governing ADR:** [ADR-0014 — IdP: Keycloak](../../../home-network-k3s/decisions/adr-0014-idp-keycloak.md)
|
|
- **Spec:** [docs/15 — Identity & Access](../../../home-network-k3s/docs/15-identity-and-access.md)
|
|
- **Serves:** R-IDP-01..06
|
|
- **Owner:** WS-SEC
|
|
|
|
## Image / tooling strategy (F-KY1 landmine avoided)
|
|
|
|
The original stub used the **Bitnami Keycloak chart + bundled Bitnami PostgreSQL**. Bitnami's free
|
|
Docker Hub images were purged (DECISIONS-LOG **F-KY1**) → that path ImagePullBackOffs. Replaced with:
|
|
|
|
- **Keycloak: the official Keycloak Operator** (`quay.io/keycloak/keycloak-operator` +
|
|
`quay.io/keycloak/keycloak`, v26.0.6) driving a **`Keycloak` CR** + a **`KeycloakRealmImport` CR** for
|
|
config-as-code (ADR-0014 permits Operator OR Helm; Operator gives clean CR-driven realm reconcile).
|
|
- **Database: CloudNativePG** (`ghcr.io/cloudnative-pg`, see [`../cnpg`](../cnpg)) — a `Cluster` on the
|
|
3-replica **`longhorn`** StorageClass (DR-critical, R-IDP-05), with declarative backup hooks.
|
|
|
|
quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release
|
|
files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.
|
|
|
|
## Layout (controllers / configs split — F-MON1)
|
|
|
|
| File | Applied by | What |
|
|
|---|---|---|
|
|
| `kustomization.yaml` (namespace, operator CRDs+Deployment, admin secret, dashboard) | **platform** | controllers |
|
|
| `keycloak-instance.yaml` (`Keycloak` CR, `${KEYCLOAK_REPLICAS}`) | **platform-config** | config |
|
|
| `postgres-cluster.yaml` (CNPG `Cluster` on longhorn) | **platform-config** | config |
|
|
| `keycloak-realm-import.yaml` (`KeycloakRealmImport`: realm/groups/clients/MFA) | **platform-config** | config |
|
|
| `client-secret-reconciler.yaml` (CronJob: client secrets ← git, every 15m) | **platform-config** | config |
|
|
| `httproute.yaml` (keycloak.home.keithjackson.dev) | **platform-config** | config |
|
|
| `servicemonitor.yaml` (Prometheus scrape) | **platform-config** | config |
|
|
|
|
Realm config (groups `platform-admins`/`media-users`/`viewers`, OIDC clients `grafana`+`oauth2-proxy`,
|
|
brute-force ON, and a **conditional MFA browser flow that REQUIRES OTP for platform-admins**, R-IDP-04)
|
|
reconciles purely from git — no console clicking (R-IDP-03).
|
|
|
|
## Admin credential + break-glass (R-IDP-04 / R-IDP-05)
|
|
|
|
The provisioned `keycloak-admin-credentials.sops.yaml` (keys `admin-user`/`admin-password`) is injected
|
|
as the temporary **bootstrap/break-glass admin** via `KC_BOOTSTRAP_ADMIN_*` env (podTemplate escape
|
|
hatch) — this consumes the secret **as provisioned, without re-keying it** (the operator's native
|
|
`bootstrapAdmin` wants `username`/`password` keys, which would require editing the encrypted file).
|
|
The Keycloak bootstrap admin is the documented break-glass path so an IdP/SSO outage does not lock the
|
|
operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO is proven.
|
|
|
|
## Operator hand-offs (fill before / during deploy)
|
|
|
|
1. **`keycloak-realm-clients.sops.yaml`** (keycloak ns) — from `.sops.yaml.example`. One
|
|
`<clientId>-client-secret` key per confidential client, `openssl rand -hex 32` each. The realm
|
|
import injects them into the Keycloak clients via `spec.placeholders`. Add to
|
|
`kustomization.yaml` once filled.
|
|
**The key name MUST be `<clientId>-client-secret`** — the reconciler below derives the clientId
|
|
from it, and CI rejects any other name.
|
|
*(This step used to read "the realm import stays PENDING until this exists". It does not — see
|
|
the next section. That false belief is what allowed the 2026-09-06 outage.)*
|
|
2. **`../oauth2-proxy/oauth2-proxy-credentials.sops.yaml`** (oauth2-proxy ns) — `client-id: oauth2-proxy`,
|
|
`client-secret` = the SAME `oauth2-proxy-client-secret` above, `cookie-secret` = `openssl rand -base64 32`.
|
|
3. **`../monitoring/grafana-oidc-credentials.sops.yaml`** (monitoring ns) — `client-secret` = the SAME
|
|
`grafana-client-secret` above.
|
|
|
|
(Cross-namespace Secrets can't be shared, so each client secret is stored twice — once for the realm
|
|
import in `keycloak`, once for the consuming app.) See the two `.sops.yaml.example` templates for the
|
|
exact keys + steps. No plaintext secret is ever committed.
|
|
|
|
## Client secrets: the realm import fails OPEN, so a reconciler owns them (2026-09-06)
|
|
|
|
**The realm import is a one-shot that fails open.** Measured on both clusters:
|
|
|
|
* When the operator **cannot resolve** a `$(env:...)` placeholder it stores **the literal placeholder
|
|
text** as the client secret — and the import Job still exits 0. There is no PENDING state, no
|
|
error condition, and `HasErrors` stays `False`.
|
|
* Once complete the CR rests at **`Done=True` / `Started=False` and never re-runs**, so a Secret key
|
|
added later is never picked up.
|
|
|
|
Together those produced a total SSO outage on prod (all 12 clients holding the literal placeholder;
|
|
every login rejected with *"Invalid client or Invalid client credentials"*) and a single broken client
|
|
on dev (`oauth2-proxy`, whose key was added after dev's import had already completed). Every cluster
|
|
signal was green throughout, including the conformance test that asserts `Done=True`.
|
|
|
|
Three layers now cover it:
|
|
|
|
| Layer | Where | What it does |
|
|
|---|---|---|
|
|
| **Prevent** | `conformance/tests/l0-static/keycloak-client-secret-reconciler` (runs on every merge, unconditionally, from `ci/l0.sh`) | Every placeholder is defined, reads `keycloak-realm-clients`, and names an **existing** key called `<clientId>-client-secret`; public clients declare no secret; and the Flux Kustomization applying the Secret is a real transitive `dependsOn` ancestor (`wait: true`) of the ones applying the import and the reconciler. A missing SOPS key is now unmergeable. |
|
|
| **Heal** | `client-secret-reconciler.yaml` — CronJob, every 15m | Compares each client's **stored** secret with the mounted `keycloak-realm-clients` value and PUTs the **minimal** `{clientId, secret}` representation only where they differ. Skips correct clients, never logs a secret, holds a hard endpoint allow-list so users/groups/roles/flows are unreachable, and needs no Kubernetes API access (`automountServiceAccountToken: false`). |
|
|
| **Detect** | `conformance/tests/l2-behavioral/keycloak-client-secrets-match-git` + alerts `KeycloakClientSecretReconcilerFailing` / `…Stale` | Runs the same script in read-only `--audit` mode against the live realm and fails on any mismatch or literal placeholder; the alerts make the reconciler's silence observable in prod. |
|
|
|
|
The reconciler's own `--selftest` (run by the L0 gate, which extracts the script from the shipped
|
|
ConfigMap) proves compare-then-write, **idempotence** (zero writes on a converged realm),
|
|
no-secret-logging, the endpoint allow-list, and fail-closed behaviour.
|
|
|
|
**Manual repair / one-off run** (the CronJob normally does this):
|
|
|
|
```sh
|
|
kubectl -n keycloak create job --from=cronjob/keycloak-client-secret-reconciler kc-fix-$(date +%s)
|
|
kubectl -n keycloak logs -l app.kubernetes.io/name=keycloak-client-secret-reconciler --tail=-1
|
|
```
|
|
|
|
**What the reconciler does NOT fix:** anything other than client secrets. A NEW client, a changed
|
|
redirect URI or a new protocol mapper still needs the realm import to actually re-run (it will not on
|
|
its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that a realm edit
|
|
landed.
|
|
|
|
## Exceptions that keep native auth (do not force SSO) — ADR-0014
|
|
|
|
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
|