jakman85andClaude Opus 5
e7c4a95c2d
The four backup-credential SOPS Secrets are SHARED BASE manifests: platform/ and apps/ are
rendered by the same Flux tiers on both clusters, and the ONLY per-environment difference in
this repo is cluster-vars substitution (R-ENV-02). So writing the R2 token into all four
(0a6a862) did not "give prod credentials" — it REPLACED DEV'S. Merging that as-is would have
handed dev's Velero and all three dev CNPG Clusters an R2 key id to sign MinIO requests with:
403 on every WAL segment, and archive_mode=on turns that into a disk-full clock on the pgdata
PVC, not a degraded backup. Dev's PROVEN DR restore (R-STATE-03) is the thing that would have
broken.
This commit splits the material. Each credential is now TWO Secret objects, both applied to
BOTH clusters, differing only in name suffix:
dev -> velero-cloud-credentials keycloak/nextcloud/semaphore-db-backup-credentials
prod -> velero-cloud-credentials-prod ...-db-backup-credentials-prod
The consumer picks one by name; that wiring is the NEXT commit, so this one changes no
rendered reference and is a no-op for both live clusters.
WHY DEV KEEPS THE UNSUFFIXED NAME. Dev is live. Renaming its Secret would change
`s3Credentials.accessKeyId.name` on three running CNPG Clusters — which projects into the
instance pods' environment and therefore ROLLS keycloak-db, nextcloud-db and semaphore-db —
and would roll the velero Deployment, all to rename an object whose contents do not change.
This repo already makes exactly this call for dev's live identifiers (EXTERNALDNS_TXT_OWNER_ID
"default", VELERO_BACKUP_PREFIX ""): the live cluster keeps the name it is running, and the
NEW environment takes the new one. The asymmetry is deliberate and documented at each
kustomization.
WHERE THE MATERIAL CAME FROM. Neither side is invented:
- the four unsuffixed files are restored byte-for-byte from `main` (the MinIO material as
it stood before 0a6a862), so dev's Secrets are IDENTICAL to what dev is running;
- the four `-prod` files carry the R2 token from 0a6a862 (already on this branch as
9739e9b), re-encrypted only because metadata.name changed.
RE-ENCRYPTION, AND WHY A COPY+SED IS NOT ENOUGH. `.sops.yaml` sets
`encrypted_regex: ^(data|stringData)$`, so metadata.name stays cleartext — but SOPS MACs the
WHOLE document (there is no `mac_only_encrypted` here), so editing that cleartext line
invalidates the MAC and `sops -d` then refuses the file. The `-prod` files were therefore
produced inside the dev toolbox (infra/devcontainer, which has sops 3.9.4 and the age key
mounted) by decrypt -> rename -> re-encrypt, with the plaintext never leaving a container
tmpfile. Note for the next person: `sops -e --output NEW OLD` matches creation rules against
the INPUT path, so encrypting a tmpfile into a repo path needs `--filename-override NEW` or
it dies with "no matching creation rules found".
Verified: all eight files `sops -d` cleanly; the four dev files hash to dev's material and the
four prod files to R2's, and the two sets differ. `.sops.yaml` needed no new rule — the
existing `(platform|apps|catalog)/.*(secret|sops)\.ya?ml$` rule already matches
`*-prod.sops.yaml`, so the new files encrypt to the same age recipient.
apps/wave-0/kustomization.yaml is GENERATED, so render.py learns the `-prod` sidecar rather
than the golden being hand-edited: `<db>-backup-credentials-prod.sops.yaml` now places into
the DB's wave alongside its unsuffixed twin (both must exist before the Cluster that names
one), and the selftest asserts it.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/keycloak
Keycloak — the platform Identity Provider (SSO via OIDC + SAML), onboarded early (Phase 6 IdP).
- Governing ADR: ADR-0014 — IdP: Keycloak
- Spec: docs/15 — Identity & Access
- Serves: R-IDP-01..06
- Owner: WS-SEC
Image / tooling strategy (F-KY1 landmine avoided)
The original stub used the Bitnami Keycloak chart + bundled Bitnami PostgreSQL. Bitnami's free Docker Hub images were purged (DECISIONS-LOG F-KY1) → that path ImagePullBackOffs. Replaced with:
- Keycloak: the official Keycloak Operator (
quay.io/keycloak/keycloak-operator+quay.io/keycloak/keycloak, v26.0.6) driving aKeycloakCR + aKeycloakRealmImportCR for config-as-code (ADR-0014 permits Operator OR Helm; Operator gives clean CR-driven realm reconcile). - Database: CloudNativePG (
ghcr.io/cloudnative-pg, see../cnpg) — aClusteron the 3-replicalonghornStorageClass (DR-critical, R-IDP-05), with declarative backup hooks.
quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.
Layout (controllers / configs split — F-MON1)
| File | Applied by | What |
|---|---|---|
kustomization.yaml (namespace, operator CRDs+Deployment, admin secret, dashboard) |
platform | controllers |
keycloak-instance.yaml (Keycloak CR, ${KEYCLOAK_REPLICAS}) |
platform-config | config |
postgres-cluster.yaml (CNPG Cluster on longhorn) |
platform-config | config |
keycloak-realm-import.yaml (KeycloakRealmImport: realm/groups/clients/MFA) |
platform-config | config |
httproute.yaml (keycloak.home.keithjackson.dev) |
platform-config | config |
servicemonitor.yaml (Prometheus scrape) |
platform-config | config |
Realm config (groups platform-admins/media-users/viewers, OIDC clients grafana+oauth2-proxy,
brute-force ON, and a conditional MFA browser flow that REQUIRES OTP for platform-admins, R-IDP-04)
reconciles purely from git — no console clicking (R-IDP-03).
Admin credential + break-glass (R-IDP-04 / R-IDP-05)
The provisioned keycloak-admin-credentials.sops.yaml (keys admin-user/admin-password) is injected
as the temporary bootstrap/break-glass admin via KC_BOOTSTRAP_ADMIN_* env (podTemplate escape
hatch) — this consumes the secret as provisioned, without re-keying it (the operator's native
bootstrapAdmin wants username/password keys, which would require editing the encrypted file).
The Keycloak bootstrap admin is the documented break-glass path so an IdP/SSO outage does not lock the
operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO is proven.
Operator hand-offs (fill before / during deploy)
keycloak-realm-clients.sops.yaml(keycloak ns) — from.sops.yaml.example. Two client secrets (grafana-client-secret,oauth2-proxy-client-secret),openssl rand -hex 32each. The realm import injects them into the Keycloak clients viaspec.placeholders. Add tokustomization.yamlonce filled. The realm import stays PENDING until this exists.../oauth2-proxy/oauth2-proxy-credentials.sops.yaml(oauth2-proxy ns) —client-id: oauth2-proxy,client-secret= the SAMEoauth2-proxy-client-secretabove,cookie-secret=openssl rand -base64 32.../monitoring/grafana-oidc-credentials.sops.yaml(monitoring ns) —client-secret= the SAMEgrafana-client-secretabove.
(Cross-namespace Secrets can't be shared, so each client secret is stored twice — once for the realm
import in keycloak, once for the consuming app.) See the two .sops.yaml.example templates for the
exact keys + steps. No plaintext secret is ever committed.
Exceptions that keep native auth (do not force SSO) — ADR-0014
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).