jakman85andClaude Opus 4.8
d1248e6778
Author the GitOps side of off-cluster backups against the on-Proxmox MinIO (http://192.168.1.9:9000, path-style; buckets velero + cnpg). Velero (R-STATE-03): - HelmRelease (chart 8.x / Velero 1.15) + velero-plugin-for-aws init container + node-agent kopia File System Backup (deployNodeAgent, uploaderType=kopia, snapshotsEnabled=false). BSL default -> bucket velero, s3ForcePathStyle. - Two daily Schedules: `daily` backs up the whole cluster's resources+PVC data (defaultVolumesToFsBackup opt-out) EXCEPT monitoring+unifi; `daily-resources-only` captures monitoring+unifi RESOURCES only (Prometheus TSDB, Loki 72h logs, and the ~7.4GiB UniFi VM disk are reconstructable — excluded from volume backup). - velero-cloud-credentials SOPS Secret (cloud INI key, ns velero). - Own non-gating Flux tier clusters/base/backup.yaml (dependsOn platform-config, wait:false) + clusters/dev SOPS decryption patch — mirrors headlamp. CNPG native Barman PITR (R-IDP-05 / R-STATE-03): - keycloak-db, nextcloud-db, semaphore-db gain spec.backup.barmanObjectStore (s3://cnpg/<cluster>, endpointURL MinIO, gzip WAL+data, 30d retention) + a daily ScheduledBackup (6-field cron, immediate) each. - Per-namespace scoped-key SOPS Secrets (<db>-backup-credentials), same-ns only. - render.py: emit-waves now recognizes <svc>-db-backup-credentials.sops.yaml as a DB-wave sidecar (+ self-test); wave-0 kustomization regenerated. Creds mirror platform/backup/minio-credentials.sops.yaml (scoped key k3s-backup), SOPS round-trip verified with the cluster age key. gate: green. Co-Authored-By: Claude Opus 4.8 <[email protected]>
platform/keycloak
Keycloak — the platform Identity Provider (SSO via OIDC + SAML), onboarded early (Phase 6 IdP).
- Governing ADR: ADR-0014 — IdP: Keycloak
- Spec: docs/15 — Identity & Access
- Serves: R-IDP-01..06
- Owner: WS-SEC
Image / tooling strategy (F-KY1 landmine avoided)
The original stub used the Bitnami Keycloak chart + bundled Bitnami PostgreSQL. Bitnami's free Docker Hub images were purged (DECISIONS-LOG F-KY1) → that path ImagePullBackOffs. Replaced with:
- Keycloak: the official Keycloak Operator (
quay.io/keycloak/keycloak-operator+quay.io/keycloak/keycloak, v26.0.6) driving aKeycloakCR + aKeycloakRealmImportCR for config-as-code (ADR-0014 permits Operator OR Helm; Operator gives clean CR-driven realm reconcile). - Database: CloudNativePG (
ghcr.io/cloudnative-pg, see../cnpg) — aClusteron the 3-replicalonghornStorageClass (DR-critical, R-IDP-05), with declarative backup hooks.
quay.io + ghcr.io are unaffected by the Bitnami purge. Operator manifests are pinned upstream release files pulled as remote kustomize bases (the kubevirt-stub pattern); Renovate maintains the version.
Layout (controllers / configs split — F-MON1)
| File | Applied by | What |
|---|---|---|
kustomization.yaml (namespace, operator CRDs+Deployment, admin secret, dashboard) |
platform | controllers |
keycloak-instance.yaml (Keycloak CR, ${KEYCLOAK_REPLICAS}) |
platform-config | config |
postgres-cluster.yaml (CNPG Cluster on longhorn) |
platform-config | config |
keycloak-realm-import.yaml (KeycloakRealmImport: realm/groups/clients/MFA) |
platform-config | config |
httproute.yaml (keycloak.home.keithjackson.dev) |
platform-config | config |
servicemonitor.yaml (Prometheus scrape) |
platform-config | config |
Realm config (groups platform-admins/media-users/viewers, OIDC clients grafana+oauth2-proxy,
brute-force ON, and a conditional MFA browser flow that REQUIRES OTP for platform-admins, R-IDP-04)
reconciles purely from git — no console clicking (R-IDP-03).
Admin credential + break-glass (R-IDP-04 / R-IDP-05)
The provisioned keycloak-admin-credentials.sops.yaml (keys admin-user/admin-password) is injected
as the temporary bootstrap/break-glass admin via KC_BOOTSTRAP_ADMIN_* env (podTemplate escape
hatch) — this consumes the secret as provisioned, without re-keying it (the operator's native
bootstrapAdmin wants username/password keys, which would require editing the encrypted file).
The Keycloak bootstrap admin is the documented break-glass path so an IdP/SSO outage does not lock the
operator out (ADR-0014); Grafana likewise keeps its local admin login until SSO is proven.
Operator hand-offs (fill before / during deploy)
keycloak-realm-clients.sops.yaml(keycloak ns) — from.sops.yaml.example. Two client secrets (grafana-client-secret,oauth2-proxy-client-secret),openssl rand -hex 32each. The realm import injects them into the Keycloak clients viaspec.placeholders. Add tokustomization.yamlonce filled. The realm import stays PENDING until this exists.../oauth2-proxy/oauth2-proxy-credentials.sops.yaml(oauth2-proxy ns) —client-id: oauth2-proxy,client-secret= the SAMEoauth2-proxy-client-secretabove,cookie-secret=openssl rand -base64 32.../monitoring/grafana-oidc-credentials.sops.yaml(monitoring ns) —client-secret= the SAMEgrafana-client-secretabove.
(Cross-namespace Secrets can't be shared, so each client secret is stored twice — once for the realm
import in keycloak, once for the consuming app.) See the two .sops.yaml.example templates for the
exact keys + steps. No plaintext secret is ever committed.
Exceptions that keep native auth (do not force SSO) — ADR-0014
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).