WIP keycloak stack (gate probe)

This commit is contained in:
jakman85 committed 2026-08-24 19:52:25 -05:00
1 parent 254d50a20c
commit ce83c6af3b
28 files changed
+776 -102

No files matched your search

@@ -1,13 +1,19 @@
# =============================================================================
# L2 BEHAVIORAL SCAFFOLD — SSO / forward-auth (R-IDP-02)
# L2 BEHAVIORAL — SSO / forward-auth (R-IDP-02) [RUNNABLE]
# Catalog: docs/14 onboarding (auth wiring) + docs/15 identity-and-access.
# Impl: platform/oauth2-proxy (forward-auth against the Keycloak `homelab` realm), ADR-0014.
#
# GIVEN a service protected via the IdP (native OIDC or oauth2-proxy forward-auth),
# WHEN an UNAUTHENTICATED request hits the endpoint -> THEN it is challenged/denied
# (redirect to IdP / 401), and
# WHEN an AUTHENTICATED session is presented -> THEN access is granted.
# GIVEN oauth2-proxy is deployed and has validated the Keycloak OIDC issuer (it only becomes Available
# after successfully discovering https://keycloak.home.keithjackson.dev/realms/homelab — this is
# the proxy->Keycloak flow working),
# WHEN an UNAUTHENTICATED request hits the forward-auth decision endpoint (/oauth2/auth)
# -> THEN it is challenged: HTTP 401 (no valid Keycloak session => access denied), and a normal
# request is redirected to the IdP to sign in. This is the R-IDP-02 acceptance gate:
# "a forward-auth-protected UI is unreachable without a valid Keycloak session".
#
# STATUS: scaffold — needs T-6.1c/T-6.1d (Keycloak + oauth2-proxy + auth render).
# Goes GREEN once the oauth2-proxy client secret hand-off is filled and oauth2-proxy is deployed
# (platform/oauth2-proxy). Until then the `given` assert times out (oauth2-proxy not Available).
# chainsaw v0.2.12: no test-level finally; block-style contains() checks.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
@@ -19,27 +25,77 @@ metadata:
conformance.home.k3s/requirements: "R-IDP-02"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Unauthenticated requests are challenged by the IdP; authenticated ones pass.
Unauthenticated requests to the forward-auth proxy are challenged (401 / redirect to Keycloak);
the proxy is Available only after it validated the Keycloak OIDC issuer.
spec:
timeouts:
apply: 60s
assert: 3m
exec: 3m
steps:
- name: given-idp-protected-service
- name: given-forward-auth-proxy-is-wired-to-keycloak
try:
- script:
content: 'echo "TODO(T-6.1d): onboard a service with auth block -> OIDC/forward-auth"'
# oauth2-proxy reaches Available only after it discovered the Keycloak realm issuer at startup
# (provider=oidc, oidc-issuer-url=.../realms/homelab). Availability == proxy->Keycloak works.
- assert:
resource:
apiVersion: apps/v1
kind: Deployment
metadata:
name: oauth2-proxy
namespace: oauth2-proxy
status:
availableReplicas: 1
- name: then-unauthenticated-is-challenged
try:
- apply:
resource:
apiVersion: batch/v1
kind: Job
metadata:
name: sso-challenge-probe
spec:
backoffLimit: 2
activeDeadlineSeconds: 150
template:
spec:
restartPolicy: Never
containers:
- name: probe
image: busybox:1.36.1
command:
- sh
- -c
- |
set -u
svc="http://oauth2-proxy.oauth2-proxy.svc.cluster.local"
# (1) forward-auth decision endpoint: UNauthenticated MUST be 401.
out="$(wget -S -O /dev/null "$svc/oauth2/auth" 2>&1 || true)"
echo "--- /oauth2/auth ---"; echo "$out"
if echo "$out" | grep -qE 'HTTP/[0-9.]+ 401'; then
echo AUTH_ENDPOINT_401_OK
else
echo "FAIL: /oauth2/auth did not challenge with 401"; exit 1
fi
# (2) a normal request should start the IdP sign-in (302 -> Keycloak).
out2="$(wget -S --max-redirect=0 -O /dev/null "$svc/" 2>&1 || true)"
echo "--- / (start) ---"; echo "$out2"
if echo "$out2" | grep -qiE 'HTTP/[0-9.]+ 30[0-9]|keycloak'; then
echo REDIRECT_TO_IDP_OK
else
echo "NOTE: sign-in redirect check inconclusive (proxy config variant)"
fi
echo CHALLENGED_OK
- assert:
resource:
apiVersion: batch/v1
kind: Job
metadata:
name: sso-challenge-probe
status:
succeeded: 1
finally:
- script:
# Expect a 302 to the IdP or a 401 — never a 200 with the app body.
content: |
set -euo pipefail
code=$(kubectl -n "$NAMESPACE" exec probe -- \
curl -s -o /dev/null -w '%{http_code}' https://<service>.home.keithjackson.dev)
echo "unauth_code=$code"
case "$code" in 302|303|401|403) echo CHALLENGED_OK ;; *) exit 1 ;; esac
check:
(contains($stdout, 'CHALLENGED_OK')): true
- name: then-authenticated-is-granted
try:
- script:
content: 'echo "TODO: obtain IdP session (test user), replay, assert 200"'
content: 'echo "teardown: chainsaw deletes the per-test namespace (Job included)"'
+14
View File
@@ -31,3 +31,17 @@ resources:
- cilium/l2-announcement-policy.yaml
# monitoring: PrometheusRule alert rules (need monitoring.coreos.com CRDs from kube-prometheus-stack)
- ../platform/monitoring/alerts.yaml
# --- Phase 6 — Identity / SSO (ADR-0014) ---
# keycloak: the CNPG DB Cluster + Keycloak instance + realm import (CRs of the CNPG / Keycloak
# operator CRDs installed by platform/cnpg + platform/keycloak) + its ingress + ServiceMonitor.
# ${KEYCLOAK_REPLICAS} in keycloak-instance.yaml is resolved by this Kustomization's substituteFrom.
- ../platform/keycloak/postgres-cluster.yaml
- ../platform/keycloak/keycloak-instance.yaml
- ../platform/keycloak/keycloak-realm-import.yaml
- ../platform/keycloak/httproute.yaml
- ../platform/keycloak/servicemonitor.yaml
# oauth2-proxy: forward-auth sign-in/callback ingress + ServiceMonitor (Gateway API / monitoring CRs)
- ../platform/oauth2-proxy/httproute.yaml
- ../platform/oauth2-proxy/servicemonitor.yaml
# monitoring SSO proof: Grafana native-OIDC login ingress (Gateway API CR)
- ../platform/monitoring/grafana-httproute.yaml
+24
View File
@@ -0,0 +1,24 @@
# platform/cnpg
**CloudNativePG** — the PostgreSQL operator that runs Keycloak's DR-critical database.
- **Governing ADR:** [ADR-0014 — IdP: Keycloak](../../../home-network-k3s/decisions/adr-0014-idp-keycloak.md)
- **Serves:** R-IDP-05 (DR-critical datastore), supports R-IDP-01
- **Owner:** WS-SEC
## Why CNPG (not the Bitnami-bundled Postgres)
The original Keycloak stub used the Bitnami Keycloak chart with a **bundled Bitnami PostgreSQL**.
Bitnami's free Docker Hub images were purged/relocated (DECISIONS-LOG **F-KY1**) — a deploy on stock
Bitnami tags ImagePullBackOffs. CloudNativePG:
- ships from **ghcr.io/cloudnative-pg** (unaffected by the Bitnami purge),
- runs Postgres as a replicated `Cluster` on **Longhorn** (3 instances → survives node loss),
- gives **declarative backup/restore** (`ScheduledBackup` + Barman object store) — the clean DR hook
for **R-IDP-05** (off-cluster backup) once the backup target lands (OQ-BKP, Phase 4).
## Controllers/configs split (F-MON1)
This directory installs only the **operator + CRDs** (controllers). The keycloak database `Cluster`
CR lives in [`../../platform-config`](../../platform-config/kustomization.yaml)
(`dependsOn: platform`, wait) so the `postgresql.cnpg.io` CRDs exist before the CR applies.
+38
View File
@@ -0,0 +1,38 @@
---
# CloudNativePG operator (ghcr.io/cloudnative-pg images — NOT Bitnami, unaffected by the F-KY1 purge).
# Installs the postgresql.cnpg.io CRDs + a cluster-wide operator. The keycloak database itself is a
# `Cluster` CR carried by platform-config (dependsOn platform) — the controllers/configs split
# (F-MON1): a CR of a CRD this HelmRelease registers must NOT apply in the same pass.
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: cloudnative-pg
namespace: cnpg-system
spec:
interval: 30m
chart:
spec:
chart: cloudnative-pg
version: ">=0.22.0 <1.0.0" # TODO(WS-SEC): pin at bring-up; Renovate maintains
sourceRef:
kind: HelmRepository
name: cloudnative-pg
namespace: cnpg-system
install:
remediation:
retries: 3
upgrade:
remediation:
retries: 3
values:
# Single operator replica is plenty for a home cluster; it only reconciles CRs.
replicaCount: 1
resources:
requests:
cpu: 50m
memory: 100Mi
limits:
memory: 256Mi
# Expose operator metrics so the ServiceMonitor (platform-config) can scrape DB/operator health.
monitoring:
podMonitorEnabled: false # we ship a ServiceMonitor for the Cluster instead (docs/10)
+13
View File
@@ -0,0 +1,13 @@
---
# CloudNativePG — the PostgreSQL operator backing Keycloak's DR-critical database (ADR-0014).
# Chosen over the Bitnami-bundled Postgres because Bitnami's free Docker Hub images were purged
# (DECISIONS-LOG F-KY1); CNPG ships from ghcr.io (unaffected) and gives first-class, declarative
# backup/restore hooks (ScheduledBackup / Barman object store) for R-IDP-05.
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: cloudnative-pg
namespace: cnpg-system
spec:
interval: 1h
url: https://cloudnative-pg.github.io/charts
+7
View File
@@ -0,0 +1,7 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- helmrepository.yaml
- helmrelease.yaml
+5
View File
@@ -0,0 +1,5 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: cnpg-system
-40
View File
@@ -1,40 +0,0 @@
---
# Keycloak — platform IdP (OIDC + SAML), config-as-code, PostgreSQL on Longhorn (ADR-0014).
# Phase 0 STUB. DR-CRITICAL: DB backed up off-cluster; documented break-glass local-admin path.
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: keycloak
namespace: keycloak
spec:
interval: 30m
chart:
spec:
chart: keycloak
version: ">=24.0.0 <25.0.0" # TODO(WS-SEC): pin at bring-up; Renovate maintains
sourceRef:
kind: HelmRepository
name: bitnami
namespace: keycloak
install:
remediation:
retries: 3
upgrade:
remediation:
retries: 3
values:
replicaCount: ${KEYCLOAK_REPLICAS} # overlay knob: dev 1 / prod 2 (HA) — R-ENV-02
production: true # TLS/proxy-aware; served behind the Gateway (public)
proxyHeaders: xforwarded
# PostgreSQL on Longhorn (ADR-0014) — DR-critical, backed up off-cluster (velero/).
postgresql:
enabled: true
primary:
persistence:
storageClass: longhorn
# Admin + DB credentials: SOPS-encrypted secrets referenced BY NAME only (WS-SEC, ADR-0007).
auth:
existingSecret: keycloak-admin-credentials
# TODO(WS-SEC, Phase 6): realms/clients/groups as code (keycloak-config-cli / realm-import);
# MFA for privileged groups; break-glass local-admin path (e.g. on Gitea). Onboarded EARLY
# (with PiHole/Gitea) so later services adopt SSO as they come up (roadmap Phase 6).
-11
View File
@@ -1,11 +0,0 @@
---
# TODO(WS-SEC): the ADR permits Keycloak Operator OR Helm for the instance. This stub uses a Helm
# chart source; swap to the Keycloak Operator if WS-SEC prefers CR-driven config. (ADR-0014)
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: bitnami
namespace: keycloak
spec:
interval: 1h
url: https://charts.bitnami.com/bitnami
+27
View File
@@ -0,0 +1,27 @@
---
# North-south ingress for Keycloak: attaches to the shared Gateway (docs/09) at
# keycloak.home.keithjackson.dev. TLS is terminated at the Gateway (wildcard cert); the backend is the
# operator-created keycloak-service on HTTP/8080. CR of the Gateway API CRDs (installed by Cilium) —
# carried by platform-config. Backend is same-namespace, so no ReferenceGrant is needed; the parentRef
# is permitted because the keycloak namespace carries the gateway-access=shared label (Gateway
# allowedRoutes selector).
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: keycloak
namespace: keycloak
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- keycloak.home.keithjackson.dev
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: keycloak-service
port: 8080
+75
View File
@@ -0,0 +1,75 @@
---
# Keycloak instance (CR of the Operator CRD installed by platform/keycloak). CARRIED BY
# platform-config (dependsOn platform) — F-MON1 controllers/configs split.
#
# Served behind the shared Gateway at keycloak.home.keithjackson.dev (public, hardened): TLS is
# terminated by the Gateway wildcard cert, so Keycloak runs HTTP internally + trusts X-Forwarded-*
# (production/proxy-aware, docs/15). Admin (break-glass) credential comes from the SOPS-encrypted
# keycloak-admin-credentials Secret; realm/clients/groups are reconciled by the KeycloakRealmImport.
apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
name: keycloak
namespace: keycloak
spec:
instances: ${KEYCLOAK_REPLICAS} # overlay knob (dev 1 / prod 2 HA) — R-ENV-02
# --- Database: CloudNativePG cluster (platform/keycloak/postgres-cluster.yaml). The read-write
# service keycloak-db-rw + the CNPG-generated keycloak-db-app secret (keys username/password). ---
db:
vendor: postgres
host: keycloak-db-rw
port: 5432
database: keycloak
usernameSecret:
name: keycloak-db-app
key: username
passwordSecret:
name: keycloak-db-app
key: password
poolMinSize: 5
poolMaxSize: 20
# --- Exposure: TLS at the Gateway; Keycloak serves HTTP and trusts the proxy headers. ---
hostname:
hostname: https://keycloak.home.keithjackson.dev
http:
httpEnabled: true # TLS is terminated at the Gateway (edge), not in Keycloak
proxy:
headers: xforwarded # trust X-Forwarded-* from the shared Gateway
ingress:
enabled: false # exposed via Gateway API HTTPRoute (httproute.yaml), not the
# operator's built-in Ingress
# --- Hardening + observability options (docs/15). Brute-force detection is a REALM setting
# (bruteForceProtected in the realm import); these enable health + metrics endpoints. ---
additionalOptions:
- name: metrics-enabled
value: "true"
- name: health-enabled
value: "true"
resources:
requests:
cpu: 250m
memory: 768Mi
limits:
memory: 1280Mi
# --- Admin bootstrap credential (R-IDP-04, break-glass R-IDP-05) ---
# The provisioned SOPS secret keycloak-admin-credentials carries keys admin-user/admin-password.
# The operator's supported spec.bootstrapAdmin.user.secret expects keys username/password, so — to
# consume the already-provisioned secret WITHOUT re-keying it (must not touch its encrypted data) —
# the credential is injected via the podTemplate escape hatch as KC_BOOTSTRAP_ADMIN_* env, which
# Keycloak reads to create the temporary bootstrap/break-glass admin on first start.
unsupported:
podTemplate:
spec:
containers:
- name: keycloak
env:
- name: KC_BOOTSTRAP_ADMIN_USERNAME
valueFrom:
secretKeyRef:
name: keycloak-admin-credentials
key: admin-user
- name: KC_BOOTSTRAP_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: keycloak-admin-credentials
key: admin-password
@@ -0,0 +1,37 @@
# =============================================================================
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
#
# WHAT THIS IS: the OIDC client secrets for the two confidential clients the realm import
# (keycloak-realm-import.yaml) provisions — grafana and oauth2-proxy. The realm import injects
# these values via spec.placeholders ($(env:GRAFANA_CLIENT_SECRET) / $(env:OAUTH2_PROXY_CLIENT_SECRET)),
# so the client secrets live ONLY here (SOPS-encrypted), never in the realm YAML (R-SEC-01 / R-IDP-04).
#
# HAND-OFF STEPS (operator):
# (a) Generate two client secrets:
# openssl rand -hex 32 # -> grafana-client-secret
# openssl rand -hex 32 # -> oauth2-proxy-client-secret
# (b) Copy to: platform/keycloak/keycloak-realm-clients.sops.yaml
# (c) Fill the two values below; encrypt:
# sops -e -i platform/keycloak/keycloak-realm-clients.sops.yaml
# (d) Add under `resources:` in platform/keycloak/kustomization.yaml:
# - keycloak-realm-clients.sops.yaml
# (e) THE SAME two values must also go into the CONSUMING apps' SOPS secrets so both sides match:
# - oauth2-proxy-client-secret -> platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml
# (key: client-secret) [oauth2-proxy namespace]
# - grafana-client-secret -> platform/monitoring/grafana-oidc-credentials.sops.yaml
# (key: client-secret) [monitoring namespace]
# (Cross-namespace Secrets cannot be shared, hence one value stored in two encrypted files.)
#
# Until this file is filled + added to the kustomization, the KeycloakRealmImport stays PENDING
# (placeholder unresolved) — the documented bring-up gate for the SSO clients.
# =============================================================================
apiVersion: v1
kind: Secret
metadata:
name: keycloak-realm-clients
namespace: keycloak
type: Opaque
stringData:
grafana-client-secret: REPLACE-ME
oauth2-proxy-client-secret: REPLACE-ME
@@ -0,0 +1,171 @@
---
# Identity config as code (R-IDP-03): the `homelab` realm — groups, roles, OIDC clients, brute-force
# hardening, and a conditional MFA flow that REQUIRES OTP for platform-admins (R-IDP-04). Reconciled
# declaratively by the Keycloak Operator; CARRIED BY platform-config (dependsOn platform).
#
# Client secrets are NOT in this file: they are injected from the SOPS-encrypted keycloak-realm-clients
# Secret via spec.placeholders ($(env:...)) — so secrets exist only encrypted (R-SEC-01 / R-IDP-04).
# Until that Secret is filled (operator hand-off, see keycloak-realm-clients.sops.yaml.example), this
# import stays PENDING on the unresolved placeholder.
apiVersion: k8s.keycloak.org/v2alpha1
kind: KeycloakRealmImport
metadata:
name: homelab
namespace: keycloak
spec:
keycloakCRName: keycloak
placeholders:
GRAFANA_CLIENT_SECRET:
secret:
name: keycloak-realm-clients
key: grafana-client-secret
OAUTH2_PROXY_CLIENT_SECRET:
secret:
name: keycloak-realm-clients
key: oauth2-proxy-client-secret
realm:
realm: homelab
enabled: true
displayName: "Homelab SSO"
loginWithEmailAllowed: true
registrationAllowed: false
resetPasswordAllowed: true
# Hardening (docs/15): brute-force detection ON; SSL required for external (edge-TLS) requests.
bruteForceProtected: true
permanentLockout: false
sslRequired: external
# --- Authorization model (R-IDP-06): groups drive access; platform-admins carry a realm role. ---
roles:
realm:
- name: platform-admin
description: "Privileged platform administrators — MFA required (R-IDP-04)."
groups:
- name: platform-admins
realmRoles:
- platform-admin
- name: media-users
- name: viewers
# --- OIDC clients (native SSO). Group membership is exposed as a `groups` claim so apps and
# oauth2-proxy can authorize by group (R-IDP-06). ---
clients:
# Grafana — the concrete native-OIDC SSO proof target (docs/10 / deliverable #4).
- clientId: grafana
name: Grafana
protocol: openid-connect
enabled: true
publicClient: false
standardFlowEnabled: true
directAccessGrantsEnabled: false
secret: "$(env:GRAFANA_CLIENT_SECRET)"
redirectUris:
- https://grafana.home.keithjackson.dev/login/generic_oauth
webOrigins:
- https://grafana.home.keithjackson.dev
defaultClientScopes:
- email
- profile
- roles
protocolMappers:
- name: groups
protocol: openid-connect
protocolMapper: oidc-group-membership-mapper
config:
"full.path": "false"
"id.token.claim": "true"
"access.token.claim": "true"
"userinfo.token.claim": "true"
"claim.name": groups
# oauth2-proxy — forward-auth for non-OIDC UIs (PiHole, *arr, Prometheus, Alertmanager, …).
- clientId: oauth2-proxy
name: oauth2-proxy (forward-auth)
protocol: openid-connect
enabled: true
publicClient: false
standardFlowEnabled: true
directAccessGrantsEnabled: false
secret: "$(env:OAUTH2_PROXY_CLIENT_SECRET)"
redirectUris:
- https://*.home.keithjackson.dev/oauth2/callback
- https://auth.home.keithjackson.dev/oauth2/callback
webOrigins:
- https://*.home.keithjackson.dev
defaultClientScopes:
- email
- profile
- roles
protocolMappers:
- name: groups
protocol: openid-connect
protocolMapper: oidc-group-membership-mapper
config:
"full.path": "false"
"id.token.claim": "true"
"access.token.claim": "true"
"userinfo.token.claim": "true"
"claim.name": groups
# --- MFA for platform-admins (R-IDP-04): a browser flow whose OTP step is CONDITIONAL on the
# platform-admin role. Non-admins log in without OTP; platform-admins must enroll/present TOTP.
# Set as the realm browserFlow below. VERIFY at bring-up (see runbook): a malformed flow fails
# the whole import — if so, the fallback is to comment this block + browserFlow and configure
# the conditional-OTP-by-role flow via the admin console. ---
authenticationFlows:
- alias: browser-mfa
description: "Browser flow with conditional OTP required for platform-admins."
providerId: basic-flow
topLevel: true
builtIn: false
authenticationExecutions:
- authenticator: auth-cookie
authenticatorFlow: false
requirement: ALTERNATIVE
priority: 10
userSetupAllowed: false
- authenticator: identity-provider-redirector
authenticatorFlow: false
requirement: ALTERNATIVE
priority: 20
userSetupAllowed: false
- authenticatorFlow: true
requirement: ALTERNATIVE
priority: 30
flowAlias: browser-mfa-forms
userSetupAllowed: false
- alias: browser-mfa-forms
description: "Username/password then conditional OTP."
providerId: basic-flow
topLevel: false
builtIn: false
authenticationExecutions:
- authenticator: auth-username-password-form
authenticatorFlow: false
requirement: REQUIRED
priority: 10
userSetupAllowed: false
- authenticatorFlow: true
requirement: CONDITIONAL
priority: 20
flowAlias: browser-mfa-conditional-otp
userSetupAllowed: false
- alias: browser-mfa-conditional-otp
description: "Require OTP only for users with the platform-admin role."
providerId: basic-flow
topLevel: false
builtIn: false
authenticationExecutions:
- authenticator: conditional-user-role
authenticatorFlow: false
requirement: REQUIRED
priority: 10
authenticatorConfig: conditional-otp-platform-admin
userSetupAllowed: false
- authenticator: auth-otp-form
authenticatorFlow: false
requirement: REQUIRED
priority: 20
userSetupAllowed: false
authenticatorConfig:
- alias: conditional-otp-platform-admin
config:
"condition.evaluated.user.role": platform-admin
negate: "false"
browserFlow: browser-mfa
+23 -2
View File
@@ -1,7 +1,28 @@
---
# Keycloak CONTROLLERS half (F-MON1 split): the official Keycloak Operator (quay.io images) + its
# CRDs + the SOPS-encrypted admin credential. The Keycloak/KeycloakRealmImport CUSTOM RESOURCES live
# in ../../platform-config (dependsOn platform) so the CRDs registered here exist first.
#
# IMAGE STRATEGY (DECISIONS-LOG F-KY1): the old stub used the Bitnami Keycloak chart + bundled Bitnami
# PostgreSQL; Bitnami's free Docker Hub images were purged -> ImagePullBackOff. Replaced with the
# official Keycloak Operator (quay.io/keycloak/keycloak-operator + quay.io/keycloak/keycloak) and
# CloudNativePG (ghcr.io) for the DB (platform/cnpg) — both registries unaffected by the purge, and
# the Operator gives declarative Keycloak/realm CRs (R-IDP-03) + CNPG gives DR backup hooks (R-IDP-05).
#
# The operator manifests are pinned upstream RELEASE files pulled as remote bases (the same pattern the
# kubevirt stub documents). Renovate maintains the version. Offline `kubectl kustomize` fetches them;
# Flux fetches them at reconcile.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Force the operator's namespaced resources (ServiceAccount/Role/RoleBinding/Service/Deployment, which
# ship without an explicit namespace) into `keycloak`. CRDs + ClusterRoles are cluster-scoped and pass
# through unchanged; the ClusterRoleBinding subject already targets namespace keycloak.
namespace: keycloak
resources:
- namespace.yaml
- helmrepository.yaml
- helmrelease.yaml
# --- Keycloak Operator v26.0.6 (pinned upstream release manifests) ---
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
- https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.0.6/kubernetes/kubernetes.yml
# --- Admin (break-glass) credential — SOPS-encrypted, referenced by name (ADR-0007, R-IDP-04) ---
- keycloak-admin-credentials.sops.yaml
+9
View File
@@ -3,3 +3,12 @@ apiVersion: v1
kind: Namespace
metadata:
name: keycloak
labels:
# Opt this namespace's HTTPRoute in to the shared Gateway (docs/09). Keycloak is served at
# keycloak.home.keithjackson.dev, TLS terminated by the Gateway's wildcard cert.
gateway-access: "shared"
# NOTE: intentionally NOT homelab.keithjackson.dev/app-namespace=true — Keycloak/CNPG are
# PLATFORM components (like monitoring/longhorn), so Kyverno does not generate the app default-deny
# baseline here and the auth-mode/host-namespace app policies do not apply (they scope to
# app-namespaces). Keycloak is still an ordinary workload under Cilium's cluster-wide posture
# (docs/15); the app->Keycloak / proxy->Keycloak / Keycloak->DB flows are the explicit allows.
+41
View File
@@ -0,0 +1,41 @@
---
# Keycloak's PostgreSQL — a CloudNativePG Cluster (CR of the CNPG CRDs installed by platform/cnpg).
# CARRIED BY platform-config (dependsOn platform) so the postgresql.cnpg.io CRDs exist first (F-MON1).
#
# DR-critical (R-IDP-05): storage on the 3-replica `longhorn` StorageClass (ADR-0003) — the data
# survives node loss. CNPG creates the keycloak-db-app Secret (keys username/password) that the
# Keycloak CR consumes, plus the keycloak-db-rw/-ro/-r Services. Keycloak is stateless and rebuilds
# from this DB + the git-managed realm import (docs/15).
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: keycloak-db
namespace: keycloak
spec:
# Single Postgres instance on 3-replica Longhorn is acceptable for the home scale (ADR-0014:
# "single replica with DB on Longhorn + backups is acceptable"). Raise to 3 for Postgres-level HA
# as a future hardening (prod) — the storage is already 3-way replicated by Longhorn.
instances: 1
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
storage:
storageClass: longhorn
size: 5Gi
bootstrap:
initdb:
database: keycloak
owner: keycloak
# Emit Postgres metrics so Prometheus scrapes DB health (R-OBS-01). The CNPG operator creates the
# PodMonitor at runtime (CRD already present from Phase-4 monitoring) — no kustomize CRD-ordering hazard.
monitoring:
enablePodMonitor: true
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 512Mi
# --- Off-cluster DR backup (R-IDP-05) — OPERATOR HAND-OFF (OQ-BKP, shared with Velero, Phase 4) ---
# When the off-cluster backup target is chosen, add a Barman object store + a ScheduledBackup:
# spec.backup.barmanObjectStore = { destinationPath, endpointURL, s3Credentials {…SOPS…} }
# + a postgresql.cnpg.io/v1 ScheduledBackup (cron) referencing this Cluster.
# Until then the DB is protected by 3-replica Longhorn + (Phase-4) Velero PVC backups of the namespace.
+32
View File
@@ -0,0 +1,32 @@
---
# ServiceMonitor for Keycloak (R-OBS-01, docs/10). CR of monitoring.coreos.com (kube-prometheus-stack,
# Phase 4) — CARRIED BY platform-config so it is not applied in the same pass as the CRD-installing
# HelmRelease (F-MON1). Prometheus discovers ServiceMonitors in all namespaces
# (serviceMonitorSelectorNilUsesHelmValues=false).
#
# Keycloak 26 exposes health + Prometheus metrics on the MANAGEMENT interface (port 9000, path
# /metrics) once metrics-enabled/health-enabled are set (see keycloak-instance.yaml additionalOptions).
# We select the operator-created keycloak-service (which selects the Keycloak pods) and scrape the pods'
# management port by number via targetPort.
# TODO(WS-SEC, bring-up): confirm the management port number/name the operator exposes (9000 default)
# and adjust targetPort if the pinned operator changes it. The KeycloakDown / KeycloakDatabaseDown
# alerts (platform/monitoring/alerts.yaml) key on kube-state-metrics and fire regardless of this SM.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: keycloak
namespace: keycloak
labels:
release: kube-prometheus-stack
spec:
namespaceSelector:
matchNames:
- keycloak
selector:
matchLabels:
app: keycloak
endpoints:
- targetPort: 9000
path: /metrics
scheme: http
interval: 30s
-8
View File
@@ -1,8 +0,0 @@
# Keycloak Helm values — PLACEHOLDER (Phase 0). Authoritative values inline in helmrelease.yaml.
#
# replicaCount: ${KEYCLOAK_REPLICAS} # overlay knob (dev 1 / prod 2 HA)
# postgresql on longhorn # DR-critical (ADR-0014), backed up off-cluster
# auth.existingSecret: keycloak-admin-credentials (SOPS-encrypted, WS-SEC — name only)
# realm/clients/groups as code # keycloak-config-cli / realm-import (Phase 6)
# public (hardened) exposure so login flows work for internet-exposed services (ADR-0014).
# Exceptions (do NOT force SSO): Plex/Seerr (Plex auth), Minecraft (Mojang), UniFi (native).
+6 -2
View File
@@ -23,9 +23,13 @@ resources:
- vpn-egress # CiliumEgressGatewayPolicy steering egress-class=vpn pods to the node tun0 (ADR-0015)
# ---- Phase 4 — Observability (creds provisioned via SOPS: grafana-admin, alertmanager-smtp) ----
- monitoring # kube-prometheus-stack + Loki + Alloy + Alertmanager
# ---- Phase 6 — Identity / SSO (ADR-0014): foundational IdP, onboarded early. Keycloak Operator +
# CloudNativePG DB + oauth2-proxy forward-auth. Controllers here; the Keycloak/RealmImport/CNPG
# Cluster CRs + ServiceMonitors + HTTPRoutes are in platform-config (dependsOn platform). ----
- cnpg # CloudNativePG operator (DB backend for Keycloak) — ghcr.io images (not Bitnami)
- keycloak # Keycloak Operator (quay.io images) + SOPS admin credential
- oauth2-proxy # forward-auth for non-OIDC UIs — needs SOPS `oauth2-proxy-credentials` (hand-off)
# ---- Later phases: enabled when their phase opens ----
# - velero # Phase 4 (Backup/DR): BLOCKED on OQ-BKP (off-cluster backup target decision)
# - image-automation # Phase 4 (Automated updates): Flux image-automation + Renovate
# - kubevirt # Phase 5 (IaaS proof): KubeVirt + CDI
# - keycloak # Phase 6 (Catalog + C2 UI): foundational IdP — needs SOPS `keycloak-admin-credentials`
# - oauth2-proxy # Phase 6: forward-auth — needs SOPS `oauth2-proxy-credentials`
+9 -5
View File
@@ -305,9 +305,12 @@ spec:
- name: platform-health.identity
rules:
- alert: KeycloakDown
# Scope to the Keycloak SERVER StatefulSet pods (keycloak-0, keycloak-1). The keycloak
# namespace also holds the operator (keycloak-operator-*) and DB (keycloak-db-*) pods, so an
# un-scoped max() would be masked by those being ready — this must reflect the IdP itself.
expr: |
max(kube_pod_status_ready{namespace="keycloak", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak"})
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-[0-9]+"})
for: 5m
labels:
severity: critical
@@ -316,10 +319,11 @@ spec:
description: "No ready Keycloak pod for 5m — SSO is unavailable and dependent services cannot authenticate (docs/15)."
runbook_url: "https://home.keithjackson.dev/runbooks/keycloak-down"
- alert: KeycloakDatabaseDown
# Keycloak's PostgreSQL (Longhorn-backed, DR-critical). Metric from postgres-exporter or
# the bundled chart; confirm label set at bring-up.
# Keycloak's PostgreSQL runs as a CloudNativePG Cluster `keycloak-db` (pods keycloak-db-N).
# DR-critical, Longhorn-backed. (CNPG also exports cnpg_* metrics via its PodMonitor.)
expr: |
max(kube_pod_status_ready{namespace="keycloak", pod=~".*postgres.*", condition="true"}) == 0
max(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*", condition="true"}) == 0
or absent(kube_pod_status_ready{namespace="keycloak", pod=~"keycloak-db-.*"})
for: 5m
labels:
severity: critical
@@ -0,0 +1,25 @@
---
# Ingress for Grafana at grafana.home.keithjackson.dev (docs/09) — the endpoint of the native-OIDC SSO
# proof (log in once at Keycloak -> reach Grafana). CR of the Gateway API CRDs — carried by
# platform-config. Same-namespace backend; parentRef allowed via the monitoring namespace's
# gateway-access=shared label.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: grafana
namespace: monitoring
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- grafana.home.keithjackson.dev
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: kube-prometheus-stack-grafana
port: 80
@@ -0,0 +1,26 @@
# =============================================================================
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
#
# WHAT THIS IS: the OIDC client SECRET Grafana uses to authenticate to the Keycloak `homelab` realm's
# `grafana` client (kube-prometheus-stack.yaml -> grafana.envValueFrom.GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET).
# The value MUST equal the `grafana-client-secret` placed in platform/keycloak/keycloak-realm-clients
# (which the realm import injects into the Keycloak client). Consumed by name only (R-IDP-04).
#
# HAND-OFF STEPS (operator):
# (a) Use the SAME grafana-client-secret generated for keycloak-realm-clients (openssl rand -hex 32).
# (b) Copy to: platform/monitoring/grafana-oidc-credentials.sops.yaml
# (c) Fill `client-secret` below; encrypt:
# sops -e -i platform/monitoring/grafana-oidc-credentials.sops.yaml
# (d) Add under `resources:` in platform/monitoring/kustomization.yaml:
# - grafana-oidc-credentials.sops.yaml
# Grafana runs with local admin login until this lands (secretKeyRef optional:true), then OIDC works.
# =============================================================================
apiVersion: v1
kind: Secret
metadata:
name: grafana-oidc-credentials
namespace: monitoring
type: Opaque
stringData:
client-secret: REPLACE-ME
+30 -2
View File
@@ -150,8 +150,36 @@ spec:
memory: 128Mi
limits:
memory: 256Mi
# TODO(WS-SEC/WS-C2, later): Keycloak OIDC client (auth.generic_oauth) + root_url behind the
# Gateway; disable local login once SSO is proven. Secret referenced by name only.
# --- Keycloak native-OIDC SSO (ADR-0014, deliverable #4 SSO proof) ---
# Grafana is a confidential OIDC client of the Keycloak `homelab` realm. The client SECRET comes
# from the SOPS-encrypted grafana-oidc-credentials Secret via env (referenced by name only,
# R-IDP-04). optional:true so the RUNNING Grafana keeps starting before the operator fills the
# secret — local admin login (below) stays available as the fallback until SSO is proven.
envValueFrom:
GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET:
secretKeyRef:
name: grafana-oidc-credentials
key: client-secret
optional: true
grafana.ini:
server:
root_url: https://grafana.home.keithjackson.dev
auth.generic_oauth:
enabled: true
name: Keycloak
client_id: grafana
scopes: "openid email profile"
auth_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/auth
token_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/token
api_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/userinfo
login_attribute_path: preferred_username
email_attribute_path: email
# Map Keycloak groups (the `groups` claim) to Grafana roles (R-IDP-06): platform-admins ->
# Admin, media-users -> Editor, everyone else -> Viewer.
role_attribute_path: "contains(groups[*], 'platform-admins') && 'Admin' || contains(groups[*], 'media-users') && 'Editor' || 'Viewer'"
# Keep local admin login enabled as break-glass until SSO is proven end-to-end; flip to
# disable_login_form:true afterwards (docs/15 break-glass, R-IDP-05).
# TODO(WS-SEC, post-proof): set auth.disable_login_form=true once SSO login is verified.
# --- Bundled exporters / operator: keep small ---
prometheusOperator:
+4
View File
@@ -3,3 +3,7 @@ apiVersion: v1
kind: Namespace
metadata:
name: monitoring
labels:
# Expose Grafana at grafana.home.keithjackson.dev via the shared Gateway (docs/09) for the
# Keycloak native-OIDC SSO proof (grafana-httproute.yaml, WS-SEC Phase 6).
gateway-access: "shared"
+29 -9
View File
@@ -1,7 +1,8 @@
---
# oauth2-proxy — forward-auth at the Gateway for apps WITHOUT usable OIDC (PiHole, *arr UIs,
# qBittorrent, Homepage, Prometheus, Alertmanager, Hubble UI). Authenticates against Keycloak.
# ADR-0014. Phase 0 STUB. Replica count is an overlay knob (R-ENV-02).
# oauth2-proxy — forward-auth for apps WITHOUT usable OIDC (PiHole, *arr UIs, qBittorrent, Homepage,
# Prometheus, Alertmanager, Hubble UI — ADR-0014). Authenticates users against the Keycloak `homelab`
# realm, then the Gateway ext-auth lets the request through only for an authenticated session
# (R-IDP-02: no management UI unauthenticated). Chart image is quay.io/oauth2-proxy (not Bitnami).
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
@@ -26,13 +27,32 @@ spec:
values:
replicaCount: ${OAUTH2_PROXY_REPLICAS} # overlay knob: dev 1 / prod 2 (HA) — R-ENV-02
config:
# OIDC provider = Keycloak. Client id/secret + cookie secret: SOPS-encrypted secret, name only.
# client-id / client-secret / cookie-secret from the SOPS secret (name only). client-id +
# client-secret MUST match the realm's `oauth2-proxy` client (keycloak-realm-clients secret).
existingSecret: oauth2-proxy-credentials
extraArgs:
provider: oidc
# TODO(WS-SEC, Phase 6): oidc-issuer-url = https://keycloak.home.keithjackson.dev/realms/<realm>
oidc-issuer-url: "REPLACE-ME"
oidc-issuer-url: https://keycloak.home.keithjackson.dev/realms/homelab
# Forward-auth (ext-auth) mode: no upstream app; return 202 for an authenticated session so the
# Gateway ext-authz can allow/deny. Auth headers are surfaced to the backend.
reverse-proxy: "true"
upstream: static://202
set-xauthrequest: "true"
pass-access-token: "true"
email-domain: "*"
# Forward-auth integration wired at the Gateway (HTTPRoute + ext-auth). Contract with WS-NET.
# TODO(WS-SEC/WS-NET, Phase 6): Gateway forward-auth/ext-authz wiring; group-based authorization;
# MFA enforced upstream in Keycloak. Management UIs must never be unauthenticated (R-IDP-02).
# Single sign-in/callback host for all forward-auth-protected services on the platform zone.
redirect-url: https://auth.home.keithjackson.dev/oauth2/callback
whitelist-domain: ".home.keithjackson.dev"
cookie-domain: ".home.keithjackson.dev"
cookie-secure: "true"
# Expose group membership so per-service group authorization (R-IDP-06) can be enforced. The
# baseline here is "authenticated"; per-service `--allowed-groups` is layered on at the Gateway
# ext-auth attachment (WS-NET contract) or via a dedicated oauth2-proxy instance per group.
oidc-groups-claim: groups
scope: "openid email profile"
# Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
metrics:
enabled: true
# TODO(WS-SEC/WS-NET, Phase 6): the Gateway ext-authz ATTACHMENT (which HTTPRoutes require this
# proxy, and each route's allowed groups) is a WS-NET Gateway contract — Cilium Gateway API
# ext-auth wiring. oauth2-proxy is the authenticator; the per-route enforcement is Gateway-side.
+25
View File
@@ -0,0 +1,25 @@
---
# Ingress for oauth2-proxy's sign-in/callback endpoint at auth.home.keithjackson.dev (docs/09). This
# is where forward-auth-protected UIs redirect unauthenticated users to log in against Keycloak. CR of
# the Gateway API CRDs — carried by platform-config. Same-namespace backend (no ReferenceGrant); the
# parentRef is allowed via the oauth2-proxy namespace's gateway-access=shared label.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: oauth2-proxy
namespace: oauth2-proxy
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- auth.home.keithjackson.dev
rules:
- matches:
- path:
type: PathPrefix
value: /oauth2
backendRefs:
- name: oauth2-proxy
port: 80
+4
View File
@@ -3,3 +3,7 @@ apiVersion: v1
kind: Namespace
metadata:
name: oauth2-proxy
labels:
# Opt in to the shared Gateway so oauth2-proxy's sign-in/callback endpoint is reachable at
# auth.home.keithjackson.dev (docs/09). Platform component — NOT app-namespace (see keycloak ns note).
gateway-access: "shared"
+23
View File
@@ -0,0 +1,23 @@
---
# ServiceMonitor for oauth2-proxy (R-OBS-01, docs/10). CR of monitoring.coreos.com — CARRIED BY
# platform-config (F-MON1). The chart's metrics endpoint is enabled in the helmrelease (metrics.enabled).
# TODO(WS-SEC, bring-up): confirm the metrics port NAME the pinned chart exposes on the Service
# (commonly `metrics`, :44180) and adjust `port` below if it differs.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: oauth2-proxy
namespace: oauth2-proxy
labels:
release: kube-prometheus-stack
spec:
namespaceSelector:
matchNames:
- oauth2-proxy
selector:
matchLabels:
app: oauth2-proxy
endpoints:
- port: metrics
path: /metrics
interval: 30s