Add Keycloak IdP / SSO (ADR-0014, docs/15, R-IDP-*)

New identity subsystem, and resolves the C2 auth open question: - docs/15-identity-and-access.md + ADR-0014: Keycloak as central IdP; native OIDC where supported (Gitea, Nextcloud, Grafana, Kasm, Semaphore, Headlamp) and oauth2-proxy forward-auth for bare UIs (PiHole, *arr, qBittorrent, Homepage, Prometheus, Alertmanager, Hubble). Group-based authz + MFA; declarative realm/clients (GitOps); PostgreSQL on Longhorn, DR-critical + break-glass; hardened public exposure. Plex/Seerr use Plex auth; Minecraft N/A. - Requirements R-IDP-01..06; ServiceDefinition gains an block. - Onboarded early (with Gitea) so services adopt SSO as they come up. - Threaded through: docs 00,01,07,10,12,13; roadmap phases+workstreams; task board (T-6.1c/T-6.1d); glossary; ADR index; README. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-15 23:33:49 -05:00
1 parent 1e0e67f902
commit fb153f0bb5
@@ -138,6 +138,23 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
 - **R-SEC-03** — All the above properties MUST be demonstrable/auditable (an operator can prove
  isolation, egress path, cert validity, and backup integrity).

+### Identity & access (SSO) — `R-IDP-*`
+- **R-IDP-01** — The platform MUST provide a central Identity Provider (SSO) as the single user
+  directory and login for platform services.
+- **R-IDP-02** — Services MUST authenticate against the IdP — natively via OIDC where supported, or
+  behind an identity-aware (forward-auth) proxy otherwise. No management UI may be exposed
+  unauthenticated on the network. (Documented exceptions: Plex/Seerr use Plex auth; Minecraft uses
+  Mojang/Microsoft.)
+- **R-IDP-03** — Identity configuration (realms, clients, groups, roles) MUST be declarative and
+  GitOps-managed, not console-only.
+- **R-IDP-04** — IdP secrets (admin credential, OIDC client secrets) MUST be SOPS-encrypted; MFA MUST
+  be available to all users and required for privileged access.
+- **R-IDP-05** — The IdP MUST be treated as DR-critical (replicated + off-cluster-backed datastore)
+  with a documented **break-glass** path so an IdP outage does not lock the operator out of recovery
+  tools.
+- **R-IDP-06** — Access MUST be least-privilege via groups/roles; sensitive surfaces (C2, cluster
+  console, alerting) require an authenticated user in an authorized group, not merely "logged in."
+
 ### Environments — `R-ENV-*`
 - **R-ENV-01** — dev/test MUST be provisioned as local KVMs via IaC: created, baselined, cluster-
  joined, and managed as code.
@@ -22,7 +22,8 @@ choice here without a superseding ADR.**
 | Observability | **kube-prometheus-stack** + **Loki** + **Grafana Alloy** + Alertmanager→SMTP | R-OBS-01..04 | — | [10](10-observability-and-alerting.md) |
 | Backup/DR | **Velero** + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
 | Node/infra provisioning | **OpenTofu** (libvirt) + **cloud-init** + **Ansible** (k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
-| Command center UI | **Homepage** + **Headlamp** + **Grafana** panels | R-C2-01/02 | — | [12](12-command-center-ui.md) |
+| Identity / SSO | **Keycloak** (OIDC + SAML) + **oauth2-proxy** forward-auth | R-IDP-01..06 | [ADR-0014](../decisions/adr-0014-idp-keycloak.md) | [15](15-identity-and-access.md) |
+| Command center UI | **Homepage** + **Headlamp** + **Grafana** panels (behind Keycloak SSO) | R-C2-01/02 | — | [12](12-command-center-ui.md) |
 | Conformance testing | **Kyverno** (L0 policy) + **Chainsaw** (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | [ADR-0010](../decisions/adr-0010-conformance-testing.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md) | [14](14-testing-and-conformance.md) |

 ## Layer cake
@@ -38,6 +39,7 @@ choice here without a superseding ADR.**
 ├──────────────────────────────────────────────────────────────────────────┤
 │  Platform services                                                         │
 │   Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API           │
+│   Keycloak (SSO/IdP) + oauth2-proxy (forward-auth)                         │
 │   kube-prometheus-stack + Loki + Alloy (observability)                     │
 │   Velero (backup) · Renovate (chart updates) · SOPS (secrets)              │
 ├──────────────────────────────────────────────────────────────────────────┤
@@ -16,6 +16,11 @@ Defense in depth with declarative, auditable controls:
 - **TLS**: trusted, auto-managed certificates end to end ([08](08-tls-and-certificates.md), R-TLS-*).
 - **Supply chain**: pin images by digest where automation allows; prefer known-good upstreams;
  updates flow through policy, not ad-hoc pulls ([11](11-automated-updates.md)).
+- **Identity / SSO**: a central IdP (**Keycloak**) provides single sign-on, one user directory,
+  group-based authorization, and MFA; services authenticate via native OIDC or an oauth2-proxy
+  forward-auth layer so no management UI is unauthenticated ([15](15-identity-and-access.md),
+  [ADR-0014](../decisions/adr-0014-idp-keycloak.md), R-IDP-*). This is the primary *authentication*
+  control, complementing network default-deny (the primary *isolation* control).
 - **Policy enforcement (admission)**: **Kyverno** runs in-cluster as an admission controller and
  rejects non-conforming resources at reconcile time — e.g. a namespace missing its default-deny
  baseline, a plaintext Secret, a workload without a required egress-class label. The *same* policies
@@ -37,6 +42,7 @@ Defense in depth with declarative, auditable controls:

 ### Secret categories to manage
 - VPN credentials (PIA username/password; WireGuard) for the egress model (ADR-0013).
+- Keycloak admin credential and per-service **OIDC client secrets** ([15](15-identity-and-access.md)).
 - DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS.
 - SMTP credentials for Alertmanager email.
 - Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, Kasm, DB passwords).
@@ -27,6 +27,8 @@ Platform health (R-OBS-04) — first-class alerts:
  NetworkPolicy reconcile errors.
 - **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures.
 - **KubeVirt**: VM crash-loops / failed launches.
+- **Identity (Keycloak)**: IdP down/unhealthy or its DB unavailable (SSO is a critical dependency —
+  [15](15-identity-and-access.md)); certificate/login-endpoint failures.

 ## Alert delivery (R-OBS-02)

@@ -49,8 +49,10 @@ kubeconfig in a browser:

 - The C2 root and its sub-UIs are `internal` exposure by default (LAN-only) unless the operator
  explicitly makes a component `public`.
- Authentication in front of the C2 (and sensitive sub-UIs like Headlamp) — SSO/reverse-proxy auth —
-  is required; the mechanism (e.g. an identity-aware proxy) is pinned during the C2 workstream.
+- Authentication is via **Keycloak SSO** ([15](15-identity-and-access.md), [ADR-0014](../decisions/adr-0014-idp-keycloak.md)):
+  Headlamp/Grafana use **native OIDC**; UIs without OIDC (Homepage, Prometheus, Alertmanager, Hubble)
+  sit behind the **oauth2-proxy forward-auth** at the Gateway. Access to the C2 and sensitive
+  surfaces requires an authenticated user **in an authorized group** with **MFA** (R-IDP-06).
 - Trusted TLS on the root and all sub-UIs ([08](08-tls-and-certificates.md)).

 ## Acceptance gates
@@ -59,6 +61,9 @@ kubeconfig in a browser:
 - Platform health (nodes, storage, Flux, certs, backups, VPN) is visible without leaving the C2.
 - Safe operational actions work and are access-controlled + audited; destructive actions confirm.

+## Resolved
+- Auth mechanism — **Keycloak SSO** (native OIDC + oauth2-proxy forward-auth), [15](15-identity-and-access.md) /
+  [ADR-0014](../decisions/adr-0014-idp-keycloak.md).
+
 ## Open questions
- Auth mechanism for the C2 (identity-aware proxy vs per-app auth) — pin in the C2 workstream.
 - Custom portal vs. Homepage-as-root long term (ADR if we diverge from Homepage).
@@ -34,6 +34,13 @@ dependencies:                      # explicit allow rules; everything else denie
 ingressRoutes:                     # hostnames/paths to expose (rendered to HTTPRoute + TLS + DNS)
  - host: <name>.home.keithjackson.dev
    port: <servicePort>
+auth:                              # SSO integration (R-IDP-02; see 15-identity-and-access.md)
+  mode: oidc | forward-auth | native | none
+  # oidc         → app configured as a Keycloak OIDC client (does its own login)
+  # forward-auth → protected by oauth2-proxy → Keycloak at the Gateway (for UIs w/o OIDC)
+  # native       → app manages its own auth (e.g. Plex/Seerr via Plex) — documented exception
+  # none         → no auth (internal-only, discouraged)
+  groups: [ <keycloak-group>, ... ]  # groups allowed access (R-IDP-06)
 update:
  channel: stable                  # never pre-release unless opted in (R-UPD-01)
  range: <semver range | tag regex># acceptable versions
@@ -53,10 +60,12 @@ backup:
 ## Default posture for every service (unless overridden)
 - `kind: container`, `exposure: internal`, `egress: internet`.
 - **Default-deny**: no `dependencies` ⇒ can reach only DNS ([03](03-networking-isolation-and-egress.md)).
+- **Auth default `forward-auth`**: any exposed UI is protected by Keycloak SSO unless it declares
+  native OIDC (`oidc`) or a documented exception (`native`) ([15](15-identity-and-access.md), R-IDP-02).
 - Longhorn-backed config PVC; off-cluster backup; trusted TLS; auto-DNS; monitored; auto-updated on
  `stable`.

-## The initial catalog (13 services)
+## The initial catalog (14 services)

 Egress and dependencies below are **normative intent**; implementers render the exact policies.
 "Bulk" = large media tier ([04](04-storage-and-state.md)). All get trusted TLS + auto-DNS + monitoring
@@ -64,7 +73,8 @@ Egress and dependencies below are **normative intent**; implementers render the

 | # | Service | kind | exposure | egress | Storage | Key dependencies (explicit allows) | Notes |
 |---|---|---|---|---|---|---|---|
-| 1 | **plex** | container | public | **internet** | config: longhorn (small, replicated); media: **bulk** | — (clients reach it; Seerr → Plex) | Largest service; media on bulk tier, metadata DB on Longhorn ([04](04-storage-and-state.md)) |
+| 0 | **keycloak** | container | public | internet | db (PostgreSQL): longhorn (irreplaceable → backed up) | → its DB; ← all SSO clients + oauth2-proxy | **IdP/SSO** ([15](15-identity-and-access.md)); **foundational**, onboarded **early**; DR-critical + break-glass path; hardened public exposure |
+| 1 | **plex** | container | public | **internet** | config: longhorn (small, replicated); media: **bulk** | — (clients reach it; Seerr → Plex) | Largest service; media on bulk tier, metadata DB on Longhorn ([04](04-storage-and-state.md)); auth: **native** (Plex) |
 | 2 | **sonarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch (R-NET-06) |
 | 3 | **radarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch |
 | 4 | **jackett** | container | internal | **vpn** | config: longhorn | ← sonarr, ← radarr | VPN + kill-switch (indexer) |
@@ -90,6 +100,15 @@ radarr → qbittorrent, jackett
 (everything → DNS only, by default)
 ```

+**SSO integration (`auth.mode`, R-IDP-02; see [15](15-identity-and-access.md)):**
+- **oidc** (native Keycloak client): gitea, nextcloud, semaphore, kasm, grafana*, headlamp*
+  (*platform UIs, not in the numbered app list).
+- **forward-auth** (oauth2-proxy → Keycloak): pihole, sonarr, radarr, jackett, qbittorrent, unifi,
+  and platform UIs (Homepage, Prometheus, Alertmanager, Hubble).
+- **native** (own auth, documented exceptions): plex, seerr (Plex accounts); minecraft
+  (Mojang/Microsoft).
+- **keycloak** itself is the IdP — its own admin login is local + MFA (the root of trust).
+
 **"…but not limited to":** the catalog is open-ended. New services follow the same ServiceDefinition
 and inherit all guarantees automatically — that is the point of the abstraction.

@@ -0,0 +1,104 @@
+# 15 — Identity & Access (SSO)
+
+Serves: **R-IDP-01..06**, supports **R-C2-02**, **R-SEC-02/03**. Related:
+[ADR-0014 (Keycloak)](../decisions/adr-0014-idp-keycloak.md).
+
+## Purpose
+
+One place to log in. A central **Identity Provider (IdP)** gives the platform single sign-on (SSO),
+one user directory, group-based authorization, and MFA — so every service is protected by the same
+identity, and no management UI sits unauthenticated on the network.
+
+## Choice: Keycloak
+
+**Keycloak** is the platform IdP ([ADR-0014](../decisions/adr-0014-idp-keycloak.md)): a mature,
+self-hostable OpenID Connect (OIDC) + SAML provider with users, groups, roles, clients, MFA
+(TOTP/WebAuthn), and brute-force protection.
+
+## Two integration patterns (behind one identity)
+
+Every service authenticates against Keycloak, via whichever pattern it supports:
+
+1. **Native OIDC** — the app is configured as a Keycloak OIDC **client** and does its own login
+   redirect. Preferred where available. Initial set: **Gitea, Nextcloud, Grafana, Kasm, Semaphore,
+   Headlamp**.
+2. **Forward-auth proxy** — for apps with **no** usable OIDC, an identity-aware proxy
+   (**oauth2-proxy**) sits in front at the Gateway, authenticates the user against Keycloak, and only
+   then forwards the request. Used for: **PiHole admin, the *arr web UIs (Sonarr/Radarr/Jackett),
+   qBittorrent UI, Homepage, Prometheus, Alertmanager, Hubble UI**, and any other bare UI.
+
+> This resolves the C2 auth open question ([12](12-command-center-ui.md)): it is **both** — native
+> OIDC where supported, forward-auth everywhere else, all backed by Keycloak.
+
+### Exceptions (documented, not oversights)
+- **Plex** and **Seerr** use **Plex's own account system** (Plex OAuth) — they do not federate to
+  Keycloak.
+- **Minecraft** authenticates players via Mojang/Microsoft — N/A.
+- **UniFi** keeps its native account model (limited SSO support); protect its UI via forward-auth if
+  desired.
+
+## Authorization model (R-IDP-06)
+
+- **Groups drive access.** Keycloak groups (e.g. `platform-admins`, `media-users`, `viewers`) map to
+  per-service access and roles. A service's ServiceDefinition declares which groups may reach it
+  ([13](13-service-catalog.md), `auth.groups`).
+- **Least privilege**: the C2 console, Headlamp, Alertmanager, and other sensitive UIs require an
+  authenticated user **in an authorized group** — not merely "logged in."
+- **MFA**: TOTP/WebAuthn available for all users and **required for privileged groups**
+  (`platform-admins`).
+
+## Identity config as code (R-IDP-03)
+
+Realms, clients, groups, and roles are **declarative and GitOps-managed** — not clicked into the
+admin console and forgotten:
+- The Keycloak instance is deployed via the **Keycloak Operator** (or Helm), and **realm/client/group
+  configuration is reconciled declaratively** (e.g. `keycloak-config-cli` or realm-import CRs). Exact
+  tool is an implementation detail; the requirement is that identity config lives in git and
+  reconciles like everything else.
+- **OIDC client secrets and the Keycloak admin credential are SOPS-encrypted** ([07](07-security-and-secrets.md)).
+
+## State, exposure & resilience
+
+- **Datastore**: Keycloak runs on **PostgreSQL**; the DB lives on **3-replica Longhorn** and is
+  **DR-critical → backed up off-cluster** ([04](04-storage-and-state.md)). Keycloak is stateless
+  itself and rebuilds from the DB + git-managed realm config.
+- **Exposure**: `keycloak.home.keithjackson.dev` is **`public`** so it can serve login/redirect flows
+  for the internet-exposed services (Nextcloud, Gitea) — but it is **security-sensitive** and
+  hardened: brute-force detection on, strong admin auth + MFA, trusted TLS ([08](08-tls-and-certificates.md)),
+  minimal exposed endpoints. If the operator prefers internal-only, public services fall back to
+  their native auth.
+- **HA**: a single replica with DB on Longhorn + backups is acceptable for the home scale; note
+  clustered Keycloak as a future hardening. **Availability caveat:** SSO becomes a dependency for
+  logging into federated services — document a break-glass path (e.g. a local admin on Gitea) so an
+  IdP outage doesn't lock the operator out of recovery tools.
+
+## Interaction with default-deny & the gateway
+
+- Keycloak and oauth2-proxy are ordinary workloads under **default-deny** ([03](03-networking-isolation-and-egress.md));
+  the specific flows they need (app → Keycloak OIDC, proxy → Keycloak, Keycloak → its DB) are
+  explicit allow rules.
+- Forward-auth is enforced at the **Gateway** ([09](09-dns-and-ingress.md)) so protection happens
+  before traffic reaches the app.
+
+## Bootstrapping & order
+
+- Keycloak is **foundational** and onboarded **early** (with PiHole/Gitea, [13](13-service-catalog.md)) so
+  later services can integrate SSO as they come up.
+- No hard circular dependency at bring-up: Gitea can start with a local admin and add OIDC once
+  Keycloak is live; the C2 and sensitive UIs gain SSO as soon as the proxy + realm exist.
+
+## Acceptance gates
+- A user logs in once at Keycloak and reaches all OIDC-integrated services without re-entering
+  credentials (SSO proven).
+- A forward-auth-protected UI (e.g. PiHole admin) is **unreachable without a valid Keycloak session**
+  and reachable with one, restricted to the allowed group.
+- Realm/client/group config applied purely from git reproduces a working SSO setup (no console
+  clicking); destroying and rebuilding Keycloak from DB + git restores identity.
+- Privileged access (C2/Headlamp) requires MFA and group membership; a non-member is denied.
+- Client secrets and admin credentials exist only SOPS-encrypted.
+
+## Open questions
+- Final declarative-config tool (`keycloak-config-cli` vs operator realm-import CRs) — implementation
+  detail for WS-SEC.
+- Confirm `public` exposure for Keycloak (recommended) vs internal-only with native-auth fallback for
+  public services.