diff --git a/runbooks/README.md b/runbooks/README.md index b580775..11fe78c 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -127,13 +127,16 @@ the 404 it replaced — it looks fixed. ## Why this route has no login -Deliberately public. Every other exposed UI here declares forward-auth via oauth2-proxy → Keycloak; if +Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr) +really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on +:4180 rather than the app port. If these pages did too, `keycloak-down` would be unreadable exactly when Keycloak is down, `keycloak-db` unreadable when its database is down, and `keycloak-client-secrets` unreadable during the precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An operator who cannot log in is this route's primary reader. The content is operational procedure — metric names, thresholds, kubectl invocations, topology — and no secrets. See -`runbooks/serving/httproute.yaml` for the full reasoning, including the instruction to leave this route -**out** of the Gateway ext-authz attachment when forward-auth enforcement lands. +`runbooks/serving/httproute.yaml` for the full reasoning, including how forward-auth is actually +enforced here and the instruction to leave this route **out** of any Gateway-level ext-authz attachment +if one is ever introduced. The running narrative of real incidents on this cluster is `DECISIONS-LOG.md`. diff --git a/runbooks/serving/httproute.yaml b/runbooks/serving/httproute.yaml index a016563..35c6869 100644 --- a/runbooks/serving/httproute.yaml +++ b/runbooks/serving/httproute.yaml @@ -41,13 +41,23 @@ # during an outage. If that trade is ever revisited, the correct fix is NOT forward-auth on this # route; it is to keep it public and move sensitive specifics out of the runbooks. # -# HONEST NOTE ON TODAY'S POSTURE. The Gateway ext-authz attachment that would ENFORCE forward-auth -# does not exist anywhere in this repo yet (platform/oauth2-proxy is the authenticator only; the one -# CiliumEnvoyConfig on the cluster is the Gateway's own). So every `forward-auth` route — homepage, -# pihole, jackett, qbittorrent, radarr, sonarr — is in practice reachable unauthenticated today. -# This route is therefore no more open than its neighbours right now; the difference is that when -# ext-authz does land, this one must be left OUT of it ON PURPOSE, and this comment is the record of -# that decision so nobody "fixes" it later and re-closes the loop. +# HOW forward-auth IS ENFORCED HERE — and why this route is genuinely different. Enforcement is NOT +# a Gateway ext-authz attachment; it is PER-POD. Each forward-auth service runs an oauth2-proxy +# sidecar, its Service publishes BOTH the app port and 4180, and the service's HTTPRoute targets +# :4180 — verified on prod for all five that declare `auth: forward-auth` in the catalog (pihole, +# jackett, qbittorrent, radarr, sonarr). Ingress traffic therefore always traverses oauth2-proxy, +# which performs the Keycloak exchange and the group check. The app port stays on the Service so +# in-cluster clients can talk service-to-service without SSO (that is how sonarr/radarr reach +# qBittorrent's API), which is deliberate, not a leak: Cilium default-deny decides who may use it. +# +# So this route is NOT "no more open than its neighbours" — its neighbours are closed and this one is +# open, on purpose, for the lockout reason argued above. It has no oauth2-proxy sidecar and targets +# its app port directly. If a Gateway-level ext-authz attachment is ever introduced, this route must +# be left OUT of it ON PURPOSE, and this comment is the record of that decision so nobody "fixes" it +# later and re-closes the loop. (Headlamp is also excluded from forward-auth, for a different and +# equally deliberate reason: it runs native Keycloak OIDC and forwards the logged-in user's id_token +# to the apiserver for per-user RBAC, with clusterRoleBinding.create=false so its ServiceAccount can +# do nothing on its own.) apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: