From bce8778096c04b831b2ed879cf8fbe1733791128 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 00:10:24 -0500 Subject: [PATCH] =?UTF-8?q?docs(runbooks):=20forward-auth=20IS=20enforced?= =?UTF-8?q?=20=E2=80=94=20correct=20the=20posture=20note?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The serving route's comment claimed that because no Gateway ext-authz attachment exists, every forward-auth route is 'in practice reachable unauthenticated today'. That is wrong, and wrong in a dangerous direction for a document an operator reads during an incident. Enforcement here is PER-POD, not gateway-level: each of the five services declaring auth: forward-auth (pihole, jackett, qbittorrent, radarr, sonarr) publishes both its app port and 4180 on its Service, and its HTTPRoute targets :4180 — the oauth2-proxy sidecar. Verified on prod. The app port remains reachable in-cluster on purpose, which is how sonarr/radarr drive qBittorrent's API, and Cilium default-deny governs that path. The public-by-design decision for this route is unchanged and still correct; what changes is that it is genuinely the exception rather than one open route among many. Headlamp's exclusion is likewise deliberate (native OIDC with per-user id_token forwarding and no ServiceAccount cluster rights). Co-Authored-By: Claude Opus 5 --- runbooks/README.md | 9 ++++++--- runbooks/serving/httproute.yaml | 24 +++++++++++++++++------- 2 files changed, 23 insertions(+), 10 deletions(-) diff --git a/runbooks/README.md b/runbooks/README.md index b580775..11fe78c 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -127,13 +127,16 @@ the 404 it replaced — it looks fixed. ## Why this route has no login -Deliberately public. Every other exposed UI here declares forward-auth via oauth2-proxy → Keycloak; if +Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr) +really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on +:4180 rather than the app port. If these pages did too, `keycloak-down` would be unreadable exactly when Keycloak is down, `keycloak-db` unreadable when its database is down, and `keycloak-client-secrets` unreadable during the precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An operator who cannot log in is this route's primary reader. The content is operational procedure — metric names, thresholds, kubectl invocations, topology — and no secrets. See -`runbooks/serving/httproute.yaml` for the full reasoning, including the instruction to leave this route -**out** of the Gateway ext-authz attachment when forward-auth enforcement lands. +`runbooks/serving/httproute.yaml` for the full reasoning, including how forward-auth is actually +enforced here and the instruction to leave this route **out** of any Gateway-level ext-authz attachment +if one is ever introduced. The running narrative of real incidents on this cluster is `DECISIONS-LOG.md`. diff --git a/runbooks/serving/httproute.yaml b/runbooks/serving/httproute.yaml index a016563..35c6869 100644 --- a/runbooks/serving/httproute.yaml +++ b/runbooks/serving/httproute.yaml @@ -41,13 +41,23 @@ # during an outage. If that trade is ever revisited, the correct fix is NOT forward-auth on this # route; it is to keep it public and move sensitive specifics out of the runbooks. # -# HONEST NOTE ON TODAY'S POSTURE. The Gateway ext-authz attachment that would ENFORCE forward-auth -# does not exist anywhere in this repo yet (platform/oauth2-proxy is the authenticator only; the one -# CiliumEnvoyConfig on the cluster is the Gateway's own). So every `forward-auth` route — homepage, -# pihole, jackett, qbittorrent, radarr, sonarr — is in practice reachable unauthenticated today. -# This route is therefore no more open than its neighbours right now; the difference is that when -# ext-authz does land, this one must be left OUT of it ON PURPOSE, and this comment is the record of -# that decision so nobody "fixes" it later and re-closes the loop. +# HOW forward-auth IS ENFORCED HERE — and why this route is genuinely different. Enforcement is NOT +# a Gateway ext-authz attachment; it is PER-POD. Each forward-auth service runs an oauth2-proxy +# sidecar, its Service publishes BOTH the app port and 4180, and the service's HTTPRoute targets +# :4180 — verified on prod for all five that declare `auth: forward-auth` in the catalog (pihole, +# jackett, qbittorrent, radarr, sonarr). Ingress traffic therefore always traverses oauth2-proxy, +# which performs the Keycloak exchange and the group check. The app port stays on the Service so +# in-cluster clients can talk service-to-service without SSO (that is how sonarr/radarr reach +# qBittorrent's API), which is deliberate, not a leak: Cilium default-deny decides who may use it. +# +# So this route is NOT "no more open than its neighbours" — its neighbours are closed and this one is +# open, on purpose, for the lockout reason argued above. It has no oauth2-proxy sidecar and targets +# its app port directly. If a Gateway-level ext-authz attachment is ever introduced, this route must +# be left OUT of it ON PURPOSE, and this comment is the record of that decision so nobody "fixes" it +# later and re-closes the loop. (Headlamp is also excluded from forward-auth, for a different and +# equally deliberate reason: it runs native Keycloak OIDC and forwards the logged-in user's id_token +# to the apiserver for per-user RBAC, with clusterRoleBinding.create=false so its ServiceAccount can +# do nothing on its own.) apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: