docs(runbooks): forward-auth IS enforced — correct the posture note

The serving route's comment claimed that because no Gateway ext-authz
attachment exists, every forward-auth route is 'in practice reachable
unauthenticated today'. That is wrong, and wrong in a dangerous direction
for a document an operator reads during an incident.

Enforcement here is PER-POD, not gateway-level: each of the five services
declaring auth: forward-auth (pihole, jackett, qbittorrent, radarr, sonarr)
publishes both its app port and 4180 on its Service, and its HTTPRoute
targets :4180 — the oauth2-proxy sidecar. Verified on prod. The app port
remains reachable in-cluster on purpose, which is how sonarr/radarr drive
qBittorrent's API, and Cilium default-deny governs that path.

The public-by-design decision for this route is unchanged and still correct;
what changes is that it is genuinely the exception rather than one open route
among many. Headlamp's exclusion is likewise deliberate (native OIDC with
per-user id_token forwarding and no ServiceAccount cluster rights).

Co-Authored-By: Claude Opus 5 <[email protected]>

runbooks/README.md

@@ -127,13 +127,16 @@ the 404 it replaced — it looks fixed.
 
 ## Why this route has no login
 
-Deliberately public. Every other exposed UI here declares forward-auth via oauth2-proxy → Keycloak; if
+Deliberately public. The UIs that declare forward-auth (pihole, jackett, qbittorrent, radarr, sonarr)
+really are closed: enforcement is per-pod, each one's HTTPRoute targeting its oauth2-proxy sidecar on
+:4180 rather than the app port. If
 these pages did too, `keycloak-down` would be unreadable exactly when Keycloak is down,
 `keycloak-db` unreadable when its database is down, and `keycloak-client-secrets` unreadable during the
 precise failure that broke all 12 OIDC clients on 2026-09-06 behind an entirely green cluster. An
 operator who cannot log in is this route's primary reader. The content is operational procedure — metric
 names, thresholds, kubectl invocations, topology — and no secrets. See
-`runbooks/serving/httproute.yaml` for the full reasoning, including the instruction to leave this route
-**out** of the Gateway ext-authz attachment when forward-auth enforcement lands.
+`runbooks/serving/httproute.yaml` for the full reasoning, including how forward-auth is actually
+enforced here and the instruction to leave this route **out** of any Gateway-level ext-authz attachment
+if one is ever introduced.
 
 The running narrative of real incidents on this cluster is `DECISIONS-LOG.md`.

runbooks/serving/httproute.yaml

@@ -41,13 +41,23 @@
 # during an outage. If that trade is ever revisited, the correct fix is NOT forward-auth on this
 # route; it is to keep it public and move sensitive specifics out of the runbooks.
 #
-# HONEST NOTE ON TODAY'S POSTURE. The Gateway ext-authz attachment that would ENFORCE forward-auth
-# does not exist anywhere in this repo yet (platform/oauth2-proxy is the authenticator only; the one
-# CiliumEnvoyConfig on the cluster is the Gateway's own). So every `forward-auth` route — homepage,
-# pihole, jackett, qbittorrent, radarr, sonarr — is in practice reachable unauthenticated today.
-# This route is therefore no more open than its neighbours right now; the difference is that when
-# ext-authz does land, this one must be left OUT of it ON PURPOSE, and this comment is the record of
-# that decision so nobody "fixes" it later and re-closes the loop.
+# HOW forward-auth IS ENFORCED HERE — and why this route is genuinely different. Enforcement is NOT
+# a Gateway ext-authz attachment; it is PER-POD. Each forward-auth service runs an oauth2-proxy
+# sidecar, its Service publishes BOTH the app port and 4180, and the service's HTTPRoute targets
+# :4180 — verified on prod for all five that declare `auth: forward-auth` in the catalog (pihole,
+# jackett, qbittorrent, radarr, sonarr). Ingress traffic therefore always traverses oauth2-proxy,
+# which performs the Keycloak exchange and the group check. The app port stays on the Service so
+# in-cluster clients can talk service-to-service without SSO (that is how sonarr/radarr reach
+# qBittorrent's API), which is deliberate, not a leak: Cilium default-deny decides who may use it.
+#
+# So this route is NOT "no more open than its neighbours" — its neighbours are closed and this one is
+# open, on purpose, for the lockout reason argued above. It has no oauth2-proxy sidecar and targets
+# its app port directly. If a Gateway-level ext-authz attachment is ever introduced, this route must
+# be left OUT of it ON PURPOSE, and this comment is the record of that decision so nobody "fixes" it
+# later and re-closes the loop. (Headlamp is also excluded from forward-auth, for a different and
+# equally deliberate reason: it runs native Keycloak OIDC and forwards the logged-in user's id_token
+# to the apiserver for per-user RBAC, with clusterRoleBinding.create=false so its ServiceAccount can
+# do nothing on its own.)
 apiVersion: gateway.networking.k8s.io/v1
 kind: HTTPRoute
 metadata: