The serving route's comment claimed that because no Gateway ext-authz attachment exists, every forward-auth route is 'in practice reachable unauthenticated today'. That is wrong, and wrong in a dangerous direction for a document an operator reads during an incident. Enforcement here is PER-POD, not gateway-level: each of the five services declaring auth: forward-auth (pihole, jackett, qbittorrent, radarr, sonarr) publishes both its app port and 4180 on its Service, and its HTTPRoute targets :4180 — the oauth2-proxy sidecar. Verified on prod. The app port remains reachable in-cluster on purpose, which is how sonarr/radarr drive qBittorrent's API, and Cilium default-deny governs that path. The public-by-design decision for this route is unchanged and still correct; what changes is that it is genuinely the exception rather than one open route among many. Headlamp's exclusion is likewise deliberate (native OIDC with per-user id_token forwarding and no ServiceAccount cluster rights). Co-Authored-By: Claude Opus 5 <[email protected]>
88 lines
5.6 KiB
YAML
88 lines
5.6 KiB
YAML
---
|
|
# North-south ingress for the runbooks at https://<apex>/runbooks/ — the exact prefix the alert
|
|
# annotations already point at (`runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"`, 42 of them
|
|
# across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml,
|
|
# platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml). Nothing about those annotations
|
|
# changes; this route is what finally makes them resolve.
|
|
#
|
|
# NO `hostnames:` — DELIBERATE, and the reason this manifest needs no Flux variable substitution.
|
|
# An HTTPRoute with no hostnames inherits the hostname of every listener it attaches to, and this
|
|
# route attaches to exactly one: `https-apex`, whose hostname is `${BASE_DOMAIN}`
|
|
# (platform/gateway/gateway.yaml). So the route is correct on dev and on prod with no
|
|
# `postBuild.substituteFrom` anywhere in this tier — which matters a great deal here, because the
|
|
# SAME Flux Kustomization renders the runbook markdown into a ConfigMap. With substitution enabled,
|
|
# Flux's envsubst would run over the runbook prose and silently EMPTY every `${...}` in every shell
|
|
# example a runbook contains (it fails open — DECISIONS-LOG 2026-09-01, the Cilium k8sServiceHost
|
|
# incident). Declaring no hostname removes the last reason to substitute, so the docs ship verbatim.
|
|
# See clusters/base/runbooks.yaml.
|
|
#
|
|
# PATH PRECEDENCE AGAINST HOMEPAGE. platform/homepage/httproute.yaml already binds `https-apex` with
|
|
# PathPrefix `/`. Gateway API resolves a conflict between rules on the same hostname by preferring
|
|
# the LONGEST path match, so `/runbooks` wins over `/` for these requests and homepage keeps
|
|
# everything else. The two routes coexist by specification, not by luck — and the live 200s recorded
|
|
# in the commit prove it on this Gateway implementation (Cilium).
|
|
#
|
|
# ============================ AUTH: THIS ROUTE IS PUBLIC ==============================
|
|
# auth-mode=none, deliberately, and it is the one route in this repo where that is the SAFE choice.
|
|
#
|
|
# THE CIRCULARITY. Every other exposed UI here declares forward-auth (oauth2-proxy -> Keycloak).
|
|
# Put these pages behind that and `runbooks/keycloak-down` becomes unreadable precisely when
|
|
# Keycloak is down; `runbooks/keycloak-db` unreadable when its CNPG database is down;
|
|
# `runbooks/keycloak-client-secrets` unreadable during the exact failure mode that broke all 12
|
|
# OIDC clients on 2026-09-06 while every health signal stayed green. An on-call who cannot log in
|
|
# is the primary reader of this route. Gating documentation on the identity provider it documents
|
|
# is a loop with no exit.
|
|
#
|
|
# WHAT IS ACTUALLY EXPOSED. Operational procedure: metric names, thresholds, kubectl invocations,
|
|
# component topology. No credentials, no secrets, no tokens, no internal IPs beyond the LAN ranges
|
|
# already written in the public spec repo. The content is the same text any reader of the git repo
|
|
# can see. The real cost is reconnaissance value — it names what runs here and how it fails — and
|
|
# that is accepted knowingly, against the certainty of a locked-out operator staring at a login page
|
|
# during an outage. If that trade is ever revisited, the correct fix is NOT forward-auth on this
|
|
# route; it is to keep it public and move sensitive specifics out of the runbooks.
|
|
#
|
|
# HOW forward-auth IS ENFORCED HERE — and why this route is genuinely different. Enforcement is NOT
|
|
# a Gateway ext-authz attachment; it is PER-POD. Each forward-auth service runs an oauth2-proxy
|
|
# sidecar, its Service publishes BOTH the app port and 4180, and the service's HTTPRoute targets
|
|
# :4180 — verified on prod for all five that declare `auth: forward-auth` in the catalog (pihole,
|
|
# jackett, qbittorrent, radarr, sonarr). Ingress traffic therefore always traverses oauth2-proxy,
|
|
# which performs the Keycloak exchange and the group check. The app port stays on the Service so
|
|
# in-cluster clients can talk service-to-service without SSO (that is how sonarr/radarr reach
|
|
# qBittorrent's API), which is deliberate, not a leak: Cilium default-deny decides who may use it.
|
|
#
|
|
# So this route is NOT "no more open than its neighbours" — its neighbours are closed and this one is
|
|
# open, on purpose, for the lockout reason argued above. It has no oauth2-proxy sidecar and targets
|
|
# its app port directly. If a Gateway-level ext-authz attachment is ever introduced, this route must
|
|
# be left OUT of it ON PURPOSE, and this comment is the record of that decision so nobody "fixes" it
|
|
# later and re-closes the loop. (Headlamp is also excluded from forward-auth, for a different and
|
|
# equally deliberate reason: it runs native Keycloak OIDC and forwards the logged-in user's id_token
|
|
# to the apiserver for per-user RBAC, with clusterRoleBinding.create=false so its ServiceAccount can
|
|
# do nothing on its own.)
|
|
apiVersion: gateway.networking.k8s.io/v1
|
|
kind: HTTPRoute
|
|
metadata:
|
|
name: runbooks
|
|
namespace: runbooks
|
|
labels:
|
|
app.kubernetes.io/name: runbooks
|
|
annotations:
|
|
homelab.keithjackson.dev/auth-mode: none
|
|
homelab.keithjackson.dev/auth-rationale: >-
|
|
Public on purpose. An operator locked out of Keycloak is this route's primary reader;
|
|
gating it behind Keycloak would make runbooks/keycloak-down unreadable exactly when it is
|
|
needed. Content is operational procedure only — no secrets. Do NOT add this route to the
|
|
Gateway ext-authz attachment when forward-auth enforcement lands.
|
|
spec:
|
|
parentRefs:
|
|
- name: shared
|
|
namespace: gateway
|
|
sectionName: https-apex # the apex listener; the wildcard `https` listener won't match it
|
|
rules:
|
|
- matches:
|
|
- path:
|
|
type: PathPrefix
|
|
value: /runbooks
|
|
backendRefs:
|
|
- name: runbooks
|
|
port: 80
|