home-network-k3s-impl/platform/oauth2-proxy/README.md

platform/oauth2-proxy

oauth2-proxy — forward-auth at the Gateway for apps without usable OIDC, authenticating against Keycloak.

• Governing ADR: ADR-0014 — IdP: Keycloak
• Spec: docs/15 — Identity & Access
• Serves: R-IDP-01..06 (esp. R-IDP-02: management UIs never unauthenticated)
• Owner: WS-SEC (Gateway wiring shared with WS-NET)

What this stub is

Flux HelmRelease for oauth2-proxy as the forward-auth provider for non-OIDC apps (PiHole, *arr UIs, qBittorrent, Homepage, Prometheus, Alertmanager, Hubble UI — ADR-0014). Replica count is an overlay knob (${OAUTH2_PROXY_REPLICAS}, R-ENV-02). Client/cookie secrets referenced by name, SOPS-encrypted (WS-SEC).

Contract (SSO/auth + Gateway)

Pairs with ../keycloak to realize the "SSO/auth contract": a ServiceDefinition auth: forward-auth service is protected by oauth2-proxy at the Gateway (ext-auth); auth: oidc services go native to Keycloak. Gateway ext-auth wiring is a WS-NET contract.

Enforcement (ext_authz forward-auth)

The authenticator (oauth2-proxy) is deployed and wired to Keycloak: provider=oidc, oidc-issuer-url=.../realms/homelab, --reverse-proxy, --upstream=static://202, --set-xauthrequest, cookie/redirect domain .home.keithjackson.dev, and — added here — --allowed-groups=platform-admins (R-IDP-06 floor) + --skip-provider-button (302 straight to Keycloak). The Keycloak oauth2-proxy client already carries the auth.home…/oauth2/callback redirect URI and a groups claim with full.path=false (bare platform-admins) — no realm change needed.

Mechanism. Forward-auth is an Envoy ext_authz HTTP filter that calls oauth2-proxy's /oauth2/auth decision endpoint: 202 for a valid session in platform-admins (request passes to the upstream, X-Auth-Request-* identity headers forwarded), 401/403 otherwise. With failure_mode_allow: false the gateway fails closed — a denied or errored check never reaches the upstream UI (R-IDP-02). The filter + oauth2-proxy cluster live in forward-auth-cec.yaml.

Why this is NOT wired into the shared Gateway (Cilium 1.16.5 constraint)

The shared Gateway's Envoy program is a machine-owned, constantly-regenerated CiliumEnvoyConfig (gateway/cilium-gateway-shared, ownerReferences->Gateway, regenerated on every HTTPRoute change). Cilium 1.16.5 exposes no supported hook to inject an ext_authz filter into that generated listener, and native Gateway-API ExternalAuth (GEP-1494) ships in no released Cilium version yet (PoC only). So there is no clean per-route ext_authz on the shared Gateway today.

Options (PM decision):

#	Approach	Pros	Cons
A	Interim self-authored CEC — a dedicated ext_authz HTTPS listener on its own L2 VIP (forward-auth-cec.yaml), protected hosts moved off the shared Gateway and DNS repointed.	Works on 1.16.5 today; per-route opt-in; reuses the wildcard TLS via SDS; verified below.	Hand-authored TLS+routes; a 2nd ingress VIP + DNS; per-app maintenance; not Gateway-API-native.
B	oauth2-proxy in reverse-proxy mode per app — app HTTPRoute backendRef -> a per-app oauth2-proxy (--upstream=http://app.svc).	Fully Gateway-API-native (just a backendRef swap); robust; no CEC.	One oauth2-proxy instance/config per app; more pods; not the single-proxy ext_authz model.
C	Upgrade Cilium to a release with native ExternalAuth (GEP-1494) and gate routes with an HTTPRoute filter.	Cleanest, truly per-route, Crafty-friendly.	Not released yet; large jump from 1.16.5; must keep the Ansible-bootstrap/Flux parity contract in lock-step.

Recommendation: ship the oauth2-proxy hardening now (done); adopt A as the interim enforcement for the sensitive UIs when the Phase-2 L2 VIP pool lands; track C as the long-term target and drop A once released.

Opt-in (per route)

A route is protected only by being listed as a virtualHost on the ext_authz listener — apps not listed keep hitting the shared Gateway. To protect an app (pihole today; Crafty next): add its Service + EDS cluster + virtualHost to forward-auth-cec.yaml, remove its HTTPRoute from the shared Gateway, and repoint its DNS at the ext_authz VIP. The ServiceDefinition already declares auth.mode: forward-auth + groups: [platform-admins] (rendered onto the HTTPRoute annotations) — the opt-in signal. Ready to extend to Crafty and the other non-OIDC UIs (PiHole/*arr/qBittorrent/Homepage).

Proof (in-cluster, cilium-envoy v1.30.8, 2026-08-28)

A throwaway standalone cilium-envoy pod ran the exact ext_authz stanza from forward-auth-cec.yaml against the live oauth2-proxy Service, with a no-filter control route on the same backend:

• Control (:8081, no ext_authz) -> 200 CONTROL-OK.
• Protected (:8080, ext_authz), unauthenticated -> 403; body is oauth2-proxy's sign-in page, not the upstream (UPSTREAM-OK never served). oauth2-proxy's own access log recorded the /oauth2/auth call from the envoy pod. failure_mode_allow: false confirmed fail-closed.

This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on this exact Envoy. The allow path (valid platform-admins session -> 202 -> upstream) needs a real browser round-trip.

Browser test (for the human, once forward-auth-cec.yaml is deployed)

1. Incognito -> https://pihole.home.keithjackson.dev -> expect a 302 to https://keycloak.home.keithjackson.dev/realms/homelab/... (not the PiHole UI).
2. Log in as a platform-admins user (MFA/OTP enforced) -> land back on the PiHole UI.
3. Log in as a non-platform-admins user -> expect 403 (authenticated but not authorized).
4. Confirm the session cookie is .home.keithjackson.dev-scoped (single sign-on across the other protected UIs).

Observability: what this component can and cannot see (measured 2026-09-30)

SUPERSEDED — read "Update 2026-10-05" at the end of this section first. The gate sidecars are scraped now and the TSDB does hold oauth2_proxy_* for them. The two subsections below are kept as the dated record of how this was diagnosed, not as current state.

servicemonitor.yaml scrapes this namespace's deployment and nothing else. On prod that is two targets, job="oauth2-proxy", both up=1. The complete set of oauth2-proxy-specific series in the TSDB is:

count by (__name__) ({__name__=~"oauth2_proxy.*"})
  oauth2_proxy_requests_in_flight => 2

The per-app gate sidecars are not scraped, and there is nothing to scrape. Six apps run an oauth2-proxy container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr — each with --http-address=0.0.0.0:4180 and no --metrics-address, so oauth2-proxy's metrics listener never starts. They declare only a http:4180 container port, no metrics port exists on their Services, and kubectl get servicemonitors,podmonitors -A selects none of them. Any alert written against oauth2_proxy_requests_total{namespace="jackett"} would be structurally inert.

This matters because the gate's characteristic failure is invisible by construction: an unauthenticated XHR gets a 302 to Keycloak rather than a 401, the browser follows it against a live SSO session with no prompt, and the call resolves 200 carrying an HTML page where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds into a 5-minute session, while jackett-down sat at 1 replica available the whole time.

auth-visibility-alerts.yaml therefore alerts on the gap, not on the incident: OAuth2ProxyGateUnobserved compares kube_pod_container_info{container="oauth2-proxy"} against up{container="oauth2-proxy"} — two already-scraped families — and names every namespace running a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each one gains a scrape. runbooks/oauth2-proxy-auth-failures.md is the by-hand substitute meanwhile, and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).

To close the gap for an app, three things are needed together, on that app's own manifest: --metrics-address=0.0.0.0:44180 on the sidecar, a metrics containerPort plus the matching Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained code="302" as a share of oauth2_proxy_requests_total, held across a window longer than the session lifetime — become writable rather than decorative.

Update 2026-10-01: the listener exists; the scrape path does not

PR #28 added --metrics-address=0.0.0.0:44180 and a named metrics container port to the five fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). podmonitor.yaml in this directory discovers them by the pod label homelab.keithjackson.dev/auth-mode: forward-auth — a PodMonitor rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the renderer.

The endpoint is real: port-forwarded straight to the jackett pod, /metrics returns http=200, 10208 bytes, exporting oauth2_proxy_requests_in_flight, oauth2_proxy_requests_total (with code labels) and oauth2_proxy_response_duration_seconds. That is verified by payload, not by the flag appearing in args — a distinction worth keeping, because the two claims came apart here.

The scrape is nevertheless dropped. From a pod in ns monitoring, both :44180 and :4180 on the jackett pod time out; the shared proxy's :44180 answers. The only difference is that ns oauth2-proxy has no NetworkPolicy at all, while each app namespace has a default-deny whose policyTypes include Ingress with no ingress rules (deny-all) plus a single CNP admitting fromEntities: [ingress] to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no clause.

So the targets read up == 0 and the TSDB is still empty for those namespaces. This is the same bug and the same fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents for Longhorn, and the same narrow fix applies: ns monitoring → the gate pod label → TCP 44180 only. It cannot ship from this directory, because oauth2-proxy and apps-wave-0 are siblings in the Flux graph (both dependsOn: [platform-config]), so a NetworkPolicy naming ns jackett would apply on today's cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.

Consequence for auth-visibility-alerts.yaml: its unless clause requires up == 1, not merely a present up series. A discovered-but-failing target publishes up with value 0, so the bare form would have cleared OAuth2ProxyGateUnobserved the moment the PodMonitor landed — reporting the gates as observed while nothing was being scraped. The unit test pins that scenario; deleting the == 1 fails it and nothing else.

Update 2026-10-05: the scrape works

PR #49 (merged 21c88b0) added the missing third piece. catalog/render.py's r_monitoring_ingress_policy now also fires for the gate sidecar's metrics port, keyed on _needs_oauth2_sidecar, rendering <app>-allow-monitoring-scrape per app: ns monitoring → that app's endpointSelector → TCP 44180 only. Five objects, one each for jackett, pihole, qbittorrent, radarr, sonarr.

Measured on prod 2026-10-05 after Flux applied it (apps-wave-0 21:37:01Z, apps-wave-1 21:37:46Z):

	before	after
target health (6)	down	up
lastError	context deadline exceeded	''
lastScrapeDuration	10.0s — exactly the scrapeTimeout	0.001–0.006s
up	0 × 6	1 × 6

count by (__name__) ({__name__=~"oauth2_proxy_.*",
                      namespace=~"jackett|radarr|sonarr|qbittorrent|pihole"})
  oauth2_proxy_requests_in_flight              => 6
  oauth2_proxy_requests_total                  => 16
  oauth2_proxy_response_duration_seconds_bucket => 108
  oauth2_proxy_response_duration_seconds_sum   => 9
  oauth2_proxy_response_duration_seconds_count => 9

sum by (namespace) (oauth2_proxy_requests_total)
  qbittorrent => 35072    radarr => 23560    sonarr => 1284

oauth2_proxy_requests_total is absent for jackett and pihole simply because that counter is not created until the gate has served its first request.

So the auth-churn rule is now writable rather than decorative — sustained code="302" as a share of oauth2_proxy_requests_total, held across a window longer than the session lifetime. That is the rule the 2026-09-30 incident above could not be caught by. It is still unwritten.

Alert outcome: all 5 TargetDown resolved, and 5 of 6 OAuth2ProxyGateUnobserved with them. At repeat_interval: 12h with group_by: [alertname, namespace], those 11 groups had been re-notifying at roughly 22 emails/day since 2026-10-02; that drops to 2/day.

crafty still fires, and that is correct. It is a hand-maintained Deployment under platform/crafty, outside the renderer, whose oauth2-proxy container has no --metrics-address at all — no listener, so no port to open and nothing for the PodMonitor's pod-label selector to match. Closing it needs the flag and a metrics containerPort on crafty's own manifest first, then a hand-written allowance beside it. Open follow-up work.

The paragraph above headed "To close the gap for an app" is superseded: a Service port and a ServiceMonitor are not required. A PodMonitor targets pod ports directly, which is why no Service exposes 44180 and why the renderer needed no Service change. The three things actually needed are --metrics-address + named container port, a PodMonitor, and the monitoring-ingress allowance.