home-network-k3s-impl/platform/oauth2-proxy

platform/oauth2-proxy

oauth2-proxy — forward-auth at the Gateway for apps without usable OIDC, authenticating against Keycloak.

• Governing ADR: ADR-0014 — IdP: Keycloak
• Spec: docs/15 — Identity & Access
• Serves: R-IDP-01..06 (esp. R-IDP-02: management UIs never unauthenticated)
• Owner: WS-SEC (Gateway wiring shared with WS-NET)

What this stub is

Flux HelmRelease for oauth2-proxy as the forward-auth provider for non-OIDC apps (PiHole, *arr UIs, qBittorrent, Homepage, Prometheus, Alertmanager, Hubble UI — ADR-0014). Replica count is an overlay knob (${OAUTH2_PROXY_REPLICAS}, R-ENV-02). Client/cookie secrets referenced by name, SOPS-encrypted (WS-SEC).

Contract (SSO/auth + Gateway)

Pairs with ../keycloak to realize the "SSO/auth contract": a ServiceDefinition auth: forward-auth service is protected by oauth2-proxy at the Gateway (ext-auth); auth: oidc services go native to Keycloak. Gateway ext-auth wiring is a WS-NET contract.

Enforcement (ext_authz forward-auth)

The authenticator (oauth2-proxy) is deployed and wired to Keycloak: provider=oidc, oidc-issuer-url=.../realms/homelab, --reverse-proxy, --upstream=static://202, --set-xauthrequest, cookie/redirect domain .home.keithjackson.dev, and — added here — --allowed-groups=platform-admins (R-IDP-06 floor) + --skip-provider-button (302 straight to Keycloak). The Keycloak oauth2-proxy client already carries the auth.home…/oauth2/callback redirect URI and a groups claim with full.path=false (bare platform-admins) — no realm change needed.

Mechanism. Forward-auth is an Envoy ext_authz HTTP filter that calls oauth2-proxy's /oauth2/auth decision endpoint: 202 for a valid session in platform-admins (request passes to the upstream, X-Auth-Request-* identity headers forwarded), 401/403 otherwise. With failure_mode_allow: false the gateway fails closed — a denied or errored check never reaches the upstream UI (R-IDP-02). The filter + oauth2-proxy cluster live in forward-auth-cec.yaml.

Why this is NOT wired into the shared Gateway (Cilium 1.16.5 constraint)

The shared Gateway's Envoy program is a machine-owned, constantly-regenerated CiliumEnvoyConfig (gateway/cilium-gateway-shared, ownerReferences->Gateway, regenerated on every HTTPRoute change). Cilium 1.16.5 exposes no supported hook to inject an ext_authz filter into that generated listener, and native Gateway-API ExternalAuth (GEP-1494) ships in no released Cilium version yet (PoC only). So there is no clean per-route ext_authz on the shared Gateway today.

Options (PM decision):

#	Approach	Pros	Cons
A	Interim self-authored CEC — a dedicated ext_authz HTTPS listener on its own L2 VIP (forward-auth-cec.yaml), protected hosts moved off the shared Gateway and DNS repointed.	Works on 1.16.5 today; per-route opt-in; reuses the wildcard TLS via SDS; verified below.	Hand-authored TLS+routes; a 2nd ingress VIP + DNS; per-app maintenance; not Gateway-API-native.
B	oauth2-proxy in reverse-proxy mode per app — app HTTPRoute backendRef -> a per-app oauth2-proxy (--upstream=http://app.svc).	Fully Gateway-API-native (just a backendRef swap); robust; no CEC.	One oauth2-proxy instance/config per app; more pods; not the single-proxy ext_authz model.
C	Upgrade Cilium to a release with native ExternalAuth (GEP-1494) and gate routes with an HTTPRoute filter.	Cleanest, truly per-route, Crafty-friendly.	Not released yet; large jump from 1.16.5; must keep the Ansible-bootstrap/Flux parity contract in lock-step.

Recommendation: ship the oauth2-proxy hardening now (done); adopt A as the interim enforcement for the sensitive UIs when the Phase-2 L2 VIP pool lands; track C as the long-term target and drop A once released.

Opt-in (per route)

A route is protected only by being listed as a virtualHost on the ext_authz listener — apps not listed keep hitting the shared Gateway. To protect an app (pihole today; Crafty next): add its Service + EDS cluster + virtualHost to forward-auth-cec.yaml, remove its HTTPRoute from the shared Gateway, and repoint its DNS at the ext_authz VIP. The ServiceDefinition already declares auth.mode: forward-auth + groups: [platform-admins] (rendered onto the HTTPRoute annotations) — the opt-in signal. Ready to extend to Crafty and the other non-OIDC UIs (PiHole/*arr/qBittorrent/Homepage).

Proof (in-cluster, cilium-envoy v1.30.8, 2026-08-28)

A throwaway standalone cilium-envoy pod ran the exact ext_authz stanza from forward-auth-cec.yaml against the live oauth2-proxy Service, with a no-filter control route on the same backend:

• Control (:8081, no ext_authz) -> 200 CONTROL-OK.
• Protected (:8080, ext_authz), unauthenticated -> 403; body is oauth2-proxy's sign-in page, not the upstream (UPSTREAM-OK never served). oauth2-proxy's own access log recorded the /oauth2/auth call from the envoy pod. failure_mode_allow: false confirmed fail-closed.

This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on this exact Envoy. The allow path (valid platform-admins session -> 202 -> upstream) needs a real browser round-trip.

Browser test (for the human, once forward-auth-cec.yaml is deployed)

1. Incognito -> https://pihole.home.keithjackson.dev -> expect a 302 to https://keycloak.home.keithjackson.dev/realms/homelab/... (not the PiHole UI).
2. Log in as a platform-admins user (MFA/OTP enforced) -> land back on the PiHole UI.
3. Log in as a non-platform-admins user -> expect 403 (authenticated but not authorized).
4. Confirm the session cookie is .home.keithjackson.dev-scoped (single sign-on across the other protected UIs).

Observability: what this component can and cannot see (measured 2026-09-30)

servicemonitor.yaml scrapes this namespace's deployment and nothing else. On prod that is two targets, job="oauth2-proxy", both up=1. The complete set of oauth2-proxy-specific series in the TSDB is:

count by (__name__) ({__name__=~"oauth2_proxy.*"})
  oauth2_proxy_requests_in_flight => 2

The per-app gate sidecars are not scraped, and there is nothing to scrape. Six apps run an oauth2-proxy container of their own — crafty, jackett, pihole, qbittorrent, radarr, sonarr — each with --http-address=0.0.0.0:4180 and no --metrics-address, so oauth2-proxy's metrics listener never starts. They declare only a http:4180 container port, no metrics port exists on their Services, and kubectl get servicemonitors,podmonitors -A selects none of them. Any alert written against oauth2_proxy_requests_total{namespace="jackett"} would be structurally inert.

This matters because the gate's characteristic failure is invisible by construction: an unauthenticated XHR gets a 302 to Keycloak rather than a 401, the browser follows it against a live SSO session with no prompt, and the call resolves 200 carrying an HTML page where the app expected JSON. Nothing in the request path reports an error. On 2026-09-30 jackett spent a burst of requests in exactly that state, re-established a session, and lost it 26 seconds into a 5-minute session, while jackett-down sat at 1 replica available the whole time.

auth-visibility-alerts.yaml therefore alerts on the gap, not on the incident: OAuth2ProxyGateUnobserved compares kube_pod_container_info{container="oauth2-proxy"} against up{container="oauth2-proxy"} — two already-scraped families — and names every namespace running a gate that nothing observes. It fires for six namespaces today and clears per-namespace as each one gains a scrape. runbooks/oauth2-proxy-auth-failures.md is the by-hand substitute meanwhile, and it reads the sidecar logs out of Loki (Alloy collects them; 72h retention).

To close the gap for an app, three things are needed together, on that app's own manifest: --metrics-address=0.0.0.0:44180 on the sidecar, a metrics containerPort plus the matching Service port, and a ServiceMonitor selecting it. Only then does the auth-churn rule — sustained code="302" as a share of oauth2_proxy_requests_total, held across a window longer than the session lifetime — become writable rather than decorative.

Update 2026-10-01: the listener exists; the scrape path does not

PR #28 added --metrics-address=0.0.0.0:44180 and a named metrics container port to the five fleet sidecars (jackett, radarr, sonarr, qbittorrent, pihole). podmonitor.yaml in this directory discovers them by the pod label homelab.keithjackson.dev/auth-mode: forward-auth — a PodMonitor rather than a ServiceMonitor because no Service exposes 44180 and adding one would mean changing the renderer.

The endpoint is real: port-forwarded straight to the jackett pod, /metrics returns http=200, 10208 bytes, exporting oauth2_proxy_requests_in_flight, oauth2_proxy_requests_total (with code labels) and oauth2_proxy_response_duration_seconds. That is verified by payload, not by the flag appearing in args — a distinction worth keeping, because the two claims came apart here.

The scrape is nevertheless dropped. From a pod in ns monitoring, both :44180 and :4180 on the jackett pod time out; the shared proxy's :44180 answers. The only difference is that ns oauth2-proxy has no NetworkPolicy at all, while each app namespace has a default-deny whose policyTypes include Ingress with no ingress rules (deny-all) plus a single CNP admitting fromEntities: [ingress] to port 4180 — the Gateway's Envoy and nothing else. Prometheus matches no clause.

So the targets read up == 0 and the TSDB is still empty for those namespaces. This is the same bug and the same fingerprint as platform/longhorn/networkpolicy-metrics.yaml documents for Longhorn, and the same narrow fix applies: ns monitoring → the gate pod label → TCP 44180 only. It cannot ship from this directory, because oauth2-proxy and apps-wave-0 are siblings in the Flux graph (both dependsOn: [platform-config]), so a NetworkPolicy naming ns jackett would apply on today's cluster and fail on a cold rebuild. It belongs with each app's own rendered policy.

Consequence for auth-visibility-alerts.yaml: its unless clause requires up == 1, not merely a present up series. A discovered-but-failing target publishes up with value 0, so the bare form would have cleared OAuth2ProxyGateUnobserved the moment the PodMonitor landed — reporting the gates as observed while nothing was being scraped. The unit test pins that scenario; deleting the == 1 fails it and nothing else.