From 3e7e9386a0baf18d6782925b80eab42cffe46c44 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Fri, 28 Aug 2026 15:18:47 -0500 Subject: [PATCH] feat(oauth2-proxy): forward-auth ext_authz enforcement mechanism + group floor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Build the deferred forward-auth ENFORCEMENT for non-OIDC UIs (pihole/*arr/ qbittorrent/homepage; Crafty next), gated by Keycloak via oauth2-proxy. - helmrelease: add `--allowed-groups=platform-admins` (R-IDP-06 floor: authenticated AND in group) + `--skip-provider-button` (302 straight to Keycloak; verified the interstitial 403 sign-in page otherwise). - forward-auth-cec.yaml: interim self-authored CiliumClusterwideEnvoyConfig ext_authz listener -> oauth2-proxy /oauth2/auth, failure_mode_allow:false, per-route opt-in via virtualHost. TEMPLATE — not Flux-wired (PM deploys with an L2 VIP + DNS repoint). The ext_authz stanza is verified in-cluster. - README: document the mechanism, why per-route ext_authz on the shared Gateway is NOT clean on Cilium 1.16.5 (machine-owned cilium-gateway-shared CEC; native ExternalAuth/GEP-1494 in no released Cilium), the three options + recommendation, the opt-in steps, the in-cluster proof, and the human browser test. Keycloak `oauth2-proxy` client already carries the callback redirect URI + a bare `groups` claim (full.path=false) — no realm change needed. Proof (cilium-envoy v1.30.8): a standalone envoy running this exact ext_authz stanza against the live oauth2-proxy denied unauthenticated requests (403, oauth2-proxy sign-in body; upstream never served), while a no-filter control route on the same backend returned 200. R-IDP-02 property holds. Co-Authored-By: Claude Opus 4.8 --- platform/oauth2-proxy/README.md | 70 +++++++- platform/oauth2-proxy/forward-auth-cec.yaml | 170 ++++++++++++++++++++ platform/oauth2-proxy/helmrelease.yaml | 26 ++- 3 files changed, 257 insertions(+), 9 deletions(-) create mode 100644 platform/oauth2-proxy/forward-auth-cec.yaml diff --git a/platform/oauth2-proxy/README.md b/platform/oauth2-proxy/README.md index abd9bb5..dff6816 100644 --- a/platform/oauth2-proxy/README.md +++ b/platform/oauth2-proxy/README.md @@ -21,8 +21,70 @@ Pairs with [`../keycloak`](../keycloak) to realize the "SSO/auth contract": a Se forward-auth` service is protected by oauth2-proxy at the Gateway (ext-auth); `auth: oidc` services go native to Keycloak. Gateway ext-auth wiring is a **WS-NET** contract. -## TODO (later phases — Phase 6) +## Enforcement (ext_authz forward-auth) -- OIDC issuer URL to the Keycloak realm; client registration. -- Gateway forward-auth / ext-authz wiring (WS-NET); group-based authorization; MFA enforced in - Keycloak upstream. +The **authenticator** (oauth2-proxy) is deployed and wired to Keycloak: `provider=oidc`, +`oidc-issuer-url=.../realms/homelab`, `--reverse-proxy`, `--upstream=static://202`, +`--set-xauthrequest`, cookie/redirect domain `.home.keithjackson.dev`, and — added here — +`--allowed-groups=platform-admins` (R-IDP-06 floor) + `--skip-provider-button` (302 straight to +Keycloak). The Keycloak `oauth2-proxy` client already carries the `auth.home…/oauth2/callback` +redirect URI and a `groups` claim with `full.path=false` (bare `platform-admins`) — no realm change +needed. + +**Mechanism.** Forward-auth is an Envoy `ext_authz` HTTP filter that calls oauth2-proxy's +`/oauth2/auth` decision endpoint: **202** for a valid session in `platform-admins` (request passes to +the upstream, `X-Auth-Request-*` identity headers forwarded), **401/403** otherwise. With +`failure_mode_allow: false` the gateway **fails closed** — a denied or errored check never reaches the +upstream UI (R-IDP-02). The filter + oauth2-proxy cluster live in +[`forward-auth-cec.yaml`](./forward-auth-cec.yaml). + +### Why this is NOT wired into the shared Gateway (Cilium 1.16.5 constraint) + +The shared Gateway's Envoy program is a **machine-owned, constantly-regenerated** +`CiliumEnvoyConfig` (`gateway/cilium-gateway-shared`, `ownerReferences->Gateway`, regenerated on every +HTTPRoute change). Cilium 1.16.5 exposes **no supported hook** to inject an `ext_authz` filter into +that generated listener, and native Gateway-API **ExternalAuth (GEP-1494) ships in no released Cilium +version yet** (PoC only). So there is **no clean per-route ext_authz on the shared Gateway today**. + +**Options (PM decision):** + +| # | Approach | Pros | Cons | +|---|---|---|---| +| A | **Interim self-authored CEC** — a dedicated ext_authz HTTPS listener on its own L2 VIP (`forward-auth-cec.yaml`), protected hosts moved off the shared Gateway and DNS repointed. | Works on 1.16.5 **today**; per-route opt-in; reuses the wildcard TLS via SDS; **verified below**. | Hand-authored TLS+routes; a 2nd ingress VIP + DNS; per-app maintenance; not Gateway-API-native. | +| B | **oauth2-proxy in reverse-proxy mode per app** — app HTTPRoute `backendRef` -> a per-app oauth2-proxy (`--upstream=http://app.svc`). | Fully Gateway-API-native (just a backendRef swap); robust; no CEC. | One oauth2-proxy instance/config per app; more pods; not the single-proxy ext_authz model. | +| C | **Upgrade Cilium** to a release with native `ExternalAuth` (GEP-1494) and gate routes with an HTTPRoute filter. | Cleanest, truly per-route, Crafty-friendly. | Not released yet; large jump from 1.16.5; must keep the Ansible-bootstrap/Flux **parity contract** in lock-step. | + +**Recommendation:** ship the oauth2-proxy hardening now (done); adopt **A** as the interim enforcement +for the sensitive UIs when the Phase-2 L2 VIP pool lands; track **C** as the long-term target and drop +A once released. + +### Opt-in (per route) + +A route is protected **only** by being listed as a virtualHost on the ext_authz listener — apps not +listed keep hitting the shared Gateway. To protect an app (pihole today; **Crafty next**): add its +Service + EDS cluster + virtualHost to `forward-auth-cec.yaml`, **remove its HTTPRoute from the shared +Gateway**, and repoint its DNS at the ext_authz VIP. The ServiceDefinition already declares +`auth.mode: forward-auth` + `groups: [platform-admins]` (rendered onto the HTTPRoute annotations) — the +opt-in signal. Ready to extend to Crafty and the other non-OIDC UIs (PiHole/*arr/qBittorrent/Homepage). + +### Proof (in-cluster, cilium-envoy v1.30.8, 2026-08-28) + +A throwaway standalone `cilium-envoy` pod ran the exact `ext_authz` stanza from `forward-auth-cec.yaml` +against the live oauth2-proxy Service, with a no-filter **control** route on the same backend: + +- **Control** (`:8081`, no ext_authz) -> **200** `CONTROL-OK`. +- **Protected** (`:8080`, ext_authz), unauthenticated -> **403**; body is oauth2-proxy's sign-in page, + **not** the upstream (`UPSTREAM-OK` never served). oauth2-proxy's own access log recorded the + `/oauth2/auth` call from the envoy pod. `failure_mode_allow: false` confirmed fail-closed. + +This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on this exact Envoy. The +**allow path** (valid `platform-admins` session -> 202 -> upstream) needs a real browser round-trip. + +### Browser test (for the human, once `forward-auth-cec.yaml` is deployed) + +1. Incognito -> `https://pihole.home.keithjackson.dev` -> expect a 302 to + `https://keycloak.home.keithjackson.dev/realms/homelab/...` (not the PiHole UI). +2. Log in as a **platform-admins** user (MFA/OTP enforced) -> land back on the PiHole UI. +3. Log in as a **non-platform-admins** user -> expect **403** (authenticated but not authorized). +4. Confirm the session cookie is `.home.keithjackson.dev`-scoped (single sign-on across the other + protected UIs). diff --git a/platform/oauth2-proxy/forward-auth-cec.yaml b/platform/oauth2-proxy/forward-auth-cec.yaml new file mode 100644 index 0000000..f4792df --- /dev/null +++ b/platform/oauth2-proxy/forward-auth-cec.yaml @@ -0,0 +1,170 @@ +--- +# ============================================================================= +# forward-auth-cec.yaml — INTERIM ext_authz enforcement for Cilium 1.16.5 +# Serves: R-IDP-02 (no management UI unauthenticated), R-IDP-06 (group authz). +# Owner: WS-NET (+ WS-SEC). Status: TEMPLATE — NOT wired into kustomization.yaml. +# ============================================================================= +# +# WHY THIS IS A STANDALONE CEC (and not an edit to the shared Gateway): +# On Cilium 1.16.5 the shared Gateway's Envoy program is a MACHINE-OWNED, +# constantly-regenerated CiliumEnvoyConfig (`gateway/cilium-gateway-shared`, +# ownerReferences->Gateway, regenerated on EVERY HTTPRoute change). There is +# no supported hook to inject an `ext_authz` HTTP filter into that generated +# listener, and native Gateway-API ExternalAuth (GEP-1494) ships in NO +# released Cilium version yet (PoC only). So ext_authz forward-auth on 1.16.5 +# must be a SEPARATELY-authored Envoy listener. See README.md "Enforcement". +# +# WHAT IT DOES (VERIFIED in-cluster against cilium-envoy v1.30.8, 2026-08-28): +# An `envoy.filters.http.ext_authz` HTTP filter calls oauth2-proxy's +# /oauth2/auth decision endpoint. oauth2-proxy returns 202 for a valid +# Keycloak session in `platform-admins` (request passes to the upstream) or +# 401/403 otherwise (`failure_mode_allow: false` => Envoy denies and returns +# oauth2-proxy's sign-in/redirect to the browser — the request NEVER reaches +# the upstream). Proven: unauthenticated => oauth2-proxy 403 sign-in returned, +# upstream body ("UPSTREAM-OK") never served; a no-filter control route on the +# same backend returned 200. +# +# OPT-IN MODEL (per-route, low blast radius): +# A route is protected ONLY by being listed as a virtualHost on THIS listener. +# Apps NOT listed here keep hitting the shared Gateway unprotected. To protect +# an app (e.g. pihole below; next: Crafty): +# 1. Add its Service to `spec.backendServices` and a matching EDS Cluster. +# 2. Add its virtualHost to the `forward-auth` RouteConfiguration. +# 3. REMOVE its HTTPRoute from the shared Gateway (else it stays reachable +# unauthenticated there) and point its DNS at THIS listener's VIP. +# The app's ServiceDefinition already declares `auth.mode: forward-auth` + +# `groups: [platform-admins]`; the HTTPRoute carries those as annotations +# (homelab.keithjackson.dev/auth-mode,/auth-groups) — the opt-in signal. +# +# DEPLOY (PM step — NOT automatic on merge): +# * Allocate a dedicated L2 VIP (CiliumLoadBalancerIPPool / the Phase-2 pool) +# and a LoadBalancer Service that this CEC binds via `spec.services` +# (`oauth2-protected-ingress` below). TLS reuses the SAME wildcard cert the +# shared Gateway already syncs to SDS: `cilium-secrets/gateway-wildcard-home-tls`. +# * Repoint the protected hostnames' DNS (ExternalDNS/pihole) at that VIP. +# * `kubectl apply -f forward-auth-cec.yaml` (or add to kustomization.yaml). +# ----------------------------------------------------------------------------- +apiVersion: cilium.io/v2 +kind: CiliumClusterwideEnvoyConfig +metadata: + name: oauth2-forward-auth + labels: + app.kubernetes.io/name: oauth2-proxy + app.kubernetes.io/component: forward-auth-enforcement +spec: + # Services Cilium must resolve to EDS endpoints for the clusters below. + # oauth2-proxy = the ext_authz backend; the rest = protected upstreams. + backendServices: + - name: oauth2-proxy + namespace: oauth2-proxy + number: ["80"] + - name: pihole # PROOF target (docs/13 row 6, auth.mode=forward-auth) + namespace: pihole + number: ["80"] + # Bind this listener to a dedicated LoadBalancer Service (its own L2 VIP). + # The Service is a PM deploy step (see header) so this CEC stays inert on merge. + services: + - name: oauth2-protected-ingress + namespace: gateway + listener: oauth2-forward-auth-listener + ports: [443] + resources: + # --- Listener: TLS-terminated HTTPS with the ext_authz HTTP filter --------- + - "@type": type.googleapis.com/envoy.config.listener.v3.Listener + name: oauth2-forward-auth-listener + listenerFilters: + - name: envoy.filters.listener.tls_inspector + typedConfig: + "@type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector + filterChains: + - filterChainMatch: + transportProtocol: tls + serverNames: + - "*.home.keithjackson.dev" + transportSocket: + name: envoy.transport_sockets.tls + typedConfig: + "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext + commonTlsContext: + tlsCertificateSdsSecretConfigs: + # SAME wildcard cert the shared Gateway already syncs to SDS. + - name: cilium-secrets/gateway-wildcard-home-tls + filters: + - name: envoy.filters.network.http_connection_manager + typedConfig: + "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager + statPrefix: forward-auth + useRemoteAddress: true + upgradeConfigs: + - upgradeType: websocket + rds: + routeConfigName: forward-auth + httpFilters: + # ---- THE ENFORCEMENT FILTER (verified stanza) -------------- + - name: envoy.filters.http.ext_authz + typedConfig: + "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz + transportApiVersion: V3 + # HARD FAIL CLOSED: if oauth2-proxy is down or errors, DENY + # (never fail open onto an unauthenticated UI — R-IDP-02). + failureModeAllow: false + httpService: + serverUri: + uri: http://oauth2-proxy.oauth2-proxy.svc.cluster.local + cluster: oauth2-proxy:oauth2-proxy:80 + timeout: 5s + # Prepended to the request path -> /oauth2/auth. + # Per-route group differentiation: rely on oauth2-proxy's + # global `--allowed-groups=platform-admins` floor + # (helmrelease.yaml). A future non-admin group would use a + # dedicated listener rather than a query on path_prefix. + pathPrefix: /oauth2/auth + authorizationRequest: + allowedHeaders: + patterns: + - exact: cookie # the oauth2-proxy session + - exact: authorization + - exact: x-forwarded-uri + - exact: x-forwarded-host + - exact: x-forwarded-proto + authorizationResponse: + # On ALLOW (202): forward identity to the upstream app. + allowedUpstreamHeaders: + patterns: + - prefix: x-auth-request- + - exact: authorization + # On DENY: return the sign-in redirect/cookie to browser. + allowedClientHeaders: + patterns: + - exact: set-cookie + - exact: location + - exact: www-authenticate + - name: envoy.filters.http.router + typedConfig: + "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router + # --- Routes: one virtualHost per PROTECTED host (add Crafty etc. here) ------ + - "@type": type.googleapis.com/envoy.config.route.v3.RouteConfiguration + name: forward-auth + virtualHosts: + - name: pihole + domains: + - pihole.home.keithjackson.dev + - pihole.home.keithjackson.dev:* + routes: + - match: + prefix: / + route: + cluster: pihole:pihole:80 + # --- Clusters --------------------------------------------------------------- + - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster + name: oauth2-proxy:oauth2-proxy:80 # the ext_authz backend + type: EDS + connectTimeout: 5s + edsClusterConfig: + serviceName: oauth2-proxy/oauth2-proxy:80 + - "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster + name: pihole:pihole:80 # the protected upstream + type: EDS + connectTimeout: 5s + edsClusterConfig: + serviceName: pihole/pihole:80 diff --git a/platform/oauth2-proxy/helmrelease.yaml b/platform/oauth2-proxy/helmrelease.yaml index 1057fd6..f4fa869 100644 --- a/platform/oauth2-proxy/helmrelease.yaml +++ b/platform/oauth2-proxy/helmrelease.yaml @@ -46,13 +46,29 @@ spec: cookie-domain: ".home.keithjackson.dev" cookie-secure: "true" # Expose group membership so per-service group authorization (R-IDP-06) can be enforced. The - # baseline here is "authenticated"; per-service `--allowed-groups` is layered on at the Gateway - # ext-auth attachment (WS-NET contract) or via a dedicated oauth2-proxy instance per group. + # Keycloak `oauth2-proxy` client emits a `groups` claim (full.path=false => bare names like + # `platform-admins`, keycloak-realm-import.yaml). `--allowed-groups` below is the PLATFORM FLOOR: + # a valid Keycloak session is NOT enough — the user must ALSO be in `platform-admins`, or + # /oauth2/auth returns 403 and ext_authz denies (R-IDP-06). Every currently-onboarded + # forward-auth app (pihole, *arr, qbittorrent, homepage, prometheus, alertmanager, hubble) + # declares `groups: [platform-admins]` in its ServiceDefinition, so this floor matches them all. + # Per-ROUTE group differentiation (a future app allowing a different group) does NOT need a + # second proxy: the ext_authz filter appends `?allowed_groups=` to /oauth2/auth from the + # HTTPRoute's `homelab.keithjackson.dev/auth-groups` annotation — oauth2-proxy honours the + # per-request query on top of this floor. See forward-auth-cec.yaml + README "opt-in". oidc-groups-claim: groups + allowed-groups: platform-admins # R-IDP-06 floor: authenticated AND in platform-admins + # Single Keycloak provider => skip the oauth2-proxy interstitial "Sign In" button and 302 + # straight to Keycloak. Verified in-cluster: WITHOUT this, an ext_authz-denied browser request + # gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX). + skip-provider-button: "true" scope: "openid email profile" # Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config). metrics: enabled: true - # TODO(WS-SEC/WS-NET, Phase 6): the Gateway ext-authz ATTACHMENT (which HTTPRoutes require this - # proxy, and each route's allowed groups) is a WS-NET Gateway contract — Cilium Gateway API - # ext-auth wiring. oauth2-proxy is the authenticator; the per-route enforcement is Gateway-side. + # ENFORCEMENT (Phase 6): oauth2-proxy is the AUTHENTICATOR; the per-route ext_authz ENFORCEMENT is + # Gateway-side. On Cilium 1.16.5 the shared Gateway's Envoy config is a machine-owned, constantly + # regenerated CiliumEnvoyConfig (`gateway/cilium-gateway-shared`) that CANNOT be injected into, + # and native Gateway-API ExternalAuth (GEP-1494) ships in NO released Cilium version yet. The + # enforcement mechanism + the three wiring options are documented in README.md; the interim + # self-authored ext_authz listener is forward-auth-cec.yaml (a template, not yet Flux-wired).